(HLSL) Cel (Toon) Shading

[klikmaster]

Ok, no errors anymore, but my model appears black, I got a feeling it's my mLightPos.

I made my lightnode a global variable (mistake?) so I could access it in my callback.

Here's my shader callback:

Code: Select all

class MyShaderCallBack : public video::IShaderConstantSetCallBack
{
public:

  virtual void OnSetConstants(video::IMaterialRendererServices* services, s32 userData)
  {
    video::IVideoDriver* driver = services->getVideoDriver();

    core::matrix4 invWorld = driver->getTransform(video::ETS_WORLD);
    invWorld.makeInverse();

    // set clip matrix
    core::matrix4 worldViewProj;
    worldViewProj = driver->getTransform(video::ETS_PROJECTION);			
    worldViewProj *= driver->getTransform(video::ETS_VIEW);
    worldViewProj *= driver->getTransform(video::ETS_WORLD);

    if (UseHighLevelShaders)
       services->setVertexShaderConstant("mWorldViewProj", &worldViewProj.M[0], 16);
    else
       services->setVertexShaderConstant(&worldViewProj.M[0], 4, 4);
		
    vector3df lightPosOS;
    invWorld.transformVect(light1->getPosition(),lightPosOS); 

    if (UseHighLevelShaders)
      services->setVertexShaderConstant("mLightPos", reinterpret_cast<f32*>(&lightPosOS), 3);
    else
      services->setVertexShaderConstant(reinterpret_cast<f32*>(&lightPosOS), 8, 1);

    // set light color 
    video::SColorf col(0.0f,1.0f,1.0f,0.0f);
	}
};

And here is my material and model setup.

Code: Select all

	s32 toonshader = 0;

	if (gpu)
	{
		MyShaderCallBack* mc = new MyShaderCallBack();
	
		if (UseHighLevelShaders)
		{
			toonshader = gpu->addHighLevelShaderMaterialFromFiles(
				vsFileName,	"vertexMain", video::EVST_VS_1_1,
				psFileName, "pixelMain", video::EPST_PS_1_1,
				mc, video::EMT_SOLID);
		}
		else
		{
			toonshader = gpu->addShaderMaterialFromFiles(vsFileName,
				psFileName, mc, video::EMT_SOLID);
		}

		mc->drop();
	}
//==============================================================================
	IAnimatedMesh* mesh = smgr->getMesh("models/goku.3ds");
	IAnimatedMeshSceneNode* node = smgr->addAnimatedMeshSceneNode( mesh );

	if (node)
	{
		node->setMaterialFlag(EMF_LIGHTING, true);
		node->setMaterialTexture( 0, driver->getTexture("models/TW_GOKU.JPG") );
		node->setMaterialTexture( 3, driver->getTexture("data/toonshader.bmp") );
		node->setMaterialType((video::E_MATERIAL_TYPE)toonshader);
	}

[BlindSide]

Klikmaster, you can show me the whole code if you want. Ive got no idea why its not working right now. Where is you light positioned? (Nothing wrong with making it a global variable.)

HOLD UP, it seems you are setting the toonshader.bmp to the 4th texture (3). Set it to (1) or (2) depending on what you set in the shader file.

You must have confused me saying "3rd" as number 3, but because it starts from 0 it is number 2! OR number 1 if you are using s1 in the shader (Which is not recommended unless you set the whole model to non filtered.)

PS: I forgot to mention, nice goku model!

@Blindside: I did wonder about the sampler thing. Wink If you need different filters on textures then I don't think you can currently do it* - a filter is set per-material. Neutral

(*unless you exploit the 1.2 bug, but that's gone in the latest svn version)

Damn I guess I can just set the material non filtered then, that should work. With celshading texture filtering isnt gonna affect it much.

[klikmaster]

Hoorah! It works! Thankyou so much BlindSide for this nice shader.
Sorry I was being noobish, but I think I've learned a lot more about shaders now

Here is how it looks:

Yes, the texture is a bit messed up, but the cel shader works

By the way, I stole the goku model, but I'm in the process of making my own.

It's looking very blocky at the moment, apart from making my polygons arranged nicer, it there a way to make it smoother? Like per-pixel shaded rather than per pixel?

[BlindSide]

Im afraid it is calculated at a vertex level. I have not come across a celshader yet that does all the light and direction calculations at a pixel level. Your best bet is to import that model in blender and sub surf it to increase polygons.

PS: I wonder why the texture is so stuffed up? That is odd.. I think i know a solution. Try to remove the lines

Code: Select all

   // Shading offset
   float offset = 0.1f;
   
   //Subtract the offset and clamp the texture to 0
   diffuse = clamp(diffuse+offset,0,1);
 

From the vertex shader part. I only used them to force model to become brighter, but it is possible the clamping is causing those artifacts. Tell me how that works out for you.

[klikmaster]

No, it's the model that's messed up, don't worry about it

[omaremad]

use averaged or "smoothed normals", that are imported per vertex (dont use .3ds irrlicht only loads face normals for that)

[BlindSide]

No, it's the model that's messed up, don't worry about it

Thats not true it looked fine in the first picture...

Oh well you seem happy

[rayxzero]

Is someone here knows how this guy achieved this nice toon shading effect? It showed the same effect as that NARUTO game on Playstation 2.
http://gpower-en.blogspot.com/2006/03/h ... ffect.html

[B@z]

hi
just a quick question.
i managed to get it work, but it only works with one light.
and i have more (around 10) lights at my map....
how should i do that every (or maybe just the ones what can be seen) light affect the node?

[BlindSide]

Just use something like this instead:

Code: Select all

//-----------------------------------------------------------------------------
// Global variables
//-----------------------------------------------------------------------------
float4x4 mWorldViewProj;  // World * View * Projection transformation
float4x4 mInvWorld;       // Inverted world matrix
float3 mLightPos[8];         // Light position

struct VS_OUTPUT
{
   float4 Position: POSITION0;
   float2 TexCoord:   TEXCOORD0;
   float diffuse: TEXCOORD1;
};


VS_OUTPUT vertexMain(
   float4 Position: POSITION0,
   float2 TexCoord: TEXCOORD0,
   float diffuse: TEXCOORD1,
   float3 Normal:   NORMAL0)
{
   VS_OUTPUT Output;

   // calculate light vector
   float3 N = normalize(Normal);
   diffuse = 0.0;
   
   for(int i = 0;i < 8;++i)
   {
      float3 L = normalize(mLightPos[i] - Position.xyz);

      // Calculate diffuse component
      diffuse += max(dot(N, L),0);
   }

   // Shading offset
   float offset = 0.1f;
   
   //Subtract the offset and clamp the texture to 0
   diffuse = clamp(diffuse+offset,0,1);

   Output.Position = mul(Position,mWorldViewProj);
   Output.diffuse = diffuse;
   Output.TexCoord = TexCoord;
   return( Output );
}
   
sampler2D tex0 : register(s0) ;
sampler1D diffuseRamp : register(s2) ;

float4 pixelMain(
   float2 TexCoord:     TEXCOORD0,
      float diffuseIn:    TEXCOORD1
   ) : COLOR0
{
   float4 color = float4(1,1,1,1);
   float4 diffuse =  float4(0.8f, 0.8f, 0.8f, 1.0f);
   // Step functions from textures
   diffuseIn = tex1D(diffuseRamp, diffuseIn).x;

      
   
   color = (diffuse * diffuseIn);
     float4 col = tex2D( tex0, TexCoord); 
     col*=2;
      return( color*col );
 
} 

And when passing the shader constant use:

Code: Select all

services->setVertexShaderConstant("mLightPos", arrayOfEightLightPositions , 3 * 8);

You can just get the light positions using driver->getDynamicLight and getDynamicLightCount.

Cheers

[B@z]

thanks! works cool now!

[sudi]

Seams like u use untransformed normal and position. does that really work? don't u have to transform the vertex position and normal into worldspace first? instead of using the modelspace...

[BlindSide]

Seams like u use untransformed normal and position. does that really work? don't u have to transform the vertex position and normal into worldspace first? instead of using the modelspace...

This shader requires you to transform lights by the inverse world transform of the scene node before passing the positions.

Eg.

Code: Select all

matrix4 invWorld;
driver->getTransform(ETS_WORLD).getInverse(invWorld);
invWorld.transformVect(lightPos);

So for the eight lights you would have to loop over the array and transform them.

Cheers

[B@z]

yup, i did it, so it worked
got it from the first source xD

[Katsumi]

Is someone here knows how this guy achieved this nice toon shading effect? It showed the same effect as that NARUTO game on Playstation 2.
http://gpower-en.blogspot.com/2006/03/h ... ffect.html

I found here this url http://www.paulsprojects.net/opengl/cel ... ading.html
looks awesome, but still a noob in irrlicht. first coming tutorials. ^^