Sampling different material layers for shaders

If you are a new Irrlicht Engine user, and have a newbie-question, this is the forum for you. You may also post general programming questions here.
Post Reply
WizardofChaos
Posts: 17
Joined: Tue Jan 11, 2022 8:43 pm

Sampling different material layers for shaders

Post by WizardofChaos »

I feel like I'm doing something *wrong* but I don't know exactly *where* with regards to texture variables in an HLSL shader.

I'm trying to get the hang of HLSL shaders by writing out a bump-map implementation for myself, but it doesn't appear to be sampling the normal map correctly. Where am I doing it wrong?

In the shader itself I declare my sampler like so:

Code: Select all

texture normalTex; //our model's normal map
sampler2D normalSampler //used to sample the norm at a given position
{
    textureSampler = (normalTex);
    Filter = MIN_MAG_MIP_LINEAR;
    AddressU = Clamp;
    AddressV = Clamp;
};
Then I set the textures for both regular color and normal map in the OnSetConstants for my shader callback like so (directly ripping off the tutorial):

Code: Select all

	s32 texLayer = 0;
	s32 normLayer = 1;
	services->setPixelShaderConstant("modelTex", &texLayer, 1);
	services->setPixelShaderConstant("normalTex", &normLayer, 1);
...and construct the shader material like so:

Code: Select all

	s32 mat = gpu->addHighLevelShaderMaterialFromFiles(
		fname.c_str(), "vertexMain", EVST_VS_4_0,
		fname.c_str(), "pixelMain", EPST_PS_4_0,
		amb, EMT_NORMAL_MAP_SOLID, 0, EGSL_DEFAULT);

	node->setMaterialFlag(EMF_LIGHTING, false);
	node->setMaterialFlag(EMF_BLEND_OPERATION, true);
	node->setMaterialType((E_MATERIAL_TYPE)mat);
This samples my regular texture, but not my normal map for some reason. If it helps I can post the shader to see if I'm literally just doing it wrong there, but I *think* I'm pretty solid on that front? Should I be using different base material when creating the shader or am I declaring the textures wrong?
Local man gets so irritated with modern space sims that he starts writing his own
Seven
Posts: 1030
Joined: Mon Nov 14, 2005 2:03 pm

Re: Sampling different material layers for shaders

Post by Seven »

I really dont know but it seems that solid uses only one texture. maybe this post will help?

viewtopic.php?f=7&t=50202
WizardofChaos
Posts: 17
Joined: Tue Jan 11, 2022 8:43 pm

Re: Sampling different material layers for shaders

Post by WizardofChaos »

Seven wrote: Wed Feb 15, 2023 5:22 pm I really dont know but it seems that solid uses only one texture. maybe this post will help?

viewtopic.php?f=7&t=50202
That's hell of weird, because the built-in materials are doing basically what I'm doing (ie EMT_NORMAL_MAP_SOLID)... or they should be. I'll try recompiling the engine with some of the changes in that thread as a last resort, I guess. I feel like that shouldn't be intended behavior, though.
Local man gets so irritated with modern space sims that he starts writing his own
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Sampling different material layers for shaders

Post by CuteAlien »

Bit of a guess, but back when we used HLSL we started with lines like that to set the texture a sampler had to use:

Code: Select all

sampler2D FirstSampler : register(s0);
sampler2D SecondSampler : register(s1);
So registers used had to be set explicitely.
Also in our case we worked with EMT_TRANSPARENT_ALPHA_CHANNEL or the _REF variant.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
WizardofChaos
Posts: 17
Joined: Tue Jan 11, 2022 8:43 pm

Re: Sampling different material layers for shaders

Post by WizardofChaos »

Hmm, that didn't really change all that much. I can only assume that my own code is bad, or that the patch that Seven mentioned is the correct fix. I'll rewrite my own code until I'm *damn* sure it works and report back with any updates... recompiling the engine is a "do that last" kinda thing.
Local man gets so irritated with modern space sims that he starts writing his own
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Sampling different material layers for shaders

Post by CuteAlien »

Wasn't that patch about EMT_SOLID_2_LAYER? You shouldn't need this for bump-maps.
Be careful that you should only use any EMT_NORMAL_MAP variant if your mesh is using S3DVertexTangents (edit: Sorry, turns out this is totally wrong for shaders, never use EMT_NORMAL as base material for those).

And please tell me which Irrlicht version you are currently working with. Trunk and 1.8 are not quite identical when it comes to shader handling, so helps me knowing what you are using to figure this out.

Lastly, I haven't worked yet with setting the state stuff like Filter, AddressU, AddressV. But maybe start as simple as possible - aka not doing that at first. It's probably fine, just helps keeping things at first as simple as possible.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
WizardofChaos
Posts: 17
Joined: Tue Jan 11, 2022 8:43 pm

Re: Sampling different material layers for shaders

Post by WizardofChaos »

Yeah, it's using S3DVertexTangents -- I was originally using a default normal-mapped material (EMT_PARALLAX_MAP_SOLID) and it worked fine. I'm on Irrlicht 1.8.5. I haven't had the chance to debug my shader yet, but I'm gonna give it a pass today.

Actually, since that was such a quick thing I just tried dropping the state stuff and set the baseline material to EMT_NORMAL_MAP_SOLID. It bugged out horribly and made the model invisible and screwed up the skybox, which I am 99% sure means I'm the idiot, lol. I think it's my shader that's wrong.

Code: Select all

struct VertexOutput //holds whatever we change about the vertexes
{
    float4 Position : POSITION0; //position of vertex
    //float4 Color : COLOR0; //color of vertex
    float3 Normal : TEXCOORD0; //normal vector
    float2 TextureCoordinate : TEXCOORD1; //texture coord
    float3 Tangent : TEXCOORD2; //vector pointing *tangential* to the surfice (along it)
    float3 Binormal : TEXCOORD3; //also pointing along the surface, but perpendicular to the tangent
};

struct PixelOutput //holds the final output for pixels
{
    float4 Color : COLOR0;
};

VertexOutput vertexMain(
    in float4 vPosition : POSITION0, //the position of the vertex in model space
    in float4 vNormal : NORMAL0, //the normal associated with the vertex
    in float2 TextureCoordinate : TEXCOORD0,
    in float3 vTangent : TANGENT0,
    in float3 vBinormal : BINORMAL0) //the texture coordinate we're at
{
    VertexOutput output;
    
    output.Position = mul(vPosition, worldViewProj); //shove the vertex into the world from model space
    
    //same op on all three of these: we adjust the normal vectors to reflect the actual world
    output.Normal = normalize(mul(vNormal, worldInverseTranspose));
    output.Tangent = normalize(mul(vTangent, worldInverseTranspose));
    output.Binormal = normalize(mul(vBinormal, worldInverseTranspose));
    
    //pass the texture coordinate along as-is
    output.TextureCoordinate = TextureCoordinate;
    
    return output;
}

texture modelTex; //our model's input texture
sampler2D textureSampler : register(s0); //used to sample the texture at a given position

float bumpConst = 1; //how bumpy the thing is

texture normalTex; //our model's normal map

sampler2D normalSampler : register(s1); //used to sample the norm at a given position

PixelOutput pixelMain(
    VertexOutput vertOut) //take in the output of our vertex shader, since that's where we're at in the pipeline
{
    PixelOutput output;
    
    //create the normalized bump map to see if a thing is... well, bumped at this pixel
    float3 bump = bumpConst * (tex2D(normalSampler, vertOut.TextureCoordinate) - (.5, .5, .5));
    float3 bumpNorm = vertOut.Normal + (bump.x * vertOut.Tangent + bump.y * vertOut.Binormal);
    bumpNorm = normalize(bumpNorm);
    
    //diffuse lighting calculations
    float diffuseIntensity = dot(normalize(DiffuseLightDirection), bumpNorm);
    if (diffuseIntensity < 0)
        diffuseIntensity = 0;
    
    //specular lighting component with the bump normal
    float3 light = normalize(DiffuseLightDirection); //get the normalized direction of the light: should already be normalized but trust nobody
    float3 reflection = normalize(2 * dot(light, bumpNorm) * bumpNorm - light); //determine the reflection (if we have one) based off normals and light directions
    float3 view = normalize(mul(normalize(viewVec), world)); //change how we *view* the model, to see whether the reflection is towards the camera
    
    float4 textureColor = tex2D(textureSampler, vertOut.TextureCoordinate); //sample our texture to get a basic color
    textureColor.a = 1; //set alpha to be opaque
    
    float visibleReflection = dot(reflection, view); //dot product between our view vector and the reflection vector gives us how much of the reflection is visible
    //intensity of our reflection is based off how shiny the object is and how much color is on the vertex
    float4 finalSpec = specIntensity * SpecColor * max(pow(visibleReflection, shiny), 0) * diffuseIntensity; 
    
    //our final pixel color is the texture coordinate * the vertex coordinate * our ambient light and whatever reflective light we have
    output.Color = saturate(textureColor * (diffuseIntensity) + (ambientColor * intensity) + finalSpec);
    
    return output;
}
}
If it helps, here's the shader in question. All the constants are what you think they are. My vertex shader works fine, the vertices and stuff are where they should be (it worked on simpler versions, anyway). I'm new to this, so I was just attempting to re-implement some normal mapping to get a handle on it.
Local man gets so irritated with modern space sims that he starts writing his own
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Sampling different material layers for shaders

Post by CuteAlien »

OK, took me a while figuring it out. I've not worked yet with Irrlicht 1.8.5, but I suppose it's mostly same as Irrlicht trunk here.

First thing: Don't use any of the EMT_NORMAL_x or EMT_PARALLAX_x. They seem to be _only_ useful for the build-in shaders. And the mess you get is likely because as soon as you use those you get some parts of the build-in stuff mixed in.
So what I used was EMT_SOLID (if you need alpha use one of the alpha base materials). And strangely enough despite code looking like it disables second texture for EMT_SOLID - it doesn't and using second texture still worked for me (maybe only does disable for fixed function pipeline and shader does not care?).

And the thing I also missed for a while: tangent and binormal are _not_ passed as TANGENT0 and BINORMAL0, but as TEXCOORD1 and TEXCOORD2. After I found that out I also noticed that's documented in S3DVertexTangents (*sigh*).

So the very simplest shader (basically original d3d9.hlsl just with added variables for tangent, binormal and samplers):

Code: Select all

float4x4 mWorldViewProj; // World * View * Projection transformation
float4x4 mInvWorld;      // Inverted world matrix
float4x4 mTransWorld;    // Transposed world matrix
float3 mLightPos;        // Light position (actually just camera-pos in this case)
float4 mLightColor;      // Light color

// Vertex shader output structure
struct VS_OUTPUT
{
	float4 Position : POSITION;  // vertex position
	float4 Diffuse  : COLOR0;    // vertex diffuse color
	float2 TexCoord : TEXCOORD0; // tex coords
	float3 Tangent   : TEXCOORD1;
	float3 Binormal  : TEXCOORD2;
};

VS_OUTPUT vertexMain(in float4 vPosition : POSITION,
					in float3 vNormal    : NORMAL,
					float2 texCoord      : TEXCOORD0, 
					in float3 vTangent   : TEXCOORD1,
					in float3 vBinormal  : TEXCOORD2
					)
{
	VS_OUTPUT Output;

	Output.Tangent = vTangent;
	Output.Binormal = vBinormal;

	// transform position to clip space
	Output.Position = mul(vPosition, mWorldViewProj);

	// transform normal somehow (NOTE: for the real vertex normal you would use an inverse-transpose world matrix instead of mInvWorld)
	float3 normal = mul(float4(vNormal,0.0), mInvWorld);

	// renormalize normal
	normal = normalize(normal);

	// position in world coordinates (NOTE: not sure why transposed world is used instead of world?)
	float3 worldpos = mul(mTransWorld, vPosition);

	// calculate light vector, vtxpos - lightpos
	float3 lightVector = worldpos - mLightPos;

	// normalize light vector
	lightVector = normalize(lightVector);

	// calculate light color
	float3 tmp = dot(-lightVector, normal);
	tmp = lit(tmp.x, tmp.y, 1.0);

	tmp = mLightColor * tmp.y;
	Output.Diffuse = float4(tmp.x, tmp.y, tmp.z, 0);
	Output.TexCoord = texCoord;

	return Output;
}


// Pixel shader output structure
struct PS_OUTPUT
{
	float4 RGBColor : COLOR0; // Pixel color
};


sampler2D myTexture : register(s0);
sampler2D normalTexture : register(s1);
	
PS_OUTPUT pixelMain(float2 TexCoord : TEXCOORD0,
					float4 Position : POSITION0,
					float4 Diffuse  : COLOR0,
					float3 Tangent  : TEXCOORD1,
					float3 Binormal  : TEXCOORD2
					) 
{
	PS_OUTPUT Output;

	float4 col = tex2D( myTexture, TexCoord );
	float4 col2 = tex2D( normalTexture, TexCoord );

	// just experimenting
//	Output.RGBColor = col * col2;
	Output.RGBColor = col2;
	Output.RGBColor.rgb *= Tangent;
//	Output.RGBColor.rgb = Binormal;

	return Output;
}
Shader callback - the one from Irrlicht shader example works - no need to change anything. Textures don't have to be initialized in there at all (not for D3D anyway, for OpenGL you would have to).
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Sampling different material layers for shaders

Post by CuteAlien »

Btw, it's possible to avoid having to pass tangents, binormals per vertex and instead calculate them directly in the pixel shader.
There's some blog about how to implement this: https://hacksoflife.blogspot.com/2009/1 ... pping.html

I have some implementation for this in our code, with some changes. It's GLSL, but probably easy to convert to HLSL (vec is float and functions are usually identical or have similar names):

Code: Select all

struct STangentSpace
{
	vec3 TangentN;
	vec3 BinormalN;
};

// Calculate a tangent space (mainly to avoid having to pass it per vertex from CPU)
// normal does not have to be normalized.
STangentSpace getTangentSpace(vec3 pos, vec2 texUV, vec3 normal)
{
	STangentSpace ts;

	// Solution from  Benjamin Supnik with Ben Houston adaption: https://hacksoflife.blogspot.com/2009/11/per-pixel-tangent-space-normal-mapping.html
	vec3 q0 = dFdx( pos );
	vec3 q1 = dFdy( pos );
	vec2 st0 = dFdx( texUV );
	vec2 st1 = dFdy( texUV );
	if ( (st0.t == 0 && st1.t == 0) || (st0.s == 0 && st1.s == 0) )
	{
		// Probably identical uv's (bad mesh, but happens)
		st0.t = 1.0;
		st0.s = 0.0;
		st1.t = 0.0;
		st1.s = 1.0;
	}

	ts.TangentN = normalize( q0 * st1.t - q1 * st0.t );
	ts.BinormalN = normalize( -q0 * st1.s + q1 * st0.s );

	// Invert when the UV direction is backwards (from mirrored faces),
	// otherwise it will do the normal mapping backwards.
	bool flipTangent = false;
	vec3 NfromST = cross( ts.TangentN, ts.BinormalN );
	if( dot( NfromST, normal ) < 0.0 )
	{
		flipTangent = !flipTangent;
	}

	// backface polygons also need to switch direction. (not in blog, added by me)
	if ( !gl_FrontFacing )
	{
		flipTangent = !flipTangent;
	}

	if ( flipTangent )
	{
		ts.TangentN *= -1.0;
	}
	else
	{
		ts.BinormalN *= -1.0;
	}

	return ts;
}
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
WizardofChaos
Posts: 17
Joined: Tue Jan 11, 2022 8:43 pm

Re: Sampling different material layers for shaders

Post by WizardofChaos »

CuteAlien wrote: Sat Feb 25, 2023 7:30 pm OK, took me a while figuring it out. I've not worked yet with Irrlicht 1.8.5, but I suppose it's mostly same as Irrlicht trunk here.

First thing: Don't use any of the EMT_NORMAL_x or EMT_PARALLAX_x. They seem to be _only_ useful for the build-in shaders. And the mess you get is likely because as soon as you use those you get some parts of the build-in stuff mixed in.
So what I used was EMT_SOLID (if you need alpha use one of the alpha base materials). And strangely enough despite code looking like it disables second texture for EMT_SOLID - it doesn't and using second texture still worked for me (maybe only does disable for fixed function pipeline and shader does not care?).

And the thing I also missed for a while: tangent and binormal are _not_ passed as TANGENT0 and BINORMAL0, but as TEXCOORD1 and TEXCOORD2. After I found that out I also noticed that's documented in S3DVertexTangents (*sigh*).
This is a HUGE help. Thank you so much for spending some time on this. The tangent / binormal thing alone is extremely helpful. EMT_SOLID bit as well. I'll report back with any updates on whether or not my code works again, but I think it should.
Local man gets so irritated with modern space sims that he starts writing his own
just_in_case
Posts: 1
Joined: Tue Jul 12, 2022 1:27 pm

Re: Sampling different material layers for shaders

Post by just_in_case »

CuteAlien wrote: Sat Feb 25, 2023 7:30 pm OK, took me a while figuring it out. I've not worked yet with Irrlicht 1.8.5, but I suppose it's mostly same as Irrlicht trunk here.

First thing: Don't use any of the EMT_NORMAL_x or EMT_PARALLAX_x. They seem to be _only_ useful for the build-in shaders. And the mess you get is likely because as soon as you use those you get some parts of the build-in stuff mixed in.
So what I used was EMT_SOLID (if you need alpha use one of the alpha base materials). And strangely enough despite code looking like it disables second texture for EMT_SOLID - it doesn't and using second texture still worked for me (maybe only does disable for fixed function pipeline and shader does not care?).

And the thing I also missed for a while: tangent and binormal are _not_ passed as TANGENT0 and BINORMAL0, but as TEXCOORD1 and TEXCOORD2. After I found that out I also noticed that's documented in S3DVertexTangents (*sigh*).

So the very simplest shader (basically original d3d9.hlsl just with added variables for tangent, binormal and samplers):

Code: Select all

float4x4 mWorldViewProj; // World * View * Projection transformation
float4x4 mInvWorld;      // Inverted world matrix
float4x4 mTransWorld;    // Transposed world matrix
float3 mLightPos;        // Light position (actually just camera-pos in this case)
float4 mLightColor;      // Light color

// Vertex shader output structure
struct VS_OUTPUT
{
	float4 Position : POSITION;  // vertex position
	float4 Diffuse  : COLOR0;    // vertex diffuse color
	float2 TexCoord : TEXCOORD0; // tex coords
	float3 Tangent   : TEXCOORD1;
	float3 Binormal  : TEXCOORD2;
};

VS_OUTPUT vertexMain(in float4 vPosition : POSITION,
					in float3 vNormal    : NORMAL,
					float2 texCoord      : TEXCOORD0, 
					in float3 vTangent   : TEXCOORD1,
					in float3 vBinormal  : TEXCOORD2
					)
{
	VS_OUTPUT Output;

	Output.Tangent = vTangent;
	Output.Binormal = vBinormal;

	// transform position to clip space
	Output.Position = mul(vPosition, mWorldViewProj);

	// transform normal somehow (NOTE: for the real vertex normal you would use an inverse-transpose world matrix instead of mInvWorld)
	float3 normal = mul(float4(vNormal,0.0), mInvWorld);

	// renormalize normal
	normal = normalize(normal);

	// position in world coordinates (NOTE: not sure why transposed world is used instead of world?)
	float3 worldpos = mul(mTransWorld, vPosition);

	// calculate light vector, vtxpos - lightpos
	float3 lightVector = worldpos - mLightPos;

	// normalize light vector
	lightVector = normalize(lightVector);

	// calculate light color
	float3 tmp = dot(-lightVector, normal);
	tmp = lit(tmp.x, tmp.y, 1.0);

	tmp = mLightColor * tmp.y;
	Output.Diffuse = float4(tmp.x, tmp.y, tmp.z, 0);
	Output.TexCoord = texCoord;

	return Output;
}


// Pixel shader output structure
struct PS_OUTPUT
{
	float4 RGBColor : COLOR0; // Pixel color
};


sampler2D myTexture : register(s0);
sampler2D normalTexture : register(s1);
	
PS_OUTPUT pixelMain(float2 TexCoord : TEXCOORD0,
					float4 Position : POSITION0,
					float4 Diffuse  : COLOR0,
					float3 Tangent  : TEXCOORD1,
					float3 Binormal  : TEXCOORD2
					) 
{
	PS_OUTPUT Output;

	float4 col = tex2D( myTexture, TexCoord );
	float4 col2 = tex2D( normalTexture, TexCoord );

	// just experimenting
//	Output.RGBColor = col * col2;
	Output.RGBColor = col2;
	Output.RGBColor.rgb *= Tangent;
//	Output.RGBColor.rgb = Binormal;

	return Output;
}
Shader callback - the one from Irrlicht shader example works - no need to change anything. Textures don't have to be initialized in there at all (not for D3D anyway, for OpenGL you would have to).
Thanks a lot for making it clear that Tangent and BiNormals are passed as TEXCOORD1 and TEXCOORD2, I spent a lot of time fiddling with them for my shader in CopperCube and ended up creating my own TBN matrix and use that for the calculations in the shader, here is a bit of code I used to create the TBN matrix, however, I am not sure if it is TBN matrix or something else but my shader still has some issues with separated "UV islands" with normal maps, it shows some faceted artifacts, but works just fine when not using separated UV islands for textures. Maybe it is a tangent space or object space issue I am not sure. I will try to use TEXCOORD1 and TEXCOORD2 to see if the issue gets resolved.

Meanwhile if someone is interested in chcking the TBN code for HLSL , I was using it in Fragment shader

Code: Select all

			float3 N = worldNormal;													
			float3 dT = ddx(vPosition.xyz);												
			float3 dB = ddy(vPosition.xyz);												
			float3 dN = ddx(TexCoord.y)*dB-dT*ddy(TexCoord.y);							        
			float3 Tan = normalize(dN-N*dot(N,dN));									
			float3 Norm = normalize(N);											      
			float3 BiTan = cross(Norm,Tan);												
			float3x3 TBM = float3x3( normalize(-Tan), normalize(BiTan), normalize(Norm) );               									
Also, the incoming world normal is mul(mInvWorld,vNormal).
Post Reply