Irrlicht Engine

The irrEffects pixel shader renderer works well, but at low angles to lights the edges of the shadow flicker even when smoothed.

I guess the steep depth map changes on oblique angles might be the reason, with the tiniest camera movement causing a large change in the shadow depth map.

Has anyone any ideas how this can be improved, or what the real cause is?

XEffects I guess. Not using XEffects myself, but was using near identical shader also written by Ahmed for a while.

I suppose you tried already to increase shadow map resolution?

Maybe check also if it's used well - I had that sometimes in the past that the render in the shadowmap wasn't filling it as well as it could (aka camera could have been closer).

Might help playing with constants in variance calculation (that stuff: float variance = min(max(E_x2 - Ex_2, 0.00001) + 0.000001, 1.0);\n")

You want a good resolution, so make sure near-plane is not too small.

All hints not very specific to this problem :-(

It's great chatting with you about these issues, you really know your stuff!
Yeah, shadow camera needs to be well focused on the required zone only, suitable resolution, etc., but I'm using a dynamic shadow-light camera positioning trick that tracks it in behind the view cam keeping the same direction of light, but zooming in for better close-up details, and of course I guess its the movement of the shadow-camera that is causing flickering shadow edges on those pixels with higher slope... if you follow me.
Guess there isn't an easy solution is there?

Sorry, no immediate idea. I should create a test-case with XEffects for myself I guess.

I improved the shader math and it's fixed.

Was the floats getting to close to zero.

Oh, cool. Can you post your fix here? Just in case someone else runs into this.

I could but it wouldn't be much use without the other 35 k lines of code that make up the vertex-based height map, shadow, lighting and bump mapping system. It's sorta got really complicated, as the shader has to read a height map to elevate the vertices for the terrain, then calculate the X, Z slope to calculate the normal, then rotate it to match the mesh normal, which might be a plane, a morphed-sphere plane, or a sphere, then apply the bump map, which again has to be rotated to match the default normal, then applied as a variation to the previously rotated height mesh... amazing that a cheap GPU can handle all those mults!

Ah OK - I thought it was just some simple bug in XEffects shader.

XEffects shader doesn't handle normals calculation in terrain vertex height processing, which was where my issue was. Its surprising how this can get exponentially complicated as you add to the code. My pixel shader is 70K when compiled now, takes several minutes to compile to, so I keep the compiled output to avoid the compile each load.

Oh wow, how do you even debug a shader when it gets that large.

Trial and error, quite painful at times!