xEffects and Fog

[dart_theg]

I'm using xEffects for my project and it's been working alright (need to change directional lights to actually be directional, but I think editing the shader code will fix this easily*), but I noticed fog in Irrlicht doesn't work which makes sense. Is there an alternative to fog that works with xEffects? I haven't found any online.

[CuteAlien]

You will probably have to adapt the main pixel shaders (SHADOW_PASS_2P in EffectShaders.h). I guess shadow effect would be somewhere between where finalCol is set and where it's applied to gl_FragColor (for glsl, similar for hlsl).

Sorry, don't know more about it yet (I just hunted for it a bit right now, I didn't even know that this shader was in a header file).

[dart_theg]

Yeah, I've been digging through the header file for a bit now because I want to add the ability for a proper directional light (I think I'm on to something but not really sure). I have zero shader knowledge but I can vaguely pick up how this is working...

[CuteAlien]

Ugh, first shader experience with a shader coded inside a c++ header file. Don't get discouraged, usually shaders are more fun (pure text files and you can reload them at runtime).
If you don't have to care about older c++ compilers, maybe at least replace it with a raw string literal. That should make it more readable.

[dart_theg]

Gotcha. I've been digging into the code for the lighting and I find it interesting that it actually projects the shadows uniformly but the lighting is still a "point" light, where the light doesn't project orthographically and does still take into account the distance from the light. I hope to resolve this but I've put this on the backburner as I'd rather find time to hone in on shaders in general soon, unless there's some resolve for this somewhere? I find it hard to believe xEffects was used in its current state because there are some bizarre design decisions after using it for some weeks.

[Elevations]

I tried xeffects and i also faced the same problem where the light is actually just a "spotlight", if i was actually going to use this i would need actual point lights that omit in all directions. I agree with what your saying dart_theg that it is hard to believe that people managed to find uses for Xeffects without some serious tweaking to it.

[CuteAlien]

Xeffect is maybe more of a starting point. I haven't used it myself yet, but I do work in my day job with code based on something Ahmed (the XEffect author) wrote which is very similar. Thought he no longer put the shader in the header then.

As long as you don't need shadows, then point-light is relatively easy to add in shaders. If you need shadows it's tricky the way Xeffect works as shadowmap algorithm used in there needs a light direction. It renders the scene from the point of each light and then uses the resulting textures in the real scene render to calculate which parts are inside a shadow.

[dart_theg]

I'm trying to look into how Irrlicht handles fog, and excluding nodes from being affected by fog because I'd like to share this system, but in xEffects. Any ideas on how this can be achieved or where in the source code for Irrlicht this logic can be found? I'm having trouble finding it.

[CuteAlien]

In Irrlicht the ::setFog functions in the drivers do enable fog. It's a screen-wide effect for the fixed function pipeline, so likely no effect on shaders.
Although some shaders in Irrlicht try to do something about it, but I'm not sure about the details either (I never learned shader assembly and internal shaders are still written in that). Irrlicht creates 4 different shaders internally for different fog modes. And then depending on the active settings of the fog in the fixed function pipeline it switches between those shaders. Not sure if that's a good idea really.

I'd just ignore whatever settings fixed function pipeline has when writing my own shader. It's your shader - you can pass any variables to it you want.

[dart_theg]

Gotcha. If I make my own fog shader, I could perhaps pass it as a post processing effect or how would I incorporate this? I suppose once everything is rendered, then it uses the depth of each pixel to determine a fog color?

[CuteAlien]

I wouldn't use post processing, especially if you want to exclude some objects. Should be in the usual pixel shader. So basically one of the last lines you modify color once more based on distance.

Thought getting the distance in the pixel shader is a bit tricky actually. It's easy to access the depth-buffer value (gl_FragCoord.z in glsl), but that isn't linear (so nearly everything except stuff very close to the camera will probably be near value 1). As a quick hack for a first test you can multiply it with itself a few times (or use whatever other math you want on it) to get something like:
gl_FragColor.rgb = mix(gl_FragColor.rgb, vec3(0.7, 0.7, 0.7), gl_FragCoord.z*gl_FragCoord.z*gl_FragCoord.z*gl_FragCoord.z*gl_FragCoord.z*gl_FragCoord.z);
Probably really bad way to do fog, but just to give you a quick first idea to try (it mixes the original color with some gray value and multiplying the value with itself straightens it out a bit in a way which will make math guys bite into the table).

To get a better (linear) distance you either calculate the distance to the camera in the vertex shader and pass it on to the fragment shader or you will need to pass near/far values of the camera to the fragment shader (as uniforms) and linearize the depth buffer (you'll find the formula for that online).

Thought if you already do post-processing and have a depth-buffer + render to texture anyway - I suppose you can also do it as post-processing effect. Irrlicht svn actually got an example a few months ago for how to do post-processsing shaders (example 27.PostProcessing).