Stencil buffer + Transparent materials = Screen goes black

[dart_theg]

Title. When I look/load in a stencil buffer shadow volume, my screen goes black. When I look away, my game goes back to normal. I'm on Irr 1.8.5.

[CuteAlien]

Stencil shadows can be tricky. First - experiment with the settings - especially zfail vs zpass in this case. Also in 1.8 directional light didn't work with shadows - only point lights.

Do you get the error also in the Example 08 Special effects? And which driver?

edit: I missed the transparent materials part (it's late...). Tried a quick test using a transparent material type, but that wasn't enough I guess. I think in theory it should simply consider this as solid. Maybe try if you set the material type or your model to EMT_SOLID you get same or different results.

[dart_theg]

Yeah, when it's not transparent, the issue does not occur. If it's even like vertex alpha type etc, issue. Also to clarify, I'm using no lights in this scene. I've made a custom Shadow Volume that acts as basically "Fill anything in this mesh with a shadow". (Trying to mimic old DS-y aesthetics). It works, but not when there is an alpha material floating around. These objects are still registered with SHADOW, so they render at the same time as the normal Irrlicht ones would.

Here is the code for it, if perhaps the issue lies in my implementation: https://pastebin.com/MBU1nBE1

And currently, this is how I render my project:

ShadowVolumeSceneNode::DrawingThisFrame = false;

i_driver->beginScene(true, true, irr::video::SColor());
i_smgr->drawAll();

if (doStencilBuffer && ShadowVolumeSceneNode::DrawingThisFrame) i_driver->drawStencilShadow(false);

guiManager->Render();

i_driver->endScene();

[CuteAlien]

OK, just had time for a quick look now. And I think the problem is that drawStencilShadowVolume doesn't do what I thought it would do. When I proposed that in last thread I assumed it would simply write the triangles to the stencil-buffer, but that doesn't seem to be the case. I realize now that function is specific setup for Shadow volumes. And Shadow volumes are not just meshes - they are planes around the outside of your mesh which extends the mesh away from the light. Bit like putting a very long blanket over your mesh and then the sun is blowing it away from itself with super strong wind. And the function doesn't simply draw them into the stencil buffer but it's doing some kinda of counting - so depending on front/backfaces it draws it sets/clears the stencil-buffer repeatingly.

Sorry about that. I saw no other function about writing into stencil buffer and assumed it would be that one. Sadly turns out - Irrlicht doesn't support stencil-buffer at all otherwise :-( Seems it only supports flipping color buffers. So the only way to currently do that is using native functions (opengl/d3d) in your node. Which I've never done with d3d. When using OpenGL I tend to use the "glad" library (thought for this case pure opengl headers would probably work).

Native gl functions needed to write your mesh into the stencil buffer should be glEnable/glDisable(GL_STENCIL_TEST) and some setting for glStencilMask. I've not done that myself, so currently I don't know more about it. And I assume D3D has something similar.

Without using native drivers Irrlicht would need another flag or two in SMaterial similar to ColorMask and ZBuffer flags.

Sorry, real bummer, I didn't expect this.

edit: Also maybe there are other solutions involving rendertarget textures and doing multiple stages or so. Or you can use the shadows in Irrlicht by using Irrlicht's shadow-volume node and enabling specific lights with a lightmanager before the shadow stage which are directly above the head of your characters (so they don't light other stuff and are only used for this).

[dart_theg]

Hm. I mean, the shadows work as long as I don't have objects that render alpha stuff...

I just moved a plane that has transparency across the map, kept my stencil shadow volume, and I could look at both of them individually but the middle where both would load/draw the black screen would come back. I wonder if I can somehow patch this myself...

Thank you for diving into this!

[dart_theg]

I changed the register type to TRANSPARENT_EFFECT and it works, but the shadows are pitch black. Not the intention but maybe right direction?

[dart_theg]

Ok I got it working. Set it to TRANSPARENT_EFFECT then set the shadow color in the drawStencilShadow func instead of the drawStencilShadowVolume one. Woohoo!

[CuteAlien]

Yeah - as long as the polygons-crossed count is ok it might work. Maybe transparency messing with the stencil count, not sure. But glad to see you have something which works for you :-)