How to generate and send cubemap in shader correctly?

[Andrey01]

I am making an omnidirectional shadow mapping for one point light so far. That requires creating a cube shadow map whose texture faces will save depth value of each fragment in each cube direction (+X, +Y, +Z, -X, -Y, -Z). As I understand the cube map support in Irrllicht is relatively a new feature (present only in 1.9) and I'm unsure whether it works stably as there's even some method 'addRenderTargetTextureCubeMap' about which the comment in IVideoDriver.h reports: "NOTE: Only supported on D3D9 so far".

So my problem is the samplerCube in the fragment shader is like empty in my case (I tried to sample from it and pass the color into gl_FragColor, it just outputs always a black color). However, I tried to draw2DImage() each texture from shadow_cubemap array, it looks as expected.

What I do:
1. create 6 render target textures of video::ECF_A16B16G16R16F color format, cube surfaces and cameras:

Code: Select all

video::ITexture* shadow_t = nullptr;

core::array<video::ITexture*> shadow_cubemap(6);
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide1RT", video::ECF_A16B16G16R16F));
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide2RT", video::ECF_A16B16G16R16F));
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide3RT", video::ECF_A16B16G16R16F));
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide4RT", video::ECF_A16B16G16R16F));
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide5RT", video::ECF_A16B16G16R16F));
shadow_cubemap.push_back(vdrv->addRenderTargetTexture(wnd_size, "ShadowSide6RT", video::ECF_A16B16G16R16F));

core::array<video::E_CUBE_SURFACE> shadow_cubemap_surfaces(6);
shadow_cubemap_surfaces.push_back(video::ECS_POSX);
shadow_cubemap_surfaces.push_back(video::ECS_POSY);
shadow_cubemap_surfaces.push_back(video::ECS_POSZ);
shadow_cubemap_surfaces.push_back(video::ECS_NEGX);
shadow_cubemap_surfaces.push_back(video::ECS_NEGY);
shadow_cubemap_surfaces.push_back(video::ECS_NEGZ);

scene::ICameraSceneNode* shadow_cams[] = {
createCameraForCubemap(lights[0]->pos, video::ECS_POSX),
createCameraForCubemap(lights[0]->pos, video::ECS_POSY),
createCameraForCubemap(lights[0]->pos, video::ECS_POSZ),
createCameraForCubemap(lights[0]->pos, video::ECS_NEGX),
createCameraForCubemap(lights[0]->pos, video::ECS_NEGY),
createCameraForCubemap(lights[0]->pos, video::ECS_NEGZ)
};

2.set a render target, set shadow material for the only meshnode in the scene to render only a depth and switch cameras in while loop:

Code: Select all

// in while loop:
rt->setTexture(shadow_cubemap, depth_t, shadow_cubemap_surfaces);
vdrv->setRenderTargetEx(rt, video::ECBF_COLOR | video::ECBF_DEPTH, video::SColor(255, 0, 0, 0));
table->setMaterialType((video::E_MATERIAL_TYPE)shadow_mat);
skybox->setVisible(false);   // skybox shouldn't be rendered to the shadow map as it is not shadowed

for (u16 i = 0; i < 6; i++)
{
	smgr->setActiveCamera(shadow_cams[i]);
        smgr->drawAll();
}

Then generte the cube map. I convert the shadow textures to video::IImage type to pass it in 'addTextureCubemap'. Afterwards I send it to the main fragment shader of 'lighting_mat' material as samplerCube type:

Code: Select all

// in while loop:
shadow_t = vdrv->addTextureCubemap(
"ShadowRT",
convertTextureToImage(shadow_cubemap[0]),
convertTextureToImage(shadow_cubemap[1]),
convertTextureToImage(shadow_cubemap[2]),
convertTextureToImage(shadow_cubemap[3]),
convertTextureToImage(shadow_cubemap[4]),
convertTextureToImage(shadow_cubemap[5])
);

vdrv->setRenderTargetEx(0, 0);
smgr->setActiveCamera(camera);   // set main camera
table->setMaterialType((video::E_MATERIAL_TYPE)lighting_mat);
table->setMaterialTexture(2, shadow_t);  // set as second layer the resulted shadow map that will be sent to 'light_mat' shader
skybox->setVisible(true);

smgr->drawAll();

How 'convertTextureToImage' function looks like:

Code: Select all

video::IImage* convertTextureToImage(video::ITexture* tex)
{
    video::IVideoDriver* vdrv = device->getVideoDriver();
    video::IImage* img = vdrv->createImageFromData(tex->getColorFormat(), tex->getSize(), tex->lock(), false);

    tex->unlock();

    return img;
}

Also in the constant setter callback I pass the shadow map to the shader:

Code: Select all

s32 base_tex = 0;
s32 normaltex_id = 1;
s32 depthtex_id = 2;

services->setPixelShaderConstant("mBaseTex", &base_tex, 1);
services->setPixelShaderConstant("mNormalTex", &normaltex_id, 1);
services->setPixelShaderConstant("mDepthTex", &depthtex_id, 1);  // uniform samplerCube mDepthTex in the fragment shader

Other problem is as I re-generate the cube map texture each time in the loop using addTextureCubemap(), that decreases FPS much. I suppose it happens of the progressive memory leak, but when I try to shadow_t->drop() after rendering the program just crashes. Is there any other way to generate and send cube maps?

[CuteAlien]

First - OpenGL can do cubemaps as well by now. I just forgot to remove that comment - it's gone now ;-) Actually it's even slightly better there because on OpenGL you can filter over cubemap borders (this single difference made me switch to OpenGL in a rather large project).

Also not sure if you noticed, but example 28 shows a bit how to use cubemaps. dynamicCubeMapRTT in there works maybe like what you want.

Maybe you can directly use the cubemap as rendertarget? You can do that in Irrlicht svn trunk. You create the rendertarget before loop with: rt = IVideoDriver::addRenderTarget(). Then you create cubeRTT = IVideoDriver::addRenderTargetTextureCubemap also before main loop. Then in rendering you can do rt->setTexture(cubeRTT, someDepthStencilTexture, target_cube_surface_from_E_CUBE_SURFACE). That should render directly into one side of the cubemap texture depending on the E_CUBE_SURFACE you pass.

Even if you need to render to other textures you should avoid ever sending textures back to CPU. So if you need a non-cubemap as your rendertarget for some reason (other shaders etc), then you can probably still use ScreenQuad (example 27). So render to your other texture. Then put the cubemap as rendertarget in a screenquad and render from other texture to ScreenQuad with CubeMapRTT.

As for the problem in your code above... I'm not that good that I can see what goes wrong from there. I probably need a full example for this so I can debug around a bit.

[Andrey01]

Thanks. That has actually fixed the problem. Like addTextureCubemap() can not accept render target textures, only ordinary ones or I converted to video::IImage's incorrectly. I took the example of the convertation from here: viewtopic.php?f=9&t=22268