RTT alpha channel blended incorrectly

[DtD]

I recently changed my GUI system from rendering the windows straight to the screen buffer to rendering. However, it appears that the alpha channel is not getting blended correctly.

Rendered to an RTT:

(The effect is very obvious where the shadow renders about the red "ribbon")

Rendered as normal to the screen buffer:


Each window is its own texture which is rendered by webkit. Here is the exported RTT alone: {link}

This is how the children windows are rendered:

Code: Select all

   driver->setRenderTarget(rtt_layer);
   irr::gui::IGUIElement::draw();//Draw children
   driver->setRenderTarget(0,false,false);

The actual windows are rendered like this:

Code: Select all

Environment->getVideoDriver()->draw2DImage(texture,drawBorders.offsetVector(AbsoluteRect.UpperLeftCorner),
     core::recti(0,0,getSize().Width,getSize().Height),0,video::SColor(255,255,255,255),true);

(I've also tried rendering it as an on-screen quad -- same results)

I'm using Irrlicht 1.7 (so not SVN) and this happens in both OpenGL and DirectX.

Any help is very much appreciated.

~Pathogen David

[BlindSide]

Hey that's a very sexy looking gui.

So is each "window" in it's own RTT or is the whole webkit environment in a single RTT? I am confused because your exported RTT contains the whole thing.

It's probably a good thing that it happens both in OGL and D3D9, atleast it's less likely that it's some nasty driver bug that's impossible to fix.

Aside from that I am just as confused as you are, the only sure way to get closer to a solution is to make a very minimal test application which demonstrates this problem, without webkit or the game code.

[hybrid]

It looks like some windows have fragements of the large font on their edges. So I guess that you either render wrong things into the RTT, or you don't clean them properly. You should be able to get the exactly same effects, though, from a technical point of view. Just a matter to find the correct settings.

[BlindSide]

Yes I noticed that too but it seems those fragments of text are what's supposed to be behind the red thing? So it's blending with the background but not with the red thing?

I think I understand the problem a little now. Even though the gray window blends correctly with the red thing in the RTT, it outputs an alpha value < 255 and hence gets blended again with the background, when it's SUPPOSED to return 255 alpha. This is assuming the red thing and the gray windows are all in their own gigantic RTT. Does that sort of make sense?

Again it's possible thats the entire issue here. The black borders fade out when the object is rendered to the RTT, and then fade out again when the RTT is rendered, because it's rendering transparent values to the RTT. I suppose if you set the UseAlpha param of the draw2DImage everything looks correct (But you can't see the background...)?

[Mel]

The problem is with the blending of the alpha-channel. The current implementation of the 2D drawing functions discards any alpha channel that were before rendering a new pixel in the alpha channel. But to correctly draw the alpha channel, it should be added instead to the previous value of the alpha channel, like this:



I think it is posible to add the values of the alpha channels while the color channels can be blended like usual.

[DtD]

@Blindside
Thanks, we're aiming for it to look nice.

Basically, Webkit gives me a pointer to the memory with the page rendered. That would be one single window. I think take this and memcpy it to a locked IImage (for an internal buffer for partial redraw regions) and then memcpy that IImage to a locked texture. So each window is a texture and then there is one RTT that they are all drawn to.

I will prepare a test application later today.

@hybrid
That is the underlying scene leaking through, our little test world has our logo on it, basically what blindside's second post says.


Mel is pretty much spot-on with his illustration. However, I've tried rendering 3D quads instead, so I don't think it is isolated just to the 2D drawing functions.

Code: Select all

     video::IVideoDriver* driver = Environment->getVideoDriver();
     core::recti oldViewport = driver->getViewPort();
     core::matrix4 oldViewMat = driver->getTransform(irr::video::ETS_VIEW);
     core::matrix4 oldProjMat = driver->getTransform(irr::video::ETS_PROJECTION);
     
     driver->setViewPort(core::recti(drawBorders.offsetVector(AbsoluteRect.UpperLeftCorner),dragResizeSize));
     driver->setTransform(irr::video::ETS_VIEW,core::matrix4());
     driver->setTransform(irr::video::ETS_PROJECTION,core::matrix4());
     driver->setTransform(irr::video::ETS_WORLD,core::matrix4());
     
     irr::video::SMaterial mat;
     mat.BackfaceCulling=true;
     mat.Lighting=false;
     mat.MaterialType=irr::video::EMT_TRANSPARENT_ALPHA_CHANNEL;
     mat.TextureLayer[0].Texture=texture;
     driver->setMaterial(mat);
     
     driver->drawMeshBuffer(&ares->getScreenMesh());
     
     driver->setViewPort(oldViewport);
     driver->setTransform(irr::video::ETS_VIEW,oldViewMat);
     driver->setTransform(irr::video::ETS_PROJECTION,oldProjMat);

Like I said, I'll get a small test program a bit later today.

[DtD]

Ok, made a small demo to replicate the problem:
Download link (With images and Win32 binary)

You can press R to see what it should be rendered without using the RTT.

Code: Select all

//RTT Alpha Test for Irrlicht
#include <irrlicht.h>

class InputManager : public irr::IEventReceiver
{
 public:
  bool renderToRtt;
  InputManager() : renderToRtt(true) {}
  virtual bool OnEvent(const irr::SEvent& event)
  {
   if (event.EventType==irr::EET_KEY_INPUT_EVENT && event.KeyInput.Key==irr::KEY_KEY_R) {renderToRtt = !event.KeyInput.PressedDown;}
   return false;
  }
};

int main(int argc,char* argv[])
{
 InputManager* input = new InputManager();
 irr::IrrlichtDevice* device = irr::createDevice(irr::video::EDT_DIRECT3D9,irr::core::dimension2du(512,512),32,false,false,false,input);
 irr::video::IVideoDriver* driver = device->getVideoDriver();
 
 irr::video::ITexture* rtt = driver->addRenderTargetTexture(driver->getScreenSize(),"rt",irr::video::ECF_A8R8G8B8);
 
 irr::video::ITexture* bg = driver->getTexture("pattern.png");
 irr::video::ITexture* one = driver->getTexture("redfade.png");
 irr::video::ITexture* two = driver->getTexture("bluefade.png");
 
 //Make the screen mesh
 irr::scene::SMeshBuffer screenMesh;
 screenMesh.Vertices.set_used(4);
 screenMesh.Indices.set_used(6);
 
 screenMesh.Vertices[0]=irr::video::S3DVertex(-1.f,-1.f,0.f,0.f,0.f,1.f,irr::video::SColor(255,255,255,255),0.f,1.f);//Bottom Left
 screenMesh.Vertices[1]=irr::video::S3DVertex(-1.f, 1.f,0.f,0.f,0.f,1.f,irr::video::SColor(255,255,255,255),0.f,0.f);//Top Left
 screenMesh.Vertices[2]=irr::video::S3DVertex( 1.f, 1.f,0.f,0.f,0.f,1.f,irr::video::SColor(255,255,255,255),1.f,0.f);//Top Right
 screenMesh.Vertices[3]=irr::video::S3DVertex( 1.f,-1.f,0.f,0.f,0.f,1.f,irr::video::SColor(255,255,255,255),1.f,1.f);//Bottom Right
 
 screenMesh.Indices[0]=0;
 screenMesh.Indices[1]=1;
 screenMesh.Indices[2]=2;
 screenMesh.Indices[3]=2;
 screenMesh.Indices[4]=3;
 screenMesh.Indices[5]=0;
 //-----------------------------
 
 while(device->run())
 {
  driver->beginScene(true,true,irr::video::SColor(255,255,0,255));
  
  driver->draw2DImage(bg,irr::core::vector2di(0,0));
  
  if (input->renderToRtt) {driver->setRenderTarget(rtt);}
  
  driver->draw2DImage(two,irr::core::vector2di(150,115) ,irr::core::recti(0,0,250,100),0,irr::video::SColor(255,255,255,255),true);
  driver->draw2DImage(one,irr::core::vector2di(300,100),irr::core::recti(0,0,100,250),0,irr::video::SColor(255,255,255,255),true);
  
  if (input->renderToRtt)
  {
   driver->setRenderTarget(0,false,false);
   
   //Draw the RTT on the screen:
   irr::core::recti oldViewport = driver->getViewPort();
   irr::core::matrix4 oldViewMat = driver->getTransform(irr::video::ETS_VIEW);
   irr::core::matrix4 oldProjMat = driver->getTransform(irr::video::ETS_PROJECTION);
   
   driver->setViewPort(irr::core::recti(irr::core::vector2di(0,0),(irr::core::dimension2di)driver->getScreenSize()));
   driver->setTransform(irr::video::ETS_VIEW,irr::core::matrix4());
   driver->setTransform(irr::video::ETS_PROJECTION,irr::core::matrix4());
   driver->setTransform(irr::video::ETS_WORLD,irr::core::matrix4());
   
   irr::video::SMaterial mat;
   mat.BackfaceCulling=true;
   mat.Lighting=false;
   mat.MaterialType=irr::video::EMT_TRANSPARENT_ALPHA_CHANNEL;
   mat.TextureLayer[0].Texture=rtt;
   mat.TextureLayer[0].TextureWrapU=irr::video::ETC_CLAMP_TO_EDGE;
   mat.TextureLayer[0].TextureWrapV=irr::video::ETC_CLAMP_TO_EDGE;
   mat.TextureLayer[0].BilinearFilter=false;
   driver->setMaterial(mat);
   
   driver->drawMeshBuffer(&screenMesh);
   
   driver->setViewPort(oldViewport);
   driver->setTransform(irr::video::ETS_VIEW,oldViewMat);
   driver->setTransform(irr::video::ETS_PROJECTION,oldProjMat);
  }
  
  driver->endScene();
 }
 
 device->drop();
 
 return 0;
}

EDIT: I was playing around with the source for the DirectX and OpenGL setRenderStates2DMode functions, I got some amusing results but nothing desirable. (I messed around with like 2830 of COpenGLDriver.cpp the most)

[stuey]

Hi, this sounds very much like the problem that had me banging my head against the wall in some non-irrlicht DX9 rendering engine (based on HGE) I've been tinkering with.

Everything was fine until I decided I needed the extra functionality of rendering to textures with alpha - a common practice for people implementing fancy GUI's.

The problem was that the alpha in the render target texture was not ending up as expected, especially where I'd blended multiple overlays. This showed itself when drawing on the backbuffer, as the alpha in the texture was being blown away leaving holes and transparent areas where they shouldn't have been so you could see the background coming through - exactly like your image where the background comes through the ribbon where the window shadow is.

The problem is to do with the way the hardware is handling the alpha, it does it cheap and nasty mixing the colour in the same proportions as it does the alpha - do a little research and you'll see it's an age old problem.

To solve this problem you can use premultiplied alpha, there are several ways to implement this.

1. Lock your source & render-target surfaces, and implement your own alpha correct blitting, pixel by pixel.

2. Get the hardware to do it for you. In DX you can set the render states/stages in the fixed function pipeline to do it or use pixel shaders. Either way will depend on the hardware CAPS a little, but it's not asking too much.

Note: you don't have to pre-process your images to make them pre-multiplied alpha, you can do this on the fly.

Oh, and you also need different render/blend states when you draw your render-target texture on the backbuffer. You'll know this when you get your head around pre-multiplied alpha.

I remember I also had a halo problem around massively enlarged images rendered to the RTT, but I eventualy solved this too.

In all, I had quite a bit of unexpected extra work to do to get RTT with alpha working so that the results were the same as they would have been if everything was always drawn to the backbuffer - but it was worth it.

[DtD]

@stuey
Thanks for the info. I was going to try fixing it with shaders a bit later, but I was hoping it could be fixed at the driver level. I will try some of your suggestions, although it would be nice if there was a better way to do it >.<

~David