Convert Image to IrrlichtTexture

If you are a new Irrlicht Engine user, and have a newbie-question, this is the forum for you. You may also post general programming questions here.
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Convert Image to IrrlichtTexture

Post by Siebenschläfer »

Hello,

I am using OpenCV to capture a video and convert it to an irrlicht texture to be able to display it. The nightsoft guys have a pretty code that shows how to do it. But it gets awefull slow for me with a resolution higher than 352x288, so thats nothing!

code is:

Code: Select all

void CameraInput::repaint( IplImage* frame )
{	
	// read the pixels directly into the texture
	char* texture = (char*)( irr_texture->lock() );
	char* ardata = (char*) frame->imageData;
	char* final_loc = frame->imageSize + frame->imageData;

	switch( irr_texture->getColorFormat() )
	{
		case ECF_R8G8B8:
			while( ardata < final_loc )
			{
				*texture = *ardata;
				texture++; ardata++;
				*texture = *ardata;
				texture++; ardata++;
				*texture = *ardata;
				texture++; ardata++;
			}
			break;
			
		case ECF_A8R8G8B8:
			while( ardata < final_loc )
			{
				// R
				*texture = *ardata;
				texture++; ardata++;
				// G
				*texture = *ardata;
				texture++; ardata++;
				// B
				*texture = *ardata;
				texture++; ardata++;			
				// A: skip 
				texture++;

				//cout<<"rgba"<<endl;
			}
			break;
	}


	irr_texture->unlock();
}
I also tried with ARToolKit, but still it is very slow. Has anyone used video capture and found a way to optimize the converting process?

Thanks
Think Tannenzäpfle!
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

any idea?
Think Tannenzäpfle!
Katsankat
Posts: 178
Joined: Sun Mar 12, 2006 4:15 am
Contact:

Post by Katsankat »

Hi, the conversion in itself looks good. Remove the switch, but this won't help much.

However as I know OpenCV quite a bit I'm afraid you can't get better performances unless with better hardware.

The slowdown could come from the rest of the code : are you writing an avi file simulataneously? It takes cycles to capture, convert, display, save at 30 frames per second... Not to mention irrlicht running, too. Hard to say more without knowing what is doing the application internally.
Nox
Posts: 304
Joined: Wed Jan 14, 2009 6:23 pm

Post by Nox »

first guess: use memcpy not a while !!!
second guess: do you use a dynamic/rendertarget-texture? If not change this.
third guess: may be two textures speed this up (write at first, show second, write at second show first).
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

ok, thank you a lot, I'll try that!
Think Tannenzäpfle!
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

hm,

first:
memcpy( &texture, &ardata, frame->height*frame->width*frame->depth );

only returns an access violation.

second:
no I don't because from opencv I don't receive power2-images...

third:
how do you mean using 2 textures? within the program all commands are executed sequential or not?!

I thought about sending the image-data to a shader and returning it as an Irrlicht texture... do you tink that's sensefull?
Think Tannenzäpfle!
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

ok, maybe this can be moved to snippets because I see that a couple of people tried to convert OpenCV Images to Irrlicht Textures, too, all the same way. Also I don't want to push-up the thread...

to first again:

I had to convert the OpenCV Image from BGR to BGRA to get it convertible with the texture. Also I had to fill the alpha channel with 255 because it is 0.
Then I can use the memcpy:

Code: Select all

void CameraManager::convertImageToTexture( IplImage* frame, ITexture* videoTexture )
{	
	IplImage* newFrame = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
	cvCvtColor( frame, newFrame, CV_BGR2BGRA );

	IplImage* mask = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
	cvSet( mask, cvScalar( 0,0,0,255 ) );

	cvAdd( newFrame, mask, newFrame );

	// read the pixels directly into the texture
	char* texture = (char*)( videoTexture->lock() );
	char* ardata = (char*) newFrame->imageData;
	char* final_loc = newFrame->imageSize + newFrame->imageData;

	memcpy( texture, ardata, newFrame->width*frame->height*newFrame->nChannels );	
            videoTexture->unlock();

            cvReleaseImage( &mask );
	cvReleaseImage( &newFrame );
}
But unfortunatly the result has the same FPS number than the code on top, even without all the conversions it's only 1 FPS difference. The reason is the lock() and unlock() - these function kill the performance!


I gonna try the second and the third thought now. Also I would try it with shader if someone could give me hints how to deliver the char* arrays to it and how to receive not a material but a texture.
Think Tannenzäpfle!
Nox
Posts: 304
Joined: Wed Jan 14, 2009 6:23 pm

Post by Nox »

lock/unlock will be faster if you use a rendertarget/dynamic texture.
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

oh yes, you are right, with an RTT it's really much faster. But I tested it and the while loop is totally as fast as the memcpy, it brings no difference!
Think Tannenzäpfle!
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

What I like to try right now is to deliver the imageData of the OpenCV Image to the shader to be returned as texture? Is that possible?

What I try is:

Code: Select all

main:
        ShadManager->convertTexture( conversionNode );

ShadManager:
        node->setMaterialType( ( E_MATERIAL_TYPE )m_ConversionMaterial ); 

ShadCallback:
        		IplImage* src = camera->getCurrentUndistortedFrameLeft();

		CvMat buffer = cvMat( src->height, src->width, CV_8U, src->imageData );

		cvNamedWindow( "lalala", 1 );
		cvShowImage( "lalala", &buffer );

		videoTexture = buffer.data.fl;
		
		services->setPixelShaderConstant("VideoTexture", (float*)videoTexture, src->height * src->width * 3 );

and the frag shader:

Code: Select all

uniform sampler2D VideoTexture;

varying vec2 TexCoord;
	
	void main()
	{		
		vec4 lalala;
		

		vec4 colorVID = texture2D(VideoTexture,lalala.xy).xyz; 

		colorVID.a = 255.0;
		
				
		gl_FragColor = colorVID;
	}

so thats the idea, but how can I get this working?
Think Tannenzäpfle!
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Post by hybrid »

No, you cannot pass texture data as a parameter. You have to create an actual ITexture in Irrlicht (which then copies the data to the GPU) and set the texture inside the material. Then your shader can work on the texture data. Instead of creating new textures all the time you can also lock the texture, upon unlocking the new content will be copied to the GPU.
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

but I have to fill the texture every loop again with the content of the image data, how should I avoid that?
Think Tannenzäpfle!
hybrid
Admin
Posts: 14143
Joined: Wed Apr 19, 2006 9:20 pm
Location: Oldenburg(Oldb), Germany
Contact:

Post by hybrid »

You can't. At least unless you cannot generate the data on the GPU directly. Even if your GPU was allowed to access the data in RAM it wouldn't be much faster than locking and unlocking the texture. PBOs would avoid one copy, because you could specify that you only want to write the data, not read. But that's a very specific case and I don't know how much trouble the FBO preparation is when locking the data, but not reading from it.
sudi
Posts: 1686
Joined: Fri Aug 26, 2005 8:38 pm

Post by sudi »

I see that u use

Code: Select all

IplImage* newFrame = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
   cvCvtColor( frame, newFrame, CV_BGR2BGRA );

   IplImage* mask = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
   cvSet( mask, cvScalar( 0,0,0,255 ) ); 
every frame. Wouldn't it be faster to create the Images once and just reuse them?
We're programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We're not excited by renovation:tinkering,improving,planting flower beds.
Siebenschläfer
Posts: 34
Joined: Thu Feb 05, 2009 11:37 am
Location: Koblenz, Germany

Post by Siebenschläfer »

@hybrid:
sorry I don't get why is is possible to set vectors or matrices as parameters in the shaders but not an array of pixel values :-/... is it simply just to big?

@Sudi:
I thought so too but doing like you say did not lead to more speed; because the memcpy method was even a little more slowly than the standard copying I just sticked with the previous method using RTTs

Code: Select all

char* texture = (char*)( videoTexture->lock() );
	char* ardata = (char*) frame->imageData;
	char* final_loc = frame->imageSize + frame->imageData;

	while( ardata < final_loc )
	{
		// B
		*texture = *ardata;
		texture++; ardata++;
		// G
		*texture = *ardata;
		texture++; ardata++;
		// R
		*texture = *ardata;
		texture++; ardata++;			
		// A: skip 
		*texture = (char) 255;
		texture++;
	}

Think Tannenzäpfle!
Post Reply