Convert Image to IrrlichtTexture

Siebenschläfer · Post by **Siebenschläfer** » Thu Feb 05, 2009 7:52 pm

Hello,

I am using OpenCV to capture a video and convert it to an irrlicht texture to be able to display it. The nightsoft guys have a pretty code that shows how to do it. But it gets awefull slow for me with a resolution higher than 352x288, so thats nothing!

code is:

Code: Select all

void CameraInput::repaint( IplImage* frame )
{	
	// read the pixels directly into the texture
	char* texture = (char*)( irr_texture->lock() );
	char* ardata = (char*) frame->imageData;
	char* final_loc = frame->imageSize + frame->imageData;

	switch( irr_texture->getColorFormat() )
	{
		case ECF_R8G8B8:
			while( ardata < final_loc )
			{
				*texture = *ardata;
				texture++; ardata++;
				*texture = *ardata;
				texture++; ardata++;
				*texture = *ardata;
				texture++; ardata++;
			}
			break;
			
		case ECF_A8R8G8B8:
			while( ardata < final_loc )
			{
				// R
				*texture = *ardata;
				texture++; ardata++;
				// G
				*texture = *ardata;
				texture++; ardata++;
				// B
				*texture = *ardata;
				texture++; ardata++;			
				// A: skip 
				texture++;

				//cout<<"rgba"<<endl;
			}
			break;
	}


	irr_texture->unlock();
}

I also tried with ARToolKit, but still it is very slow. Has anyone used video capture and found a way to optimize the converting process?

Thanks

Siebenschläfer · Post by **Siebenschläfer** » Sat Feb 07, 2009 9:08 am

any idea?

Katsankat · Post by **Katsankat** » Sat Feb 07, 2009 9:33 am

Hi, the conversion in itself looks good. Remove the switch, but this won't help much.

However as I know OpenCV quite a bit I'm afraid you can't get better performances unless with better hardware.

The slowdown could come from the rest of the code : are you writing an avi file simulataneously? It takes cycles to capture, convert, display, save at 30 frames per second... Not to mention irrlicht running, too. Hard to say more without knowing what is doing the application internally.

Nox · Post by **Nox** » Sat Feb 07, 2009 10:35 am

first guess: use memcpy not a while !!!
second guess: do you use a dynamic/rendertarget-texture? If not change this.
third guess: may be two textures speed this up (write at first, show second, write at second show first).

Siebenschläfer · Post by **Siebenschläfer** » Sat Feb 07, 2009 11:04 am

ok, thank you a lot, I'll try that!

Siebenschläfer · Post by **Siebenschläfer** » Thu May 28, 2009 5:04 pm

hm,

first:
memcpy( &texture, &ardata, frame->height*frame->width*frame->depth );

only returns an access violation.

second:
no I don't because from opencv I don't receive power2-images...

third:
how do you mean using 2 textures? within the program all commands are executed sequential or not?!

I thought about sending the image-data to a shader and returning it as an Irrlicht texture... do you tink that's sensefull?

Siebenschläfer · Post by **Siebenschläfer** » Fri May 29, 2009 10:17 am

ok, maybe this can be moved to snippets because I see that a couple of people tried to convert OpenCV Images to Irrlicht Textures, too, all the same way. Also I don't want to push-up the thread...

to first again:

I had to convert the OpenCV Image from BGR to BGRA to get it convertible with the texture. Also I had to fill the alpha channel with 255 because it is 0.
Then I can use the memcpy:

Code: Select all

void CameraManager::convertImageToTexture( IplImage* frame, ITexture* videoTexture )
{	
	IplImage* newFrame = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
	cvCvtColor( frame, newFrame, CV_BGR2BGRA );

	IplImage* mask = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
	cvSet( mask, cvScalar( 0,0,0,255 ) );

	cvAdd( newFrame, mask, newFrame );

	// read the pixels directly into the texture
	char* texture = (char*)( videoTexture->lock() );
	char* ardata = (char*) newFrame->imageData;
	char* final_loc = newFrame->imageSize + newFrame->imageData;

	memcpy( texture, ardata, newFrame->width*frame->height*newFrame->nChannels );	
            videoTexture->unlock();

            cvReleaseImage( &mask );
	cvReleaseImage( &newFrame );
}

But unfortunatly the result has the same FPS number than the code on top, even without all the conversions it's only 1 FPS difference. The reason is the lock() and unlock() - these function kill the performance!

I gonna try the second and the third thought now. Also I would try it with shader if someone could give me hints how to deliver the char* arrays to it and how to receive not a material but a texture.

Nox · Post by **Nox** » Fri May 29, 2009 1:55 pm

lock/unlock will be faster if you use a rendertarget/dynamic texture.

Siebenschläfer · Post by **Siebenschläfer** » Fri May 29, 2009 2:54 pm

oh yes, you are right, with an RTT it's really much faster. But I tested it and the while loop is totally as fast as the memcpy, it brings no difference!

Siebenschläfer · Post by **Siebenschläfer** » Wed Jun 24, 2009 1:43 pm

What I like to try right now is to deliver the imageData of the OpenCV Image to the shader to be returned as texture? Is that possible?

What I try is:

Code: Select all

main:
        ShadManager->convertTexture( conversionNode );

ShadManager:
        node->setMaterialType( ( E_MATERIAL_TYPE )m_ConversionMaterial ); 

ShadCallback:
        		IplImage* src = camera->getCurrentUndistortedFrameLeft();

		CvMat buffer = cvMat( src->height, src->width, CV_8U, src->imageData );

		cvNamedWindow( "lalala", 1 );
		cvShowImage( "lalala", &buffer );

		videoTexture = buffer.data.fl;
		
		services->setPixelShaderConstant("VideoTexture", (float*)videoTexture, src->height * src->width * 3 );

and the frag shader:

Code: Select all

uniform sampler2D VideoTexture;

varying vec2 TexCoord;
	
	void main()
	{		
		vec4 lalala;
		

		vec4 colorVID = texture2D(VideoTexture,lalala.xy).xyz; 

		colorVID.a = 255.0;
		
				
		gl_FragColor = colorVID;
	}

so thats the idea, but how can I get this working?

hybrid · Post by **hybrid** » Wed Jun 24, 2009 1:50 pm

No, you cannot pass texture data as a parameter. You have to create an actual ITexture in Irrlicht (which then copies the data to the GPU) and set the texture inside the material. Then your shader can work on the texture data. Instead of creating new textures all the time you can also lock the texture, upon unlocking the new content will be copied to the GPU.

Siebenschläfer · Post by **Siebenschläfer** » Thu Jun 25, 2009 8:08 am

but I have to fill the texture every loop again with the content of the image data, how should I avoid that?

hybrid · Post by **hybrid** » Thu Jun 25, 2009 11:27 am

You can't. At least unless you cannot generate the data on the GPU directly. Even if your GPU was allowed to access the data in RAM it wouldn't be much faster than locking and unlocking the texture. PBOs would avoid one copy, because you could specify that you only want to write the data, not read. But that's a very specific case and I don't know how much trouble the FBO preparation is when locking the data, but not reading from it.

sudi · Post by **sudi** » Thu Jun 25, 2009 2:55 pm

I see that u use

Code: Select all

IplImage* newFrame = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
   cvCvtColor( frame, newFrame, CV_BGR2BGRA );

   IplImage* mask = cvCreateImage( cvGetSize( frame ), IPL_DEPTH_8U, 4 );
   cvSet( mask, cvScalar( 0,0,0,255 ) );

every frame. Wouldn't it be faster to create the Images once and just reuse them?

Siebenschläfer · Post by **Siebenschläfer** » Thu Jun 25, 2009 3:49 pm

@hybrid:
sorry I don't get why is is possible to set vectors or matrices as parameters in the shaders but not an array of pixel values :-/... is it simply just to big?

@Sudi:
I thought so too but doing like you say did not lead to more speed; because the memcpy method was even a little more slowly than the standard copying I just sticked with the previous method using RTTs

Code: Select all

char* texture = (char*)( videoTexture->lock() );
	char* ardata = (char*) frame->imageData;
	char* final_loc = frame->imageSize + frame->imageData;

	while( ardata < final_loc )
	{
		// B
		*texture = *ardata;
		texture++; ardata++;
		// G
		*texture = *ardata;
		texture++; ardata++;
		// R
		*texture = *ardata;
		texture++; ardata++;			
		// A: skip 
		*texture = (char) 255;
		texture++;
	}