Webcam Capture and Display in Irrlicht [solved]

[jack]

Hi guys,

I've been looking for weeks trying to find a pre-written package that will do this for me. With no luck I've tried implementing it myself.

I am running Linux.

I was able to capture a frame from the webcam. I save this to disk as a jpeg file in my webcam_capture thread.

My main drawing thread then loops deleting a texture and reloading from the same pathname so it will have the updating image. This is a stupid way to do it I know, but I am having major trouble thinking how to do it differently.

I don't know how to update a texture of a node without removing it and reloading from disk.

Also, the format of the frame I capture from my webcam is in the form of an array of RGB values. Does anyone know how I can directly create a texture from this without going through the SAVE AS JPEG TO DISK, LOAD JPEG FROM DISK THROUGH DEVICE TO TURN INTO TEXTURE method?

I'm really stuck.

Thanks.

[BlindSide]

Haha, I did this the other day using ARToolKit, what a coincidence:

(Uses marker detection to display models.)

Anyway, it is very easy to copy the data from the image pointer your camera software will give you.

This is the code that I use:

Code: Select all

u8* Pointer = (u8*)camtex->lock();

// Saves us a multiply...or something, lol.
const int cam_width_const_quad = cam_width_const * 4;

int source_offset=0; int dest_offset=0;
for(int y=0; y < cam_height_const; ++y)
{
	// Memcpy or loop, which one is better?

	// Memcpy : Allows for better compiler optimization, maps better to hw.
	// Loop : 25% less copying since I can miss out the alpha component.

	//memcpy(Pointer + dest_offset,cam_image_const + source_offset,cam_width_const_quad);

	for(int x=0; x < cam_width_const_quad; x += 4)
	{
		Pointer[dest_offset + x]	 = cam_image_const[source_offset + x];
		Pointer[dest_offset + x + 1] = cam_image_const[source_offset + x + 1];
		Pointer[dest_offset + x + 2] = cam_image_const[source_offset + x + 2];
	}

	source_offset	+= cam_width_const_quad;  
	dest_offset		+= cam_width_const_quad;
}

camtex->unlock();

"cam_image_const" is your image pointer obtained from your webcam, I can obtain this from ARToolKit using:

Code: Select all

const u8* cam_image_const = (u8*)arVideoGetImage();

Offcourse this is for RGBA format (I am missing out the alpha though because it is completely unnecessary ). It shouldnt be hard to do for other formats, eg ARGB, just increment pointers by one. or ABGR, flip the order of transfer. For RGB you have to change the offset, but I recommend just looking into seeing if your cam software can provide an image in RGBA format as it is easier to copy to an Irrlicht texture.

Cheers

PS: Just curious, what kind of framerate did you get with that silly method ???

[drac_gd]

here is how I update my tile textures directly without creating new textures each time

first create a empty texture of the correct dimensions

Code: Select all

	char as8Buf[ 30 ];
	sprintf( as8Buf, "TileTexture%d", iTileNodeCnt );
    m_poTileTexture = m_poMgr->m_poDriver->addTexture(core::dimension2d<s32>(128, 128), as8Buf, video::ECF_A1R5G5B5);
	sprintf( as8Buf, "TileNormaMap%d", iTileNodeCnt );
    m_poTileNormalMap = m_poMgr->m_poDriver->addTexture(core::dimension2d<s32>(128, 128), as8Buf, video::ECF_A1R5G5B5);

then update it directly

Code: Select all

	// read the pixels directly into the texture
    u8* pixels;
    pixels = (u8*)(m_poTileTexture->lock());
	if( pixels )
	{
		poCache->LoadData(	pixels,								// buffer to read into
							u64FileOffs + sizeof( VxTileCell ),	// offset in file to read at
							TILE_PIXEL_ARRAY_SIZE );		// number of bytes to read
		m_poTileTexture->unlock();
	}
	else
	{
		log_msg( 0, "Could not lock pixels tile %d, col %d row %d\n", this->m_iDisplayNum, m_iTerCol, m_iTerRow ); 
	}
    pixels = (u8*)(m_poTileNormalMap->lock());
	if( pixels )
	{
		poCache->LoadData(	pixels,								// buffer to read into
							u64FileOffs + sizeof( VxTileCell ) + TILE_PIXEL_ARRAY_SIZE,	// offset in file to read at
							TILE_PIXEL_ARRAY_SIZE );		// number of bytes to read
		m_poTileNormalMap->unlock();
	}
	else
	{
		log_msg( 0, "Could not lock normal map tile %d, col %d row %d\n", this->m_iDisplayNum, m_iTerCol, m_iTerRow ); 
	}

[jack]

Is the memcpy function copying a row of the webcam image to the texture memroy?

I.e. if I'm using RGB values I would be copying (width*3) bytes per memcpy? (where width = number of pixels per row)

[hybrid]

memcpy can copy any amount of bytes. You tell it the destination and source pointers, and the number of bytes to copy. Note, though, that it's bytes, so you have to multiply by 3 or 4, depending on whether you use RGB8 or RGBA8.
however, it's by far more efficient than the color-per-pixel access, I'd guess that you can easily make it 10 times faster using memcpy.

[jack]

memcpy can copy any amount of bytes. You tell it the destination and source pointers, and the number of bytes to copy. Note, though, that it's bytes, so you have to multiply by 3 or 4, depending on whether you use RGB8 or RGBA8.
however, it's by far more efficient than the color-per-pixel access, I'd guess that you can easily make it 10 times faster using memcpy.

Thanks. :p I was trying to figure out why the first example is using a for loop to copy bytes, when you could just remove the loop and memcpy number of bytes * y instead....?

[hybrid]

If you're sure that dimension, color format, and padding match you can safely replace the for loop with a memcpy. But even if you have to pad or shorten lines you will get better results by looping over a memcpy of the lines than doing a per-pixel transaction.

[BlindSide]

I use the for loop so that I don't have to copy the alpha byte, I noticed no noticable speed difference from memcpy.

[jack]

I managed to get it working the way I want. Here is the source if anyone else wants it.

Code: Select all

void updateFrame()
{
    // Get a new frame
    webcam->takePicture();

    // Initialise buffers
    source_buffer = webcam->dumpPicture();
    u8* dest_buffer = (u8*) webcamTexture->lock();

    // Copy pixel data from the webcam buffer to the texture buffer
    memcpy(dest_buffer, source_buffer, WEBCAM_TOTALSIZE);

    // Unlock the webcamTexture for drawing
    webcamTexture->unlock();
}

This function is called in my main threads drawing loop, which makes the framerate a little laggy I think.

webcamTexture is the texture for the node I want to update.

webcam is a videoObject* using the linux-webcam package from http://www.stillhq.com/opensource.html

I had to edit the package to set my webcam to use RGB32 values.

WEBCAM_TOTALSIZE is a global variable which is the pixel height * pixel width * bytes per pixel.

I also edited the webcam package so that it didn't malloc new memory for each framegrab.

Hopefully someone can find this useful in future.

[hybrid]

Make sure that you disable mipmaps if you can do so (otherwise each unlock will call the mipmap autoupdate). Also, adding a test for the proper color format of the texture could make this code somewhat more safe You can also force the texture to become RGBA8 on creation, which seems to be advisable (check the texture creation flags for how to change those things).

[FreakNigh]

With thanks to this thread I would like to post the learning code I developed to make the lib I am about to make.

This is my whole test.cpp, it is a merge of ARToolKit's simpletest.c and irrlicht's Hello World. Minus the girl I made a billboard node display the webcam input from the artoolkit.

a Youtube video of this code in action: http://www.youtube.com/watch?v=tk8xPPeqi40

Code: Select all

#include <irrlicht/irrlicht.h>
#include <AR/video.h>
#include <AR/param.h>
#include <AR/ar.h>
 
using namespace irr;

using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;

#ifdef _WIN32
char			*vconf = "Data¥¥WDM_camera_flipV.xml";
#else
char			*vconf = "";
#endif

int             xsize, ysize;
int             thresh = 100;
int             count = 0;

char           *cparam_name    = "ardata/camera_para.dat";
ARParam         cparam;

char           *patt_name      = "ardata/patt.hiro";
int             patt_id;
double          patt_width     = 80.0;
double          patt_center[2] = {0.0, 0.0};
double          patt_trans[3][4];

IrrlichtDevice *device;
IVideoDriver* driver;
ISceneManager* smgr;

void initAR();
ITexture* create_ITexture_from_ARimage(ARUint8 *ardata, int width, int height);
ITexture* update_ITexture_from_ARimage(ITexture *dest, ARUint8 *ardata, int width, int height);

int main()
{
	//init code ripped from ARToolkit's simpleTest.c
	initAR();
	
	device = createDevice( video::EDT_OPENGL, dimension2d<s32>(640, 480), 16, false, false, false, 0);
	
	device->setWindowCaption(L"Hello World! - Irrlicht Engine Demo");
	
	driver = device->getVideoDriver();
	smgr = device->getSceneManager();
	
	//our billboard
	IBillboardSceneNode* our_bill_node = smgr->addBillboardSceneNode(NULL, dimension2d<f32>(20.0f, 20.0f), vector3df(10,0,0));
	our_bill_node->setMaterialFlag(video::EMF_LIGHTING, false);
	
	ICameraSceneNode* camera = smgr->addCameraSceneNodeFPS();
	
	camera->setPosition(vector3df(40,0,0));
	camera->setTarget(vector3df(0,0,0));
	
	int lastFPS = -1;
	ARUint8         *dataPtr;
		
	if( (dataPtr = (ARUint8 *)arVideoGetImage()) == NULL ) 
		printf("no image loaded\n");
	
	ITexture* ARimage = create_ITexture_from_ARimage(dataPtr, xsize, ysize);
	
	our_bill_node->setMaterialTexture( 0, ARimage );
	
	while(device->run())
	{
		if(dataPtr = (ARUint8 *)arVideoGetImage()) 
			update_ITexture_from_ARimage(ARimage, dataPtr, xsize, ysize);
		
		driver->beginScene(true, true, SColor(255,100,101,140));

		smgr->drawAll();
		//guienv->drawAll();

		driver->endScene();
		
		int fps = driver->getFPS();

		if (lastFPS != fps)
		{
			core::stringw str = L"Irrlicht Engine [";
			str += driver->getName();
			str += "] FPS:";
			str += fps;

			device->setWindowCaption(str.c_str());
			lastFPS = fps;
		}
	}
	
	device->drop();

	return 0;
}

ITexture* update_ITexture_from_ARimage(ITexture *dest, ARUint8 *ardata, int width, int height)
{
	u8* pixels;
	pixels = (u8*)(dest->lock());
	if( pixels )
	{
		int max_pixels = width * height;
		
		for(int i=0;i<max_pixels;i++)
		{
			*pixels = *ardata;
			pixels++; ardata++;
			*pixels = *ardata;
			pixels++; ardata++;
			*pixels = *ardata;
			pixels++; ardata++;
			
			pixels++;
		}
		
		dest->unlock();
	}
	else
	{
		printf("some hellish error\n");
	}

	return dest;
}

ITexture* create_ITexture_from_ARimage(ARUint8 *ardata, int width, int height)
{

	ITexture* m_poTileTexture = driver->addTexture(core::dimension2d<s32>(width, height), "TileTex", video::ECF_A1R5G5B5);

	ITexture* m_poTileNormalMap = driver->addTexture(core::dimension2d<s32>(width, height), "NormTex", video::ECF_A1R5G5B5); 
	
	   // read the pixels directly into the texture
	u8* pixels;
	pixels = (u8*)(m_poTileTexture->lock());
	if( pixels )
	{
		int max_pixels = width * height;
		
		for(int i=0;i<max_pixels;i++)
		{
			*pixels = *ardata;
			pixels++; ardata++;
			*pixels = *ardata;
			pixels++; ardata++;
			*pixels = *ardata;
			pixels++; ardata++;
			
			pixels++;
		}
		
		m_poTileTexture->unlock();
	}
	else
	{
		printf("some hellish error\n");
	}

	return m_poTileTexture;
}

void initAR()

{
	//nearly verbatum rip from ARToolKit's simpleTest.c, minus the "g" stuff that is used for integrating with glut
	ARParam  wparam;
	
	/* open the video path */
	if( arVideoOpen( vconf ) < 0 ) exit(0);
	/* find the size of the window */
	if( arVideoInqSize(&xsize, &ysize) < 0 ) exit(0);
	printf("Image size (x,y) = (%d,%d)\n", xsize, ysize);

	/* set the initial camera parameters */
	if( arParamLoad(cparam_name, 1, &wparam) < 0 ) {
		printf("Camera parameter load error !!\n");
		exit(0);
	}
	arParamChangeSize( &wparam, xsize, ysize, &cparam );
	arInitCparam( &cparam );
	printf("*** Camera Parameter ***\n");
	arParamDisp( &cparam );

	if( (patt_id=arLoadPatt(patt_name)) < 0 ) {
		printf("pattern load error !!\n");
		exit(0);
	}
	
	arVideoCapStart();
}

[/url]

[Dragonazul]

Hi all,

I'm using too Artoolkit but i´d like a better method for unlock the image.
This process (glTexSubImage2D) that is the final call of unlock waste "a lot of time". Textures are not supposed to be change very often and so this is a not very speed call.

Is possible to show the image directly as the background on a win32 window?
I should too put the background of irrlicht without any texture or color or better with a transparent color.

I think then I can only need a memcpy to copy my image as background of the win32 window.

This can be done?

Thanks

[FreakNigh]

if you check out this post (which has very similar code) a few people have suggested similar solutions with a bit of insight on how to do it.

http://irrlicht.sourceforge.net/phpBB2/ ... hp?t=29389

[Dragonazul]

Hi FreakNigh,

What i'm trying to do is: not to use a texture for show my Image (or the video into the code you show) because the unlock call take lots of time.

I'm looking the Win32Window sample.
I want to see into the irrlicht window the background of the win32 window. Is possible to create the window of irrlicht transparent??

driver->beginScene(true, true, irr::video::SColor(255,0,0,0));¿?