Standby mode issues | [WAS:] "Cleaning up properly"

mdeininger · Post by **mdeininger** » Mon Mar 01, 2010 3:10 pm

Hello all,

I'm having a bit of a problem at the moment regarding irrlicht and its memory use... or probably my understanding of it. I'm using a separate Thread in an application that will summarise a simple graph using the terrain mesh and associated height maps. The code basically boils down to this: (I'm omitting some parts for legibility)

Code: Select all

	irr::scene::ISceneManager* smgr;
	video::IVideoDriver* driver;
	bool wasPaused = true;
	AnsiString apath = ExtractFilePath(Application->ExeName);
	scene::ITerrainSceneNode* terrain = NULL;

	Pause3D = true;

	_control87( PC_64|MCW_EM, MCW_PC|MCW_EM );

		irr::SIrrlichtCreationParameters param;
		param.WindowId = hIrrlichtWindow;
		param.DriverType = DriverType;

		GlobalIrrlichtData.device = irr::createDeviceEx(param);
		if (GlobalIrrlichtData.device == NULL)
		{
			ShowMessage("Internal error: IRR3D createDeviceEx failed");
			return;
		}

		GlobalIrrlichtData.device->getTimer()->stop();

		smgr   = GlobalIrrlichtData.device->getSceneManager();
		driver = GlobalIrrlichtData.device->getVideoDriver();

		scene::ICameraSceneNode* cam = smgr->addCameraSceneNode();
		cam->setTarget(core::vector3df(0,150,0));

		scene::ISceneNodeAnimator* anim = smgr->createFlyCircleAnimator(core::vector3df(100,280,0), 500.0f, 0.0007);
		cam->addAnimator(anim);

		anim->drop();

		cam->drop();

		AnsiString fn_image_sky = apath + "sky.jpg";
		if (FileExists( fn_image_sky))
		{
			smgr->addSkyDomeSceneNode( driver->getTexture(fn_image_sky.c_str()),
									32,
									32,
									1.0,
									1.2
								 );
		}

  reloadHeightMap:

            /* generate texture for terrain, otherwise boring */

	if (terrain == NULL)
	{
		terrain = (scene::ITerrainSceneNode*)smgr->getSceneNodeFromId (TERRAIN_NODE_MAGIC_COOKIE);
	}

	if (terrain == NULL)
	{
		terrain = smgr->addTerrainSceneNode(
			(fn_heightmap).c_str(),
			0,					// parent node
			TERRAIN_NODE_MAGIC_COOKIE, // node id
			core::vector3df(-516.f, -50.f, -516.f),		// position
			core::vector3df(0.0f, 80.9f, 0.f),		// rotation
			core::vector3df(8.0f, 0.675f, 8.0f),	// scale
			video::SColor ( 255, 255, 255, 255 ),	// vertexColor
			1,					// maxLOD: 5, ok 3, 0 ... Full details
			scene::ETPS_17,				// patchSize
			0					// smoothFactor: 4 ok 2
			);
	}
	else
	{
		terrain->loadHeightMap(GlobalIrrlichtData.device->getFileSystem()->createAndOpenFile((fn_heightmap).c_str()),
							   video::SColor ( 255, 255, 255, 255 ),
							   0);
	}

	ReloadHeightMap = false;

	terrain->setMaterialTexture(0, driver->getTexture(fn_texture.c_str()));

	if ((DriverType != video::EDT_OPENGL) && (DriverType != video::EDT_DIRECT3D9))
	{
		terrain->setMaterialTexture(1, driver->getTexture(fn_texture.c_str()));
	}

	terrain->setMaterialType(video::EMT_LIGHTMAP);

	terrain->scaleTexture(1.0f, 1.0f /*20.0f*/);

	DeleteFile(fn_texture);
	DeleteFile(fn_heightmap);

	while (!Terminated)
	{
		if (ReloadHeightMap)
		{
			Pause3D = true;
			goto reloadHeightMap;
		}

		if (wasPaused != Pause3D)
		{
			if (wasPaused)
			{
				GlobalIrrlichtData.device->getTimer()->start();
			}
			else
			{
				GlobalIrrlichtData.device->getTimer()->stop();
			}

			wasPaused = !wasPaused;
		}

		GlobalIrrlichtData.device->getTimer()->tick();

		driver->beginScene(true, true, video::SColor(255, 0x45,0xAB,0xDB));
		smgr->drawAll();
		driver->endScene();

		if (Screenshot)
		{
			driver->writeImageToFile(driver->createScreenShot(), FileNameScreenshot.c_str());
			Screenshot = false;
		}

		GlobalIrrlichtData.device->yield();
	}

now, to regenerate the heightmap the (volatile) ReloadHeightMap is set from another thread and other relevant parameters are adjusted, so that in the big while loop it jumps back up and will then modify the heightmap... however, each time this happens, the memory use will rise by 1-2 megabytes, thus eventually causing lots of problems (such as missing textures...). My question is thus... do I need to drop() something somewhere? I mean I tried, I also tried removeTexture() and removeAllTextures() with some restructuring to get it all back in, as well as remove() on all SceneNodes and even remove the screenshot capabilities, but no matter what I do, it always rises by 1-2 megs each time. :/ Btw, the logic is working fine in and of itself, so there's no missing volatiles on booleans... it's just the mem consumption that I can't seem to handle...

Anyone have any idea what I'm missing here?

(btw i'm using irrlicht 1.7.1, the same patches that I posted on the other subforum and compiled with borland...)

thanks in advance,
Magnus

zillion42 · Post by **zillion42** » Mon Mar 01, 2010 4:38 pm

It's not that I really understand your multithreading thing or that I actually have experience, neither with your multithreading nor much with the engines inner workings, BUT:
http://irrlicht.sourceforge.net/docu/cl ... unted.html

When you create an object in the Irrlicht engine, calling a method which starts with 'create', an object is created, and you get a pointer to the new object. If you no longer need the object, you have to call drop(). This will destroy the object, if grab() was not called in another part of you program, because this part still needs the object. Note, that you only need to call drop() to the object, if you created it, and the method had a 'create' in it.

A simple example:

If you want to create a texture, you may want to call an imaginable method IDriver::createTexture. You call ITexture* texture = driver->createTexture(dimension2d<u32>(128, 128)); If you no longer need the texture, call texture->drop().

If you want to load a texture, you may want to call imaginable method IDriver::loadTexture. You do this like ITexture* texture = driver->loadTexture("example.jpg"); You will not have to drop the pointer to the loaded texture, because the name of the method does not start with 'create'. The texture is stored somewhere by the driver.

So what I can see and know is, that you should at least drop the FlyCircleAnimator, the device once you dont need it anymore probably the heightmap and I'm really unsre with the screenshot.
Hope that answeres your question a little.

mdeininger · Post by **mdeininger** » Mon Mar 01, 2010 4:48 pm

zillion42 wrote:It's not that I really understand your multithreading thing or that I actually have experience, neither with your multithreading nor much with the engines inner workings, BUT:
http://irrlicht.sourceforge.net/docu/cl ... unted.html
When you create an object in the Irrlicht engine, calling a method which starts with 'create', an object is created, and you get a pointer to the new object. If you no longer need the object, you have to call drop(). This will destroy the object, if grab() was not called in another part of you program, because this part still needs the object. Note, that you only need to call drop() to the object, if you created it, and the method had a 'create' in it.

A simple example:

If you want to create a texture, you may want to call an imaginable method IDriver::createTexture. You call ITexture* texture = driver->createTexture(dimension2d<u32>(128, 128)); If you no longer need the texture, call texture->drop().

If you want to load a texture, you may want to call imaginable method IDriver::loadTexture. You do this like ITexture* texture = driver->loadTexture("example.jpg"); You will not have to drop the pointer to the loaded texture, because the name of the method does not start with 'create'. The texture is stored somewhere by the driver.
So what I can see and know is, that you should at least drop the FlyCircleAnimator, the device once you dont need it anymore probably the heightmap and I'm really unsre with the screenshot.
Hope that answeres your question a little.

hi there, and thanks for the swift reply.

the thing is, basically I already had that, but for some reason it doesn't make any difference whatsoever. i trimmed the code a little to make sure it's somehow understandable. i tried with completely disabling the screenshot capability, i tried with ->drop()'ing the createAndOpenFile() result, and with ->remove() and ->removeTexture() for the things i did get with ->add* and ->get* functions, but none of it seemed to change memory usage characteristics at all (including doing all of that at the same time.). being desperate, i even tried ->drop()-ing things that were explicitly marked as not to be ->drop()'d, like the texture...

i'm really in a bind here... i suspect that maybe the terrain mesh is not cleaning something or that maybe i need to make it throw stuff away?

mdeininger · Post by **mdeininger** » Mon Mar 01, 2010 5:16 pm

it seems that, upon even closer inspection, omitting the loadHeightMap() seems to cut the memory loss by roughly 90%...

this is kind of weird alright... i suppose not many people would be using that function a lot though, so i'll dig into there and see if i can come up with something. (doesnt seem to be the file handle, btw, the extended version does also drop that).

zillion42 · Post by **zillion42** » Mon Mar 01, 2010 8:35 pm

well, like I said earlier I dont know a thing about multithreading nor Borland and not a lot about reference counting.
However, if I use a visual studio project in debug and exit the device before dropping whatever object I created with a method that started with "create" as opposed to "add" or "load" I will see the output complaining about memory leaks.
And like I said earlier this applies to your animator and device at least. Also the getFileSystem()->createAndOpenFile looks very suspicious, so just put a drop() on those objects and you should be fine.

//! Opens a file for read access.
/** \param filename: Name of file to open.
\return Returns a pointer to the created file interface.
The returned pointer should be dropped when no longer needed.
See IReferenceCounted::drop() for more information. */
virtual IReadFile* createAndOpenFile(const path& filename) =0;

so try:

Code: Select all

io::IReadFile *heightmap = device->getFileSystem()->createAndOpenFile("test.tga");
//and later
heightmap->drop();

mdeininger · Post by **mdeininger** » Tue Mar 02, 2010 1:49 pm

well, as i said i had tried that... i tried it again but it didn't seem to fly either :S. btw, the animator already had this:

Code: Select all

		anim->drop();

or did i misunderstand that and i have to drop it again?

what did help was doing this:

Code: Select all

		if (sky)
		{
			sky->remove();
			sky = NULL;
		}

		if (terrain)
		{
			terrain->remove();
			terrain = NULL;
		}

		driver->removeAllTextures();
		driver->removeAllHardwareBuffers();

i added an extra "sky" variable for the skydome, and did this after the jump label... removeAllTextures() didn't actually do all that much, but removeAllHardwareBuffers() did the trick, so I'm guessing, based on the docs, that somewhere along the way a mesh is being lost?

i did try the loadHeightMap() like this, verbatim:

Code: Select all

			io::IReadFile* file = GlobalIrrlichtData.device->getFileSystem()->createAndOpenFile((fn_heightmap).c_str());

			terrain->loadHeightMap(file, video::SColor ( 255, 255, 255, 255 ), 0);

			file->drop();

but it was still using more and more memory, but commenting it out entirely was basically keeping a constant mem (or rather, it did use an extra 100k or so each time, probably the texture which did still change). since the removeAllHardwareBuffers() + recreating it all fixes is, it would seem that that loadHeightMap() is somehow losing a mesh?

it's working for now, but i'd still kinda like to know where i went wrong...

mdeininger · Post by **mdeininger** » Tue Mar 02, 2010 4:59 pm

Something else my colleague noticed: when the computer goes into standby mode, and irrlicht is in Direct3D or OpenGL mode (Windows XP machine), the output "vanishes" and the next action to manipulate it all will create woeful exceptions. This only appears to happen when irrlicht is getting a separate thread; no errors appeared in another project where I used the same thread to do irrlicht's rendering. Also the problem does not occur when I'm using Burnings software renderer. Is this by any chance a known issue?

I tried to work around it by catching windows' standby notice and trying to close the irrlicht device over it, roughly by doing this:

Code: Select all

		driver->removeAllTextures();
		driver->removeAllHardwareBuffers();

		device->closeDevice();
		device->run();
		device->drop();
		device = NULL;

this appears to work, however when the next call to CreateDeviceEx() occurs, an AccessViolation exception is thrown by the runtime (in the OpenGL case, I believe it was in a function that compiles an OGLSL programme?).

It's not necessarily a critical problem at the moment, we might just as well block suspension for that particular programme. I'll try to make a test case in vc++. The regular demo programmes don't seem to be affected, presumably due to lack of threading, and as I said it doesn't happen in a different single-threaded programme I wrote either...

EDIT: speaking of it, the fragment above doesn't seem to suffice to get rid of all things irrlicht yet either. I added a function to the programme to switch between renderers on the fly, which works fine and uses the above fragment to close irrlicht, but on each iteration it consumes about 3-5 megs more memory... that part is actually quite weird... especially since programme termination proper WILL get rid of it all.