[fixed]Memory leak

[Iaro]

Hello,

I'm having some problems regarding memory management in an application I'm working on. The world is structured in locations, and each location can contain triggers, objects, etc. For now only triggers are implemented. The problem is that when a location is cleared and a new one is loaded I'm loosing some memory.

First I don't know if this technique for releasing locations is good:
- the scene node is removed with remove()
- for every meshbuffer in the locations mesh the textures are removed:

Code: Select all

ITexture *toRemove;
toRemove = location->mesh->getMeshBuffer(i)->getMaterial().getTexture(0);
//remove textures
if (toRemove) driver->removeTexture(toRemove);
toRemove = location->mesh->getMeshBuffer(i)->getMaterial().getTexture(1);
//remove lightmaps
if (toRemove) driver->removeTexture(toRemove);

- the mesh is removed from the mesh cache:

Code: Select all

location->smgr->getMeshCache()->removeMesh(location->mesh);

Other possible leaks I could think of would be in the structures defining locations and triggers, so I've tried only loading/removing location nodes without the use of any structures. For this the application just crashed. Running is in debug mode the crash location is line 484 of COpenGLDriver.cpp :

Code: Select all

const bool isRTT = Material.getTexture(i) && Material.getTexture(i)->isRenderTarget();

I am not using anything related to RTT.
I've tried to find the leak using the the VC built in debugger, but when the application closes, the memory dump is very very large, and also shows data from the engine, so finding the bug for my tiny application would be very hard.

The structures in question:

Code: Select all

typedef struct
{
	CTriggerNode *node;		
	s32	      majorType;
	s32	      minorType;	
	stringw	     action;
	stringw     antiAction;
	bool	     actionState;		
	stringw   description;
	stringw     	     name;
	bool		     state;
} STrigger;

typedef struct
{
	ISceneNode *node;
	ISceneManager *smgr;
	IMesh	   *mesh;
	ISceneNode* skyBox;
	irr::core::array < STrigger* > triggers;		
} SLocation;

CTriggerNode if basically a cube scenenode. No new memory is declared in it, vertices/indices are normal predefined arrays.

Unrelated to Irrlicht:
Should I avoid using stuff like string/wstring/vector from the std library, or can they be used without risk of memory loss. I've tried replaceing std::vector with irr::core::array in the location stucture but the problem persists.
And lastly, what approach would be better: Focus on actually building the application, implementing memory management as best I can or continually check for memory leaks at each step and do not advance until every problem is solved.
This is a very frustrating problem, so any help would be greatly appreciated.

[hybrid]

The problem with removing textures from the cache is that you cannot guarantee (easily), that the textures are not used elsewhere. Only if you, as the programmer and designer, know for sure that the textures are only used once in this location, you're safe to remove it. Otherwise you might have other nodes which still refer to the (now deleted) texture and crash at the first position where the pointer is dereferenced.

[rogerborg]

I've tried to find the leak using the the VC built in debugger, but when the application closes, the memory dump is very very large, and also shows data from the engine, so finding the bug for my tiny application would be very hard.

There shouldn't be any leaks on exit, so that's either a bug in usage, or you've discovered a bug in the engine. If so, you win a cookie![1]

INSUFFICIENT DATA FOR MEANINGFUL ANSWER, I'm afraid. If you're able and willing to make a compilable, runnable project available, I'd be happy to have a poke at it though.

[1] Disclaimer: winners must supply own cookie.

[Iaro]

At this point the code is pretty messy, spread around in a couple of classes and largely uncommented, so I don't know how much help it would be. But I'll try something else: make a very simple test program not connected to the application, and see if the problem persists.
About the textures, yes for now they are used only for one location. But if in the future maybe some textures would need to be stored and reused. A solution I've been thinking about: temporarily load two meshes, for the two locations, then go over the textures of the first mesh and delete only those that have a reference count of 1 - meaning they are unused by any other mesh.

[Iaro]

This is a simple program for loading/releasing a location:

Code: Select all

#include <iostream>
using namespace std;

#include "irrlicht.h"
#pragma comment(lib, "Irrlicht.lib")
using namespace irr;
using namespace core;
using namespace gui;
using namespace io;
using namespace video;
using namespace scene;

bool exitApp = false;
IMesh* mesh;
ISceneNode *node;
bool keys[KEY_KEY_CODES_COUNT];

class MyEventReceiver : public IEventReceiver
{
public:
	virtual bool OnEvent(const SEvent& event)
	{
	if (event.EventType == EET_KEY_INPUT_EVENT)
	{
		//********************************************
		if (event.KeyInput.Key == KEY_KEY_W && !event.KeyInput.PressedDown)
		{
				keys[KEY_KEY_W] = true;
		}
		//********************************************

		//********************************************
		if (event.KeyInput.Key == KEY_KEY_Q && !event.KeyInput.PressedDown)
		{
				keys[KEY_KEY_Q] = true;
		}
		//********************************************

		//********************************************
		if (event.KeyInput.Key == KEY_ESCAPE && !event.KeyInput.PressedDown)
		{
				exitApp = true;
		}
		//********************************************
	}

		return false;
	}
};

int main()
{
	MyEventReceiver receiver;
	IrrlichtDevice *device;
	IVideoDriver* driver;
	ISceneManager* smgr;
	
	device = createDevice(EDT_OPENGL, dimension2d<s32>(800,600),32,false, false, false,&receiver);
	driver = device->getVideoDriver();
	smgr = device->getSceneManager();
	smgr->addCameraSceneNodeFPS();

	while(device->run() && !exitApp)
	if (device->isWindowActive())
	{
		driver->beginScene(true, true, video::SColor(255,25,155,35));

		if (keys[KEY_KEY_W])
		{
			keys[KEY_KEY_W] = false;

			mesh = smgr->getMesh("data/home.b3d");
			node = smgr->addMeshSceneNode(mesh);
			node->setScale(vector3df(1000,1000,1000));
			node->setMaterialFlag(EMF_LIGHTING,false);
		}

		if (keys[KEY_KEY_Q])
		{
			keys[KEY_KEY_Q] = false;

			//remove node
			node->remove();
			//remove textures
			s32 mbCount;
			mbCount = mesh->getMeshBufferCount();
			
			for (s32 i=0;i<mbCount;i++)
			{
				ITexture *toRemove = NULL;
				toRemove = mesh->getMeshBuffer(i)->getMaterial().getTexture(0);
				if (toRemove != NULL) driver->removeTexture(toRemove);
				toRemove = NULL;
				toRemove = mesh->getMeshBuffer(i)->getMaterial().getTexture(1);
				if (toRemove != NULL) driver->removeTexture(toRemove);					
			}
			//remove mesh
			smgr->getMeshCache()->removeMesh(mesh);
		}

		smgr->drawAll();
		driver->endScene();
	}
	device->drop();
	return 0;
}

W - loads a mesh and node and Q - releases it. I get a crash when the textures are removed. Looking at the mesh in IrrEdit the first three meshbuffers have textures like so:
data/texture1.bmp
data/texture2.bmp
data/texture1.bmp
and the program crashes on the third meshbuffer. I preinitialized the "toRemove" texture with NULL, and check afterwards, so if a texture has already been removed it should skip it.

Another thing: In the example the program flow is something like this:
begin scene
if something load scene
if something remove scene
draw
end scene

In the main application, which somehow works the flow after I compress all the intermediary functions is like this:

Current frame:
begin scene
if boolLoading
{
remove scene and reload another
boolLoading = false
}
if something boolLoading = true
draw
end scene

Next frame (with boolLoading true):
begin scene
if boolLoading
{
remove scene and reload another
boolLoading = false
}
if something boolLoading = true
draw
end scene

I tried this program flow for the test program but is still crashes at texture removal.

[pera]

I tried your program, it works fine under DirectX on my machine (I cant run OpenGL). I had to provide b3d file (I had one with only one texture), and put it in the right link (data/home.b3d) and all works fine, no crash on Irrlicht 1.4.

[JP]

He's not getting a crash, he's getting a memory leak

[pera]

first he was leaking, but then he got a crash.

[JP]

I should think the code he posted is the non-crashing version

[Iaro]

The code is for the crashing version. I still can't make it to work. I've put the texture names in an array, extracted the unique names and only removed those. So now it removes the textures, but still crashes on the next frame. I've also tried node->setMesh instead or removing the node but it doesn't work.

[rogerborg]

Let's try and take some variables out, and do a deterministic test.

Code: Select all

#include <windows.h>
#include <psapi.h>
#pragma comment(lib, "psapi.lib")

#include "irrlicht.h"
#pragma comment(lib, "Irrlicht.lib")
using namespace irr;
using namespace core;
using namespace gui;
using namespace io;
using namespace video;
using namespace scene;

// Windows only!
void showMemoryUsage()
{
    static size_t firstWorkingSetSize = 0;

    HANDLE myProcess;
    if(DuplicateHandle(GetCurrentProcess(), 
                      GetCurrentProcess(), 
                      GetCurrentProcess(),
                      &myProcess, 
                      0,
                      FALSE,
                      DUPLICATE_SAME_ACCESS))
    {
        PROCESS_MEMORY_COUNTERS counters;
        if(GetProcessMemoryInfo(myProcess, &counters, sizeof(counters)))
        {
            (void)printf("\nPeak working set  : %d\n", counters.PeakWorkingSetSize);
            (void)printf("Working set       : %d\n", counters.WorkingSetSize);

            if(0 == firstWorkingSetSize)
            {
                firstWorkingSetSize = counters.WorkingSetSize;
            }
            else
            {
                (void)printf("Working set change: %d\n",
                    counters.WorkingSetSize - firstWorkingSetSize);
            }
        }

        CloseHandle(myProcess);
    }
}

class MyEventReceiver : public IEventReceiver
{
    virtual bool OnEvent(const SEvent & theEvent)
    {
        if(theEvent.EventType == EET_LOG_TEXT_EVENT)
            return true;

        return false;
    }
};

int main()
{
    MyEventReceiver receiver;
    IrrlichtDevice *device;
    IVideoDriver* driver;
    ISceneManager* smgr;

    device = createDevice(EDT_OPENGL, dimension2d<s32>(800,600),32,false, false, false, &receiver);
    driver = device->getVideoDriver();
    smgr = device->getSceneManager();
    smgr->addCameraSceneNodeFPS();


    device->run();
    driver->beginScene(true, true, SColor());
    smgr->drawAll();
    driver->endScene();

    for(int iterations = 0; iterations < 100; ++iterations)
    {
        // Add it...
        IAnimatedMesh * animatedMesh = smgr->getMesh("../../media/dwarf.x");
        IAnimatedMeshSceneNode * node = smgr->addAnimatedMeshSceneNode(animatedMesh);

        // Take it out...
        animatedMesh = node->getMesh();
        IMesh * mesh = animatedMesh->getMesh(0);
        node->remove();
        node = 0;

        u32 meshBufferCount = mesh->getMeshBufferCount();

        for (u32 mb = 0; mb < meshBufferCount; mb++)
        {
            IMeshBuffer * meshBuffer = mesh->getMeshBuffer(mb);

            for(int texture = 0; texture < MATERIAL_MAX_TEXTURES; ++texture)
            {
                ITexture * toRemove = meshBuffer->getMaterial().getTexture(texture);
                if(toRemove)
                    driver->removeTexture(toRemove);
            }
        }

        smgr->getMeshCache()->removeMesh(animatedMesh);

        showMemoryUsage();
    }

    device->drop();
    return 0;
}

Running this, there's an apparent upwards trend in the working set, although I wouldn't 100% trust this figure. Perhaps one of the linux chaps can valgrind it?

The meshes and textures do appear to be being dropped (and deleted) and the cache entries removed, so it's not immediately apparent to me where the memory is leaking (if it is). I'll try to investigate further.

=====
Update
=====

Doing the same test but with sydney.md2 - i.e. not loading any textures - shows the working set peaking very quickly. That indicates that any problem might be with textures, rather than meshes or scene nodes.

=======
Update 2
=======

Loading sydney.md2 and adding textures:

Code: Select all

        IAnimatedMesh * animatedMesh = smgr->getMesh("../../media/sydney.md2");

        IAnimatedMeshSceneNode * node = smgr->addAnimatedMeshSceneNode(animatedMesh);
        SMaterial material;
        material.setTexture(0, driver->getTexture("../../media/sydney.bmp"));
        material.setTexture(1, driver->getTexture("../../media/faerie2.bmp.bmp"));
        node->getMaterial(0) = material;

Shows the working set jumping around, but with no significant upwards trend.

I'm open to ideas. I'd suspect the .x loader, but MSVC is reporting that nothing is leaking on exit, so that seems unlikely. Still, in the absence of anything else to check...

[hybrid]

Please note that even if you just use one texture in a certain area of a mesh, it could be put into several meshbuffers by Irrlicht. Please go through the meshbuffers and print all texture adresses, becasue I still assume that you're double freeing textures. Note that this heavily depends on the mesh file loader and mesh file you're using.
Still testing rogerborg's app...

[Iaro]

A little update: even after I removed exacly the needed textures and lightmaps, it would still crash when loading my test location, which is a b3d file with a lot of textures and lightmaps, so a lot of mesh buffers. The dwarf mesh with one texture and so one lightmap worked fine. I don't know exacly why but I thought about setting the driver material manually, maybe it was somehow used even if the texture is deleted, and the good news is that it works now. I don't know if this is something in my code, the location mesh or the b3d loader.
The next step would be to use Rogerborg's memory function to test for memory leaks, because the Task Manager maybe isn't the best debugging tool.

Here's the code:

Code: Select all

#include <iostream>
using namespace std;

#include "irrlicht.h"
#pragma comment(lib, "Irrlicht.lib")
using namespace irr;
using namespace core;
using namespace gui;
using namespace io;
using namespace video;
using namespace scene;

bool exitApp = false;
IMesh* mesh = NULL;
IMeshSceneNode *node = NULL;
bool keys[KEY_KEY_CODES_COUNT];

class MyEventReceiver : public IEventReceiver
{
public:
	virtual bool OnEvent(const SEvent& event)
	{
	if (event.EventType == EET_KEY_INPUT_EVENT)
	{
		//********************************************
		if (event.KeyInput.Key == KEY_KEY_W && !event.KeyInput.PressedDown)
		{
				keys[KEY_KEY_W] = true;
		}
		//********************************************

		//********************************************
		if (event.KeyInput.Key == KEY_KEY_Q && !event.KeyInput.PressedDown)
		{
				keys[KEY_KEY_Q] = true;
		}
		//********************************************

		//********************************************
		if (event.KeyInput.Key == KEY_ESCAPE && !event.KeyInput.PressedDown)
		{
				exitApp = true;
		}
		//********************************************
	}

		return false;
	}
};

int main()
{
	MyEventReceiver receiver;
	IrrlichtDevice *device;
	IVideoDriver* driver;
	ISceneManager* smgr;
	
	device = createDevice(EDT_OPENGL, dimension2d<s32>(800,600),32,false, false, false,&receiver);
	driver = device->getVideoDriver();
	smgr = device->getSceneManager();
	smgr->addCameraSceneNodeFPS();

	while(device->run() && !exitApp)
	if (device->isWindowActive())
	{
		
		driver->beginScene(true, true, video::SColor(255,25,155,35));
		smgr->drawAll();
		driver->endScene();

		if (keys[KEY_KEY_W])
		{
			keys[KEY_KEY_W] = false;

			mesh = smgr->getMesh("data/home.b3d");
			node = smgr->addMeshSceneNode(mesh);
			node->setScale(vector3df(1000,1000,1000));
			node->setMaterialFlag(EMF_LIGHTING,false);				
		}

		if (keys[KEY_KEY_Q])
		{
			keys[KEY_KEY_Q] = false;

			mesh = node->getMesh();
			//remove node
			node->remove();
			node = 0;
			//remove textures
			s32 mbCount;
			mbCount = mesh->getMeshBufferCount();

			array<stringc> textures;
			array<stringc> lightmaps;
			
			for (s32 i=0;i<mbCount;i++)
			{
				if (mesh->getMeshBuffer(i)->getMaterial().getTexture(0))
				textures.push_back(mesh->getMeshBuffer(i)->getMaterial().getTexture(0)->getName());
				if (mesh->getMeshBuffer(i)->getMaterial().getTexture(1))
				lightmaps.push_back(mesh->getMeshBuffer(i)->getMaterial().getTexture(1)->getName());
			}
			
			//make the texture names array unique, and remove textures
			for (u32 i=0;i<textures.size();i++)
			{
				if (textures[i] != "removed")
				{
					for (u32 j=0;j<textures.size();j++)
					{
						if ((i!=j) && (textures[i] == textures[j]))
						{
							textures[j] = "removed";							
						}
					}
				driver->removeTexture(mesh->getMeshBuffer(i)->getMaterial().getTexture(0));	
				cout<<"Removed: "<<textures[i].c_str()<<endl;
				}				
			}

			
			
			//make the lightmaps names array unique, and remove lightmaps
			for (u32 i=0;i<lightmaps.size();i++)
			{
				if (lightmaps[i] != "removed")
				{
					for (u32 j=0;j<lightmaps.size();j++)
					{
						if ((i!=j) && (lightmaps[i] == lightmaps[j]))
						{
							lightmaps[j] = "removed";							
						}
					}
				driver->removeTexture(mesh->getMeshBuffer(i)->getMaterial().getTexture(1));	
				cout<<"Removed: "<<lightmaps[i].c_str()<<endl;
				}



			}

			textures.clear();
			lightmaps.clear();

			SMaterial s;
			driver->setMaterial(s);

			//remove mesh
			smgr->getMeshCache()->removeMesh(mesh);		
		}
	}
	device->drop();
	return 0;
}

[Iaro]

I've done some tests using Rogerborg's memory function:

1. b3d mesh, 100 iterations: the memory increases. Data

2. different b3d meshes, 100 iterations, same problem. Data

3. b3d mesh, 1000 iterations. Noticeable leak. Data

4. x mesh (dwarf.x), 1000 iterations. Stable. Data

5. x mesh (the b3d file converted to .x),lots of textures, no lightmaps, 1000 iterations. Stable. Data

So either there's a problem with the way I remove lightmaps, but they are removed exacly the same way as the textures, or there's something wrong with the b3d loader. The data is exported from Blender. Another thing I'll try is use the IrrBlend exporter and see if the problem persists for a mesh with textures and lightmaps.

[rogerborg]

Actually, I'm not convinced that we are seeing genuine upwards trends, since I'm also seeing the working set drop, and it does seem to peak eventually for either .b3d or .x. The working set isn't really an accurate summary of allocs/deallocs, it's just a general indication.

Ideally we'd accurately trap and record all allocations and deallocations. I normally use fortify for this, but I haven't tried shoehorning it into Irrlicht, since it requires getting it included in every file that does memory allocation. I might take a poke at it later. Or I might not. I'm a creature of whim.