Bug? Memory release depends on some order !?

You are an experienced programmer and have a problem with the engine, shaders, or advanced effects? Here you'll get answers.
No questions about C++ programming or topics which are answered in the tutorials!
Post Reply
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Bug? Memory release depends on some order !?

Post by Sinsemilla »

Hello,

First of all i am not sure if am experiencing a bug here or not. If not, i apologize for the wrong blame...
As it seems, the memory is not completely freed when scene nodes and their meshes are not released in the reverse order of creation. I made a small testprogram to reproduce this:

Code: Select all

 
#include <irrlicht.h>
 
using namespace irr;
using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;
 
int main(int argc, char **argv)
{
    IrrlichtDevice *device = createDevice(video::EDT_OPENGL, dimension2d<u32>(640, 480), 32, false, false, false, 0);
 
    if (!device) {
        return 1;
    }
 
    device->setWindowCaption(L"Hello World! - Irrlicht Engine Demo");
    IVideoDriver* driver = device->getVideoDriver();
    ISceneManager* smgr = device->getSceneManager();
    IGUIEnvironment* guienv = device->getGUIEnvironment();
 
    guienv->addStaticText(L"Hello World! This is the Irrlicht Software renderer!",
        rect<s32>(10,10,260,22), true);
 
    IAnimatedMesh* mesh1 = smgr->getMesh("../../sydney.md2");
    IAnimatedMesh* mesh2 = smgr->getMesh("../../sydney(copy).md2");
    IAnimatedMeshSceneNode* node1 = smgr->addAnimatedMeshSceneNode(mesh1);
    IAnimatedMeshSceneNode* node2 = smgr->addAnimatedMeshSceneNode(mesh2);
 
    smgr->getMeshCache()->removeMesh(mesh1->getMesh(0));
    node1->remove();
 
    smgr->getMeshCache()->removeMesh(mesh2->getMesh(0));
    node2->remove();
 
    while (device->run()) {
        driver->beginScene(true, true, SColor(255,100,101,140));
 
        smgr->drawAll();
        guienv->drawAll();
 
        driver->endScene();
    }
 
    device->closeDevice();
    device->drop();
    return 0;
}
 
 
On my machine, this program takes 30.8Mb in the main loop when node2 is removed first. If node1 is removed first the program takes 31.4Mb.
CuteAlien
Admin
Posts: 9734
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Bug? Memory release depends on some order !?

Post by CuteAlien »

Meshes don't use up much memory anyway (probably less than 1 MB even both together).
You could also clean-up textures with driver->removeAllTextures();
But most memory in this case will simply be the dll's - especially when compiling in debug.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

Thanks for your reply CuteAlien.

Besides, in the example i used animated meshes but it also applies for static meshes...

Anyway, i think i need to provide some more details...
I am hunting a sort of memory leak in my project which is quite hard to detect and which unfortunately is really big problem for me.

In my project, i need to load and unload a few hundred static meshes during the game. By static i mean that the flag EHC_STATIC is set on the meshbuffer. The meshes are loaded, and reloaded in a more or less random manner since they represent terrain tiles which are loaded when they are in the frustum aka. sight-radius of the player and unloaded if they fall out of that radius again. So if my terrain consists of 100 tiles, let's say for example that the tiles 28 - 59 are loaded, tile 0 - 27 stay unloaded also as 60 - 99, since they wouldnt be in the radius.

The problem i have, is that the memory doesn't go down completely when the tiles are deleted. The remaining memory is only freed when my game exits, e.g. after device->drop, it is never freed completely during the game. Over time the ram usage goes up and up and ends in gigabytes because with each load and unload of a tile i loose a little bit of memory until the game is ended.

The reason i am suspecting that some order is relevant, is that i can get the memory completely freed when i delete my meshes directly after they have been loaded or if i delete them in the reverse order. Imagine the following loading scenarios:

Code: Select all

 
void loadTiles() 
{
    for (int i = 0; i < 100; ++i) {
        loadTile();
        unloadTile();
    }
}
 
or

Code: Select all

 
void loadTiles() 
{
    for (int i = 0; i < 100; ++i) {
        loadTile();
    }
}
 
void unloadTiles() 
{
    for (int i = 99; i >= 0; --i) {
        unloadTile();
    }
}
 
Then there is no memory lost.

When the situation is changed to a more real world situation e.g. the tiles are just loaded one after another, and then deleted later in some other order or in single manner, some memory is lost as until device->drop is hit.
Imagine something like:

Code: Select all

 
void loadTiles() 
{
    for (int i = 0; i < 100; ++i) {
        loadTile();
    }
}
 
void unloadTiles() 
{
    for (int i = 0; i < 100; ++i) {
        unloadTile();
    }
}
 
or

Code: Select all

 
void loadTiles() 
{
    for (int i = 0; i < 100; ++i) {
        loadTile();
    }
}
 
void unloadTiles(int nTile) 
{
    for (int i = 0; i < 100; ++i) {
        if (nTile == 37) {
            unloadTile();
        }
    }
}
 
The meshes are loaded with my own function, not via smgr->getMesh(). I do this for efficiency because i need to get triangles for a bullet mesh and i don't want to go through all indices two times, resp. to loop through the indices after getMesh() since this is double work. If i understood things correctly this also means that no Mesh cache is used at all. The meshes have the Flag EHC_STATIC set on their meshbuffer, therefore also removeHardwarebuffer is called on removal.

The meshes are added to IMeshSceneNodes, i call drop after adding them so that the reference count is decreased to 1 so that they get deleted when m_pNode->remove is called.

Code: Select all

 
smgr->addMeshSceneNode(mesh);
mesh->drop();
 

The removal looks the following way:

Code: Select all

 
if (m_pnode != NULL) {
    scene::IMeshBuffe* mb = m_pNode->getMesh()->getMeshBuffer(0);
    device->getvideoDriver()->removeHardwareBuffer(mb);
 
    m_pNode->remove();
    m_pNode = NULL;
}
 
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

Some additional information: I use Ubuntu 16.04, GCC 4.9.0, and stock Irrlicht 1.8.3
CuteAlien
Admin
Posts: 9734
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: Bug? Memory release depends on some order !?

Post by CuteAlien »

I think hardware buffers are only removed 20 seconds after they got used for the last time (see CNullDriver::updateAllHardwareBuffers()).
Maybe it's something about that.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

I thought if i use removeHardwareBuffer() i would free the hardware buffers explicitly and therefore immediately, or not?..., in this thread: http://irrlicht.sourceforge.net/forum/v ... er#p277288, it is told so.
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: Bug? Memory release depends on some order !?

Post by hendu »

The GL driver may also cache things. You can't tell if you have a leak by the used memory number alone, use valgrind.
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

@hendu: Well, valgrind reports nothing at all since the memory is released on the ending of my game, resp. after device->drop. The problem i have, is that the memory never gets released before, resp. during the game. So, if my game is running it keeps using more and more ram over time.
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: Bug? Memory release depends on some order !?

Post by hendu »

In that case, use one of the valgrind tools, massif.
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

Thanks for the advice hendu, i didn't knew massif. :oops:

I did some runs with it over the little example i posted, and for the sake of simplicity i took away the two or three lines about gui stuff.

Code: Select all

 
#include <irrlicht.h>
 
using namespace irr;
using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;
 
int main(int argc, char **argv)
{
    IrrlichtDevice *device = createDevice(video::EDT_OPENGL, dimension2d<u32>(640, 480), 32, false, false, false, 0);
 
    if (!device) {
        return 1;
    }
 
    device->setWindowCaption(L"Hello World! - Irrlicht Engine Demo");
    IVideoDriver* driver = device->getVideoDriver();
    ISceneManager* smgr = device->getSceneManager();
 
    IAnimatedMesh* mesh1 = smgr->getMesh("../../sydney.md2");
    IAnimatedMesh* mesh2 = smgr->getMesh("../../sydney(copy).md2");
    IAnimatedMeshSceneNode* node1 = smgr->addAnimatedMeshSceneNode(mesh1);
    IAnimatedMeshSceneNode* node2 = smgr->addAnimatedMeshSceneNode(mesh2);
 
    smgr->getMeshCache()->removeMesh(mesh2->getMesh(0));
    node2->remove();
 
    smgr->getMeshCache()->removeMesh(mesh1->getMesh(0));
    node1->remove();
 
    while (device->run()) {
        driver->beginScene(true, true, SColor(255,100,101,140));
        smgr->drawAll();
        driver->endScene();
    }
 
    device->drop();
    return 0;
}
 
This code represents the following massif diagram:

Code: Select all

 
--------------------------------------------------------------------------------
Command:            ./IrrMemTest
Massif arguments:   (none)
ms_print arguments: massif.out.18795
--------------------------------------------------------------------------------
 
 
    MB
7.324^               ##                                                       
     |               #                                                        
     |             ::#                                                        
     |            :: #                                                        
     |          :::: #                                                        
     |          : :: #                                                        
     |        @:: :: #                                                        
     |        @:: :: #                                                        
     |      ::@:: :: #                                                        
     |      : @:: :: # :::: :  ::@::@ :  :::: : :: ::::: : :  : ::::     :::@:
     |    :@: @:: :: # ::: ::::: @: @::::: : :::::::: ::::::::::::::@:::::::@:
     |    :@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |    :@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |    :@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |   ::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |   ::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |   ::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     |   ::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     | : ::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
     | ::::@: @:: :: # ::: ::: : @: @::::: : :::::::: :: :::: ::::::@:::::::@:
   0 +----------------------------------------------------------------------->Mi
     0                                                                   707.1
 
And here is the diagram for the opposite order of deletion, aka. node1 first, then node2:

Code: Select all

 
--------------------------------------------------------------------------------
Command:            ./IrrMemTest
Massif arguments:   (none)
ms_print arguments: massif.out.18889
--------------------------------------------------------------------------------
 
 
    MB
7.324^               ##                                                       
     |               #                                                        
     |              :#                                                        
     |            :::#                                                        
     |           :: :#                                                        
     |           :: :#                                                        
     |         @::: :#                                                        
     |        @@::: :#                                                        
     |      ::@@::: :#                                                        
     |      : @@::: :#                                                        
     |      : @@::: :# ::::::::::::::@::::::::@:::::::::::::@:::::::::@::::::@
     |    ::: @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |    : : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |    : : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |   @: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |   @: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |   @: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     |   @: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     | ::@: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
     | : @: : @@::: :# : :: :: :::: :@:: ::: :@:::::::: : ::@: :::::::@::::::@
   0 +----------------------------------------------------------------------->Mi
     0                                                                   679.4
 
Sinsemilla
Posts: 38
Joined: Mon Jan 09, 2012 5:07 pm

Re: Bug? Memory release depends on some order !?

Post by Sinsemilla »

Besides, here is the diagram of my project, which means that about 700 tiles are loaded, and unloaded:

Code: Select all

 
    MB
687.8^                                                                     :  
     |                                                                  #:::  
     |                                                                  #:::  
     |                                                                @@#:::  
     |                                                             @@@@@#:::  
     |                                                         @@@@@@@@@#:::  
     |                                                     @@@@@@@@@@@@@#:::  
     |                                                 @:@:@ @@@@@@@@@@@#:::: 
     |                                              :@:@:@:@ @@@@@@@@@@@#:::: 
     |                                          @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                                      @:@@@@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                                   @@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                              @@@:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                           :@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                       @@@::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |                   ::::@ @::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |               ::@@: ::@ @::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |           @@:@::@@: ::@ @::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |        :::@ :@::@@: ::@ @::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
     |    @:@:: :@ :@::@@: ::@ @::@@@ @:@@@@@:@ @@:@:@:@:@:@ @@@@@@@@@@@#:::: 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   40.31
 
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: Bug? Memory release depends on some order !?

Post by hendu »

The graphs show memory being freed in the simple cases, but not in your program. Try to find out what's the difference.
Post Reply