How big is too big?

g0bl1n · Post by **g0bl1n** » Sun Jun 26, 2011 6:13 am

I was wondering what everyone thinks is too big when it comes to using a computer's memory? Because I'm getting up there, when I make my map pretty big my program gets bigger as well (200000k). So how big is too big? The majority of "small games" I've seen run at around 5000k-10000k. But when it comes to games that use the Frostbite engine and the Unreal engines and such I can imagine they can get quite large. If I wanted to port my game to another system (don't even know if you can with Irrlicht, but just figured I'd put it out there) my game wouldn't be able to run on a ds or psp (unless they have some kind of memory paging system). So in any case, just figured I'd throw this question out, how much memory usage is too much?

On a side note, I wonder how much memory these games use:
Starcraft
Diablo 2:LOD
Minecraft (Client)

I imagine Minecraft could get quite large. same with the server side for Diablo 2:LOD. But Starcraft came out way back when I was clippin' with 256 mb of ram and 250 mb of harddrive memory, all I remember was that it was 32 mb of hard drive space. Just curious though!

Goblin

hendu · Post by **hendu** » Sun Jun 26, 2011 7:58 am

Minecraft leaks like crazy, not a good comparison. That, and java causing it's starting RAM consumption to already be a magnitude higher than an app written in a compiled language.

200mb is not much on today's computers. Yeah, it won't run even on current consoles, but neither does irrlicht AFAIK.

Also, the professional games do paging as you guessed. They only keep data as cache if there's more than the minimum ram, otherwise there are loading times.

g0bl1n · Post by **g0bl1n** » Sun Jun 26, 2011 8:59 am

I figured as much (with Irrlicht not running on handhelds and such, but jirr could probably run on them through a web browser...maybe). Thing is my game is 2D and I just figured it might be cool to at one point have a handheld version, I bet it wouldn't be hard to switch it over to opengl (in fact I know it wouldn't).

Prepare yourself, there be a giant wall of text ahead...

Maybe I can cut down on the memory I'm using somehow. My biggest problem is my endless use of vectors (std ones). I have a parent object called Object (I know, creative!). In this object I have a vector called Mask. A Mask contains a vector of Polygon pointers. And a Polygon contains a vector of Vertex pointers, and a Vertex contains two f32s. To do collisions I check the two Object's Mask objects against each other to determine a collision. Which is reasonable I think. But since my game has a custom ship builder, each piece is an Object, but they also contain a Polygon of their own. When one of these pieces is on the ground the player can collide with it, so the Polygon's pointer is added it's Mask vector. But when it is connected to a ship I remove that polygon from it's piece's Mask vector, and add it to the ship vector, that way I can check each ship to the other ship, and pieces belonging to the same ship do not collide with one another. But I did a test with this, when there are 999999 ship pieces created, the game takes up 201 mb of memory. I don't know if I would have 999999 pieces on the screen (each with 1 Mask containing 1 Polygon containing 4 Vertices). Not sure how I could do this differently though, I've been thinking about it for a long time, and this seems to be the only logical way to do this. Thinking about collisions makes my brains hurt...

Radikalizm · Post by **Radikalizm** » Sun Jun 26, 2011 11:55 am

Memory management in handheld devices and consoles is completely different from memory management on a pc, different types of architectures will provided different ways of handling your memory

Before you go thinking about complex schemes on reducing memory usage, first look at the obvious points many people seem to forget:
Are your running a debug build or a release build to measure memory usage?
Did you check for any memory leaks?
Are you using any overly complex resources where they're not needed? (eg. hi-res textures for small details)

Also, I have absolutely no idea of the concept of your game, so I really don't have a clue on what you're trying to accomplish with your whole collision system, maybe give some more details about your game?

By the way: handheld devices like the PSP and DS aren't exactly the easiest platforms out there, I believe the DS even requires you to write in assembler (and development for these platforms is almost impossible anyway since you'll need to be a licensed developer)

hendu · Post by **hendu** » Sun Jun 26, 2011 2:31 pm

Worst case eh? Well, outside of removing unnecessary data (collision data only for the whole ship like you said), you could check whether you really need dynamic vectors everywhere. Do the vertices of a part change for example?

Interesting how I did a similar calculation yesterday

I measured the worst case of recording all movement, and came up with 600mb per hour. I feel like that's perfectly ok to keep in ram in my case.

ent1ty · Post by **ent1ty** » Sun Jun 26, 2011 2:54 pm

Lol, at first I thought you are a girl who is asking advice on.. eh.. men's machinery

g0bl1n · Post by **g0bl1n** » Sun Jun 26, 2011 11:12 pm

@Radikalizm My game is kind of like Garry's Mod if you've ever played it, except it's more restrictive (can't create monsters or things out of thin air). It's a top-down shooter where you build ships out of small blocks and fly around in space killing other ships made out of blocks. I think a picture helps a bit:

I plan to have two controllers: a cockpit and a chair. You can build ships that you can walk inside of (this is where you need a chair) and you can build ships like in the picture above with a cockpit (because there is really 3 layers of each ship: floor, walls, top). When you shoot a ship it's where ever the mouse (x,y) is, so you can destroy the cockpit of a ship and disable it. But since you cannot see the chair controller that is inside of a ship it is a little harder to destroy. Same thing with engines and weapons, each part can be targeted individually. So that's why I need a polygon for each part, so I can target each one individually. Every ship has to have a controller, this serves as a base for building. So a controller has three vectors, one for each "level" of the ship (floor, walls, top). Also NPCs have a circle mask, and I haven't quite figured out how to include this inside a mask yet, that's on my things to do (although I do have circle to polygon collisions figured out).

I have no memory leaks and I'm only using one image (the one above). And with 100 blocks in the game I have 400k in memory (I think the blocks are only 250k or so of that).

I was thinking about making my own vector class, because an empty vector is 16k (I think it's like three iterators and an accessor? I'm not quite sure). I'm pretty sure I could get an empty iterator to 8k (a pointer and an int). But I'm not quite sure how I would go about doing it, something like:

Code: Select all

template <typename T>
class vector
{
     public:
          vector():_array(NULL),_size(0)
          {}
          ~vector()
          {
               if(_array!=NULL)
               {
                    delete[] _array;
               }
          }

          void push_back(T item)
          {
               if(_array==NULL)
               {
                    size++;
                    _array=new T[_size];
                    _array[size-1]=item;
               }
               else
               {
                    T *temp=_array;
                    _array=new T[_size+1];

                    for(unsigned int ii=0;ii<_size;ii++)
                    {
                         _array[ii]=temp[ii];
                    }

                    delete temp;
                    _size++;
                    _array[_size-1]=item;
               }
          }

          void erase(const unsigned int ind)
          {
               if(ind>=_size)
               {
                    throw runtime_error("Index passed is out of bounds!!!");
               }

               T *temp=_array;
               _array=new T[_size-1];

               T *swapper=temp[ind];
               temp[ind]=temp[_size-1];
               temp[_size-1]=*swapper;

               _size--;

               for(unsigned int ii=0;ii<_size;ii++)
               {
                    _array[ii]=temp[ii];
               }

               delete[] temp;
          }

          T operator[](const unsigned int ind)
          {
               if(ind>=_size)
               {
                    throw runtime_error("Index passed is out of bounds!!!");
               }

               return _array[ind];
          }

     private:
          T *_array;
          int _size;
};

Well, I ended up writing a class when I was doing this reply, dunno if it works though. It'll be a bit inefficient (because there is a lot of copying going on). But it would be smaller in memory.

@ent1ty Yea I have that problem...ahhh...I just made myself sad...

Brkopac · Post by **Brkopac** » Mon Jun 27, 2011 3:56 am

g0bl1n wrote:@Radikalizm My game is kind of like Garry's Mod if you've ever played it, except it's more restrictive (can't create monsters or things out of thin air). It's a top-down shooter where you build ships out of small blocks and fly around in space killing other ships made out of blocks. I think a picture helps a bit:

I plan to have two controllers: a cockpit and a chair. You can build ships that you can walk inside of (this is where you need a chair) and you can build ships like in the picture above with a cockpit (because there is really 3 layers of each ship: floor, walls, top). When you shoot a ship it's where ever the mouse (x,y) is, so you can destroy the cockpit of a ship and disable it. But since you cannot see the chair controller that is inside of a ship it is a little harder to destroy. Same thing with engines and weapons, each part can be targeted individually. So that's why I need a polygon for each part, so I can target each one individually. Every ship has to have a controller, this serves as a base for building. So a controller has three vectors, one for each "level" of the ship (floor, walls, top). Also NPCs have a circle mask, and I haven't quite figured out how to include this inside a mask yet, that's on my things to do (although I do have circle to polygon collisions figured out).

I have no memory leaks and I'm only using one image (the one above). And with 100 blocks in the game I have 400k in memory (I think the blocks are only 250k or so of that).

I was thinking about making my own vector class, because an empty vector is 16k (I think it's like three iterators and an accessor? I'm not quite sure). I'm pretty sure I could get an empty iterator to 8k (a pointer and an int). But I'm not quite sure how I would go about doing it, something like:
Code: Select all
template <typename T>
class vector
{
     public:
          vector():_array(NULL),_size(0)
          {}
          ~vector()
          {
               if(_array!=NULL)
               {
                    delete[] _array;
               }
          }

          void push_back(T item)
          {
               if(_array==NULL)
               {
                    size++;
                    _array=new T[_size];
                    _array[size-1]=item;
               }
               else
               {
                    T *temp=_array;
                    _array=new T[_size+1];

                    for(unsigned int ii=0;ii<_size;ii++)
                    {
                         _array[ii]=temp[ii];
                    }

                    delete temp;
                    _size++;
                    _array[_size-1]=item;
               }
          }

          void erase(const unsigned int ind)
          {
               if(ind>=_size)
               {
                    throw runtime_error("Index passed is out of bounds!!!");
               }

               T *temp=_array;
               _array=new T[_size-1];

               T *swapper=temp[ind];
               temp[ind]=temp[_size-1];
               temp[_size-1]=*swapper;

               _size--;

               for(unsigned int ii=0;ii<_size;ii++)
               {
                    _array[ii]=temp[ii];
               }

               delete[] temp;
          }

          T operator[](const unsigned int ind)
          {
               if(ind>=_size)
               {
                    throw runtime_error("Index passed is out of bounds!!!");
               }

               return _array[ind];
          }

     private:
          T *_array;
          int _size;
};
Well, I ended up writing a class when I was doing this reply, dunno if it works though. It'll be a bit inefficient (because there is a lot of copying going on). But it would be smaller in memory.

@ent1ty Yea I have that problem...ahhh...I just made myself sad...

Don't fall into this trap, why are you attempting to recreate something that has been done and done and done and done. STL is there for a reason, its going to be much better than any container you think you can "replicate", there are even third party options (Boost, QT). I believe the only reason Irrlicht used its own custom containers is because of VS 6 "back in the day", please don't take that as a "proper" example.

Xaryl · Post by **Xaryl** » Mon Jun 27, 2011 5:49 am

ent1ty wrote:Lol, at first I thought you are a girl who is asking advice on.. eh.. men's machinery

Dirty dirty mind, ent1ty.

But back to the serious side:

g0bl1n wrote:How big is too big?

First of all, I have to admit that I didn't read through everything above carefully, and I am unable to answer your question directly, but please allow me to make a suggestion.

Somehow, I have a feeling that the limiting factor isn't going to be the memory, but the cpu usage from collision detection between bullets/ships.

Of course, the actual limiting factor depends on the exact mechanism of the game, the system it runs on, and many many other things, so I don't know for sure.

However, my suggestion is that you make the game work first, keep everything modular, and optimize at the end (and it should be obvious what the limiting factor is at that point).

Again, I didn't read the posts above too carefully, so I might be totally wrong, but the same idea applies.

Eigen · Post by **Eigen** » Mon Jun 27, 2011 8:58 am

[Not Irrlicht/Game related at all]
During the short period I've been working as a developer for mobile devices (iOS, Android), I've done some memory sensitive apps for the iOS and the older devices pretty much crash and burn when memory usage goes above 10-11 MB which is very little if you think about it. Of course, interanlly, it gives you warnings before that, which you can use to release stuff and so on.
It was a tv-guide app with over 150 channels, each with around 40 programmes a day and the client required that the whole guide should be instantly scrollable with no data loading after first load. Storing the data for that is one thing. Eventually the server response was cut to smallest pieces of data I needed, which was a variable length string of title, an integer of channel ID and 2 doubles for start and end timestamps (20 bytes + size of the string .. 1 byte per char, 15-40 characters per title + string object overhead). That's at least 40 bytes per programme. That's almost 2 megs just in programme data. Getting all that data to render with an adequate speed and storing it in memory was another thing of course, I went with tiled layers which, together with memory, is handled by iOS internally but still, it's not like it uses no resources, oh no.

Anyway, once the app started and reached the splash page with nothing really loaded, memory usage was at 1.6 megs already (Android version of the app in that state used around 3 mb, just sitting there having done nothing.) Then moving into the guide it reached around 5 megs once all the visible channel rows had been rendered. Moving down of course started increasing that bit by bit. If you clicked on the programme, a new view was shown the rest of the programme data was loaded. Again, more memory used up. So on and on.

The way iPhone handles moving between different views is that each view is loaded into the memory when it's first shown but is not removed once it disappears. It's only removed when a memory warning is received and it's not the currently visible one. Still, a pain to handle that and release all the used resources which never seemed to decrease memory usage as much as they had used up .. go figure.
The thing is, once you've been to most of the screens in the app, the memory usage is so high (with all the UI elements and images loaded) the guide view kept crashing when moving up and down a lot. So, yeah, I spent ages trying to keep it from doing that. Luckily the iPhone Simulator has a "Simulate memory warning" functionality. Superb.

Keeping stuff in memory, most of the time is a good thing, unlike Android which pretty much destroys the screen once you move away from it and have to reload it the next time. Funny thing about that, happened to the other Android developer here during the project .. Android *sometimes but not always* before removing the previous screen makes an internal image of that screen so it can quickly show it the next time before drawing the real thing. It does that on most devices I think, but it only causes problems on small devices like HTC Wildfire, where the memory usage is very limited. Anyway, the image needed like 4 megs of memory and the app simply caused a "force close" because Android could not figure out that there's not enough room to fit it. Yeah, instead of not storing the image which is not that necessary anyway it decided crash. So, next time you use some Android app on one of the smallest devices and the app suddenly crashes, it's not always the developers fault. Most of the time it is, but not always

Not sure what the point of all this was, but oh well. Related to that, I'll be very impressed once I see Minecraft run on iOS and Android, seeing that the PC version leaks as hell.

g0bl1n · Post by **g0bl1n** » Mon Jun 27, 2011 9:27 pm

@Eigen I had no idea that developing mobile applications was this complicated (in terms of memory management). I figured there would be a "standard" to keep things in memory (delete when memory is freed, allocate when possible). Is there a mobile version of Minecraft coming out? It barely runs for a couple hours on my gaming desktop before crashing (as you said, it leaks like crazy), I wonder how the developers of Minecraft will handle all the memory for every block? That would be interesting to see.

Kojack · Post by **Kojack** » Tue Jun 28, 2011 12:52 am

and 2 doubles for start and end timestamps

That's a very accurate tv guide, considering 64 bit (if used as an int) would let you specify start and end times accurate to microseconds within the next 584,554 years.

Eigen · Post by **Eigen** » Tue Jun 28, 2011 4:58 am

Kojack wrote:
and 2 doubles for start and end timestamps
That's a very accurate tv guide, considering 64 bit (if used as an int) would let you specify start and end times accurate to microseconds within the next 584,554 years.

Right, I'll have to look at the source and see if they were standard integers instead, would make more sense, yes. My bad. Not that it would matter, because the data is not stored in any custom object, but a standard object containing key-pair values, because that's what the JSON parser parses into and is the fastest to store. So that internally has quite more overhead than doubles anyway.

About Minecraft, I don't know much about it, just what I've been reading in different places but they say it's in development. Not that I care about the game nor want to play it, it's just the techincal side that interests me

g0bl1n wrote:I had no idea that developing mobile applications was this complicated (in terms of memory management). I figured there would be a "standard" to keep things in memory (delete when memory is freed, allocate when possible).

It shouldn't be. And most of the time it isn't, because a lot of apps are really simple (like the basic news apps and so on). It's just some very specific apps that require a rethink and don't allow you to just hack something together. Also, having bunch of different phones to support does not help. Compared to Android, iPhones are much better to develop for as there are mostly iPhone4-s and very small segment of older iPhone3G-s (and iPod Touches which basically have the same hardware and capability), which if you have to support, might need some special attention, again depending on the type of the app. And there always odd little kinks between the supposedly identical versions of phones or pads. There also issues with the different versions of iOS and so on.

Yes, that's how memory is managed but it's not enough sometimes. You simply can't just unload stuff at a random point because eventually you have to load it in again and the user will not tolerate a loading spinner on every screen. Of course, things that are not necessary are removed the second they become useless.
But I think it's fun tackling these issues because on the PC you tend to forget about these things as you have pretty much all the resources you need available and don't really have to think about it.