I didn't think it was polite to spam the forum with a bunch of posts so I'll just summarize my general design questions here.
Octrees with voxels are slow, what data structure would be good for this? (goals: traversing the data faster than raw access and compression, run length encoding a flat array could work but there must be some more efficient solution (b-trees?)).
How would I select blocks? I read about the triangle selector but I don't know if that's the feature I'm looking for (what if I want to select multiple blocks?), in worst case I could probably hack it by raycasting and determining the location where the ray hit, then calculate the position in the tree or flat array.
If I want to create meshes in a second thread and then make them into nodes in the main thread, how would I do that?
if I grab() a mesh, add that pointer to a vector and then when the main thread turns it to a node I drop it, would that drop the mesh or just that reference of the mesh but still keep the reference used by the node? Do I even need to use grab or can I just plonk a pointer in a vector and then delete nuke that vector element? ( the reason for this is because there's very noticeable microstutter and delays when generating new chunks due to doing it in a single thread)
If I wish to check items in a vector to determine what their neighbors are, what's an efficient way to do that?
I could use a big if/else if/else block or a large switch statement but that seems inelegant and potentially slow since I have to do multiple comparisons (at least three to determine if the block is at the edge of the vector to not get an out of memory exception and then up to six to check the neighbors).
Does irrlicht support texture atlases? I can't use texture arrays as my minimum target platform does not support them, how would I optimize meshes with texture atlases (currently I have a few hundred thousand verts per chunk (completely unoptimized), VBOs do nothing. I read that VBOs get slower if you have too many vertices and are inefficient if you have too few, I'd like to hit a target of... 4000 or so, seems reasonable)? (I read that you can't do greedy meshing with them because it messes up UVs, this works for plain texture files since they can repeat but not for atlases and I need an atlas due to being pretty severely drawcall limited), I could probably figure out how to implement them if Irrlicht does not support them (from what I can tell the easiest method is loading a large texture and setting the UVs to fractions of 1, 0.25f for 25% of the image in one direction, correct me if I'm wrong)
Overall just a bunch of questions on topics I have little or no understanding of, data structures baffle me. There are so many and I wouldn't even begin to know which ones are suitable for what. I know that octrees aren't fast for voxels for two reasons, tried them and read a bunch of posts about it explaining exactly why they're slow for voxels).
On that matter, what setting is optimal for VBOs? Seeing the mesh is modifiable I'd assume EHM_DYNAMIC
EDIT: Oh and how would I create threads programatically? I know how to get the core, hardware thread count and various other data but I don't know how to create n threads based on hardware availability. (specifically the more threads that can build meshes the better)
Various general questions
-
Cube_
- Posts: 1010
- Joined: Mon Oct 24, 2011 10:03 pm
- Location: 0x45 61 72 74 68 2c 20 69 6e 20 74 68 65 20 73 6f 6c 20 73 79 73 74 65 6d
Various general questions
"this is not the bottleneck you are looking for"
Re: Various general questions
It's posted in beginner questions, but those aren't beginner topics. I know currently just one programmer personally who works on a voxel-engine (http://steamcommunity.com/sharedfiles/f ... =350265788). And he spend a long time on that already working full-time on it. Your questions are specific detail-problems for several huge topics.
There is for example no optimal setting for VBO's - there is only a optimal setting for a situation. The reason there are different settings is that this depends on the context it's used it. The fastest is EHM_STATIC, but you can't always use that.
Or there is no general telling about optimal data structures without knowing details about your level-layout. And what it's used for (rendering, collisions?). And even if we have that information it's not easy to tell. The real answer is - implement them all and then compare.
Even optimal way of selecting a block is not a trivial questions. I suppose you can find a bunch of articles online about that - and likely none of them will be easy to read.
And multi-threading... even 90% of the professional games avoid that except for the sound/music. When you are still struggling with vectors it will be hell for you to wrap your mind around multithreaded programming. Basically your first step is to read about data-races and the techniques to avoid them. You can't write multithreaded code that works correct until you did that. I was lucky to have a good university course about that, but maybe there are good tutorials about it online as well. The good news is - in most cases it's not worth it anyway.
You seem to have a very difficult game-concept and which tries to do it all at once. And it's stuff that takes even experienced programmers a long time to figure out. But maybe I can tell you how to get there. Start by splitting up your idea into seperate hard technical problems. And then do minor games for each of those which allow you to learn the technical challenges. For example - start by writing a voxel-floor. Not yet with cubes. Not optimized like hell. Just simple enough that it works and can maybe used for a voxel-pacman. Then write another game which uses blocks. Maybe a simple shooter which just has many blocks around the scene which you can shoot. That way you learn to work with them and line-collisions etc.
For multi-threading - not sure. Writing some demo might be nice to learn about that.
Such games might also take a while. I think my first "snake" like game took nearly a year until I was happy with the result. But that way you get way faster to your target than if you try to learn everything at the same time. And it has the advantage that you learn in each game to work with data-structures and you learn about code architecture way faster. For example after doing 2-3 small games you already start to see patterns about which parts of a game look more or less identical even thought you wrote different games. So those are architecture components which can be made re-usable.
Divide and conquer. Tackle the challanges one by one instead of all together. You will be able to write your dream-game way faster that way in the end.
There is for example no optimal setting for VBO's - there is only a optimal setting for a situation. The reason there are different settings is that this depends on the context it's used it. The fastest is EHM_STATIC, but you can't always use that.
Or there is no general telling about optimal data structures without knowing details about your level-layout. And what it's used for (rendering, collisions?). And even if we have that information it's not easy to tell. The real answer is - implement them all and then compare.
Even optimal way of selecting a block is not a trivial questions. I suppose you can find a bunch of articles online about that - and likely none of them will be easy to read.
And multi-threading... even 90% of the professional games avoid that except for the sound/music. When you are still struggling with vectors it will be hell for you to wrap your mind around multithreaded programming. Basically your first step is to read about data-races and the techniques to avoid them. You can't write multithreaded code that works correct until you did that. I was lucky to have a good university course about that, but maybe there are good tutorials about it online as well. The good news is - in most cases it's not worth it anyway.
You seem to have a very difficult game-concept and which tries to do it all at once. And it's stuff that takes even experienced programmers a long time to figure out. But maybe I can tell you how to get there. Start by splitting up your idea into seperate hard technical problems. And then do minor games for each of those which allow you to learn the technical challenges. For example - start by writing a voxel-floor. Not yet with cubes. Not optimized like hell. Just simple enough that it works and can maybe used for a voxel-pacman. Then write another game which uses blocks. Maybe a simple shooter which just has many blocks around the scene which you can shoot. That way you learn to work with them and line-collisions etc.
For multi-threading - not sure. Writing some demo might be nice to learn about that.
Such games might also take a while. I think my first "snake" like game took nearly a year until I was happy with the result. But that way you get way faster to your target than if you try to learn everything at the same time. And it has the advantage that you learn in each game to work with data-structures and you learn about code architecture way faster. For example after doing 2-3 small games you already start to see patterns about which parts of a game look more or less identical even thought you wrote different games. So those are architecture components which can be made re-usable.
Divide and conquer. Tackle the challanges one by one instead of all together. You will be able to write your dream-game way faster that way in the end.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
-
Cube_
- Posts: 1010
- Joined: Mon Oct 24, 2011 10:03 pm
- Location: 0x45 61 72 74 68 2c 20 69 6e 20 74 68 65 20 73 6f 6c 20 73 79 73 74 65 6d
Re: Various general questions
Yes, I figured some of these aren't beginner general beginner level questions, however I default to beginner help if I don't know where to post them.
For VBOs I'll just try them all, that's easy enough.
Implementing all data structures sounds like a massive pain, I can already exclude octrees, b-trees are a likely candidate as they work well for large datasets. (my level data is used for collision and rendering, seeing it's a reasonably simple mesh... might generate a simpler convex hull for that at some point if necessary). Either way, I'm not looking for "the best" data structure or the most optimal one, just one that's better than a flat array compressed using RLE (octrees can have up to linear performance per branch, given that it needs to traverse many branches it actually gets noticeably slower for a lot of cases with voxels, especially the more unique voxels you have).
As for block selection, I'm not looking for an optimal method. Merely a starting point (I assume that casting a ray, getting the location of the hit and then converting that to an array index is a valid way to solve the issue)
Multi threading is a topic I've read about extensively, at least on a conceptual level I understand how to avoid data races through various ways (locking, but I've also read about nonlocking methods), however what I primarily wanted was to know how to create a variable amount programmatically without just coding a list of how many to use, generating chunk meshes does not in itself pose a risk of data races as they're only added to the list of meshes to turn to nodes at the point of the chunk class terminating (and as such there can only be one concurrent user of that data, the only thing that could potentially have a race condition is the vector to store them as far as I can tell).
As for it not being worth it, in my case it is as no matter how much I optimize my chunk generator it will cause a noticeable freeze every time it loads a chunk unless it happens faster than a frame can draw (which won't happen), even when the chunks generate at optimal speed we're looking at an extra 20ms per frame per chunk, assuming I generate at most one per frame, that leaves a measly 13.333 (recurring) milliseconds for everything else, it's not possible. Even on a reasonably high end CPU (i5-3330S) they don't generate fast enough to allow proper rendering, it would cause noticeable slow frames whenever a chunk generates (which, if that can be delegated to a background thread that issue is solved). And do note that's assuming 30FPS is acceptable, which incidentally is within my margin of what I consider acceptable for the minimum recommended specs. at 60 FPS I can't even generate one chunk per frame because one chunk at optimal speed (not counting empty or near empty chunks) would be 5 milliseconds above acceptable limits, I guess I'll have to take the hard route and read research papers and dig through source code then.
My game concept is difficult, at least in some areas (specifically generating and processing stupid large amounts of data in realtime), however most of the modules are built in the way you suggested (in separate programs, that's mainly because it makes the testing faster).
Really though, is there any interface for modifying meshes that already exist and are part of scene nodes? (first of all I don't wish to regenerate a mesh every time I remove a block, I'd rather just add or remove faces, second of all I wish to optimize it using greedy meshing (an algorithm which I understand, however it requires the entire chunk to be generated first), I don't know how to deal with texturing issues as I can't tile textures from an atlas (without hacks at least)).
Now... on the topic of other projects, I could probably scavenge my simple water simulator and my physics engine (not realistic, good enough for my purposes, it does gravity and basic collision. It can throw blocks around which is all I care about).
I've been thinking, my game would be a lot easier to complete if I hadn't stubbornly decided to optimize everything so it runs on as low end hardware as possible, ah well. I enjoy this insanity.
I guess I have a long week of research ahead of me then
(not to mention I intend to implement more advanced physics features like structural failure and whatnot, insanity is a very good descriptor).
For VBOs I'll just try them all, that's easy enough.
Implementing all data structures sounds like a massive pain, I can already exclude octrees, b-trees are a likely candidate as they work well for large datasets. (my level data is used for collision and rendering, seeing it's a reasonably simple mesh... might generate a simpler convex hull for that at some point if necessary). Either way, I'm not looking for "the best" data structure or the most optimal one, just one that's better than a flat array compressed using RLE (octrees can have up to linear performance per branch, given that it needs to traverse many branches it actually gets noticeably slower for a lot of cases with voxels, especially the more unique voxels you have).
As for block selection, I'm not looking for an optimal method. Merely a starting point (I assume that casting a ray, getting the location of the hit and then converting that to an array index is a valid way to solve the issue)
Multi threading is a topic I've read about extensively, at least on a conceptual level I understand how to avoid data races through various ways (locking, but I've also read about nonlocking methods), however what I primarily wanted was to know how to create a variable amount programmatically without just coding a list of how many to use, generating chunk meshes does not in itself pose a risk of data races as they're only added to the list of meshes to turn to nodes at the point of the chunk class terminating (and as such there can only be one concurrent user of that data, the only thing that could potentially have a race condition is the vector to store them as far as I can tell).
As for it not being worth it, in my case it is as no matter how much I optimize my chunk generator it will cause a noticeable freeze every time it loads a chunk unless it happens faster than a frame can draw (which won't happen), even when the chunks generate at optimal speed we're looking at an extra 20ms per frame per chunk, assuming I generate at most one per frame, that leaves a measly 13.333 (recurring) milliseconds for everything else, it's not possible. Even on a reasonably high end CPU (i5-3330S) they don't generate fast enough to allow proper rendering, it would cause noticeable slow frames whenever a chunk generates (which, if that can be delegated to a background thread that issue is solved). And do note that's assuming 30FPS is acceptable, which incidentally is within my margin of what I consider acceptable for the minimum recommended specs. at 60 FPS I can't even generate one chunk per frame because one chunk at optimal speed (not counting empty or near empty chunks) would be 5 milliseconds above acceptable limits, I guess I'll have to take the hard route and read research papers and dig through source code then.
My game concept is difficult, at least in some areas (specifically generating and processing stupid large amounts of data in realtime), however most of the modules are built in the way you suggested (in separate programs, that's mainly because it makes the testing faster).
Really though, is there any interface for modifying meshes that already exist and are part of scene nodes? (first of all I don't wish to regenerate a mesh every time I remove a block, I'd rather just add or remove faces, second of all I wish to optimize it using greedy meshing (an algorithm which I understand, however it requires the entire chunk to be generated first), I don't know how to deal with texturing issues as I can't tile textures from an atlas (without hacks at least)).
Now... on the topic of other projects, I could probably scavenge my simple water simulator and my physics engine (not realistic, good enough for my purposes, it does gravity and basic collision. It can throw blocks around which is all I care about).
I've been thinking, my game would be a lot easier to complete if I hadn't stubbornly decided to optimize everything so it runs on as low end hardware as possible, ah well. I enjoy this insanity.
I guess I have a long week of research ahead of me then
(not to mention I intend to implement more advanced physics features like structural failure and whatnot, insanity is a very good descriptor)."this is not the bottleneck you are looking for"