Irrlicht's OWN file formats?

[Etory]

There was already a loader for this format, using libblend or something like that. I guess you'll find some heads up when searching for blend

I suppose that you're talking about readblend. It seems that it's deprecated with Blender 2.5

it may be useful for a converter or 3d application but not for a game

[hybrid]

Well, any loader for this format would probably be deprecated as well with the new version. That's the problem with binary dumps of program states. They change too fast and too much, and are sparsely documented if at all.

[REDDemon]

Requirements of the new format:

-Binary/text(fast reading/easy debbuging)
-Format wich requires a very simple loader/exporter. If writing an exporter is simple people will write more exporter for 3D editing tools.
-Fast loading. Some nice enconding for the binary format will be great..

I think that first of all there's the need to decide what to include in the file. (only vertices, and indices + bones? or also material, lights etc.?)
Maybe just think also to problems wich are caused by Irrlicht API, some little changes with animated meshes will help a lot. (both for the serialization problem, but also for skinning. It is not very nice to have 2 separate structures 1 wich uses bone scene nodes and another structure for GPU skinning with more blend weights for each vertex).

Before making a new file format every other option must be checked. Else after implementing the file format is too late for major changes.

[Radikalizm]

@REDDemon:
You're kind of repeating what was already said in the thread

I've been working on some basic designs, I've discussed it with xirta some time ago
I have exams right now, so I can't actively work on anything, and so that's why I haven't been able to show anything yet

[REDDemon]

@REDDemon:
You're kind of repeating what was already said in the thread

I've been working on some basic designs, I've discussed it with xirta some time ago
I have exams right now, so I can't actively work on anything, and so that's why I haven't been able to show anything yet

Sure I was just compressing previous posts. . I worked in past with several file types (both 3d formats and images. in detail 3ds,obj,bmp,targa). Maybe I can help a bit too.

Code: Select all

 
//start of text header
irr       //'i'  'r'  'r'  '\n'   magic number to detect endiannes.
text     //for a text file else bin.
buffers num    //number of meshbuffer states in this file. "num" is a text number wich needs to be parsed.
vertices  num    //expected nunmber of vertices
triangles  num   //expected number of indices = triangles*3
bones   num  //expected number of bones
frames num  // expected number of frames
description  //vertex description. Just a list of attributes pairs (type-name)
float X  //pos
float Y
float Z
float NX //normal
float NY
float NZ
float U  // text coord
float V
byte R  //color
byte G
byte B
byte A 
byte IDX1 //bone index
byte IDX2
byte IDX3
float W1  //bone weight
float W2  
float W3 
irr_header_end
//start here with the list of all the vertices data in binary or text format.
//1st state of the mesh buffer (needed at least for static meshes or skeletal animations)
//2nd state of the mesh buffer (only for morph animations)
//3rd state of the mesh buffer (only for morph animations)
etc.
 
// indices list (link vertices into triangles. the same for all the states of the meshbuffer)
 
//frame 1 list of bones transformations
//here raw data of all transformation matrices for all the bones in the 1st frame
 
// 2nd frame transformations..
 
//3rd frame transformations.
 
 

that's a single mesh and a single file. So 1 vertex type allowed per file. this is also a very simple file type to handle. It allows both for morph or skeletal animation or for both morph+skeletal animations (maybe some improvement can be done here to save HD space. For example enumerating vertices. The 1st state will have all the vertices, other states just have the number of the vertex wich was changed from the last frame.).
The engine can decide to split the mesh into several meshes using vertices weight infos or to use GPU skinning if available (maybe providing the interface for a GPU skinning class so that someone can implement that. This will have great performance advantages).

The lenght of every element is known in the header. So detect corrupted files is very easy and there will not be crashing loaders etc.

[Radikalizm]

I've done some work on designing a binary format, and I've assumed the inclusion of flexible vertex formats in the engine so we're not limited to a single vertex format
Formats could be described by a simple vertex signature, and each mesh buffer should be able to have its own vertex format (since each mesh buffer can have a different material)
What the engine does with different meshbuffers could be left to the engine itself or user-defined options

Having 1 vertex type per file would be extremely limiting, so I wouldn't really take that route

I'll post my design in a couple of days when I have the time to post a decent explanation

[REDDemon]

I agree here. almost thinked the same.

Formats could be described by a simple vertex signature, and each mesh buffer should be able to have its own vertex format (since each mesh buffer can have a different material)
What the engine does with different meshbuffers could be left to the engine itself or user-defined options

that's why I used only 1 vertex type! because there's just 1 mesh buffer in 1 file:). There's no sense in using the same vertex format for all the meshbuffers. And I never will do that. But each vertex buffer need its own vertex description And I just did that. Instead of packing different meshbuffers(with different vertex formats) in a single file I'm just taking 1 mesh buffer for each file (and each file its vertex description. Maybe it is a bit less elegant or complex, but I think it is more effective.

disagree

Having 1 vertex type per file would be extremely limiting, so I wouldn't really take that route

that's exactly the opposite from my point of view. The file I proposed is just the abstraction of (ideally) a single draw call wich is enough for an animated character. (if the engine have GPU skinning, else it will decide to split the mesh.). Something more complex will require a very complex mesh exporter. Of course just matter to decide if it is more worth a scene description format or a mesh description format. The advantages of abstracting a single draw call is that you have a mesh description format, and you can add later a scene description format and more materials format. The user can decide to change the material, or the scene composition without having to touch the mesh file. I think that this is very consistent with pre-existing irrlicht design. I can't wait to see also different opinions.

[3DModelerMan]

If we have to choose which platform endian conversion happens on then I think Intel/x86 should have the conversion. That way on iOS and Android (where performance is usually more important) you don't get the overhead.

[Radikalizm]

If we have to choose which platform endian conversion happens on then I think Intel/x86 should have the conversion. That way on iOS and Android (where performance is usually more important) you don't get the overhead.

I think it'd be best to find a way to avoid the endian conversion alltogether. Doing those conversions is just asking for a really hard time while debugging


@REDDemon: Why would you want 1 meshbuffer per file? For larger meshes you would get a shitload of small files which somehow should be loaded into a single mesh. More complex meshes can easily have in the hundreds of meshbuffers, and just dumping out hundreds of files to disk just to represent a single mesh just doesn't sound good in my ears. Also, how would you handle coherency in your buffers? How would you know which buffer belongs to which mesh, and how can you guarantee that all buffers will be loaded in correctly? Of course you could use some sort of mesh definition file to group buffers into a mesh, but IMO that's worse than having a single file per mesh

[REDDemon]

@Radikalizm: You can still group more mesh buffers in one file. Instead of putting the mesh info into 30 files you can just write each file after the other in one single file. And only materials info still need to be added. that's why defining the format of a single mesh is important. Making it muliple mesh requires no cost. But the assumpion you are working on is that people will make complex scenes/meshes(a hundred of drawcalls for a single mesh sounds like more a scene to me ). But for doing that they are required to work with some complex tool with some complex exporter(wich is what is done in commercial games i suppose). It would not be very nice to have a own file format but no exporters at all for that.

Endian conversion is not a problem. MAKE_IRR_ID is enough for that. that macro is perfect for check endian match. another macro MAKE_IRR_ID_e can be added with just inverted usage of parameters (so instead of abcd we can use dcba) for endian mismatch. A second version of the macro with 16/ bit lenght can be easily added by just copy cutting MAKE_IRR_ID with some changes. The code should be trasparent so that both the PC and Android can do endian conversion at runtime. The only way to avoid endian conversion is to use a binary reader wich already does that, but this will be slower than doing that manually. Well another way is to export different files for every platform but is not a very portable solutions since users have to mind about that.

So if something more than a preprocessor directive is needed.. it can be:

Code: Select all

 
//endian conversion
u32 test = MAKE_IRR_ID('i' , 'r', 'r', '\n');
u32 endian = readfirst32bitfromfile(file);
if(endian == test)
    read1();
else if (endian == MAKE_IRR_ID_e('i' , 'r', 'r', '\n'))
    read2();
else
    return false; //not a irr mesh file
 
 

another way, with a little extra overhead but wich has more enphasis on code reuse (just 1 loading function instead of 2)

Code: Select all

 
u32 test = MAKE_IRR_ID('i' , 'r', 'r', '\n');
u32 endian = readfirst32bitfromfile(file);
bool flag;
if(endian == test)
    flag = true;
else if (endian == MAKE_IRR_ID_e('i' , 'r', 'r', '\n'))
    flag = false;
else
    return false; //not a irr mesh file
 
// parsing header info
 
// starting reading vertices info
 
while () //some kind of loop
{
    u32 next32bit = readnext32bitfromfile(file);
    f32 value = flag? reinterpret_cast<f32>next32bit: reinterpret_cast<f32> TURN_ENDIAN (next32bit); //just few more cycles for endianness conversion if required.
   
}
 

[Radikalizm]

You know what, since I have some time right now I believe I could do a description of what I had in mind for a binary format:

I focused on a couple of points:

• 1. Minimal administrative overhead
  2. Good loading times
  3. Minimal size on disk -although this could definitely be improved in the current design, at a cost of speed-
  4. Non-limiting design -the user can define any type of vertex per mesh buffer, as long as the engine supports the used datatypes-
  5. Good error detection/prevention -serious errors get spotted before the actual loading starts-

Now, to realize this I had to make some assumptions as to what irrlicht would support in the future, flexible vertex formats being the most important.

First of all we have a traditional binary header and footer, our first checks for file validity happen here.
A minimal header would include:

• -A small constant magic number or file identifier (first validity check)
  -File size (second validity check; should be careful with this one)
  -Offset to footer (third validity check)

The footer would only contain the same magic number or identifier specified in the header. To check whether the file hasn't been tampered with you could just do a simple jump to your footer, read in the magic number and compare it with the expected magic number. If any of these 3 validity checks fail you can assume your file is corrupt or has been tampered with, all before you have started loading in actual data.

The header structure could also include additional information about which engine version it's compatible with, author information, file revision, etc.

Next up would be a global mesh header, a minimal header would only need to contain a mesh buffer count, but could also store debug data or additional error checks (eg. complete vertex and triangle counts)
After that we get a sequence of mesh buffer headers + buffer data. Mesh buffer headers would include:

• -Vertex count
  -Triangle/index count
  -Vertex signature string -I interpreted this as being similar to vertex description in the wavefront obj format, but more extensive-
  (-additional debug data)

The engine would use the vertex signature to determine the byte-width per vertex. We could do a raw data dump here which could be loaded in directly into a vertex and index array, or we could use some form of compression first. This is just a speed/size trade-off

I have not included skeletal information in here for a good reason. When you tie skeletal information directly to a mesh it can get harder to re-use, and each mesh using the same skeletal data would require to hold a copy of the information inside its own file, creating a memory and loading-time overhead. Defining these as external files (maybe even as simple xml files) could avoid this problem.
Same goes for materials.

Now remember, this is all very basic and nowhere near where I want it to be. Criticism is always welcome

@REDDemon: I see where you are coming from, but don't you think storing different files each with their own administrative overhead in an archive would slow down loading times? Also, I don't think it's up to us to determine how much information anyone would want to store in a mesh. If someone wants to push a scene's worth of data into a mesh (which is done regularly), that should be completely possible IMO and we should make the format work just like it would for smaller files.
Also I find the assumption that large files would require heavy and complex exporters kind of strange. The complexity of writing out a single mesh buffer is exactly the same as writing out hundreds of them, it's just repeating the same procedure over and over again until every buffer has been handled. Since buffers do not depend on eachother and since they don't overlap I don't see why this would be a problem.

[hendu]

iOS is BE? Most of the current ARM are LE, just like x86...

[REDDemon]

I have not included skeletal information in here for a good reason. When you tie skeletal information directly to a mesh it can get harder to re-use, and each mesh using the same skeletal data would require to hold a copy of the information inside its own file, creating a memory and loading-time overhead. Defining these as external files (maybe even as simple xml files) could avoid this problem.
Same goes for materials....

@REDDemon: I see where you are coming from, but don't you think storing different files each with their own administrative overhead in an archive would slow down loading times? Also, I don't think it's up to us to determine how much information anyone would want to store in a mesh. If someone wants to push a scene's worth of data into a mesh (which is done regularly), that should be completely possible IMO and we should make the format work just like it would for smaller files.
Also I find the assumption that large files would require heavy and complex exporters kind of strange. The complexity of writing out a single mesh buffer is exactly the same as writing out hundreds of them, it's just repeating the same procedure over and over again until every buffer has been handled. Since buffers do not depend on eachother and since they don't overlap I don't see why this would be a problem.

That's exactly what I sad I defined a chunk for a single mesh buffer. No one forbids to to put more chunks in one file, the scalability to multiple mesh buffers is implicit. Header of each chunk contains enough info for move the file pointer directly to the next chunk if desired. Or you can just load all chunks and that's all. I always sad to use a list of mesh buffers instead of a scene with its hierarchy. I was thinking that you wanted to handle the hierarchy also in your file format. If not we are just trying to do the same using a different naming convention and file format XD

Once you have a procedure for a single mesh buffer you have just to repeat that for each mesh buffer I have already writed that. And if an user doesn't want skeletal info it can just decide to skip loading since the skeletal buffer can be skipped because its lenght is known thanks to the header. I have just posted a very simple, and fast to load, file type wich is able to store a meshbuffer. Then if an user want to pack more mesh buffers it has just to append another "chunk" to the file. Every chunk is a mesh buffer . If the user doesn't want skeletal info it can just decide to set the option "no bones" and the mesh will be loaded without frames. Or if he set "no animated" the mesh will be also loaded without bone weights so just as a static mehs (I can't figure a reason for removing weights from vertices, But this is still possible!)

There's no point in keeping skeletal info separed from the file. The file format I posted is able also to contain just bones transformations without vertices. Just need to set "buffers" "vertices" and "triangles" to zero and the file becomes magically just a skeleton with its animation(so you can store a motion capture result in this file without declaring a single vertex!). I don't understand why you would like to put all mesh buffers together and keep skeleton separed. Skeleton is specific to a mesh. If you remove bone weights you have to give again weights using a 3D editing tool like blender. There's no reuse in that. And if you refer just to keep the bones, but change frames. This is already possible with the file format I provided:)

Given a mesh with its bones and weights, a skeletal animation (Specific to that set of bones) can be loaded from different files if wanted, or just from the original file. this is already the maximum reusability . The format file I posted is as flexible as simple and effective.

The only missing things is as you say a footer for check integrity of the file. Never thinked to that. What do you thinked to use? A checksum system?


revision 2:

Code: Select all

 
//start of text header
irr       //'i'  'r'  'r'  '\n'   magic number to detect endiannes.
text     //for a text file else bin.
buffers num    //number of meshbuffer states in this file. "num" is a text number wich needs to be parsed.
vertices  num    //expected nunmber of vertices
triangles  num   //expected number of indices = triangles*3
bones   num  //expected number of bones
frames num  // expected number of frames
description  //vertex description. Just a list of attributes pairs (type-name)
float X
float Y
float Z
irr_header_end
/*Binary data*/
irr       //I don't think that writing again "irr\ntext\n" will add to much administrative overhead to the loading of the file.
text
buffers 0
vertices 0
triangles 0
bones num
frames num
description  //in this example we have 1 mesh buffer with its animation and a skeletal animation without a mesh buffer
irr_header_end
/*Binary data*/
 
footer CHECKSUM //as you suggest a footer here would be great. A even/odd test will be great for detecting missin bytes. Something more complex like a SHA i don't think is worth. So every time we read 32 bits we XOR them with last loaded 32 bits and if something is missing at the end of the file check sum will be different. A lenght measure of the file is not needed since it can be checked thanks to the headers.
 

Another interesting thing is your idea to put debug data. What kind of data do you think to use? Something like names? because most of the debug data are already drawable by irrlicht (normals, AaBB etc.) without the need of a specific file for that.. Yeah I suppose you are speaking about names. I have seen lots of engines using to show the name of a node or a mesh buffer in debug mode This is a great idea. (already possible in irrlicht, but the user has to set debug name. But using an external file the name can be setted from the file instead of from the code (don' know if already possible with irrlicht that. I will check now)

So actually possible things:
*Fast loading (minimum header and binary econded data.)
*Flexible vertex description (many 3D formats use that)
*Skeletal info (bones and frames)
*parsing (file pointer can be moved to the next section of the file if some info is not needed)
*Multiple meshes in one file. (implicitly possible. Instead or writing a hierarcy file we have just a list of meshbuffer. Very stable solution)
*ability to store different kind of info with 1 file (so you can separe indices, bones and frames into different files if you want to reuse them. Or you can just put them all togheter)
*morphed animation
*skeletal animation
*lenght checks (possible with binary format only)
*morphed + skeletal animation.
*detection of errors (just parse headers and check if the file lenght is ok is enough to see if there are missing vertices etc.)

Missing things suggested by you
*debug info
*footer
*eventually more error check test? ("checksum"?)

Ah yeah! We are still missing the material description. Any Idea? Both adding the material as an external link or as a propierty of the mesh in the file have pro/contros. A hybrid solution would be great too.

I think that irrlicht is already able to serialize materials (not sure at all) maybe a link to the serialized material file would be enough.

Code: Select all

 
//start of text header
irr       //'i'  'r'  'r'  '\n'   magic number to detect endiannes.
material "material01.xml" 
text     //for a text file else bin.
buffers num    //number of meshbuffer states in this file. "num" is a text number wich needs to be parsed.
 

something like that is very easy to do.
Pros:
reuse irrlicht materials.

sideeffects:
How can a 3D tool export a "material01.xml"?

so maybe is better doing something different. Ideas for that?


please wait some minutes to reply I will check for errors. I'm really tired ATM.



EDIT: readed twice the post should be ok. XD

[Radikalizm]

I was aiming at names with debug info yes

I think we both have the same ideas, but there's just some miscommunication going on

About materials, I believe we should first analyze how irrlicht's material system could actually be improved before defining a file format. Since we're already depending on some yet to be implemented features I think it's safe to assume an updated material system as well

The footer would not require any checksum, just the same magic number as stored in the header; I included a global header in my design with a magic number and footer offset so you could immediately jump to the file footer to check whether the magic number stored there matches the one in the header. If it does not, you immediately know that your file is corrupt and that no attempt should be made at loading it, which also makes debugging a lot easier since the chances of the engine crashing while loading data are greatly reduced.
The file size stored in the global header could also be checked with the file size in memory to again check for any corruption.

In your design, you assume that the engine immediately starts loading the mesh buffers after the initial magic number check validates. This makes it extremely easy to inject new data into an already compiled file, and the engine won't complain at all about loading this new data, potentially resulting in a crash, undefined behaviour or unwanted behaviour. This would create a perfect engine exploit.

I separated the skeleton from the mesh itself because it is possible for multiple meshes to use the same skeletal data with the same set of poses. Vertex weights would still be included in the mesh file, but there would be no need to copy the bone data into every single file. A skeleton file would just have to be loaded in once and could be assigned to any mesh requiring it in-engine.

[REDDemon]

In your design, you assume that the engine immediately starts loading the mesh buffers after the initial magic number check validates. This makes it extremely easy to inject new data into an already compiled file, and the engine won't complain at all about loading this new data, potentially resulting in a crash, undefined behaviour or unwanted behaviour. This would create a perfect engine exploit.

I separated the skeleton from the mesh itself because it is possible for multiple meshes to use the same skeletal data with the same set of poses. Vertex weights would still be included in the mesh file, but there would be no need to copy the bone data into every single file. A skeleton file would just have to be loaded in once and could be assigned to any mesh requiring it in-engine.

You are assuming that corrupted files are only files with no footer. Corrupted data can also be altered vertices info, but for check if a vertex is altered you need a checksum system else your file can always be corrupted also with a footer or other check systems. Can you tell me how the PC can know that a vertex coordinate was changed without a checksum/hash system?

You are speaking to exploiting the header architecture. So you are saying that is possible appending more headers to my file? Maybe is possible. This would need to add an additional parameter to the header like "expected number of headers" and "expected file lenght" so "headers. lenght" . This should be enough to prevent this. But this kind of assumption "assumes" that there is a very buggy loader wich stupidly read mesh data into mesh buffers without doing any kind of checks. I will never do such loader, and no one will do . That assumption also assumes that there is someone in your pc wich is corrupting data of the files that you are loading (At this point you PC is ended at well XD, while with revision 2 was still not possible creating a mesh file wich crashes the loader, with revision 3 is also not possible to add more headers at run time).

revision 3:
should prevent code injection.

Code: Select all

 
irr
bin
headers 3    // expected number of headers
lenght <NUM> //expected file lenght. Possibly NUM is a 32 bit unsigned integer. This way it can be updated freely if lenght changes.
irr_identifier_end
/*header*/
/*binary data*/
 
/*header*/
/*binary data*/
 
/*header*/
/*binary data*/
 
/*footer*/
 

the header becomes simpler

Code: Select all

 
//header
buffers num    //number of meshbuffer states in this file. "num" is a text number wich needs to be parsed.
vertices  num    //expected nunmber of vertices
triangles  num   //expected number of indices = triangles*3
bones   num  //expected number of bones
frames num  // expected number of frames
description
float X
float Y
float Z
byte R
byte G
byte B
irr_header_end