Irrlicht's OWN file formats?

[xirtamatrix]

I think what we need to do is to list down all features we need and then only the format can be structured properly to fit with the requirements before start coding it.

Exactly!

I was doing a bit of research and what I have concluded is this:

1. Our core team should only "define" file formats and develop and maintain "loaders" in the core. Once the formats are defined we publish them publicly and let the community create, sell and maintain exporters for DCC tools like Maya, Max, Blender etc. This would reduce burden on the core team and promote healthy competition among community members ańd teams, also providing them a steady source of income. Of course, the core-developers could themselves be part of any such team, if they so wish.


2. The most obvious features, in my opinion, which the file formats need to support are:
- 2.1 Skeletal and morph animation.
- 2.2 Materials, with or without shaders.


3. Based on the above, my initial suggestion would be to start with the following:
- 3.1 A binary mesh file format: This should contain obviously the mesh data, as well as morph data. I'm not sure if it needs to contain something more, perhaps others can advise about it. This would require complimentary loader in the engine core.
- 3.2 A text or xml-based skeleton file format which encodes all the data about the skeleton. We would need to write the parser for it, perhaps even as part of the core mesh-loader above.
- 3.3 An text or xml-based material file format: this could contain data about environment, materials for each mesh, textures etc. We would need to write the parser for this.


These are my initial thoughts and meant to serve as a starting point for more concrete discussion.

Questions that need to be answered:

1. In regards to 3.1, how the mesh-data should be encoded, and exactly what kind of morph data the file should contian? This probably needs to support how morphing is currently implemented. I have never looked deeply into why we cannot have skeletal and morph animation at same time in Irrlicht, can someone enlighten me please? Nevertheless, we need to solve this issue also.

2. In regards to 3.2, same as 3.1. What is needed to be encoded which is required for skeletal system to work? Also, how to ensure skeletal animation would work together with moprh animation?

3. In regards to 3.3, how exactly the materials would be referenced? By filename or by material name? How much and what detail about each material will we encode? How could the materials utilize shaders?

4. How would all this interact? Which file will access which file and how?

Please comment and share your thoughts/opinion/knowledge.

[hendu]

I kind of doubt there's a big enough market for someone to sell more than a couple Max etc exporters to this format.

[Virion]

irrlicht itself is quite well-known in the 3D field so this format would definitely attract a lot of users especially those who long wanted such format in Irrlicht.

by the way, I think skeleton file should be in binary as well since we can hardly edit it by hand and mostly will be done in the 3D modelling software. not sure about the others though need to wait someone who has higher experience to comment.

[Radikalizm]

Having XML representations of your data would always come in handy for manual debugging, but I'd suggest to do as much as possible with binary files except for those files which should be humanly readable and editable, like materials as mentioned

To answer some of the points made by xirtamatrix with examples from my own file formats:

1. I think we should keep mesh data encoding as simple as possible in the beginning, see how efficient we can import and export data in a more or less sequential model with uncompressed data storage
ie.

Code: Select all

-Binary header
    -Data descriptor 1
        -Data 1
    -Data descriptor 2
        -Data 2
    ...
-Binary footer
 

Or a similar model to this

From there on we could look for optimizations, both in terms of memory and speed; I'm a firm believer of the KISS principle, so I believe we should start out from something really simple like this and tune it from there to see what works and what doesn't

2. I'm still playing around with the inclusion of animation data in my own formats, so I can't really tell you what has worked for me yet, but this is also partially because my animation system is not what it's supposed to be yet, would be interesting to sit down and discuss the options with some people for the irrlicht formats

3. In my formats materials are referenced by filename, but not in such a way where I pass the entire path of the material file itself. My engine expects a certain file structure on-disk in debug builds, and in release builds it requires a small set of binary files (mostly just 2 or 3 files) which act as an archive for all files needed to run the engine itself and the map it is currently rendering. The engine knows where to find files of a certain type (ie. meshes, materials, etc.), so the only thing I need to store is the name of the file itself for the engine to load the data; in-engine each set of data gets assigned a resource handle for later use. Since irrlicht works completely different from this, this idea becomes rather worthless

4. We'll need to map everything out somewhat, put it all in a graph and see where the dependencies are to determine the best possible interaction

[greenya]

I like how ODT (an Open Office Writer) format is done.
It is a zip file which contains own folder tree with xmls and binaries.

[Radikalizm]

I like how ODT (an Open Office Writer) format is done.
It is a zip file which contains own folder tree with xmls and binaries.

This approach is used rather often, and it's a good way of storing multiple data files into 1 large file, but it doesn't really have that much value when it comes to storing separate meshes, skeletons, materials, etc.

Also, I try to avoid commonly used compression formats for resources since there are tools readily available for decompression which means people can easily access your resources (and because I like reinventing the wheel)

[CuteAlien]

I would count easy accessible resources as a definite advantage for any format. The more accessible the better... if you want to protect resources don't do it by obfuscating a common format but use a protection layer around it.

I also like to note that there is already a serialization mechanism in Irrlicht for many classes, although so far it's only used for xml-serialization. It misses support for several types - which is a big open questions since a long time - anyone who really knows what's better there is welcome to give input. Basically the decision is either to reduce number of factories (like IGUIElementFactory, ISceneNodeAnimatorFactory , ISceneNodeFactory, etc) and go back to a more basic-class for that or to add a lot more of those (IParticleAffectorFactory, IParticleEmitterFactory, IFontFactory, ISkinFactory, etc) which on average means around 3-4 function per additional factory ( addFactoryOfType, getNumFactoriesOfType, removeFactoryOfType). And then the serialization class must have access to factories so for example gui-elements can for example create fonts in serialization (also skins, sub-guielements, etc). Another solution would be adding more attribute-types for all those, but not sure how sub-gui-elements would then be handled (my suspicion is that Irrlicht already uses too many attribute-types to avoid giving serialization access to the factory system - which might be a good or bad thing - I have not enough experience with different serialization systems to know about that).
I suspect serialization might profit from a few more features (p.E. inheritance and/or binding attributes to variables), but the stuff mentioned above is what I see as the biggest blocking obstacles for now.

edit: TL/DR: We have an attribute-value serialization, but it fails in all places where we would need to create sub-objects (or entities) although we sometimes work around it by ugly hacks (like writing another complete serializing for known sub-elements in a parent and avoiding name-collisions with pre-fixes in the attribute-name for the child).

Not sure if the same serialization can be used for binary as well - but the attribute-system mentions type, name and value which is basically also all you need for binary serialization (could kick out name+type probably in some cases).

Edit: When reworking serialization please keep in mind that the attributes are not only used for serialization to disc, but also as access mechanism to objects for example in irrEdit and in script-wrappers.

[Radikalizm]

I would count easy accessible reasources as a definite advantage for any format. The more accessible the better... if you want to protect resources don't do it by obfuscating a common format but use a protection layer around it.

I'm not talking about obfuscating a common format, I'm talking about rolling your own. Password protection would definitely work, but that's where the whole reinventing the wheel thing comes in
Back when I was active in OS development I've designed some simple filesystems (mostly VFSes, since none of my kernels actually got to the stage where a lot of disk IO happened), and I quite enjoyed it, so that's why I prefer to build custom formats for file archiving so I can cater it to the needs of the application which is going to use them

[CuteAlien]

I'm not talking about obfuscating a common format, I'm talking about rolling your own.

I expect an Irrlicht format to be common - or that's what I meant ;-)

[xirtamatrix]

@virion: hmm, you're right, the chances anyone would want to fiddle with the skeleton data by hand are remote, so yes it should also be binary. Though Radikalizm is also right, for debugging purposes, both mesh and skeleton files should also have corresponding xml formats.

@Radikalizm: I agree with you, we should start as simple as possible and then add to it as needed. But I would still like to do a bit of research to understand what should be our general direction and what should be our important considerations.

Basically I see two issues.

Regarding mesh and skeleton files:
1. These file formats need to support the engine, not the other way round. To be specific, I mean supporting the animation system of Irrlicht. Thus, we need some input and suggestions from the core devs about what the files should and should not encode when it come to supporting morph and skeletal animations. My key goal here would be to facilitate mixing of morph-skeletal animations, something which Irrlicht lack at present.

Regarding materials:
2. Regarding materials, are we going to support Irrlicht's pre-defined FFP material types? Or just shaders?
Regarding naming and access, I rather like the way Ogre's material system works and there is no reason why we could not implement something similar, except that Irrlicht does not have a similar resource loading system so we have no pre-defined system of specific "locations". Nevertheless, what we could do is implement our own.

We have two options:
Option 1: We expect all material scripts to be present in a specific folder. At startup, our parser reads all the material files in that location, and loads all "materials" into memory. Materials are identified by name and thus have a unique name. The mesh file then only needs to encode the materail name for a mesh or face or a set of triangles and not worry about where to load the script from.

Option 2: The mesh file encodes the name of the corresponding material file. Path is not important, it is understood the material file should be always present together with mesh file. In such case, we only process material files which are referenced by mesh files. If a referenced material file is not found where it should have been, we output a warning message and continue. Its the user's responsibility to ensure files are present.

In both cases, we make sure it is possible to use external shaders. In other words, the material scripts should have an option to reference external shader scripts and our parser should accordingly process them.



Regarding mesh data, I was trying to search for some examples, but only found this: http://en.wikipedia.org/wiki/Wavefront_.obj_file
Perhaps there is some interesting useful information here for Radikalizm?

[hybrid]

Ok, I think I have now merged all posts here that belong to this thread. If not, please PM me. Sorry again for the inconvenience, I'm just getting used to the new functions of the forum.

[Radikalizm]

I'm very familiar with the wavefront obj format, to the point where I wrote an importer for it in the early days of my engine
obj meshes are text-based though and are only able to describe actual geometry, no animations or additional vertex properties can be stored besides positions, normals and uv coords

I think you'd be messing with backwards compatibility for a lot of irrlicht-based projects if you were to enforce a location for material files and if you were to preload all materials; we should try to maintain compatibility with older projects as much as possible since irrlicht has never enforced a certain way of loading in resources, which resulted in a lot of people writing varying mechanisms for resource management which suited their projects

I think our best option would be in fact to assume that a material file is provided alongside the specified mesh, or that a developer can specify a material file to be used

The serialization interface definitely will be important for a clean and portable implementation of the mentioned formats, but as stated it needs some work
We should try to clean up and improve it (without breaking too much), and use it more thoroughly throughout the engine where needed. Since serialization of meshes and materials would be a must for designing these formats we could maybe look into serialization for the entire set of resource types used in irrlicht while we're at it? I don't really have a good view on what is serializable right now and what isn't, nor do I know what's already in the works, so I don't know what you guys (CuteAlien, hybrid, etc.) think of this

[3DModelerMan]

If you made it possible to serialize IAttributes to a binary file, then we could have binary scenes too. I think materials should be binary as well though. We could just have a material editor that could save and load them. That way you don't have to worry about having it in XML for hand editing.

[xirtamatrix]

I would count easy accessible resources as a definite advantage for any format. The more accessible the better...

How come? Could you kindly elaborate on the "advantage"?
From all the experience of the gamming industry I have, the biggest headache for professional artists and gamming studios is protection of thier copy-righted assets. The only advantage I see here would be for potential thiefs who lack the skills (and any sense of morality) or just can't be bothered to create their own art. It is ok to use art thats specifically been created for the public-domain. It is not ok to extract art assets from someone's game or application and start using them without permission. If we make our formats too open we would be promoting exactly that and be working opposite to what our potential AA clients actually need. I don't think that is going to benefit Irrlicht.

I also like to note that there is already a serialization mechanism in Irrlicht for many classes, although so far it's only used for xml-serialization.

Thanks for pointing this out. Actually I have never used Irrlicht's serialization interface and also a bit lost on why we are discussing this at all. Would it help with writing the loaders? Because I cant foresee any scenario where a game or application would actually start saving meshes. Once again, please forgive my ignorance on the topic.

I think materials should be binary as well though. We could just have a material editor that could save and load them. That way you don't have to worry about having it in XML for hand editing.

Materials NEED to be in XML, so they can be easily hand-edited when required. And why on earth would you want to force people to using another editor just for materials? Automatic generation of material scripts will be handled by exporters for DCC tools.

[xirtamatrix]

Ok, I think I have now merged all posts here that belong to this thread. If not, please PM me. Sorry again for the inconvenience, I'm just getting used to the new functions of the forum.

Thank You so much hybrid. That was very nice and helpful of you. =)