Improving Ogre Mesh Loading

[Nox587]

I decided to start this project because I ran into one to many problems with irrlicht animated mesh system. Specifically this one ->http://irrlicht.sourceforge.net/phpBB2/ ... hp?t=16829

I understand that I'm quite new to this community and this engine, but I'm no stranger to game programming, or render technologies, and I gladly jump at the opportunity to aid an open source project. Especially one that is organized and has a good following.

I intend to create a mesh file format just for irrlicht. There are multiple reasons I think this is a big requirement for this engine, but the two biggest I see are speed, and control. Having a native file format would give irrlicht more control over a mesh, and would make it quite easy to serialize the file in and out of irrlicht. The feature list I have come up with is as follows:

• Basic Mesh data - That is vertices and indices, and important vertex information such as position, format, color, and normal information
• Frame Transformation Data - Most likely stored in matrices as its the easiest to work with (at least in my experience) and would allow for mesh defomration animations.
• Material information - That is U-V coordinates and a material script that can be used to "template" different material "effects" or maps
• Serializer
• Access to vertex/index buffers
• Bounding box/ellipse
• Sub-Meshes - These would have to be created by the modeling program by spliting a mesh into different polygons, which varies depending on the program.
• Skeleton - Structure for storing bone information. Not sure if I would take the same approach as Ogre did by using a seperate file for skeleton structures, or just embed the information into one file.

Most of my inspiration for this came from the Ogre engine. I just think that the .mesh format they use is fast, and easy. For this reason, the structure I came up with is pretty similar to the one Ogre uses.

First of all I intend to implement this format using xml, because its easy to read, write, and serialize (at least in my experience). The root of the file will be an irrMesh. An irrMesh consits of subObjects, which consits of faces, and geometry. A subObject has a material parameter.

Each geometry object has vertex buffers, which of corse hold vertices. Vertices contain position, color, normal, and texture coordinate information.

Code: Select all

<irrMesh>
   <subObject material = "some.mat" >
         <geometry numVertices = "1">
            <vertexBuffer XYZ = "true" normals="true" texCoords = "1">
                <vertex>
                     <Position>
                           <X>50</X>
                            <Y>0</Y>
                            <Z>20</Z>
                     </Position>
                     <Normal>
                            <X>0.0308391</X>
                            <Y>-0.998866</Y>
                            <Z>0.0362719</Z>
                     </Normal>
                     <texture>
                           <U>1.0</U>
                           <V>1.0</V>
                     </texture>
                  </vertex>
              </vertexBuffer>
          </geometry>
      </subObject>
</irrMesh>

I have not worked out how I'm going to fit the transformation data in there. Perhaps each subObject will have frame entries that hold transformation data, similar to the way .X stores its transformation data.

I also plan to make a bone representation similar to the Ogre skeleton format, but I want to get the above working first.

You can see how information can easily be imported into the engine with this layout. The biggest challenge of corse is writing all the exporters for modeling programs. There is the option of making a converter similar to the one LightFeather uses, but I'd rather start with all the information from a modeling program, rather then use the limited information of an exported file.

At any rate, I'll keep you all updated on my progress, and if there are features I left out, which I'm sure there are, please do let me know. Of corse I could always use a hand, so if anyone is interested in helping drop me a line.[/code]

[JP]

Sounds pretty cool and useful, good luck with it!

[needforhint]

if you want to bring a mesh format to a life you must create a converter of common meshes to the new format. Irrlicht actualy knows a lot of meshes and loads them so you should find inspiration in its loaders.

[Saturn]

Doesn't IMeshBuffer already do what your subObject does?

Generally I like the idea of a custom Irrlicht Mesh format, your format is almost an exact copy of Ogre's mesh format. Instead of creating yet another format you could use this one and improve the loader.

Imho the problem is not so much the file format but the internal vertex format / underlying animation system in Irrlicht. Whatever vertex informations you store in your file format, you have to get it into Irrlicht somehow to render it. Irrlicht has three fixed vertex formats.
That's why the Ogre mesh loader is still basic.

And even more important here is, that you had to invent yet another animation system in Irrlicht. With the new getIrrMeshBone() next to the getMS3DBone and getXBone()...

Instead of introducing a new format just yet, I believe it is more useful to design a new animation API / vertex API. Then the animation loading can be unified and the tangent problem be solved.
And then your need for a new format is maybe gone already.

[Nox587]

if you want to bring a mesh format to a life you must create a converter of common meshes to the new format. Irrlicht actualy knows a lot of meshes and loads them so you should find inspiration in its loaders.

Yes I would have to agree with this, after doing some investigation it seems that most large engines have converters and exporters for there format.

Doesn't IMeshBuffer already do what your subObject does?

Generally I like the idea of a custom Irrlicht Mesh format, your format is almost an exact copy of Ogre's mesh format. Instead of creating yet another format you could use this one and improve the loader.

As simple as this is, its not something I thought of as a solution. It would certainly take less work, but raises one small concern. It would mean that the plugin I write for the irrlicht engine, would be dependent upon the format of the Ogre engine. I don't suppose this is a terrible thing considering Ogre is a very reputable engine, and I doubt they would overhaul there mesh format.

It also raises the concern of material scripts, and bone structures. The mesh format works best (imho) when used in conjunction with the material script and the skeleton structure. This would of corse mean that my system would have to parse in the skeleton file and material script, and use them with irrlicht.

I don't suppose that is such a bad thing at all...just a feature I think I'd have to implement.

Instead of introducing a new format just yet, I believe it is more useful to design a new animation API / vertex API. Then the animation loading can be unified and the tangent problem be solved.
And then your need for a new format is maybe gone already. Wink

After doing a bit of investigation this seems like a more sound solution. The first thing I took a look at was the COgreMeshFileLoader class, and how it worked. The first thing I noticed was that the loader ignores skeleton entries, bone assignments, LOD levels, submesh names, and edge lists.

Code: Select all

//...snip...
                        case COGRE_SKELETON_LINK: 
			case COGRE_BONE_ASSIGNMENT: 
			case COGRE_MESH_LOD: 
			case COGRE_MESH_SUBMESH_NAME_TABLE:
			case COGRE_MESH_EDGE_LISTS:
				// ignore chunk
				file->seek(data.header.length-data.read, true);
				data.read += data.header.length-data.read;
				break;
//...snip...

Seems to me that you can not use a bone structure with an Ogre mesh file in irrlicht.

It also seems that when constructing the object from the buffers, irrlicht just gives all the information into an IAnimatedMesh...which I suppose makes sence, but the animated mesh does not support subMeshes...which the loader is clearly reading to get the geometry.

Why not just have an inherited mesh class called IAnimatedMeshOgre that stored the data of each sub-objects geometry. You could build the animated mesh the same exact way, but the mesh would also be aware of the submesh data.

At any rate, I believe this is the path that I will take with this project as its more time efficient and seems to make more sence. Especially considering there are already exporting and converting tools for the mesh file format.

Thanks for the tips and information Saturn, I appreciate it!


[edit]
P.S I changed the title of this thread so its won't mislead people into thinking that I am creating a custom format.
[/edit]

[Saturn]

Why not just have an inherited mesh class called IAnimatedMeshOgre that stored the data of each sub-objects geometry.

That's not what I meant. That's actually a problem right now, that each loader implements its own animation system and even has its own interfaces.

A new file format, or improvements of the ones that are there, need a proper foundation inside Irrlicht. So that all the loaders just create the common Bone/Skeleton/Mesh/Meshbuffer/Animation/Frame/etc classes. The missing functionality for instance for multiple meshbuffers can be implemented there, so other loaders can use this too. If those classes are shared, then future improvements were much easier to do. Alos things like animation blending etc. had to be implemented only once and automatically work for all file types.
Also this would allow runtime created animations and other things.

And the same is true for vertex formats. While the three we have cover most cases, things like 3d texture coords or two texture coords + tangents were nice too. At least X and Ogre mesh would profit from this.

Nox587, I am not saying *you* should do all this or something, and your Ogre mesh improvements are much welcome here.
Just some random rambling...

[rogerborg]

Ditto to everything that Saturn just said. Adding another animated format to Irrlicht will make it harder to unpick the evil later, and move all of this functionality to IAnimatedMesh, or new abstract ISkeleton (etc) interfaces, where it belongs.

Something that strikes me about a custom format is: if you don't have an editor for it, how do you create a file in this format? Load and convert an existing file format and save it out? Then why not just use the existing file format?

But, again, kudos on taking a swing at this.

[UPDATE]

Ouch, I just made the mistake of looking into the current objects and loaders. I can see why nobody's tried to fix this; it's a real mess.

While I'm hesitant to sound hysterical, it does seem like this is something that if it's not fixed soon (and likely by one of the core developers), then it will become an increasingly heavy millstone around Irrlicht's neck, and one that's more difficult to remove as each new format is added.

[hybrid]

I don't understand what you mean There are not really large methods to factor out of the file loaders. Most are special to one format - mainly due to the interwoven loading methods.

[Saturn]

hybrid, the whole animation is done inside the CAnimatedMeshXYZ classes.
IAnimatedMesh::getMesh(...) is implemented seperately for each animated mesh type. For X, for MS3D, for B3D..
And these details are not special to one format. What is special to bone animations? They work the same in all formats. Blend indices, blend weights, bone matrices and transforms of those. The same for all.

We have a complete parallel code structure in those classes. You even see how the code got simply copy/pasted and then changed. There are ugly casts in the code, sometimes I am surprised how(if?) they even work. If you want to add animation blending, you have to implement it seperately for each format, eventhough they do the all the same things. Coding horror.

Look at it. Look at all what CAnimatedMeshB3D::getMesh does and what CAnimatedMeshMS3D::getMesh does and what XAnimationPlayer::getMesh does. All the same more or less only done differently because the data is locally stored differently. But this can be unified.

I really don't understand that you don't seem to see the problem in the current approach.

[rogerborg]

Most [file-specific IAnimatedMeshFoo methods] are special to one format - mainly due to the interwoven loading methods.

Well, that's the problem right there. The methods for an animated mesh should not be specific to the file type that it was loaded from. There should not be, and should never have been, an IAnimatedMeshMD2 / IAnimatedMeshMS3D / IAnimatedMeshX, not should there be any bastard methods like IAnimatedMeshSceneNode::setMD2Animation (leaving aside the issue of having an IAnimatedMeshMD2, why is that a method of IAnimatedMeshSceneNode instead of IAnimatedMeshMD2?)

Let's be clear, I'm not criticising anyone who wrote the current implementations; it's crystal clear how this happened, as it's always quicker to copy and paste a new format than to go back and abstract out the common methods. I'm only using md3d models, and the md3d objects give me all the functionality that I need, and I'm very happy and grateful to the author.

However, it is a really messy design, and it's just going to get worse every time someone adds support for a new file type.

IAnimatedMesh/CAnimatedMesh should (I suggest) have all of the functionality to support the features of any file format, and the file-specific loaders should do just that, be tools to load file data into a CAnimatedMesh, not be implementors of IAnimatedMesh themselves. I note that CXMeshFileLoader/CXFileReader has started to hive off a separate loader, but still operates on X file specific objects, rather than a general one.

Sorry if I sound negative, but it just seems painfully obvious that this design is accidental rather than purposeful, and is going to be pig to fix, if for no other reason than that most user/developers have no particular incentive to fix it, and will get rightfully upset if the interfaces get changed under their feet this late in the game.

I guess I'm looking for an clear indication from niko about whether this is ever going to be addressed. If so, the sooner the better. If not, then a clear statement to that effect (in the source code, ideally) would save anyone from wasting time trying to fix it.

Again, none of this is meant negatively, it's just that I see an opportunity to improve Irrlicht, and I'm wondering if it's worthwhile pursuing it.

[hybrid]

Ah you meant the AnimatedMesh thing. As this thread was on Ogre meshes and on vertex types I did not get this relation. A generalized animation node would definitely help.

[Electron]

For the love of all things good, do not use xml as the sole format for an internal engine filetype. Not for storing vertex and index data, anyway. It's far too slow to load. Collada is XML because it's an interchange format. .obj is similarly used as an interchange format. An xml format is good in that it can be easily exported from modeling programs, but imo you need a binary (or partially binary) format as well for fast loading. In lightfeather, we split up ascii and binary data, storing material information and basic model structure as ascii (in a format similar to .ini) while storing index,vertex, and animation data in binary form. I'm not saying that's the only way, but I think that for speed reasons you want binary data for fast loading in a production-ready project.

with regard to converters vs exporters: I think both are important. Converters are often easier to write (as long as what you're converting from has open specs) and are necessary to allow people to use all the free models found online in common formats such as .x, .obj, etc. I'll readily agree though that exporters are nice for a simple, full-featured workflow.

[Nox587]

For the love of all things good, do not use xml as the sole format for an internal engine filetype. Not for storing vertex and index data, anyway. It's far too slow to load. Collada is XML because it's an interchange format. .obj is similarly used as an interchange format. An xml format is good in that it can be easily exported from modeling programs, but imo you need a binary (or partially binary) format as well for fast loading. In lightfeather, we split up ascii and binary data, storing material information and basic model structure as ascii (in a format similar to .ini) while storing index,vertex, and animation data in binary form. I'm not saying that's the only way, but I think that for speed reasons you want binary data for fast loading in a production-ready project.

with regard to converters vs exporters: I think both are important. Converters are often easier to write (as long as what you're converting from has open specs) and are necessary to allow people to use all the free models found online in common formats such as .x, .obj, etc. I'll readily agree though that exporters are nice for a simple, full-featured workflow.

Thought changing the title of the post would help, but I just will make it clear now...I'm NOT creating a format, I'm extending the functionality of the Ogre mesh format with irrlicht. This format is only xml when its converted from a modeling program, it will ONLY work when its in binary format. If you look at the loader for an ogre mesh file, either in irrlicht or in ogre you can see that all the data is being serialized... I don't think I'd ever run any game data using native XML...its just something thats convenient to use between the modeling program and the engine.

[Electron]

Ah, I'm sorry. I must have read too quickly and misunderstood.