DFF mesh format loader

[Acki]

this looks interesting...
I think this is the dff loader I tried to help you with, some time ago, isn't it ???

well, I tried it out and therefore I installed gta3 and also vc...
but I don't get it to work !!!
it always crashes if I try to load one of the dff files...
I'm sure it's not the loader, but how I try to use it...

so my question now is if you could do a small example for how to use it, I mean what gta files and how I have to load with the loader ???
I'm curious, because there are only a few and very small dff files in gta...

[loki1985]

I think this is the dff loader I tried to help you with, some time ago, isn't it ???

i guess you are reffering to this: http://irrlicht.sourceforge.net/phpBB2/ ... highlight=
and no, that was not me. but i also posted in this thread because of the topic.

so my question now is if you could do a small example for how to use it, I mean what gta files and how I have to load with the loader ???
I'm curious, because there are only a few and very small dff files in gta...

well, as a quick test you can try to load loplyguy.dff from \models\Generic from GTA3. that works for me.

but the real world data DFFs are stored in the archive gta3.img. there are a lot of tools for extracting this format, search google for "IMGTool".

in there you will find a huge lot of DFF and TXD files. most DFF files can be loaded directly (tho i did only use world data for now, no cars or similar). the textures for the models are stored inside the TXD files, these must be extracted to TGA or JPG first (for this i recommend using TXDWorkshop).

[Acki]

but the real world data DFFs are stored in the archive gta3.img.

ahhhh, got them !!!

well, first I found a bug with the textures, they are all mirrored, you can see this on surfaces with text on it:

I only tested it with GTA3 though...

to load the maps I wrote a map-loader class that parses the ipl files (GTA3, GTAVC and GTASA)...
maybe you can use it, too...

cGTALoader.h

Code: Select all

#ifndef CGTALOADER_H
#define CGTALOADER_H

#include <irrlicht.h>
#include "CDFFMeshFileLoader.h"
using namespace irr;
using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;

enum gtaVersion{
  gta3 = 0,
  gtaVC,
  gtaSA
};

struct datMesh{
  s32 id;
  stringc file;
  vector3df pos;
  vector3df scl;
  vector3df rot;
};

class cGTALoader{
  private:
    ISceneManager* sceneManager;
    IrrlichtDevice* irrDevice;

    bool loadMesh(datMesh mesh, ISceneNode* root, bool asOctTree);
    datMesh parse3(stringc s);
    datMesh parseVC(stringc s);
    datMesh parseSA(stringc s);

  public:
    cGTALoader(IrrlichtDevice* dvc);
    ISceneNode* loadMap(char* iplFile, gtaVersion v, bool asOctTree = true);
};

#endif // CGTALOADER_H

cGTALoader.cpp

Code: Select all

#include "cGTALoader.h"

cGTALoader::cGTALoader(IrrlichtDevice* dvc){
  irrDevice = dvc;
  sceneManager = irrDevice->getSceneManager();
  CDFFMeshFileLoader* dffLoader = new CDFFMeshFileLoader(irrDevice);
  sceneManager->addExternalMeshLoader(dffLoader);
}

datMesh cGTALoader::parse3(stringc s){
  datMesh neu;
  int p;
  /*! GTA3
    inst
      id, name, x, y, z, sx, sy, sz, rx, ry, rz, rw
    end
  */
  // get ID
  p = s.findFirst(',');
  s[p] = '\0';
  stringc ID = s.c_str();
  ID.trim();
  s = &s[p + 1];
  neu.id = atoi(ID.c_str());
  // get Name
  p = s.findFirst(',');
  s[p] = '\0';
  neu.file = s.c_str();
  neu.file.trim();
  neu.file += ".dff";
  s = &s[p + 1];
  // get posX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pX = s.c_str();
  pX.trim();
  s = &s[p + 1];
  neu.pos.X = atof(pX.c_str());
  // get posY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pY = s.c_str();
  pY.trim();
  s = &s[p + 1];
  neu.pos.Y = atof(pY.c_str());
  // get posZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pZ = s.c_str();
  pZ.trim();
  s = &s[p + 1];
  neu.pos.Z = atof(pZ.c_str());
  // get sclX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sX = s.c_str();
  sX.trim();
  s = &s[p + 1];
  neu.scl.X = atof(sX.c_str());
  // get sclY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sY = s.c_str();
  sY.trim();
  s = &s[p + 1];
  neu.scl.Y = atof(sY.c_str());
  // get sclZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sZ = s.c_str();
  sZ.trim();
  s = &s[p + 1];
  neu.scl.Z = atof(sZ.c_str());
  // get rotX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rX = s.c_str();
  rX.trim();
  s = &s[p + 1];
  neu.rot.X = atof(rX.c_str());
  // get rotY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rY = s.c_str();
  rY.trim();
  s = &s[p + 1];
  neu.rot.Y = atof(rY.c_str());
  // get rotZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rZ = s.c_str();
  rZ.trim();
  s = &s[p + 1];
  neu.rot.Z = atof(rZ.c_str());

  return neu;
}
datMesh cGTALoader::parseVC(stringc s){
  datMesh neu;
  int p;
  /*! GTAVC
    inst
      id, name, int, x, y, z, sx, sy, sz, rx, ry, rz, rw
    end
  */
  // get ID
  p = s.findFirst(',');
  s[p] = '\0';
  stringc ID = s.c_str();
  ID.trim();
  s = &s[p + 1];
  neu.id = atoi(ID.c_str());
  // get Name
  p = s.findFirst(',');
  s[p] = '\0';
  neu.file = s.c_str();
  neu.file.trim();
  neu.file += ".dff";
  s = &s[p + 1];
  // get Interior Number
  p = s.findFirst(',');
  s = &s[p + 1];
  // get posX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pX = s.c_str();
  pX.trim();
  s = &s[p + 1];
  neu.pos.X = atof(pX.c_str());
  // get posY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pY = s.c_str();
  pY.trim();
  s = &s[p + 1];
  neu.pos.Y = atof(pY.c_str());
  // get posZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pZ = s.c_str();
  pZ.trim();
  s = &s[p + 1];
  neu.pos.Z = atof(pZ.c_str());
  // get sclX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sX = s.c_str();
  sX.trim();
  s = &s[p + 1];
  neu.scl.X = atof(sX.c_str());
  // get sclY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sY = s.c_str();
  sY.trim();
  s = &s[p + 1];
  neu.scl.Y = atof(sY.c_str());
  // get sclZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc sZ = s.c_str();
  sZ.trim();
  s = &s[p + 1];
  neu.scl.Z = atof(sZ.c_str());
  // get rotX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rX = s.c_str();
  rX.trim();
  s = &s[p + 1];
  neu.rot.X = atof(rX.c_str());
  // get rotY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rY = s.c_str();
  rY.trim();
  s = &s[p + 1];
  neu.rot.Y = atof(rY.c_str());
  // get rotZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rZ = s.c_str();
  rZ.trim();
  s = &s[p + 1];
  neu.rot.Z = atof(rZ.c_str());

  return neu;
}
datMesh cGTALoader::parseSA(stringc s){
  datMesh neu;
  int p;
  /*! GTASA
    inst
      id, name, int, x, y, z, rx, ry, rz, rw, lod
    end
  */
  // get ID
  p = s.findFirst(',');
  s[p] = '\0';
  stringc ID = s.c_str();
  ID.trim();
  s = &s[p + 1];
  neu.id = atoi(ID.c_str());
  // get Name
  p = s.findFirst(',');
  s[p] = '\0';
  neu.file = s.c_str();
  neu.file.trim();
  neu.file += ".dff";
  s = &s[p + 1];
  // get Interior Number
  p = s.findFirst(',');
  s = &s[p + 1];
  // get posX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pX = s.c_str();
  pX.trim();
  s = &s[p + 1];
  neu.pos.X = atof(pX.c_str());
  // get posY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pY = s.c_str();
  pY.trim();
  s = &s[p + 1];
  neu.pos.Y = atof(pY.c_str());
  // get posZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc pZ = s.c_str();
  pZ.trim();
  s = &s[p + 1];
  neu.pos.Z = atof(pZ.c_str());
  // get rotX
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rX = s.c_str();
  rX.trim();
  s = &s[p + 1];
  neu.rot.X = atof(rX.c_str());
  // get rotY
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rY = s.c_str();
  rY.trim();
  s = &s[p + 1];
  neu.rot.Y = atof(rY.c_str());
  // get rotZ
  p = s.findFirst(',');
  s[p] = '\0';
  stringc rZ = s.c_str();
  rZ.trim();
  s = &s[p + 1];
  neu.rot.Z = atof(rZ.c_str());
  // get sclX (always 1,1,1)
  neu.scl = vector3df(1,1,1);

  return neu;
}

ISceneNode* cGTALoader::loadMap(char* iplFile, gtaVersion v, bool asOctTree){
  array<datMesh> lstMeshes;
  char inp[1024];
  bool inINST = false;
  if(FILE* fTMP = fopen(iplFile, "r")){
    while(!feof(fTMP)){
      if(fgets(inp, 1024, fTMP) != NULL){
        stringc test = inp;
        test.trim();
        if(!inINST){
          if(test.equals_ignore_case("inst")) inINST = true;
        }else{
          if(test.equals_ignore_case("end")) break;
          datMesh neu;
          switch(v){
            case gta3:
              neu = parse3(test);
              break;
            case gtaVC:
              neu = parseVC(test);
              break;
            case gtaSA:
              neu = parseSA(test);
              break;
          }
          lstMeshes.push_back(neu);
        }
      }
    }
    fclose(fTMP);
  }
  ISceneNode* root = sceneManager->addEmptySceneNode();
  for(u32 t = 0; t < lstMeshes.size(); ++t)
    loadMesh(lstMeshes[t], root, asOctTree);

  return root;
}

bool cGTALoader::loadMesh(datMesh mesh, ISceneNode* root, bool asOctTree){
  IAnimatedMesh* m = sceneManager->getMesh(mesh.file.c_str());
  if(!m) return false;
  ISceneNode* node;
  if(asOctTree){
    node = sceneManager->addOctTreeSceneNode(m, root, mesh.id);
  }else{
    node = sceneManager->addAnimatedMeshSceneNode(m, root, mesh.id);
  }
  if(!node) return false;
  node->setPosition(mesh.pos);
  node->setRotation(mesh.rot);
  node->setScale(mesh.scl);

  return true;
}

and a test program:

Code: Select all

#include "cGTALoader.h"

int main(){
	IrrlichtDevice *device = createDevice(EDT_DIRECT3D9, dimension2d<s32>(640, 480), 16, false, false, false, 0);
	IVideoDriver* driver = device->getVideoDriver();
	ISceneManager* smgr = device->getSceneManager();
	device->setWindowCaption(L"GTA-Loader");

  // for easyness I packed all mesh and texture files in an archive
  device->getFileSystem()->addZipFileArchive("gta3img.zip");

  // load a map via ipl file
  cGTALoader gtaLoader(device);
  ISceneNode* map = gtaLoader.loadMap("maps\\comSW.ipl", gta3);
  map->setRotation(vector3df(-90,0,0));
  map->setScale(vector3df(2,2,2));

	// setup and run Irrlicht render loop
  smgr->addCameraSceneNodeFPS(0,50,50);
  device->getCursorControl()->setVisible(false);
	while(device->run()){
		driver->beginScene(true, true, video::SColor(0,0,0,200));
		smgr->drawAll();
		driver->endScene();
	}
  device->drop();
	return 0;
}

[loki1985]

wow, not bad.
yesterday i converted my own script parsing classes into one SceneNode class, and now i am doing some cleaning up. then this can also be used to load GTA cities without much hassle.

the problem with the texture flipping i suspected also some days ago, i will look into this later.

[loki1985]

well, 2 new files:

http://b66883.com/projects/dffloader/CD ... 081001.zip

updated version of the loader, textures get flipped and should be correct now. plus a small stability fix applied.



http://b66883.com/projects/dffloader/CG ... 081001.zip

my parsing routines combined into one scene node (tho i am not sure whether it qualifies as a conforming scene node).

this works pretty well, but it is still a little rough, and it uses some standard C++ classes as well as some C functions (atoi, strtok).

sample code to use this scene node:

Code: Select all

int main(int argc, char *argv[])
{
    int gameId = 1; // GTA3

    IrrlichtDevice *device = NULL;
    IVideoDriver *driver = NULL;
    ISceneManager *smgr = NULL;

    // use EDT_OPENGL or EDT_BURNINGSVIDEO
    device = createDevice(video::EDT_OPENGL, dimension2d<s32>(640, 480), 32, false, false, false, 0);

    driver = device->getVideoDriver();
    smgr = device->getSceneManager();

    CDFFMeshFileLoader dffloader = CDFFMeshFileLoader(device);
    smgr->addExternalMeshLoader(&dffloader);

    CGTASceneNode *gtanode = new CGTASceneNode(device, gameId, NULL, smgr, -1, core::vector3df(0.0f, 0.0f, 0.0f), core::vector3df(0.0f, 0.0f, 0.0f), core::vector3df(1.0f, 1.0f, 1.0f));
    gtanode->initialise();

    // we must manually render at least once
    gtanode->render();

    smgr->addCameraSceneNodeFPS(NULL, 100.0f, 50.0f, -1, 0, 0, false, 0.0f);

    while(device->run())
    {
        if (device->isWindowActive())
        {
            driver->beginScene(true, true, SColor(255,100,101,140));

            smgr->drawAll();

            driver->endScene();
        }
    }

    device->drop();

    return 0;
}

[Acki]

the archive with the mesh file loader seems to be corrupted...

[loki1985]

maybe try reloading. for me it works, i can download and extract successfully.

[Acki]

I tried it several times, also cleared the cache before each try !!!


EDIT: hmm, strange, just for testing I installed 7zip and it can open the archive !!!
with all other archives from you I had no problem with PowerAdchiver, I wonder what could be cause it to not open this archive !?!?!

[loki1985]

hm. i downloaded and tested on another PC right now, now 2 times with winrar and 7zip. all worked perfectly.

hope it works now using 7zip. this is a really strange behaviour.

[bitplane]

I really wanna find some time to play with this! I'll start downloading the GTA collection again tonight
(Hopefully steam will let me access the files)

[loki1985]

i found one more bug in the parser of the Scene Node, which loaded corrupt positions in VC and SA. fixed version:

http://b66883.com/projects/dffloader/CG ... 081006.zip

[sio2]

Is there a way to batch extract the textures? I've a quick go with TXDWorkshop but it doesn't seem to want to extract all textures in one go. I don't want to do all 700 one-by-one!

[loki1985]

Is there a way to batch extract the textures? I've a quick go with TXDWorkshop but it doesn't seem to want to extract all textures in one go. I don't want to do all 700 one-by-one!

there is: you can use txdworkshop on the command line, i think running "txdworkshop /extract somefile.txd" extracts all contained textures to TGA files into a new subfolder of the same name. with a bug tho: this new folder gets created in the root of the drive where the TXD file is. this means you will end up with a huge list of new folders in e.g. C:\.
personally i tried to avoid this problem by creating a virtual hard drive using TrueCrypt, and did all the converting in there. that worked pretty good.

[sio2]

Is there a way to batch extract the textures? I've a quick go with TXDWorkshop but it doesn't seem to want to extract all textures in one go. I don't want to do all 700 one-by-one!

there is: you can use txdworkshop on the command line, i think running "txdworkshop /extract somefile.txd" extracts all contained textures to TGA files into a new subfolder of the same name. with a bug tho: this new folder gets created in the root of the drive where the TXD file is. this means you will end up with a huge list of new folders in e.g. C:\.
personally i tried to avoid this problem by creating a virtual hard drive using TrueCrypt, and did all the converting in there. that worked pretty good.

Thanks for that info! I'll give it a try. It's very simple to create a virtual drive in Windows, though, and map it to a folder.

My biggest problem, perhaps, will be when I release a demo - getting end-users to do all this extracting will be a real mess. I guess it would be useful if we could use an .img file as an archive in Irrlicht in the same way as we do for zip files, and add code for Irrlicht to load .txd texture files directly. Some ideas for the future...first thing is to get something working!

[loki1985]

i already asked for this feature here: http://irrlicht.sourceforge.net/phpBB2/ ... highlight=

plus a TXD format loader (which is some work, but absolutely possible) and then your idea works, loading the entire city from the IMG file.