SQLite Based Filesystem? -- Updated Second Release

[Zeuss]

The second revision of my SQLite Filesystem is complete, and can be downloaded from here: My Irrilcht Projects

All files are now compressed with zLib.
All files can now be encrypted with XTEA algorithm.

Implementation is kept simple as always. Here is the readme for more information:


Readme for Caswal "Zeussy" Parker's SQL Filesystem Extension for Irrlicht.

Copyright Caswal Parker 2007.

Free for Public and Commercial use. All I ask for is a simple credit somewhere,
even if it is just within a readme.txt.


This Package Includes:
------------------------------------------------------------------------------

Irrlicht.dll ( Irrlicht 1.3.1 dll recompiled with SQLite file system support.
Also compiled without Directx8). The lib folder contains a new lib to build with.
New Includes, the only file that has actually changed is IFileSystem.h so that is the only one I provided.
sqlite folder contains a prebuilt lib, and header file for sqlite3.
As you have to build these yourself. Irrlicht folder contains the changed and additional files.
Check Changelog.txt for changes. Just incase you want to do a custom build.


Description:
------------------------------------------------------------------------------

This is an SQL Filesystem extension for Irrlicht, to allow files to be packed easily
into an archive for efficient storage, but also some degree of protection.

The SQL database has single table, with the follow columns:

Key, Filename, Data, FileSize

All data within the database has been compressed with zlib.
Currently no option is given over the compression.
All files are compressed with the Z_BEST_COMPRESSION option.

Data can also be optionally encrypted, with the XTEA Algorithm.
The functions are straightly extracted from the Wikipedia article:
http://en.wikipedia.org/wiki/XTEA#Implementations

My XTEA implementation uses an 128bit key, and 32 rounds.


How to Install:
------------------------------------------------------------------------------

Replace these files with mine, within your Irrlicht source code:
CFileSystem.cpp & CFileSystem.h as well as IFileSystem.h.

Or check Changelog.txt for the specific changes, if you want to add them yourself,
they are not many about 10.

Also add CSQLReader.h and CSQLReader.cpp to your source files.

You also need to add the sqlite.3 header and library files I have provided
to your environment (currently MSVC only).


Usage:
------------------------------------------------------------------------------


The usage is pretty simple. After you have built the new Irrlicht dll,
you can load a SQLite file archive with code similar too:


irr::u32 Key[4] = { 0xFFEA3FA2, 0xDEADBEEF, 0x145AF48B, 0xDC4E2F63 };

device->getFileSystem()->addSQLiteDatabase("./art.db", true, true, Key );


The key can either be a constant like so. Which is not very secure,
and can be retrived from the source.

Or you can pass a function reference into the function using the following syntax and example:

//Defined elsewhere in the program

const irr::f32 Gravity = 9.81f;

const irr::u32* GenKey(void)
{
irr::u32 Seed = 0;

memcpy( &Seed, &Gravity, 4 );

u32 Key[4];

Key[0] = Seed *= Seed >> 7;

Key[1] = Seed *= Seed >> 9;

Key[2] = Seed *= Seed >> 11;

Key[3] = Seed *= Seed >> 13;

return Key;
}

//Before you start your file loading
device->getFileSystem()->addSQLiteDatabase("./art.db", GenKey, true, true );



You will have to record the result of GenKey yourself using the debugger or
some kind of Debug output that you can use for passing into the Exporter.

The SQL FileSystem only creates a temporary copy of the key as it decrypts
the freshly loaded data, and immediately deletes the key when completed.

The Exporter has a full readme, just run it from command line
and follow its instructions.


Known Issues/Bugs:
------------------------------------------------------------------------------

It currently does not ignore paths.
It currently stores all filenames in lower case.
Upto the last 7 bytes of a file remain unencrypted. But those 7 bytes
are still compressed.

[Praetor]

I'm definitely interested, the encryption thing would be really helpful for distributing a project or demo without having to worry about someone ripping off all the art files.

[Klasker]

I think this sounds awesome. It is often faster to read and decompress data than to read uncompressed data, since the bottleneck is the almost always the harddisk, so I'm not worrying about speed.

By the way, can you give an example of what an SQL query might look like if I wanted, say, a texture?

[Zeuss]

My current prototype doesn't require you to give it a query, instead you just pass it a filename.

The table for the data I was using basically just has some simple fields:

ID (integer Primary Key field)
Filename (text)
TextureData (Blob - binary data)

A function to get some data out:

Code: Select all

bool FileSystem::GetTextureData(const char *FileName, char **Data, unsigned int &SizeInBytes)
{
	//Lets read some data out
	const char *SELECTBLOB = "SELECT TextureData FROM Textures WHERE FileName = ?";
	sqlite3_stmt *pStmt = 0;

	mRC = sqlite3_prepare(mDB, SELECTBLOB, -1, &pStmt, 0);

	do
	{
		
		if( mRC != SQLITE_OK )
		{
		  return mRC;
		}

		sqlite3_bind_text( pStmt, 1, FileName, strlen(FileName), SQLITE_STATIC  );

		mRC = sqlite3_step(pStmt);

		if( mRC == SQLITE_ROW )
		{
			SizeInBytes = sqlite3_column_bytes( pStmt, 0);
			
			*Data = new char[SizeInBytes];

			memcpy( *Data, sqlite3_column_blob(pStmt, 0), SizeInBytes );
		}
	} while ( mRC == SQLITE_SCHEMA );

	return true;
}

[BlindSide]

Whoa Im very interested too, you are really amazing you know that? I am probably going to be using both your BrittleNode and this thing in my project.

By the way something very similar was done by evo:
http://irrlicht.sourceforge.net/phpBB2/ ... ght=sqlite

But I think your system is more easy to use and I really just want it (mostly for now...) to store assets and encrypt them.

[Zeuss]

evo's wrapper sounds a lot like what I am talking/thinking about.

Probably extend of the IReadFile interface.

I don't know what evo's system was like but this is what I am thinking about :

Code: Select all

	//Load art assets from database if possible
	//												Database name		Key
	device->getFileSystem()->setDatabase("./artassets.db", "ÙB0ØH<ÞÁÙY" );

After you have compiled a new Irrlicht.dll or a precompiled one, its just 1 line to actually use it, after that its just as normal. If the file physically exists in the standard file system it is used, if not the database is searched. This would allow for easy developing.

I would produce some tool too (command-line), that you give the folder of the assets you wish to add and a key. And it will produce your database.

[rooly]

i'm really liking this idea. lets see if we can pitch it to the devs

[Pazystamo]

Yesterday I started planing map system with Evo's SQLite wrapper, I hope that it works and i can use it for my dynamic in game editing map system I want to store info about all objects in SQL database and then load visible objects.

[Praetor]

This sounds great, one question though: will it have the same license as irrlicht (or if a diffent license will it be free for commercial use)?

[Zeuss]

SQLite is in public domain.

For my wrapper, a credit in a readme or something is all i am after.

[bitplane]

I think the option to add different archive loaders/types to filesystem would be the best idea. It would give the opportunity for 7z, bz2, rar, etc archives as well as things like external database filesystems, web folders, etc.

The big question I guess, is how to fit it in with the way things currently work.

[Zeuss]

Well, I have finished my first iteration of a patch.

Have a basic command line packing tool. And just have a function that loads the database, and tested, all my arts assets are loaded out of it.

@bitplane, how about a abstract class that you can derive off? Would not take much work, I could do it if needs be. I noticed that CPakReader and CZipReader are derived from IUnknown but basically have 2 core member functions:

openFile
findFile

There is also a seperate array for both Pak and Zips. Why not an irr::array of an abstract class* so both can be stored in 1 irr::array?

Then add a function to the FileSystem to add your own instance of a derived filesystem class, to the irr::array.

[BlindSide]

Wow I cant wait. Can you also maybe add some basic data storage functions like getAttributeAsInt() similar to XML? Or wait I guess we can load/write to XML files inside the archive right?

Hmm im confused now is it possible to write inside the archive too or only load things? Can this be used for secure storage of configuration/custom ascii based map files/ etc.

[Zeuss]

Currently, the Irrlicht implementation can only read. I could make it write if wanted.

There is a command line exporter, to write the DB File.

When I get home (currently at work ). I will upload the changes to Irrlicht, compiled dll, new includes, new lib, and the exporter.

As well as my BrittleEngine demo using the database.

[Virion]

Cool. I am kinda interested with this. Can it do encryption as well?