SQLite Based Filesystem? -- Updated Second Release

[bitplane]

There is also a seperate array for both Pak and Zips. Why not an irr::array of an abstract class* so both can be stored in 1 irr::array?

Yes, just looking at this now

Ok here's what I'm thinking.. We already have IFileList which is the logical place to hold file info. maybe findFile should be moved there too.

I guess archives should have a name and perhaps also a path, so you can optionally access files like "file.zip/path/to/the/file".
It will also need a priority and a < operator for sorting the search order. I suppose it should have an isWritable method and createAndWriteFile to be able to create files inside an archive, and getType.

createFileList should return all the files, folders, archives and files within archives accessible from the current directory.

should changeWorkingDirectory be able to pwd into an archive?

[Zeuss]

sounds good.

The great thing about Irrlicht to me, is that is has all this complex functionality, if you need it but not thrust upon you.

Priority is an interesting point.

To me if the file can be found outside any archive, it should be used. The contents of an archive should be in the global file space, and treating archives as a folder as an option, defaultly off.


The irr::array containing the resource files should probably be organised by how many files it contains.

Large resource files are more likely to contain the file. So search them first.
Although the search time is at least a couple of factors faster than hdd access.

A < operator is interesting. How do you compare file lists? especially if they are a used defined type. Its like comparing apples to cucumbers.

Or do we just give the user option of defining an operator so theirs is always the best? hehe

[bitplane]

To me if the file can be found outside any archive, it should be used. The contents of an archive should be in the global file space, and treating archives as a folder as an option, defaultly off.

Yep it's currently off, wouldn't want to change current behaviour. Same with priorities. Default priorities should be based on the current system, which is: zips, paks, folders then working dir.

A < operator is interesting. How do you compare file lists? especially if they are a used defined type. Its like comparing apples to cucumbers.

Actually I take that back.. It was to sort the array by priority, but the < operator doesn't work on pointers.

[BlindSide]

Amazing! Thank you Zeuzzy

But your changelog was missing some small things:

- You must also add an include to CFileSystem.h (not just the .CPP).

- You must add

Code: Select all

	//Caswal -- Array of Databases
	core::array<CSQLReader*>  SQLFileSystems;

in the private members of CFileSystem in CFileSystem.h.

Cheers and thanks again this is great.

EDIT: Oops or maybe it is better to add "class CSQLReader;" to line 17 of CFileSystem.h to avoid having to add an include to an essentially "clean" file. <<< Please do this instead of the first thing I mentioned.

[Zeuss]

@Blindside

Didn't i include an already changed CFilesystem.h though? That already works in the ways mention

Anywho, I have updated my own Changelog.

I am going to look into AES encryption today, and using zlib.

So i can compress and encrypt the data going into the DB if needs be.

[BlindSide]

@Zeuss

The included CFileSystem.h would offcourse work, but I don't trust simply swapping files over in my modified irrlicht distribution as this usually leads to errors (And I am using the skinned mesh branch too). So I followed your ChangeLog, which I have to say, apart from those little things about CFileSystem.h, it was very helpful and informative, and stopped me from having to use "diff" on all source files and scratch my head while comparing them in a vertical split text display. (Which I had to do afterwards to implement some other more complicated things).

Btw, I thought there is already encryption because you can set an encryption key in the function, or is this only in the interface?

[Zeuss]

Only the interface currently.

Infact the only parameter that actually works is the filename, for opening the db, the rest arnt currently used. (such as case sensitivity, everything is lower case).

AES encryption looks interesting, a problem though that encryption is only as good as the key.

I'm thinking the implementation might look something like this:

Code: Select all

const u32 Key[4] = { 0x00010203, 0x04050607, 0x08090A0B, 0x0C0D0E0F };

device->getFileSystem()->addSQLiteDatabase("./art.db", false, false, reintpret_cast<char*>(Key) ); 

Then when you pack a file, using the commad-line tool

SQLExporter ./art/ art.db 000102030405060708090A0B0C0D0E0F

You can never make your art perfectly secure. Skilled people could still HexEdit your compiled binary and find the key. And write an Exporter to export the art assets out of the db. I was thinking of never releasing one myself.

Another option I thought of, is writing a pseudo-random number generator, that you just simply pass a number to seed. Which then generates the key.

Some thoughts on this would be nice.

[evo]

Hi Zeuss,

I am curious as to how your library extension helps in accessing SQLite's database functions? A quick glance at your code reveals only read-only archive's. Without encryption that's about the same as a zip archive.
Or am I missing something ?

[Zeuss]

Nope, you arnt missing anything.

It currently is about the same as a zip archive, just a bit more secure.

The idea is to keep assets secure more than just zipping them up into an archive.

I never started this project as a total SQL wrapper, more as secure file system for games, that would require a user to do more than just winrar the archive to get the content out.

Patches could be simple appends of the database too.

What I have written so far has taken me less than 2 hours. Just smashing a bit of code out to get the concept working.

I know that you can't keep something perfectly secure, but I can try to get it close.

[evo]

Ok. I understand now.

Securing the contents of your game data is complicated matter. We had a discussion about here: http://irrlicht.sourceforge.net/phpBB2/ ... hp?t=12345
Nice topicnumber by the way

Some important aspects are:
- Keep the decryption key secure. Do not store it as a constant. Erase the variable after use.
- Keep unprotected data in memory for a short period only. Erase after its not used any more.
- Check if important variables are not being altered by user/h acker

I am sure there are many more factors. And like you said you can not make a 100% secure protection.

[Zeuss]

You can never make your assets totally secure when they are outside of a controlled environment running on a hackers system.

The idea is more to make trying to hack out the assets not worth while.

I guess I could make the implementation take a function pointer to a char* Key(void) function.

That should procederally generate a key on call, but up to the user to implement it.

but to me the number 1 important Aspect is: KISS (Keep is simple, stupid).
The next aspect is to keep it secure beyond worth.

[evo]

Well, that is a dilemma then. Keeping it simple to use but complex to hack at the same time.

[Dorth]

*cough* KISS (Keep i't' simple, stupid) *cough*

[BlindSide]

*cough* KISS (Keep i't' simple, stupid) *cough*

No its KISS (Keep Interface Simple for Stupids)

[Zeuss]

Bump for update, check first post. Been editted.