IFileSystem improvement

[wetfire]

why don't add a function to remove an archive file from virtual file system.

now we have add*FileArchive only.

I propose to add rem*FileArchive so we can have many archive with differents resources but with the same name.

example:

Level1.zip :
level.irr
models/room.3ds
tex/roomTex.tga

Level2.zip :
level.irr
models/room.3ds
tex/roomTex.tga

and so on...

with rem*FileArchive we could remove Level1.zip and add Leve2.zip.

I see many advantages using this approach in dowloading levels from internet.

PS: sorry for my bad english.

[Ico]

That has been discussed before. If the contained files are always the same (or at least the 100% used files are always replaced) there shouldn't be any problem in doing it - afaik the last added zip file is always prefered. But I never tried what happens if you add file 1, add file 2 and then again add file 1 (that's already loaded).

[wetfire]

thank you for answer me.
I tried to add differents zip files but I got always the texture from first inserted archive.
I think that I will try to add the remove function by myself and then I will comment the results.

bye

[bitplane]

I don't think adding another bunch or archive handling functions is the way to go, the filesystem really needs to be more general.

Zeus and I were discussing this a while back, here's a list of annoying problems with the filesystem at present-

1) No easy way to add new archive formats, remove archives or organise the priority of them.
2) You can't cd into archives or use them in paths.
3) Some directories are case sensitive, while others are not. Consider "X.TAR\Test.X", "x.tar\Test.X" and "x.tar\test.x" on different platforms.
4) The caches have no idea where files are located. Open "test.x", change working dir, you can't open a second "test.x" because it already exists in the cache.
5) No concept of volumes in windows-style filesystems, so for example you can't get a list of drives.

Here's my first attempt at a solution, it allows pluggable archive loaders, proper URI's as fake volumes, the same file names in different paths, getting files from encrypted databases and other fun stuff like that. It doesn't really address the closing archives problem, or setting the priority of them.
It's also my first attempt at UML so I might not have used the right arrows for interfaces etc.


source here, opens in this

No promises about implementing this anytime soon, but I'd like feedback on the design.

Is it all too much bloat and hassle given Irrlicht's size? Is the SPath idea flawed? Will shallow links result in cyclic reference nastiness? Is it complete rubbish and if so, do you have a better idea?
Don't pull your punches people, I can take it!

[Ico]

Might be only a little bit related but could prevent even more to be changed some time later on: IReadFile/IWriteFile still use signed long for positions/offsets (together with the 'old' ansi c functions). As far as I know this would require different implementations at least for Windows/Linux (not sure about MaxOS and others).

-----

I'd like to see some kind of URL/protocol handling instead of file extensions.

Replace IFileSystem->addFolder() with addURL() and specify URLs like this:

"zip://./Graphics.pak"
"ftp://myserver/upload/"
"file://c|/windows/"
"ram://cachefile"

Similar goes for SPath.

Replace ArchiveLoader with ProtocolLoader.

Replace iFileSystem->getRootDirectory() with iFileSystem->getRoots() returning an array of iFileSystemDirectory objects representing all added URLs.

Add an attribute bSearchAllowed to iFileSystemDirectory so you could keep the texture loader from searching for files on a ftp server (just to name an example).

I'm missing a GetSubDirectories() returning an array of iFileSystemDirectory objects in iFileSystemDirectory.

Only problem I'm facing right now: How to access a zip file located on a ftp server?

[Midnight]

I'm not familure enough to speak about the technical side.

but I really like the concepts of sqlite loaders and zips within zips/paks...

the system "seems" fundimental and I hope there are no issues like I said though I'm not familure with OS filesystems well enough to see flaws.

I really like this idea I could think of a few more systems that need more design work in irrlicht aswell.

nice work as always bitplane. I'll try to help as much as I can bro just send me the samples for review.

[MasterGod]

Oh and the addFolderFileArchive() is totally bugged too.
Although I don't want to promise anything I might get enough time to look at it next month cause I've been working with such code/algorithm before. Its just that I don't have the right amount of time..

[bitplane]

I'd like to see some kind of URL/protocol handling instead of file extensions.

Replace IFileSystem->addFolder() with addURL() and specify URLs like this:

"zip://./Graphics.pak"
"ftp://myserver/upload/"
"file://c|/windows/"
"ram://cachefile"

This is possible with the current design, you'd just add a special "ftp:" dir to the root, which would deal with de-escaping paths, setting logins and passwords, doing comparisons with file names etc. There would be no need for protocol handlers as such, as they'd just be folders with their own rules for breaking down paths.

Replace ArchiveLoader with ProtocolLoader.

Replace iFileSystem->getRootDirectory() with iFileSystem->getRoots() returning an array of iFileSystemDirectory objects representing all added URLs.

Yes I think maybe "directory" is the wrong name, it suggests that all directories exist as IFileSystemDirectory objects. How about renaming it to "volume" instead? Also I think volumes need their own cwd (each drive in windows has a cwd, as does an FTP)

Add an attribute bSearchAllowed to iFileSystemDirectory so you could keep the texture loader from searching for files on a ftp server (just to name an example).

Yes I've missed the search path completely. Need a solution where explicitly added archives become part of the search path, but implicitly opened ones do not.

I'm missing a GetSubDirectories() returning an array of iFileSystemDirectory objects in iFileSystemDirectory.

how about getChildren for volumes, and createFileList for a list of files and folders.

Only problem I'm facing right now: How to access a zip file located on a ftp server?

Here's how I see it working-
I do getMesh("ftp://user:pass@host:port/zip%20file.zip/test.x"), which causes filesystem->getPath() which sees the first ":" and passes control to the root, which finds the "ftp" volume. The ftp volume doesn't find the user/pass/host/port child so getPath causes a new ftp-connection child volume, which resolves "zip%20file.zip" to "zip file.zip". This isn't a directory in the ftp connection, so it's passed to the archive handlers, one of which says it can open a file called "zip file.zip". It is then told to add it, so the zip archive loader requests the file, causing it to be downloaded, then adds it as a child volume of the ftp connection. "ftp://user:pass@host:port/zip%20file.zip/test.x" has now become an SPath made of the deepest volume (zip file.zip) and the path to the file (test.x).
I hope that makes sense.. it does seem rather complex and possibly slow at times though.

[wetfire]

I have given a glance at bitplane uml project and I've found It very interesting, But now I have a question: Who will develope these features?
I can work at main implementation but I need an help for additionals archives, networks protocols management.

...so...

I'm waiting for coo-workers...

see you

[wetfire]

This is my personal idea :

We can use protocols like file://, http://, ftp://, ( I think zip:// is not needed).
We may use the application directory as root (/) and let to mount several devices in our virtual file system.
In this way we are able to access to files in archives (like zip, rar, ..., etc ),
in memory and on physical file system friendly.
when we don't need an archive any more, we can unmount it from our "virtual file system" and free references to it.

[Ico]

Volumes would be ok for me.

But replace the bIsReadOnly flag on directories and use some kind of bitmask instead (readonly, writeonly, mixed) as there could be paths that are write only. Imageine some kind of smtp handler allowing you to submit bug reports. (path similar to mailto:)

[bitplane]

I have given a glance at bitplane uml project and I've found It very interesting, But now I have a question: Who will develope these features?

I would do the IFileSystem work and convert the archive types, but first I need input from everyone who's interested, find flaws and revise the design. It would mean changes to lots of Irrlicht's files, so I want to get it right first time.

Implementations of new loaders aren't important, only their possibility. We can't add network folders, rars, 7z, sqlite or anything else that needs external libraries into Irrlicht, they would have to be external extensions.

I'll update the design this week, rename IFileSystemFolder to something to do with volumes and add path info.. then I'm gonna start nagging everyone to review it, repeatedly

[wetfire]

ok, but I think that current IFileSystem Interface could be remain the same,
we can add new needed members only, so we don't have to change all the files in Irrlicht.

I'm waiting for your review.

[FuzzYspo0N]

me too, i will help if i can

[CuteAlien]

I still haven't found time to understand the new design. But I see one thing which is worrying me. That's that you wrote in SPath that it allows case insensitive comparison of paths.

<rant>
Making filenames and pathes case-insensitive is for me currently one of the biggest bugs which the current Irrlicht versions do have. The reason is that this simply prevents me from using it on linux systems. "Path" and "path" on linux are two different paths which are as different as "Path" and "AnOdeToCaseSensivity". And I can't get around that by just being careful because I often have to work with user paths. So any application which currently has to work with user paths will no longer work on linux if any user uses paths with capital letters. And many users do that...
</rant>

So please do not try to hide case-sensitive filenames or pathes for systems where they are needed. You can't - it's not possible - it's a bug.