Files or Databases - Dateien oder Datenbanken

[zyclop]

Hi,
need help. Don't know whether I should use Databases or XML-files to store my data.

When I will use databases, there are big disadvantages e.g. hacking, performance issues and of course you will need an extra server.

And my most important question is: is it possible to handle with, let's say, 5000 files simultaniously in C/C++? My questions are not especially related on the Irrlicht engine.

Greez zyclop

p.s. answers in german are welcome ^^
--------------------------------------------------
Hi,
bin grad am überlegen ob es sinnvoller ist, meine Daten in eine Datenbank oder in XML-Dateien zu verfrachten. Kann mir da jemand helfen?

Ich hab halt bei Datenbanken den Nachteil im Augen, dass es nicht so ganz sicher (Hackbarkeit) ist und man einen extra Server laufen lassen muss (Performance & Ausfallsicherheit).

Und meine wichtigste Frage ist es möglich mehrere Dateien Gleichzeitig zu bearbeiten mit C/C++ (?), ich werf jetzt mal ne Zahl in den Raum: 5000

Greez zyclop

[rogerborg]

Please help us to help you.

What sort of data are you storing?

What kind of performance do you require?

Why platform(s) are you targeting?

As to the file handles question, please Search The Fine Web.

[randomMesh]

Don't know whether I should use Databases or XML-files to store my data.

Vielleicht hilft dir die Tabelle der Vor- und Nachteile von beiden Möglichkeiten.

http://www.devmaster.net/articles/building-mmorpg/

[zyclop]

To answer your questions:

I want to store values from different variables, the values are positive (maybe negative) integers and perhaps strings.

I need very fast read/write performance and highest security. I want to try to update my files every 100ms or even more less.

My program should run first on Windows, for testing and later as an server on Linux (but this is in far future)

@randomMesh: danke für den Tip allerdings weiß ich jetzt immer noch nich ob man bis zu 5000(können auch weniger sein, allerdings min. 2000) Dateien gleichzeitig bearbeiten kann.

@randomMesh: thanx for your tip, but I don't know yet, whether it is possible to handle that much files simultaniously.

[bitplane]

structs in memory are the only way you're going to be able to update a "file" every 0.02ms. That's 50,000 write transactions per second, time to either rethink your design or take out a very big loan

[zyclop]

ok 0,02ms is very fast, I think thats not necessary for me ^^

[rogerborg]

See, this is where we get to the meat of it. You said "5000 files simultaniously" and then "update my files every 100ms".

If "my files" means every one of those 5000 files, than that's 50000 write transactions per second = 0.02ms per transaction.

This is why it's so important to be clear about what operations you actually want to perform.


Sorry, I can't help on the "highest security" issue. I don't know the best practice in that area, but I'm pretty sure that security (i.e. whole-database encryption) is going to eat into your performance.

[Dorth]

Most of the time, the database is inside a secured networked, with the game servers, so having unsecured transfers have no effects. The danger is the entry and exit point of the whole network.

[zyclop]

Ups, I had made a mistake ^^ and I missunderstood you with the 0,02ms... The 5000 files were an example. I think I never will need more than 2000 files or even more less files.

Sorry for my bad english xD

[Ondninja]

Why is it important to store the values in files or database that often?
Is that because it should be read as often?

If it is not read that often, why not just store the variables in memory, and at a given interval dump the contens of the memory?

Your example sounds like you are hitting the physical IO limits on current hardware.

String operations is pretty expensive even given the fastest read write procedure.
I recently had to parse 1 terrabyte of data, extracting information and writing the data to new files. Few string concatenations could mean the difference in hours per gigabyte.

What security issues are you reffering to with regards to database?
Normally it is pretty har to access a database if you don't have the username / password to pass along the connection string.


Can you tell a bit about why it is important to write that often?