NEED HELP: BOOST THREAD

[Dareltibus]

I'm tryiing to doing irrlicht multithreaded. yes. i tried first with SFML::thread. I made a version of irrlicht wich run on 2 threads, the problem is that using only mutex allow me to use up 1 cpu at 100% and the second cpu is used only at 50%.

The solution for doing that is very simple but mutexes give not any time control. I tried to use the "sleep" function and the result is that sometimes the second thread run 10 time faster than the first thread (using sleep seems only to reduce cpu usage. So cpu cycles freed by sleep are not used by the other thread).

I've seen that boost allows to set CPU affinity, but the problem is that i can't get it working. It's quite embarassing that problem.

Does anyone can help me make a simple project with code::blocks that can be cross-compiled on both win32 and linux using only boost::thread?

[Radikalizm]

Just out of interest, which parts of the engine are you trying to multithread exactly?

I don't have any experience with boost, I set up my multithreading system and scheduler using native windows threads, so can't help you there

[Dareltibus]

I'm not trying. I'have already done it using SFML, I threaded the whole engine, rendering included (and you will be surprised on how much less code is needed for doing that), of course I have no total control about threads and this results in a not -uniform cpu usage. I have a quad-core, the first core is used correctly at the 100% of cycles . But other threads run on the 2nd and 3rd cpus, using them at no more than 50%. (so the problem is that i cannot set cpu affinities, after did that i will work to fit every CPU at 100% of its potential, but i don't think this will be too hard).

For doing that you need only mutexes no scheduler.


The problem is that SFML lack of advanced features, and when I'm trying to compile a simple console app using Boost::thread i get the following error:

imp__ZN5boost6threadD1 EV.

I prefer to avoid native windows code since i hope that this will be included in the irrlicht SDK so it must be cross-platform and i thinked that boost is the way to go (more functionalities than SFML).


But before doing that i need a C::B project that can be compiled on both linux and windows using Boost::thread, and i'm embaracced about this but i'm not able to doing that.

I took a look at other libraries such as tinythread. The problem is that most of the libraries don't allow to set the cpu affinity. (wich is the feature that I need to be cross-platform).

[Radikalizm]

Only using mutexes here might not be an ideal situation, my multithreading system is task based so it can scale to any amount of CPUs, and it uses work-stealing so tasks are always evenly spread over all available CPUs, the amount of mutexes used is at an absolute minimum since the task-system makes sure data is handled safely

The system does not include rendering though, but there's a good reason for that since I know for a fact that OGL and D3D do not like being run in more than one thread (I believe D3D11 is the first D3D revision to actually support multithreaded rendering)

Also, keep in mind that irrlicht runs on more platforms than just a pc platform, and it is used for a varying amount of games/applications, and sadly enough there is no 'one solution fits all' multithreading system, eg. some systems might prefer async multithreading where certain components can run at higher frequencies than the main update loop, so don't hope for SDK integration too much

Setting up multithreading itself is easy for very specific cases, getting it to run optimally for a global purpose like you would expect in a library like irrlicht would be a major pain in the ass (trust me, I've been there)

[Dareltibus]

This does not help me. Actually mutexes works very well for me. And since you are already doing work -stealing i think someone else must do something else. I used only mutexes for threading almost the whole engine.

I think i have done all the do-able using mutexes so now i need more features. And i'm get stucked of that boost file that don't compile

Ideally also my work can scale to any amount of CPUs, the problem is that i cannot set CPU affinity using SFML

[mongoose7]

I wonder why you are obsessed with CPU affinity. That would seem to prevent the application from scaling with more CPUs. I also think you have a mutexing issue or a scheduling issue - if all threads can run continuously and continually, they should all be at 100% all the time. Although Irrlicht does have a 'yield' function, but I don't know when that is called. Could count the numbers of times it is called, though.

[serengeor]

Although Irrlicht does have a 'yield' function, but I don't know when that is called.

I think its for the user of lib to use, and not used anywhere internally.

[Dareltibus]

yup profiling the function shows that yield, sleep are never called (i tried using also some sleep somewhere at a first try but abandoned it).


I'm not obsessed with cpu affinity. Simply i have four cores and only 2/3 of them are used for a total cpu usage that goes from 33 to 50% (change randomly, using the same test code and get a fps boost from 30 to 45%. I have also shutted down all other processes such as skype).
The problem is that ideally all threads should gather the whole cpu power available. I can increase number of threads and will get the same cpu usage.(and a small drop down of frame rate)

I'm pretty sure no pauses occurs during execution since a thread pauses only when transfering data to another thread.(mutexes grant this due to a little locking time). There is no reason apparently for cause this...


Is so hard get some help compiling Boost.thread? Threading can be done in lots of differents ways so I don't expect someone show me how to thread (never seen a tutorial a bout that here). but the problem now is that i can't compile a simple library :-/.

[serengeor]

Is so hard get some help compiling Boost.thread? Threading can be done in lots of differents ways so I don't expect someone show me how to thread (never seen a tutorial a bout that here). but the problem now is that i can't compile a simple library :-/.

Are you using bjam for compiling?
Because bjam is the way to go with boost. I found it the easiest way to do it, though at the first time it looked ridiculously overcomplicated, but later it was easy enough.

[Dareltibus]

this is the log for bjam:

###
### Using 'msvc' toolset.
###

Downloads\boost_1_46_1\boost_1_46_1\tools\build\v2\engine\src>if exist bootstrap rd /S /Q bootstrap

Downloads\boost_1_46_1\boost_1_46_1\tools\build\v2\engine\src>md bootstrap

Downloads\boost_1_46_1\boost_1_46_1\tools\build\v2\engine\src>cl /nologo /GZ /Zi /MLd /Fobootstrap/ /Fdbootstrap/ -DNT -DYYDEBUG kernel32.lib advapi32.lib user32.lib /Febootstrap\jam0 command.c compile.c debug.c execnt.c expand.c filent.c glob.c hash.c hdrmacro.c headers.c jam.c jambase.c jamgram.c lists.c make.c make1.c newstr.c option.c output.c parse.c pathunix.c regexp.c rules.c scan.c search.c subst.c timestamp.c variable.c modules.c strings.c filesys.c builtins.c md5.c pwd.c class.c w32_getreg.c native.c modules/set.c modules/path.c modules/regex.c modules/property-set.c modules/sequence.c modules/order.c

it doesn't work.. i need powershell?

EDIT:

tried also with powershell

[Sylence]

The problem is that ideally all threads should gather the whole cpu power available.

That's everything but ideal. Threads should take the processing time they need. Not more and not less.

What do you want with all the power available if you only need 10% of it?
And what about users with scalable cpu frequency like laptop users? The CPU will always run at the highest clock available, producing the most heat and taking the most energy possible.

Simply let the OS decide how much computing time it wants to give a thread. Beause the task scheduler knows better what is best

[Dareltibus]

cannot simply help me using boost thread?

if what you say is true a work stealing method is useless since OS already uses a task scheduler.

anyway i checked the code and tested very much in those days.. the problem seems to be CPU affinity. Since when the main thread exchange data with a thread it pauses it for a little while. If all the 3/4 threads are running in the same cpu, this little while become multiplied by 3/4 and becomes bigger.

Actually i'm testing with 500/1000 dwarves. i'm splitting them amongst 7 scene manager (1 scene manager in the main thread and 2 smgr for each other thread).

There is no performance boost since all threads runs at 50% of cpu power(generally in that case dwarves on other threads are updated only 1-3 times every second).

The best performance boost i got were done using only 2 threads (main thread and a second thread) and giving 3/4 smgr to each thread (35% performance boost).

Using 2 threads is the only way for me for get almost 100% the computing power but only from 2 cpus. (not 100% but 98% on the first cpu and 89% on the second cpu, but it is great anyway).

If only i were able to move every thread to another cpu i will have 2 scene managers for every cpu (1 scene manager only for the first thread that must coordinate everything).

Changes to irrlicht was minimanl. Scene manager now can be splitted (as were possible already but this time you can split to several thread).

Scene nodes now can be created also if no textures or mesh exist, they simply ask for loading another thread and wait for the resource becoming available), (loading a file requires an additional thread for now, but i hope i'll fix this).

with performance I'm referring not at frame rate, but to the updating rate of scene nodes. With my system the game always run at 60 or more Frame Per Sec, but some scene nodes are updated only 2/3 times every second if using 4 threads.

[Sylence]

if what you say is true a work stealing method is useless since OS already uses a task scheduler.

You haven't understand a word I've written, do you ?

[Dareltibus]

The problem is that ideally all threads should gather the whole cpu power available.

That's everything but ideal. Threads should take the processing time they need. Not more and not less.

What do you want with all the power available if you only need 10% of it?
And what about users with scalable cpu frequency like laptop users? The CPU will always run at the highest clock available, producing the most heat and taking the most energy possible.

Simply let the OS decide how much computing time it wants to give a thread. Beause the task scheduler knows better what is best

You haven't understand a word I've written, do you ?

I read it.

Do you read what I wrote?

[Radikalizm]

There is a difference between scheduling threads (which is what the OS scheduler does) and letting a scheduler in your application spread tasks over available threads Do some reading before you mock other people's work/techniques

I have to agree with what Sylence said, why are you so intent on getting 100% CPU power at all times while your system needs a lot less?
The idea that you want your CPU to eat power like that and as a result let it heat up your system doesn't reallly sound too healthy