3000th commit - IrrlichtBAW (GIT repo, v 0.3.0-gamma1)

Discuss about anything related to the Irrlicht Engine, or read announcements about any significant features or usage changes.
Nadro
Posts: 1648
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by Nadro »

devsh wrote:a) setting shader uniforms by name (lookup with a binary search in an array best case, but implemented as a call to OpenGL glGetUniformLocation) - possibly 10% FPS penalty
b) cant access depth buffer as a texture, no fine tuned control over this process
A) - it's already implemented in trunk.
B) - it require just small modification in core, however without breaking an interface (it will be possible to get depth texture via TextureName).

Be patient for other changes. When I will commit new OGL driver we'll be able to merge some missing stuff.
Library helping with network requests, tasks management, logger etc in desktop and mobile apps: https://github.com/GrupaPracuj/hermes
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

@Hendu: Hybrid is the current project lead. It's a problem if he has as little time as he had for the last year, but unless anyone has a time-machine hidden in the backyard there isn't anything we can do about that. There's isn't anyone else with his qualifications right now. And the target he set for 1.9 was also pretty clear - inclusion of the GL ES branch. And right now everyone of us is working on that.

Responding to bug- and patchtracker entries is in general not something you can do in a few minutes spare-time. You have to figure out what it is about (and nearly none of those coming with examples doesn't help). And nearly all entries are _not_ without problems - that much is often visible on a first view. But you can't just go on and write that something is bad (and it often isn't, it might just be a minor problem) - in that case the original sender expects you to write what's better - and that is usually harder to tell than seeing that there _is_ a problem. And then you get into discussion like the one here and more time flies by...
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by hendu »

Just a reply "I'll take a look at this in two weeks" would be better than status quo. If you notice a problem, say so. Any dialogue > radio silence.

If the problem is the existing team not having enough time, the solutions are
a) make time
If you need finances to do that, what would lead there? Are you getting royalties from the book? Are you putting ads to the front page?

b) more people
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

I already know I won't have time for another half year, so why tell someone I will look at it in 2 weeks? I won't. If I find time in the next weeks I already got several things on my todo-list which I already know need to be done. Actually I do browse the patch and bugtrackers every once in a while as well as getting a mail for every new entry - which I always read. As well as every bugreport on the forum (and some more I'm getting per mail, chat & pm). If anything pops up which I consider more important than the stuff I already have on my list then I work on that. Or sometimes if I see something that will likely cost so little time that I can squeeze it in (that nearly never works out - stuff _always_ takes longer than expected even if it looks like it wouldn't - I already got way too many half-finished patches around where I run into trouble at one point).

a) If I could figure out how to make time ... well - it's a secret I've not learned yet. Same with money. Anyone sponsoring development is always welcome to contact me (I have a daily rate, Yoran did sponsor some features and sometimes I do sponsor them myself by just taking off few days for Irrlicht). Right now I'm working on some Android project to get the branch there ahead (on my own finances - just doing it for now until I'm broke again), but yeah - sponsors are welcome.

b) I think Hybrid didn't want to add more people before the 1.9 release (and I aggree somewhat as there's this counter-intuitive rule in software - "Adding People To A Late Project Makes It Later"). Anyway it's something I've proposed a few times by now. But we do have a BD (not sure about FL) and for me that means accepting such decisions.
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

Hendu:
If the problem is the existing team not having enough time, the solutions are
a) make time
If you need finances to do that, what would lead there? Are you getting royalties from the book? Are you putting ads to the front page?
From my experience I can tell you that an occasional donation here and there, or a sponsorship is not going to make extra time... for that you need a constant stream of money that can replace your day-job. I think people at irrlicht are already giving it all the spare time that is reasonable, and tempting them with money to do more work would just amount to them whoring out their social and family life and would be plainly morally wrong to rob them of their lives.

b) more people
definitely need more people with SVN write permissions

Nadro:
B) - it require just small modification in core, however without breaking an interface (it will be possible to get depth texture via TextureName).
You may as well not bother then XD.. better no implementation and a good patch than some half baked stuff like occlusion queries.

The reason why I use the above so viciously is because I can SHARE the depth buffer between render targets!
So I dont lose depth when doing post processing.
So I can do weighted average transparency (2 MRTs) while only writing to one RTT before for the solids in order to not kill my fillrate
So I can do all sorts of depth-based filters and effects without having to render my scene with 2 MRTs and more complex shaders

with the above approach none of these ACTUAL uses of the feature would not work


CuteAlien:

If you get any bug reports or patches in the tracker that would affect/benefit my irrlicht branch... give me links and I will fix/integrate them into the BaW above.


Update of Engine:
1) the IOcclusionQuery interface is up and running
2) we got Conditional Rendering going both as explicit optional argument "IOcclusionQuery* query" to 3D IVideoDriver draw functions and implicit as attaching the query object to ISceneNodes and setting automaticculling to EAC_CONDITIONAL_RENDER

Technically Speaking:
Water reflections take 6.5ms, Drawing Solid Objects (Z-Write) takes 8ms, Occlusion Queries take 6ms, in the typical game scenario.
The amount of time spent on occlusion queries does not include the time spent waiting for the results to be available (we chose to use the blocking query as it gave higher FPS), so imagine the sync stall!!!
Now with the conditional rendering we may avoid this stall, BUT!!!
OcclusionQueries are un-batchable, just one counter of pixels, so need to issue a separate draw command for each test mesh. With our 900+ regions and possible many water planes, this easily explains the 7ms for occlusion queries (despite low polycount!)
So now I'm looking at quasi Hi-Z very fine-grained batch culling GPU-based approaches... obviously I'm not stupid enough to try and approximately render the Z-buffer on the CPU and create max-mipmaps of it. But rather generate the Hi-Z map and test bounding boxes on the GPU by either:
A) using a vertex shader and rendering points to RTT or using transform feedback and then downloading the results to the CPU
B) doing the above but either using the compute shader, image_load_store or other extension to modify the contents of a QUERY_BUFFER OpenGL object, or typecasting/copying from a RenderBuffer/TransformFeedbackBuffer Object to a QueryBuffer Object and using ConditionalRendering to do the culling (all on the GPU, but many draw calls and GL 4.0 required)
C) using GL_draw_indirect and use the GPU to basically fill out the render commands with the shader that does the occlusion testing (definitely GL 4.0 required)
Nadro
Posts: 1648
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by Nadro »

devsh wrote:
B) - it require just small modification in core, however without breaking an interface (it will be possible to get depth texture via TextureName).
You may as well not bother then XD.. better no implementation and a good patch than some half baked stuff like occlusion queries.
It's not half baked stuff. This texture reside in memory as ITexture, just user doesn't have access for them, because all depth textures uses the same name. When you will use different names for them eg. RTT_NAME + "_DEPTH", you can get pointer to depth texture without problem.
devsh wrote:So now I'm looking at quasi Hi-Z very fine-grained batch culling GPU-based approaches
Geometry Shaders + features from GL3.x / GL4.x make GPU really nice for scene management. Occlusion Queries are really bad, of course if you use occlusion queries you should use results eg. from previous frame to avoid too much perf drop. The worst scenerio for occlusion queries are:
- Draw bounding box with OQ active.
- Check result from upper OQ and if passed draw standard node.
Library helping with network requests, tasks management, logger etc in desktop and mobile apps: https://github.com/GrupaPracuj/hermes
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

One has choice of creating the depth attachment as either a texture of a renderbuffer, some drivers or GPUs might have the renderbuffer being faster

plus you'd still have to update the RTT and MRT interface to include the ability to specify the depth buffer to be able to do all the scenarios I described

Question...

Would moving the VCS to git make the management any better?

i.e. so all these branches would be doable
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

Just letting everyone know that Conditional Rendering doesnt work too well if you have many drawcalls (10,000+)

it actually made the whole thing slower by 30% (lol)
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

Things I'm onto now:
- removing CPU-GPU syncpoints

Basically I already removed glFinish() from the end of the frame, thats the thing that most probably slows down current 1.8.1 irrlicht by a factor of 3 or 4 on basic examples


Now I'm looking into if there are more instances of using functions known to cause implicit synchronisation

I know that for example, now when we update/swap meshbuffers for landscapes as you drill or when lighting is recalculated... the "dirty" mesh buffer gets reuploaded to OpenGL in a not-asynchronous manner.

I've put in some GPU memory uploads/downloads (radiosity texture) of my doing which I must take a look at as well

After that we'll test conditional rendering again
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by hendu »

There is no glFinish call in trunk nor in my tree, so either it was removed before 1.7 or you've added it yourself ;)
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by devsh »

I have the argument... "but the trunk has X,Y,Z"

I'm not going to checkout and use the latest svn revision for a commercial project or any "live" project or active development

there is a reason why we develop with STABLE libraries
hendu
Posts: 2600
Joined: Sat Dec 18, 2010 12:53 pm

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by hendu »

1.7 is stable (well oldstable by now). If no such call was in 1.7, then where did it come from?
CuteAlien
Admin
Posts: 9643
Joined: Mon Mar 06, 2006 2:25 pm
Location: Tübingen, Germany
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by CuteAlien »

You claim it slows down our examples by factor 3 or 4 - but as far as I can see Irrlicht wasn't using a glFinish call in any of the releases (I can only check back to 1.0 - but that's over 7 years old - if your Irrlicht version is based on an even earlier release you should _really_ update at some point).
IRC: #irrlicht on irc.libera.chat
Code snippet repository: https://github.com/mzeilfelder/irr-playground-micha
Free racer made with Irrlicht: http://www.irrgheist.com/hcraftsource.htm
Nadro
Posts: 1648
Joined: Sun Feb 19, 2006 9:08 am
Location: Warsaw, Poland

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by Nadro »

Before v1.8.1 Irrlicht used glFinish call only on OSX.
Library helping with network requests, tasks management, logger etc in desktop and mobile apps: https://github.com/GrupaPracuj/hermes
christianclavet
Posts: 1638
Joined: Mon Apr 30, 2007 3:24 am
Location: Montreal, CANADA
Contact:

Re: To The Rescue of Your FPS - Build A World's Irrlicht

Post by christianclavet »

Hi!
devsh wrote:We are finally giving back to the community and here is our irrlicht

BEHOLD THE GODLY && UNDOCUMENTED
http://sodan.dk/irrbaw/irrlicht.zip

Quick diff between directory of our irrlicht and irrlicht-1.8.1 will reveal all the new features

OVERVIEW:
- OpenGL only
- Reworked Shader Constant Callback and setting uniforms with a more performance friendly mechanism (looking up shader uniforms by name every time you want to set them is REALLY bad)
- Tessellation Shaders (DX11 feature)
- 3D Texture Support
- 2D Texture Arrays Support
- DDS Texture Support (look below for explanation)
- Manual Mip Map specification and loading (useful when you generate your own with the superior sinc filter or load DDS DXT compressed files with explicit mip maps))
- FBO Blitting (super fast rendertarget copy)
- Quasi OpenGL 3.2 core compliance
- No fixed function materials
- All shader materials
- Some other stuff I cant remember but is soo epic
Thanks for giving stuff back! BAW visuals are really impressive, and I will try to build this to see; but already I'm sure I will have several problems:
1 - UNDOCUMENTED
2 - "Quick diff between directory of our irrlicht and irrlicht-1.8.1 will reveal all the new features", I'm using TortoiseSVN, how can you do this? (Can diff from a source dir with it's SVN server version but how with 2 local version?)
3- Will the Irrlicht demos will work? Do you have "little demos"?
4- All the materials have to be defined with shaders. I assume there is not even a "solid" shader in there to test. (It should be not too hard to create one but it would have been nicer to have some shaders with it, since it use only shaders)
Post Reply