Streamed IO and LOD City Demo

Steel Style · Post by **Steel Style** » Sat Oct 25, 2008 10:50 pm

Really cool stuff omaremad ^^, I would wonder how much it's increase the fps.

BlindSide · Post by **BlindSide** » Sun Oct 26, 2008 3:44 am

omaremad wrote:

   
    "   if(color.r>1.0){color.gb+=color.r-1.0;}"
    "   if(color.g>1.0){color.rb+=color.g-1.0;}"
    "   if(color.b>1.0){color.rg+=color.b-1.0;}"

Watch out, that doesn't look ATI friendly. It will probably complain about adding a single component float to a vec2.

On windows texture creation in opengl/dx is very slow so i added a third thread to handle slow driver commands.

OpenGL manuals state that OpenGL is not threadsafe and all OpenGL commands must be called from the same thread, did you have to do any fancy hacks to get this to work? Was a mutex sufficient?

Hold on, you're using MSVC, what brought on the change?

EDIT: Ok I just tested it, it crashes on my PC (On start up). NVidia 6800GT, AMD Athlon 2000 XP. I'll try it on my laptop later, it'll probably run fine on there.

omaremad · Post by **omaremad** » Sun Oct 26, 2008 12:24 pm

OpenGL manuals state that OpenGL is not threadsafe and all OpenGL commands must be called from the same thread, did you have to do any fancy hacks to get this to work? Was a mutex sufficient?

Multiple rendering contexts (thats whats causing crashing on winxp even though its is possible). Each context can be run its own thread and can share its data with other contexts. The contexts can share "lists" like textures vbo etc... using a single conext will need a context switch which will cause rendering slowdown.

I could have avoided this third thread by preallocating(this is the slow step) a pool of textures and just coping the data into them using the main thread (bit by bit to reduce transferload) but that would limit me to predefined texture sizes.

Hold on, you're using MSVC, what brought on the change?

No its still gcc.

Thanks for testing it, but im pulling my hair out trying to make it xp compatible.

BlindSide · Post by **BlindSide** » Sun Oct 26, 2008 12:42 pm

omaremad wrote:
OpenGL manuals state that OpenGL is not threadsafe and all OpenGL commands must be called from the same thread, did you have to do any fancy hacks to get this to work? Was a mutex sufficient?
Multiple rendering contexts (thats whats causing crashing on winxp even though its is possible). Each context can be run its own thread and can share its data with other contexts. The contexts can share "lists" like textures vbo etc... using a single conext will need a context switch which will cause rendering slowdown.

That's pretty fascinating. I was about to suggest that second idea but yeah it sounds icky. I hope this whole thing gets fixed up and goes somewhere, like that racing game you were making, give justice to the file you called GTA

EDIT: Ok I tested it on my laptop, it doesn't crash (Also Windows XP, my desktop pc is just tends to crash more than this one, lol, especially with Bullet), but I get alot of "opps" in the console, and I think it works for like a second (Slowly) then it gives me a garbled RTT as the screen, just a bunch of randomly coloured blocks, it also beeps alot when I press something, pretty fascinating. Anyway maybe in this particular case you have a non power of 2 rtt, in OGL they wreak havoc on built in ati chipsets.

omaremad · Post by **omaremad** » Sun Oct 26, 2008 1:05 pm

Hehe you noticed the filename, sadly i have so little time like last year to do anything requireing dedication.

Screenshots to encourage win vista testerstesters

Mirror · Post by **Mirror** » Sun Oct 26, 2008 1:43 pm

nice clouds! dynamic perlin noise or baked photo?

omaremad · Post by **omaremad** » Sun Oct 26, 2008 5:26 pm

Im not using any RTT's, i guess its because i create multiple contexts without making multiple windows is confusing xp or something?

Anyway when i get time to work on this i will go with the pool method since the whole point of texture streaming is to handle limited memory situations.

Halifax · Post by **Halifax** » Sun Oct 26, 2008 5:31 pm

Well it worked on my XP machine, but with a few problems, as expected. And it ran at like 3 FPS. But since it's not supposed to run on XP, then I guess that's pretty good.

fmx · Post by **fmx** » Sun Oct 26, 2008 11:51 pm

inspired by Halifax's comment, I decided to run it on my XP machine too, and it seems to work flawlessly (fast and smooth, shaders working as expected)!

well, maybe not entirely flawlessly, but that's something I can't confirm for myself until i see it in action on vista

the pavements do feel like a seamless transition from less-detail -> detailed (just a texture change?), but the buildings seem to transition quite rapidly with the detailed versions "popping" into place at a certain distance. Streaming seems to work as expected at any rate.

quite interesting nevertheless, looking forward to seeing this in a more comprehensive demo / project!

omaremad · Post by **omaremad** » Mon Oct 27, 2008 12:21 am

Thank for testing fmx, im glad it worked for you.

The transition should be pretty strong i think the low res textures are 0.125 the big ones. I guess i can add more lod levels but the download size will be bigger. As long as you didn't get any stutter or slowdown when moving then what you see is 100% functional.

Its just weird why its not working on some machines.

Halifax · Post by **Halifax** » Mon Oct 27, 2008 1:36 am

omaremad wrote:Its just weird why its not working on some machines.

Well seeing as it worked on fmx's machine, could it possibly be the bug with 1.4.1 coming to the forefront once again. Could you try building it with 1.4.2, because if you are using OpenGL with 1.4.1, then that is definitely the problem with my computer, and why it is running so slow.

omaremad · Post by **omaremad** » Sat Apr 18, 2009 8:19 pm

I reverted back to a loading only thread with the upload on the mian thread due to incompatibility issues.

The slow step in texture creation was not the allocation but the mipmap generation, i have made it so this happens over sevral frames to make things seamless.

The new demo should run on all new machines and be very stable (fixed a few bugs).

The perfect solution would be

DDS

since the mipmaps would be precomputed using high quality minification filters rather than having to be computed during loads. Non streamed texture will also load faster due to this even when negating the size of normal vs compressed dds textures.

http://www.mediafire.com/?sharekey=4db4 ... f6e8ebb871

Might use this tech for a nice tech demo i have planned, i just need the time.

EDIT: VRAM number displayed is an estimate that is updated every few seconds.

devsh · Post by **devsh** » Sat Apr 18, 2009 9:23 pm

excuse me but HOW THE gently caress DID you manage to load and offload textures in another stream???

Cloudef · Post by **Cloudef** » Sat Apr 18, 2009 9:58 pm

Worked really nice! You dint have linux binary anymore so i ran it trough Wine. I have quad core so implementing this to my server would be awesome!

christianclavet · Post by **christianclavet** » Sat Apr 18, 2009 11:22 pm

Hi, I tested your last version on Vista 64 french edition.

The sky is not visible and show a purple background.
When I close the application with ALT-F4, the SHELL(DOS) window is still open like the application is still running (It still run until I close the console window).

Here is the console output from that window:

Code: Select all

Irrlicht Engine version 1.5
Microsoft Windows Vista Personal Service Pack 1 (Build 6001)
Using renderer: OpenGL 3.0.0
GeForce 9800 GX2/PCI/SSE2: NVIDIA Corporation
OpenGL driver version is 1.2 or better.
GLSL version: 1.3
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\road.tga
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\pavement.tga
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\window10.tga
Loaded mesh: data/low/Mesh.000.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\road_n.tga
Loaded texture: c:\users\christian\desktop\gta2\data\low\pavement_n.tga
Loaded texture: c:\users\christian\desktop\gta2\data\low\window10_n.tga
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\windows.tga
Loaded mesh: data/low/Mesh.001.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\windows_n.tga
Loaded mesh: data/low/Mesh.002.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\windows3.tga
Loaded mesh: data/low/Mesh.003.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\windows3_n.tga
Loaded mesh: data/low/Mesh.004.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\window6.tga
Loaded mesh: data/low/Mesh.005.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\window6_n.tga
Loaded mesh: data/low/Mesh.006.irrmesh
Loaded mesh: data/low/Mesh.007.irrmesh
Loaded mesh: data/low/Mesh.008.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\window4.tga
Loaded mesh: data/low/Mesh.009.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\window4_n.tga
Loaded mesh: data/low/Mesh.010.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\window8.tga
Loaded mesh: data/low/Mesh.011.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\window8_n.tga
Loaded mesh: data/low/Mesh.012.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\low\windows2.tga
Loaded mesh: data/low/Mesh.013.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\low\windows2_n.tga
Loaded mesh: data/low/Mesh.014.irrmesh
Loaded mesh: data/low/Mesh.015.irrmesh
Loaded texture: C:\Users\Christian\Desktop\gta2\data\hi\road.tga
Loaded texture: C:\Users\Christian\Desktop\gta2\data\hi\pavement.tga
Loaded texture: C:\Users\Christian\Desktop\gta2\data\hi\window6.tga
Loaded mesh: data/hi/Mesh.006.irrmesh
Loaded texture: c:\users\christian\desktop\gta2\data\hi\road_n.tga
Loaded texture: c:\users\christian\desktop\gta2\data\hi\pavement_n.tga
Loaded texture: c:\users\christian\desktop\gta2\data\hi\window6_n.tga

Here 2 screen of the application:
1: When first loaded - low version is loaded

2: Here after navigating for a while in the scene, high version is loaded