Irrlicht unit tests

CuteAlien · Post by **CuteAlien** » Mon Apr 23, 2012 10:01 pm

This thread is for problems with the Irrlicht unit tests. The tests can be found in the tests folder and are generally there to prevent accidentally breaking working functionality. They should in generally work on all systems and this thread has the target of getting us there.

Anyone can help by running the tests once in a while on his own system and giving feedback about the results. Tests must be run from within the tests folder, which means that on Linux you have to copy the tests binary after running the Makefile to the tests folder before starting it. The results are then written to tests.log. When there had been failing tests it helps us if the corresponding tests.log is posted as well, as it contains additional information. Post it for example to a pastebin side or put it on some server.
- When you post test-results please also give your configuration: OS, Graphic-Card, Compiler, 32-bit or 64-bit (system as well as target), Irrlicht version (svn revision), debug/release
- Please update your graphic card drivers to the newest version.

CuteAlien · Post by **CuteAlien** » Mon Apr 23, 2012 10:17 pm

Currently many tests are failing (15 today on Linux gcc 32 bit, more or less on Windows and/or 64-bit platforms).
I could fix a few over the last months, but spend today a few hours on figuring out _when_ stuff got broken.
Unfortunately results were somewhat ambigious as it seems the tests broke on different revision on Windows and Linux.
Additionally there is the problem that on my Linux system the majority of tests broke between some revisions which fails completely to work. Still I hope it brings us again a little closer to finding those problems, so here my results.

I couldn't really test revisions between 3729 and 3755 as those failed on my Linux system completely (meaning I could only outcomment them to finish at all).
The tests in this table are those failing in the current revision (as of this morning) r4137.
ok = worked.
f = failed
- = test didn't exist yet
crash = test crashed
// = had to be outcommented to run tests through (usually because of asserts)

Some revisions which seem to be obviously missing here couldn't be tested because Irrlicht didn't compile (3400, 3650, 3700).
Also mosty some more tests needed to be outcommented to run through on Linux at all

Code: Select all

 
SVN Revision                3089     3200     3500      3600      3675      3725    3726/VS32      3728    3729/VS32     3756     3762     3800     4137    4137/VS32
'drawPixel'                  ok        ok       ok        ok        ok        ok        f            ok        f            f        f        f        f        f
'material'                    -         -        -         -        ok        ok        f            ok        f            f        f        f        f        f
'renderTargetTexture'        ok        ok        f         f         f         f        f            f         f            f        f        f        f        f
'textureFeatures'            ok        ok       ok        ok        ok        ok        ok           ok        ok          ok        f        f        f        f
'textureRenderStates'        ok        ok        f        ok        ok        ok        ok           ok        f            f        f        f        f        f
'transparentMaterials'                                                                  ok                     f                                                f
'antiAliasing'                -        ok        f         f         f         f        f            f         f            f        f        f        f        f
'draw2DImage'                 -        ok       ok        ok        ok        ok        f            ok        f            f        f        f        f        f
'lights'                      -         -       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'twodmaterial'                -         -        f        ok        ok        ok        f            ok        f            f        f        f        f        f
'viewPort'                    -         -       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'mrt'                         -         -    crash        //     crash        ok        ok           ok        ok           f        f        f        f        ok
'projectionMatrix'           //        //       //        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'lightMaps'                  ok        ok       ok        ok        ok        ok        ok           ok        ok           f        f        f        f        ok
'triangleSelector'            -         -        -         -        -          -        ok           -         -            -        -        -        f        ok

Certainly more tests are needed to find where exactly it breaks - this is just how far I got today (I had hoped they would break on same revisions on Windows-Linux, only found out in the evening that this wasn't the case).

I'm not sure yet if I continue with that table - bascially I'll just use this table as startingpoint when I work on figuring out the next failing tests. And I hope others can also use this info maybe to get faster an idea around when something started going wrong (although results might be different on other systems obviously).

hybrid · Post by **hybrid** » Mon Apr 23, 2012 11:46 pm

Yeah, especially RTTs have many problems on my systems. Sometimes they are not even run due to missing support, sometimes they give strange results. But at least on one system they gave the correct results

I've also fixed drawPixel - at least I hope so. Also, from time to time, I add test cases without actually fixing the cause of the failure. Test first so to say. This is usually not possible with rendered results, though.
On one of my Windows systems I get grey GUI element background instead of white - not really knowing how this can happen. Also make sure that no driver settings override the mipmap or anti-aliasing settings, as this results in severe drifts in the comparisons.

Could someone tell me how to run the tests under recent Linux systems with compiz? Seems that the colors are all messed up then.

hybrid · Post by **hybrid** » Tue Apr 24, 2012 8:24 am

For me, all tests above transparentMaterials do work under VS32 now with rev 4142. Would be also good to know, which subtest fails, and if the problem is merely a threshold one (i.e. result is largely correct) or is a major one with completely failing rendering.

CuteAlien · Post by **CuteAlien** » Tue Apr 24, 2012 8:54 am

Ok, I just run a quick-test, drawPixel is now fixed on VS32 with r4142.
My other results there: http://pastebin.com/7XiyJLBS

edit: Hm.. failing tests only write "failure" to the log. Guess we have to add more info first.
edit2: Also for Info: I compiled with Debug on VS2010 and XP and graphic card is Nvidia GeForce 6200.
edit3: I just run it once more with updated drivers, but same results: http://pastebin.com/fTfh6vgN
edit4: My testresults for VS/Win32 can now be found in http://www.michaelzeilfelder.de/pub/IrrTestR4142VS32/
The failing screenshots are always mentioned in the tests.log, so for example:
http://www.michaelzeilfelder.de/pub/Irr ... onBack.png

Yoran · Post by **Yoran** » Tue Apr 24, 2012 10:18 am

I have run a test with SVN rev 4142 on my system on 2 graphic cards:

Geforce GT330M:
50 tests of 65 passed http://pastebin.com/qSPfWUvL

and now the surprise, Intel HD graphics
62 test of 65 passed http://pastebin.com/Fd7JxKZW

- debug dll
- system: windows 7 pro 64 bit 6.1.7601 Service Pack 1, build 7601
- core i5, 4gb ram

Yoran · Post by **Yoran** » Tue Apr 24, 2012 12:54 pm

D3D fails to capture the correct screenshots, explaining some failed tests

hybrid · Post by **hybrid** » Tue Apr 24, 2012 1:32 pm

Ok, for the GForce it looks indeed as if the d3d driver had severe problems with capturing, while the Intel graphics seems to have basically just one major bug. Because the draw2DImage failure is basically because the file size differs, the difference is negelctable, though. It seems that this is again caused by the 1pixel offset in 2d rendering under d3d. The reflection2textures in transparent materials is a weird bug under OpenGL, which seems to produce different results on each system I've ever run this material. And the Burnings bug seems to vanish from time to time. Just run the test two or three times and it may disappear. So basically the textureMatrixInMixedScenes is a bug, which could be caused by wrongly created reference outputs, though. We should make sure that the colors used in the GUI are as seen in the outputs, because the reference images of the different drivers are also not equal.
@CuteAlien: GForce 6200, whoa

Anyway, looks like the OpenGL driver has some severe problems, as even mipmap locking does not work. I'll check if I can see some more things in the screenshots.

CuteAlien · Post by **CuteAlien** » Tue Apr 24, 2012 1:38 pm

hybrid wrote: @CuteAlien: GForce 6200, whoa ;-) Anyway, looks like the OpenGL driver has some severe problems, as even mipmap locking does not work. I'll check if I can see some more things in the screenshots.

Haha, will hopefully get a new system the next days - but for my 32-bit system I'll have to live with that card, there's not many betters cards for AGP ;-)

In case screenshots just had permission errors please try again, I tried allowing to view the whole folder, but seems my provider doesn't like the .htaccess file. Will put screenshots on another server next time where I have more control over permissions.

hybrid · Post by **hybrid** » Tue Apr 24, 2012 4:56 pm

Ok, draw2DImage test is now also fixed. Well, actually it's only relaxed with comparison. The problem is that in case of down scaling (dest rect is smaller than source rect), d3d driver seems to move the texture by one pixel to the top left. In all other cases, the exact same call does place the texture exactly aligned with OpenGL and the other drivers. I couldn't see any obvious reason for this, and one pixel shift is for now acceptable.

RdR · Post by **RdR** » Tue Apr 24, 2012 5:57 pm

OS: Ubuntu 10.10 (32-bit)
Compiler: GCC 4.7.0
Revision: 4143

Tests finished. 49 tests of 65 passed.

Log: http://pastebin.com/Wx4ebRSd
Results: http://ferket.net/results.zip

hybrid · Post by **hybrid** » Wed Apr 25, 2012 8:06 am

Yeah, just as I already wrote. Without special care taking it does not seem to make sense to run the tests under Linux. Every render test case fails under OpenGL. Red and blue seem to be exchanged, which of course leads to very large differences. Not sure if this is just a problem with screenshots, or general rendering.

Nadro · Post by **Nadro** » Wed Apr 25, 2012 2:09 pm

It's my log:
http://pastebin.com/tz43Abvc

Hardware: AMD Phenom X3 8650 + AMD Radeon HD2400 PRO
Software: Windows 7 x64 + Catalyst 12.1

I will show a results from a test under MacOSX soon (on GF9 series card)

I see a problem with a OpenGL floating point texture support on my machine. I'll fix it

CuteAlien · Post by **CuteAlien** » Wed Apr 25, 2012 4:19 pm

@Nadro: Which svn version? (svn already received some fixes today)

hybrid · Post by **hybrid** » Wed Apr 25, 2012 4:43 pm

Latest, because texture pointer test is running

Irrlicht Engine

Irrlicht unit tests

Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests

Re: Irrlicht unit tests