Run your test here

[Fury]

I guess if it is a good idea to create some test app (can be a demo or a 3D mark) that everyone can run. After the app end a xml file is generated and every one runned the app can post here the xml file. In wich way every one can easily erase data that don't want to post on a forum. (or just post the source and everyone can compile and run it)

such kind of data can be for example

get maximum texture size
processor speed
ram
driver version etc.
Operative system ecc.
used ram memory
hardware features ecc. (VBO, shaders ecc)

more complex tests. for example a non interactive demo that run for 5-10 minutes and after that a FPS table is printed (or only average fps)

all data must be queried via irrlicht engine so no external lib are needed.
and in this way you can see cpu and ram detected by engine correctly.
(for example on a dual core machine with 2x2GB ram irrlicht engine will detect only 1 cpu and 1 ram stick (usually on 2 GB only 1,5 are free).)
what is detected by irrlicht is also what can be used by irrlicht.

My idea is to make a irrlicht features compability profile (or something similiar)

if some one will help me i will make the statistical data available in xls format (or opencalc format) for any developer.
and if someone have already a test i can run it on 3 different pcs (one very old, one new, and one middle age)

[Fury.]

In a particular way. will be usefull for all a mark for know the fastest way for loading data I.E.

is faster in loading a zip archive with compressed bmp images?

is faster a zip archive with uncompressed png/jpeg images?

is faster a pk3 archive?

wich is the overhead for cripted archives?

is fastest a xml file loaded from an archive or not?

ways to haste xml files (short names ecc.)

Irrlicht has built-in archives and xml reader/writer. they are powerfull but wich is the worth in using such features?


the same can be done for 3d models, audio and sound of irrlklang etc..

[Simpe]

I really like this idea and framework.

I guess something like this could be implemented using google's test-framework "google-test".

I'd like to add things such as number of memory allocations (which I noticed is quite a lot at startup), memory fragmentation statistics, etc.

If someone is up for the task to do something like this, I'd be happy to code the statistical stuff for memory-statistics.

[Fury.]

Sure, any help is wellcome. I'm working on some tests. I need only high precision timers to measure time required by each test.

Memory allocation and fragmentation tests are very usefull. your Idea is great. I'll try google test and see what appen:)

[REDDemon]

Can someone run this test?

Code: Select all

#include <irrlicht.h>
#include "driverChoice.h"

using namespace irr;

using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;

#ifdef _IRR_WINDOWS_
#pragma comment(lib, "Irrlicht.lib")
//#pragma comment(linker, "/subsystem:windows /ENTRY:mainCRTStartup")
#endif

int main()
{
	// ask user for driver
    video::E_DRIVER_TYPE driverType=driverChoiceConsole();
    if (driverType==video::EDT_COUNT)
            return 1;


	IrrlichtDevice *device=0;
	device = createDevice(driverType, core::dimension2d<u32>(640, 480));

   if (device == 0)
        return 4; // could not create selected driver.



	video::IVideoDriver* driver = device->getVideoDriver();
    scene::ISceneManager* smgr = device->getSceneManager();
    gui::IGUIEnvironment* gui = device->getGUIEnvironment();


	irr::core::stringw strxml("");
	irr::core::stringw filename("");
	switch(driver->getDriverType())
	{
		case EDT_BURNINGSVIDEO:	filename=L"burn.xml"; break;
		case EDT_DIRECT3D8:		filename=L"d3d8.xml"; break;
		case EDT_DIRECT3D9:		filename=L"d3d9.xml"; break;
		case EDT_OPENGL:		filename=L"opengl.xml"; break;
		case EDT_SOFTWARE:		filename=L"software.xml"; break;
		default: device->drop(); return 7;
	}
	irr::io::IFileSystem *OS=device->getFileSystem();
	if(!OS)
		return 2;  //could not get file system
	irr::io::IXMLWriter *xml = OS->createXMLWriter(filename);
	if(!xml)
		return 3; //could not get XML writer

	xml->writeXMLHeader();

	stringw a(""); //a string in wich we store the result of querying HW features

   if(driver->queryFeature(EVDF_RENDER_TO_TARGET))
      a+="YES: EVDF_RENDER_TO_TARGET\n";      
   else         
      a+="NO:  EVDF_RENDER_TO_TARGET\n";

   if(driver->queryFeature(EVDF_HARDWARE_TL))
      a+="YES: EVDF_HARDWARE_TL \n";      
   else         
      a+="NO:  EVDF_HARDWARE_TL \n";

   if(driver->queryFeature(EVDF_MULTITEXTURE))
      a+="YES: EVDF_MULTITEXTURE\n";      
   else         
      a+="NO:  EVDF_MULTITEXTURE\n";

   if(driver->queryFeature(EVDF_BILINEAR_FILTER))
      a+="YES: EVDF_BILINEAR_FILTER\n";      
   else         
      a+="NO:  EVDF_BILINEAR_FILTER\n";

   if(driver->queryFeature(EVDF_MIP_MAP))
      a+="YES: EVDF_MIP_MAP\n";      
   else         
      a+="NO:  EVDF_MIP_MAP\n";

   /*if(driver->queryFeature(EVDF_MIP_MAP_AUTO_UPDATE))
      a+="YES: EVDF_MIP_MAP_AUTO_UPDATE\n";      
   else         
      a+="NO:  EVDF_MIP_MAP_AUTO_UPDATE\n";

   if(driver->queryFeature(EVDF_STENCIL_BUFFER))
      a+="YES: EVDF_STENCIL_BUFFER\n";      
   else         
      a+="NO:  EVDF_STENCIL_BUFFER\n";*/

   if(driver->queryFeature( EVDF_VERTEX_SHADER_1_1 ))
      a+="YES: EVDF_VERTEX_SHADER_1_1\n";      
   else         
      a+="NO:  EVDF_VERTEX_SHADER_1_1\n";

   if(driver->queryFeature( EVDF_VERTEX_SHADER_2_0  ))
      a+="YES: EVDF_VERTEX_SHADER_2_0 \n";      
   else         
      a+="NO:  EVDF_VERTEX_SHADER_2_0 \n";

   //if(driver->queryFeature(  EVDF_VERTEX_SHADER_3_0))
   //   a+="YES: EVDF_VERTEX_SHADER_3_0\n";      
   //else         
   //   a+="NO:  EVDF_VERTEX_SHADER_3_0\n";

   if(driver->queryFeature(EVDF_PIXEL_SHADER_1_1 ))
      a+="YES: EVDF_PIXEL_SHADER_1_1\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_1_1\n";

   if(driver->queryFeature( EVDF_PIXEL_SHADER_1_2 ))
      a+="YES: EVDF_PIXEL_SHADER_1_2\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_1_2\n";

   if(driver->queryFeature( EVDF_PIXEL_SHADER_1_3 ))
      a+="YES: EVDF_PIXEL_SHADER_1_3\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_1_3\n";

   if(driver->queryFeature(EVDF_PIXEL_SHADER_1_4  ))
      a+="YES: EVDF_PIXEL_SHADER_1_4\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_1_4\n";

  /* if(driver->queryFeature( EVDF_PIXEL_SHADER_2_0 ))
      a+="YES: EVDF_PIXEL_SHADER_2_0\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_2_0\n";

   if(driver->queryFeature( EVDF_PIXEL_SHADER_3_0 ))
      a+="YES: EVDF_PIXEL_SHADER_3_0\n";      
   else         
      a+="NO:  EVDF_PIXEL_SHADER_3_0\n";*/

   if(driver->queryFeature(EVDF_ARB_VERTEX_PROGRAM_1  ))
      a+="YES: EVDF_ARB_VERTEX_PROGRAM_1\n";      
   else         
      a+="NO:  EVDF_ARB_VERTEX_PROGRAM_1\n";

   if(driver->queryFeature( EVDF_ARB_FRAGMENT_PROGRAM_1 ))
      a+="YES: EVDF_ARB_FRAGMENT_PROGRAM_1\n";   
   else         
      a+="NO:  EVDF_ARB_FRAGMENT_PROGRAM_1\n";

   if(driver->queryFeature( EVDF_ARB_GLSL ))
      a+="YES: EVDF_ARB_GLSL\n";      
   else         
      a+="NO:  EVDF_ARB_GLSL\n";

   if(driver->queryFeature( EVDF_HLSL ))
      a+="YES: EVDF_HLSL\n";      
   else         
      a+="NO:  EVDF_HLSL\n";

   if(driver->queryFeature(EVDF_TEXTURE_NSQUARE  ))
      a+="YES: EVDF_TEXTURE_NSQUARE\n";      
   else         
      a+="NO:  EVDF_TEXTURE_NSQUARE\n";

   if(driver->queryFeature( EVDF_TEXTURE_NPOT ))
      a+="YES: EVDF_TEXTURE_NPOT\n";      
   else         
      a+="NO:  EVDF_TEXTURE_NPOT\n";

   if(driver->queryFeature( EVDF_FRAMEBUFFER_OBJECT ))
      a+="YES: EVDF_FRAMEBUFFER_OBJECT\n";      
   else         
      a+="NO:  EVDF_FRAMEBUFFER_OBJECT\n";

   if(driver->queryFeature( EVDF_VERTEX_BUFFER_OBJECT ))
      a+="YES: EVDF_VERTEX_BUFFER_OBJECT\n";      
   else         
      a+="NO:  EVDF_VERTEX_BUFFER_OBJECT\n";

   if(driver->queryFeature( EVDF_ALPHA_TO_COVERAGE ))
      a+="YES: EVDF_ALPHA_TO_COVERAGE\n";      
   else         
      a+="NO:  EVDF_ALPHA_TO_COVERAGE\n";

  /* if(driver->queryFeature( EVDF_COLOR_MASK ))
      a+="YES: EVDF_COLOR_MASK\n";   
   else         
      a+="NO:  EVDF_COLOR_MASK\n";*/

   if(driver->queryFeature( EVDF_MULTIPLE_RENDER_TARGETS ))
      a+="YES: EVDF_MULTIPLE_RENDER_TARGETS\n";      
   else         
      a+="NO:  EVDF_MULTIPLE_RENDER_TARGETS\n";

   if(driver->queryFeature( EVDF_MRT_BLEND ))
      a+="YES: EVDF_MRT_BLEND\n";      
   else         
      a+="NO:  EVDF_MRT_BLEND\n";

   //if(driver->queryFeature( EVDF_MRT_COLOR_MASK ))
   //   a+="YES: EVDF_MRT_COLOR_MASK\n";      
   //else         
   //   a+="NO:  EVDF_MRT_COLOR_MASK\n";

   u32 primCount  = driver->getMaximalPrimitiveCount();

   dimension2du maxTSize = driver->getMaxTextureSize();

   stringc vendorInfo = driver->getVendorInfo();


   ////// start writing things into XML file
   //
   //
   //
   xml->writeElement(L"Test001");
   xml->writeLineBreak ();
   xml->writeLineBreak ();
   strxml=L"\n\tYour System Info:"; //print "Your System Info: "

	strxml+=L"\n\t\tOS version: ";  //print os version
	strxml+=device->getOSOperator()->getOperationSystemVersion();

	strxml+=L"\n\t\tCPU speed: ";   //print the speed of the cpu in wich the game run
	{
		irr::u32 cpu=0;
		if(device->getOSOperator()->getProcessorSpeedMHz(&cpu))
		{
			strxml+=cpu;
			strxml+=L" MHz";
		}
		else
			strxml+="not available";
	}

	{                             //print the available RAM memory/Total RAM memory
		irr::u32 tot=0,av=0;
		strxml+=L"\n\t\tRam Memory available: ";
		if(device->getOSOperator()->getSystemMemory(&tot,&av))
		{
			
			strxml+=av;
			strxml+=" / ";
			strxml+=tot;
			strxml+=" KiloBytes";
		}
		else
			strxml+="not available\n";

	}

	{								//print the irrlicht engine version
		strxml+=L"\n\t\tIrrlicht version: ";
		strxml+=device->getVersion();
		strxml+=L" (Irrlicht is Copyright (C) by Nikolaus Gebhardt)";
	}

	strxml+=L"\n";

	strxml+=L"\n\t Maximal Primitive Count: ";
	strxml+=primCount;
	strxml+=L"\n\t Maximum Texture Size supported: ";
	u32 x=maxTSize.Width;
	u32 y=maxTSize.Height;
	strxml+=x;
	strxml+="x";
	strxml+=y;

	strxml+=L"\n\t VideoCard Vendor Info: ";
	strxml+=vendorInfo;
	strxml+=L"\n";
	strxml+=L"\n\t Other Infos:";
	strxml+=L"\n";
	strxml+=a;
	strxml+=L"\n";


	xml->writeComment(strxml.c_str());
	xml->writeLineBreak ();
	xml->writeLineBreak ();

	xml->writeClosingTag(L"Test001");
	xml->drop();

	device->drop();



	return 0;
}

I need to know some things for my world editor.
Please try different drivers (especially openGl).

every time you run the app you must select a driver and a xml file with the name of the file is generated. you can send all your xml files via pm to me. I will merge the result togheter. thanks you very much.

[Fury.]

that's the test is used for measure time needed by file reading.
updated the first post with timing results.

Code: Select all

#include <irrlicht.h>
#include <time.h>
#include <iostream>

using namespace irr;

using namespace core;
using namespace scene;
using namespace video;
using namespace io;
using namespace gui;

using namespace std;


int main(int argc, char** argv)
{
    IrrlichtDevice *device =
        createDevice(EDT_SOFTWARE, dimension2d<u32>(640, 480), 16,
            false, false, false, 0);


    device->setWindowCaption(L"Hello World! - Irrlicht Engine Demo");


    IVideoDriver* driver = device->getVideoDriver();
    ISceneManager* smgr = device->getSceneManager();
    IGUIEnvironment* guienv = device->getGUIEnvironment();
    IImage* image=0;

    device->getFileSystem()->addFileArchive ("test.zip",
        true, true, EFAT_UNKNOWN);

    unsigned tick,tack,elapsed;

    tick=clock();
    for(int i=0;i<300;i++)
    {
        image=driver->createImageFromFile("test/irrtest01.bmp");
        image->drop();
    }
    tack=clock();
    elapsed=tack-tick;
    cout<<elapsed<<endl;

    tick=clock();
    for(int i=0;i<300;i++)
    {
        image=driver->createImageFromFile("test/irrtest01.PNG");
        image->drop();
    }
    tack=clock();
    elapsed=tack-tick;
    cout<<elapsed<<endl;

    tick=clock();
    for(int i=0;i<300;i++)
    {
        image=driver->createImageFromFile("test/irrtest01.JPG");
        image->drop();
    }
    tack=clock();
    elapsed=tack-tick;
    cout<<elapsed<<endl;



    while(device->run())
    {

        driver->beginScene(true, true, SColor(0,200,200,200));

        smgr->drawAll();
        guienv->drawAll();

        driver->endScene();
    }


    device->drop();

    return 0;
}

the test was run using 7zip compressor. Only zip files where used for this test. I used different compression level, and in some tests i used AES-256 encryption. I runned the test on 3 different files (the same 512x512 terrain texture saved in 3 different formats).

A more complete test will try to load multiple files organized in folders instead of only 1 file.

bmp file size: 768 KB
JPG file size: 53,7 KB
PNG file size: 481 KB (that was a bad image for png compression)

All the numbers are time needed in milliseconds (isntead of look at absolute time it is interesting notice the ratio between different file formats).

Compression level can go from 0 (uncompressed) to 5 (ultra compression)

reading 300 files. no archive. average of 3 tests
1458 bmp
5146 png
3812 jpeg

reading 10 files. aes-256 encription. no compression. average of 3 tests
687 bmp
661 png
416 jpeg

reading 10 files. aes-256 encription.ultra compression. 1 test
625 bpm
671 png
422 jpeg

reading 300 files. uncompressed archive (compression 0). average of 3 tests
1406
5140
3745

reading 300 files. compressed archive (compression 2).average of 3 tests
5229
5146
3927

reading 300 files. compressed archive (compression 5)average of 3tests

4984
5786
3927

the fastest way for loading files seems to be a uncompressed zip archive.
and that was i expected.

but now. all the results are a sum of hardware overheads and software overheads. It would be nice to know the time needed for reading a file of size X. Since all the 3 files I used have different sizes i cannot quantify the time needed for loading and the time needed for processing. the test i used is very simple and don't treat the case of multiple files (different files readed). all the test I used were only the reloading of the same file (this is not very good since the hardware "cursor" will be in the same place without changin position)..

any suggestion on how to improve the test?

NOTE: no software optimizaton (compiler optimization) is used here.

Good tests to try with next time:
Uncompressed zip archive (or pk3 that is the same)
All compression levels with a zip AES encrypted file
All compression levels with a zip file.

Repeat tests with full compiler optimization switched on.

[hybrid]

You should also note that the memory consumption is very different. Uncompressed zip files will create memory read files, which can speed up file loading significantly. However, some loaders do this internally as well. This means that you can have up to three copies of the same file in memory. which might not be what you really want.
And tests without compiler optimizations make no sense at all.