get full pixel array from IImage instead of one at a time

[AlexAzazel]

I'm planning to make a simulation in Irrlicht and I expect to have a high performance. What I need is taking screenshots for doing computation upon it at every frame. What I found out that could be done is demonstrated in the following pseudocode

Code: Select all

 
video::IImage* image = driver->createScreenShot();
 
// next three lines are looped with i and j
captured_image[i][j][0] image->getPixel(i, j).getRed();
captured_image[i][j][1] image->getPixel(i, j).getGreen();
captured_image[i][j][2] image->getPixel(i, j).getBlue();
 
image->drop();
 
// do some stuff on captured_image
 
 

To be more specific I will be doing numeric 2 dimensional convolution operation on it, i.e. applying filters with custom kernels (https://en.wikipedia.org/wiki/Kernel_(image_processing)).

I have few ideas on how this might be done in an optimal way. The obvious way is to find out how to get the full pixel data from IImage directly instead of getting each color of each pixel individually, I'm not good at c++ but I sense calling getPixel() for each pixel is suboptimal. Any ideas how to do that? The other thought I have is, maybe Irrlicht has it's own function of applying filters (it should not be just standard filters like blur, I should be able to give it any kernel), but then I will still need to return the resulting image that will be indexable the same way captured_image is.

For anyone who's wondering what I'm doing, I want to feed the captured image into convolutional neural network of virtual creature A.I. and train/evolve it.

[CuteAlien]

You can access IImage data by IImage::lock() (or IImage::getData() if you use Irrlicht trunk which replaced the lock()).
Thought it won't be fast as screenshot creates an image each time and then copies over the texture data.

Slightly faster - render your result to a texture. And then use ITexture::lock(). Just be careful as texture-data can have padding on the right-side (so you have to work with texture pitch and not texture width when calculating the pixel position). Also you need to know your bit-depth and your color-layout (rgba in memory might be argb etc...).
Also if you work with Irrlicht trunk be careful that default behavior of textures has been changed and it can affect speed a lot. Previously textures always had a copy of graphic-card texture-memory in the computer memory as well. Which made lock() very fast (but increased memory-usage). Now we no longer have a memory-copy unless you request it explicitely when creating textures with:
driver->setTextureCreationFlag(video::ETCF_ALLOW_MEMORY_COPY, true)
Don't know if you need it in your situation, but it might make a huge speed-difference in some cases (thought likely not for render-target textures, I doubt those have a copy in memory as it wouldn't make sense... but I'm not really certain right now how those are handled). For all other textures set it for now (also... I think there still is a a bug when you do not set that flag - I run into that 2-3 weeks ago, but still trying to reproduce it more clearly... just having something with alpha's going wrong unless I set the above line).

And last solution which is a lot faster - work with shaders in all intermediate steps. They are more or less perfect for working with kernels. And that way you remove the slow part of moving texture to/from the graphic-card but can do all calculations on the gpu. And they even run all in parallel.

[AlexAzazel]

You can access IImage data by IImage::lock()

Code: Select all

void* source = image->lock();

And how do I use source now to access the data?

And last solution which is a lot faster - work with shaders in all intermediate steps. They are more or less perfect for working with kernels. And that way you remove the slow part of moving texture to/from the graphic-card but can do all calculations on the gpu. And they even run all in parallel.

That's seems like a better option though will reading this tutorial http://irrlicht.sourceforge.net/docu/example010.html be enough for me to figure out how to do that?

[hendu]

Cast it to u8*. The format depends on what your texture format is, maybe it's rgba, maybe something else.

[CuteAlien]

Tutorial get's you started with shaders. When you have questions the forum always helps :-)

[AlexAzazel]

Cast it to u8*. The format depends on what your texture format is, maybe it's rgba, maybe something else.

Did it, thanks!

Code: Select all

 
u8* source = (u8) image->lock();
 

Tutorial get's you started with shaders. When you have questions the forum always helps

Thank you very much. I'm thinking of going with the first approach for now since I have integrated Intel GPU which uses the same RAM as CPU does (probably?). But I don't exactly know how to do indexing in the most optimal way, pointer returns a single row array which has a following structure BGRABGRABGRA.... as I figured out by trial and error. If I write a function that converts x y indices into the single row 'language' indices I doubt it would be any fast so maybe I should look first how irr::video::IImage::getPixel works. I tried to search its implementation in the source code but I have no idea how to search for anything in it. Can you explain in brief how do I look up functions in the source code?

[CuteAlien]

Usually the file is named the same just with C at the start. So CImage.cpp in this case.
If you have the project file of Irrlicht open in VisualStudio you can also do a global search on the project (ctrl+shift+f usually, but they made it somewhat flexible so you can change those settings (which you usually do accidently and it's really annoying)). In modern VS version you can also click the function with right-mouse and then select - go to definition (will give you several for option if there's more than one implementation).
Other IDE's also have such functions to search globally.
Or on Linux I tend to use the tool ack-grep on the console (bash) and have the following alias in my .bashrc:
alias ack-cpp='ack-grep --type=cpp --type=cc'
That way I type "ack-cpp getPixel" and will search for that in all header and source files for c/c++ in my current folder + all sub-folders.

[AlexAzazel]

Earlier you mentioned

Slightly faster - render your result to a texture. And then use ITexture::lock().

What exactly do you mean?

[CuteAlien]

Create a rt texture for example with: irr::video::ITexture * rt = videoDriver->addRenderTargetTexture(texdim, "rtTxt", irr::video::ECF_A8R8G8B8);
Then you use IVideoDriver::setRenderTarget with that texture and Irrlicht will render to that texture instead of the screen (if you need screen additionally then set the rendertarget back to 0 and call your rendering like smgr->drawAll() once more - no need for new begin/endscene in between).
I never actually compared the speed, but I suspect that it might be faster than doing screenshots as you avoid creating a IImage each time and also avoid copying texture to IImage.

Also in case you need to draw that texture itself again back to the screen in full size you might want to search the forum (or code-snippets sub-forum) for "screenquad" (or look at XEffects which has a CScreenQuad class I think).

[AlexAzazel]

Create a rt texture for example with: irr::video::ITexture * rt = videoDriver->addRenderTargetTexture(texdim, "rtTxt", irr::video::ECF_A8R8G8B8);

As I checked, that already makes it slower than using IImage even when I create rt outside the game loop.

By the way, is it possible to disable alpha channel in the entire engine all together? So that I get only three channels in IImage and maybe a better performance for the whole engine?

[Foaly]

So that I get only three channels in IImage and maybe a better performance for the whole engine?

At least graphics cards are usually faster with four channels.