Textures compression tool (BINARY RELEASED)

[REDDemon]

Nadro I was 100% precise in my example. If the first thread is "any driver" is implicitly also OpenGL, you are not forced to use 2 gl context if you want just decompressed data you can use that data also in software driver (software rendering thread) if decompression is done in the Gl thread.

Hm, this patch would force two GL contexts? Bad idea, it should be left up to the user. Some drivers are not happy with multiple contexts.

The patch is not forcing 2 gl contexts. If you want to do texture compression in a second thread like Nadros sad. You'll need 2 gl context. There is nothing you can do about that. My example was specific to show to Nadro that my patch has no issues with MT. If you just need streaming for HD then you don't need 2 gl contexts.


Your summary is not good:

both solutions:
-If you want decompression to happen in a 2nd thread YOU NEED a 2nd gl context. That's a issue specific to GL and probably to DX design.
-If you want to load data from hard disk in a second thread you can. And YOU DON'T NEED a 2nd gl context.
-PBO and streaming can be added or not. That's not a problem for now and is not a hard problem.

REDDemon's solution:
1) is faster. If users create the texture, there's even no need for creating a IImage object. So my solution is extremly faster when creating textures
2) If users want a IImage, then users want that IIMage DECOMPRESSED. otherwise he can't do nothing with that image. So i decompress the image immediatly ( this can seem slow but you should note that else data is useless, and there also the fact that data is already cached saving few cache misses)
3) no api changes


Nadro's solution:
1) You create a IImage with compressed data.. Wich is useless. because users have to decompress that data if they want to use that data.
2) Users have to decompress the image manually. When users do that then your IImage loading is fast exactly the same of my IImage loading. (a bit slower in real)
3) api changes (always bad if they can be avoided in a consistent way)


Why should users want to access a IImage wich has compressed data? If they want a IImage they necessary want that image uncompressed, else they just need to create a texture (my solution). If they want texture streaming the ITexture->lock is 100 times better and faster.

IN your solution all users are likely to do that

Code: Select all

 
IImage* image = createImageFromFile( file)
if(!image)
    return ErrorCode;
 
if (image ->isCompressed())
    image->decompress();
 
 

with my solution:

Code: Select all

 
IImage* image = createImageFromFile( file)
if(!image)
    return ErrorCode;
 

.. @current patch (waiting for opinion of hybrid about that):
http://irrlicht.sourceforge.net/forum/v ... =2&t=46041

[Nadro]

OK now I see Your concept of two devices, but what is sense of mix 2 different drivers? You can't mix OpenGL and D3D and there is no sense to mix Software and HW accelerated driver, so in Your example You have got 2 OpenGL drivers.

So let's go:

-If you want decompression to happen in a 2nd thread YOU NEED a 2nd gl context. That's a issue specific to GL and probably to DX design.

Only small percent of users want a decompression images, but if someone will want it for specific reason, nobody should do it while game is run, but in loading data phase etc, so short freeze from single glGetTexImage call per frame isn't a problem, so in my solution You don't need a second OGL context.

-If you want to load data from hard disk in a second thread you can. And YOU DON'T NEED a 2nd gl context.

In an Irrlicht we have two methods for loading images/textures, createImageFromFile and getTexture (which include first method) and in Your solution both of this methods require access to an OpenGL, in my solution createImageFromFile in OpenGL calls free, so I can safe run it in another thread

My solution is faster. If users create the texture, there's even no need for creating a IImage object. So my solution is extremly faster when creating textures

In which place? Eg. You load texture from a separate thread (thread B), which already have own GL context. In thread B You read data from a disc, upload it to a GPU and download uncompressed data from a GPU to a system memory. Next in a thread related to all OpenGL rendering things (mainly, thread A) You have to create and upload data to a GPU second time for thread A. You don't have a separate methods for read data from a file and texture creation process. In my solution in a thread B you just read a data from a file and put them in a IImage class (this is only container handled by Irrlicht for a compressed data) and in thread A You will only call addTexture method, so we have only one uploading texture to a GPU. If user doesn't need an Image after texture was created, he can drop an image. Summary: Your solution is slower in MT environments.

You create a IImage with compressed data.. Wich is useless. because users have to decompress that data if they want to use that data.

Yes, but more users doesn't need them so we can't enable automatic decompression data, because it will be waste of time.

In my solution IImage is only a pure data container and thats why I suggest to add automatic drop image flag in getTexture method, but in a manual texture creation process it looks like that:

Code: Select all

IImage* image = createImageFromFile(file)
 
ITexture* texture = addTexture(image) // when texture exist we can decode an image.
image->drop();
 
IImage* uncompressed_img = createImage(texture->lock(ECF_R8G8B8A8), size_etc);

Short summary:
For an efficient MT texture handling we need two separate methods from a data read from a hard drive (this method must be OpenGL calls free) and texture creation process.

[REDDemon]

In an Irrlicht we have two methods for loading images/textures, createImageFromFile and getTexture (which include first method) and in Your solution both of this methods require access to an OpenGL, in my solution createImageFromFile in OpenGL calls free, so I can safe run it in another thread

nope because if you want to use that IImage you need to decompress it passing data back to OpenGL . At this point is simpler creating a threaded IReadFile and send its data as parameter for "createImageFromFile". This eliminate the cost of hardware access.

OK now I see Your concept of two devices, but what is sense of mix 2 different drivers? You can't mix OpenGL and D3D and there is no sense to mix Software and HW accelerated driver, so in Your example You have got 2 OpenGL drivers.

There is no sense in that. Just an example showing that even mixing drivers is possible if required. That's the most generic possible example. (and that's not true mixing. those are at the end 2 process wich exanghe some data. nothing really extraordinary)

In which place? Eg. You load texture from a separate thread (thread B), which already have own GL context. In thread B You read data from a disc, upload it to a GPU and download uncompressed data from a GPU to a system memory. Next in a thread related to all OpenGL rendering things (mainly, thread A) You have to create and upload data to a GPU second time for thread A. You don't have a separate methods for read data from a file and texture creation process. In my solution in a thread B you just read a data from a file and put them in a IImage class (this is only container handled by Irrlicht for a compressed data) and in thread A You will only call addTexture method, so we have only one uploading texture to a GPU. If user doesn't need an Image after texture was created, he can drop an image. Summary: Your solution is slower in MT environments.

No is not slower. If someone want a texture opened in a thread He just need a threaded hardware access (a simple IReadFile). Then the data can be sent only from the Thread holding the context. The speed is the same in that case. If he want only the Image the speed is the same again.

Yes, but more users doesn't need them so we can't enable automatic decompression data, because it will be waste of time.

why ? what can users do with compressed data? decompress it later? see next point.

Only small percent of users want a decompression images, but if someone will want it for specific reason, nobody should do it while game is run, but in loading data phase etc, so short freeze from single glGetTexImage call per frame isn't a problem, so in my solution You don't need a second OGL context.

So what's the problem with my solution from that point of view? You guessed how to decompress Data in a 2nd thread. My answer was to have a 2nd GL context. You can't call glGetTexImage from an arbitrary thread, since textures are not shared in GL (well in my engine you can enable sharing of textures and other object between different gl context, but that will be really a messup for irrlicht). So if you want to call glGetTextImage from1 thread that must be the same thread who loaded the texture. where's the threading in that? That's obvious that you don't need a second GL context if you do everything from 1 thread ^^. And if you think that the threading is in "HardDisk access" well that' the same for my solution..

What users want is some data that can be used. I doubt they will load compressed image just for keeping it in Ram compressed

1) they need uncompressed data => already decompressed by creating the image. Done
2) they need a texture => "addTexture" . Done.
3) they want threading? => IReadFile.
4) decompression is fast. Very fast. only compression is slow.
5) i can reverse the statement: "only few users will really need compressed data".

For an efficient MT texture handling we need two separate methods from a data read from a hard drive (this method must be OpenGL calls free) and texture creation process.

Agree, but there's no need for API changes. IReadFile can handle that.

[Nadro]

Why You change Your approach in disscussion?... You didn't say about IReadFile before or something else, but You mention about CImageLoaderDDS/KTX (this class inherit from IImageLoader and create IImage object) etc integrated with getTexture and/or createImageFromFile. If we'll have two separate method for read data from a hard drive and create texture all is ok, but:
We can create class like a CImageLoaderDDS/KTX atop IReadFile class (and put these class available to an user, if somebody will want just load a data, what will require API changes...). Without it loaded data isn't parsed so isn't usefull for us. Your solution with IReadFile is better than previous, but require many changes in current unified interfaces inside an engine and also in API, so it's still worse solution for me.

why ? what can users do with compressed data? decompress it later? see next point.

Upload it to a GPU in compressed form eg. in main thread.

"only few users will really need compressed data".

I said that only few users will require decompress a data and next manipulate over it.

[hybrid]

IImage and all decendants are not knowing about IVideoDriver at all. And we won't introduce a relation to IVideoDriver into those classes. The texture loading in Irrlicht (I think I already stated this some posts ago) is always file->IImage->ITexture, and only ITexture deals with the GPU. This is the way to go and will stay untouched. All solutions which break this scheme won't make it into the engine.
And we can't add a compression/decompression directly into CImage, because we might have multiple IImage implementations and we won't add these dozens of compression formats. Currently OpenGL offers about 10 different compressed image formats. That needs to be separated into different methods which can be used from multiple image implementations.
We just need to check all places where IImages are used and how processing compressed formats can be handled there. Check the API functions used and how to fix return values etc. Make sure that it is safe to deal with all image formats in all places.

[Nadro]

@Hybrid
I 100% agree with You, current loading texture scheme should be untouched.

[REDDemon]

I said that only few users will require decompress a data and next manipulate over it.

so where's the problem then if that operation is slower(not slower than your solution anyway)? anyway those "few users" are the ones using "createImageFromFile".. I really don't understand you. I didn't changed my approach anyway. Hard disk access is related to IReadFile. So if you want really doing threading with that IReadFile is 100 times there better than with IImage. What I did in my terrain manager for threading is ugly. But that code was not writed for being nice. but just for working.

@Hybrid
I 100% agree with You, current loading texture scheme should be untouched.

Well if you agree so you agree with my solution.. but how can you agree 100%? You are the one who asked for GPU related methods in IImage ^^.


My IImage is not using GPU . "createImageFromFile" is in IVideoDriver, so it's ok if in that method GPU is used. After that the returned image is no longer GPU related :* to hardware specific stuff.

the only point I disagree (other points are already present in my patch)

The texture loading in Irrlicht (I think I already stated this some posts ago) is always file->IImage->ITexture

Well, that will be performance hit at loading time forcompressed texture. If you are fond to that model there will be some problems.I'm not speaking about API changes. that can be achieved without API changes (wich is my warhorse). The problem is internally then. because you are just copying raw data from file to IImage for compressed textures. At this point is just 1 redundant operation (the main advantage of IImage is data decoding, DDS and KTX can't be decoded on CPU). Since mipmap levels are packed with DDS and KTX files or you will use multiple IImages for keep that data. or you just end up using IImage as a IReadFile (no one forbids, but this step can be simply skipped). It's not a specific issue to irrlicht but a issue with how KTX and DDS are designed.

@Hybrid: I think you can see easily all advantages of my solution. No API changes is always good (well this is file->texture not file-image-texture. but as you can see in my patch users will not have problems caused by this):
http://irrlicht.sourceforge.net/forum/v ... =2&t=46041

Are you totally sure to force KTX and DDS loaders to create a IImage when and where creating a IImage can be skipped Hybrid? one of the advantages of compressed formats is loading speed.

anyway API changes can be avoided anyway even if you forc to use File->Image->Texture model.
HOW TO AVOID API CHANGES:
-compressed images are used only internally.
-createImageFromFile will return anyway a UNCOMPRESSED image.
-users will not headhacke about Images.

-Only cons is slower loading time.

[Nadro]

Well if you agree so you agree with my solution.. but how can you agree 100%? You are the one who asked for GPU related methods in IImage ^^.

No I don't agree with Your solution, I told many times why. I also don't want GPU related methods in an IImage (in my first concept I wanted prepare software methods for compress and decompress but in my second concept I removed these methods) so please don't lie... You just used getTexture method (which require HW accel -> OGL calls) in a createImageFromFile method (this method must stay fully software inside) with and this is big fault of Your concept.

If You still don't understand my concept this is short info:
- We add lock flag to a some IImage methods, when image is compressed (as Hybrid said require some API changes or checks)
- You can decompress(created uncompressed image)/compress(created compressed texture) loaded data manually by call ITexture::lock/IVideoDriver::addTexture

Thats all!

IReadFile is bad idea, because You must parse this file before upload it to a GPU and upload only important parts...

Well, that will be performance hit at loading time forcompressed texture.

No, there will be not perf hit, because in these case IImage is only a container... In this sheme You:
1. Read file from a hard drive (Standard IReadFile operation)
2. Parse a file (You put a pointer to a data prepared for an upload to a GPU into an IImage container)
3. Upload texture data to a GPU (draw IImage data, so pointer to a parsed data in previous step)
This is the fastest possible way, so don't talk about perf hit etc, from performance view nobody care if You will left a pointer to a compressed image data somewhere eg. in IImage class (Irrlicht design require it)...

I said everything what I wanted. For me topic (disscussion about how compressed texture support in Irrlicht should look) is closed, everybody know what should do. Compressed textures should be available in a v1.9.

[REDDemon]

Well, my patch is finished and tested. I need only to apply the few changes suggested by hybrid (just to don't waste that..). You can do what you want with the patch even nether looking at it. ^^ (no one asked me to write. that just only a contribution that you can freely reject).

so please don't lie

If you call me lier, (that was ironical if you can see that. I agreed 95% with hybrid, you agreed 100%. we were disagreeing for a 5%?). Anyway I not and I not lied. I don't mind what you think at this point. Well discussion closed. Thanks for being polite.


good easter

[fmx]

Thanks for the KTX texture tool, interesting insight into compressed textures

[Fury.]

I'd love the exporter to export 3D textures. I already have a KTX loader. Is there any chance to see that feature? thx for the loader

[REDDemon]

@fmx: THX

@fury: that's on my todolist from some time.