[RESOLVED] OpenGL and ITexture->lock() for movie player

[Shomaa]

Hello,

I've created my VideoLibrary for display a Video in a texture.
I've encountered few problem...

with the Driver BURNINGDVIDEO, my video is well displayed on my cube...
but with the OPENGL Driver, my cube is desperately blue when my library play the sound

I suppose that's the fault of ITexture->Lock() but I'm not sure of that.

Doesnt exist an another method for apply a buffer of pixel on my texture for the OpenGL Driver ?

Help !!

[hybrid]

No, there's only lock/unlock to manipulate texture pixels. Make sure that you have the correct color format, sizes, no mipmapping, and other settings correct.

[Shomaa]

Hi ,
I conduct some research and test after read your post...
Unfortunaly the bug is alway here.
so I'll post the source here ...
Am I forget anything ?

#define WIDTH 640
#define HEIGHT 480
#define TEXTUREWIDTH 512
#define VIDEOWIDTH 320
#define VIDEOHEIGHT 240
#define PITCH 640

using namespace irr;


video::ITexture* rt;
uint16_t* pixel;
pthread_mutex_t mutex;


static void *lock(void *data, void **p_pixels)
{
pthread_mutex_lock(&mutex);

*p_pixels = rt->lock();

return NULL;
}

static void unlock(void *data, void *id, void *const *p_pixels)
{

rt->unlock();
pthread_mutex_unlock(&mutex);
assert(id == NULL);
}

static void display(void *data, void *id)
{
(void) data;
assert(id == NULL);
}

int main()
{
libvlc_instance_t *libvlc;
libvlc_media_t *m;
libvlc_media_player_t *mp;
char const *vlc_argv[] =
{
/*"--no-audio", /* skip any audio track */
"--no-xlib", /* tell VLC to not use Xlib */
};
int vlc_argc = sizeof(vlc_argv) / sizeof(*vlc_argv);

libvlc = libvlc_new(vlc_argc, vlc_argv);
m = libvlc_media_new_path(libvlc, "DJ Max - OP.mp4");
mp = libvlc_media_player_new_from_media(m);
libvlc_media_release(m);

// creation of mutex

mutex = PTHREAD_MUTEX_INITIALIZER;

// ask user for driver
//video::E_DRIVER_TYPE driverType=driverChoiceConsole();
//if (driverType==video::EDT_COUNT)
// return 1;

// create device and exit if creation failed

IrrlichtDevice *device =
irr::createDevice(video::EDT_OPENGL, core::dimension2d<u32>(640, 480),
16, false, false);

if (device == 0)
return 1; // could not create selected driver.

video::IVideoDriver* driver = device->getVideoDriver();
scene::ISceneManager* smgr = device->getSceneManager();
gui::IGUIEnvironment* env = device->getGUIEnvironment();

rt = 0;
// add fps camera
scene::ICameraSceneNode* fpsCamera = smgr->addCameraSceneNodeFPS();
fpsCamera->setPosition(core::vector3df(-50,50,-150));

// disable mouse cursor
device->getCursorControl()->setVisible(false);

// create test cube
scene::ISceneNode* test = smgr->addCubeSceneNode(60);

test->setPosition(core::vector3df(-100,0,-100));
test->setMaterialFlag(video::EMF_LIGHTING, false); // disable dynamic lighting

if (driver->queryFeature(video::EVDF_RENDER_TO_TARGET))
{
rt = driver->addRenderTargetTexture(core::dimension2d<u32>(TEXTUREWIDTH,TEXTUREWIDTH),"RTT1");

test->setMaterialTexture(0, rt); // set material of cube to render target

// add fixed camera
}



device->setWindowCaption(L"Irrlicht Engine - Render to Texture and Specular Highlights example");

int lastFPS = -1;

pixel = (uint16_t*)rt->lock();
rt->unlock();

//libvlc_video_set_callbacks(mp, lock, unlock, display, &rt);
libvlc_video_set_callbacks(mp, lock, unlock, display, &rt);
libvlc_video_set_format(mp, "RV32", VIDEOWIDTH, VIDEOHEIGHT,PITCH);

libvlc_media_player_play(mp);

while(device->run())
if (device->isWindowActive())
{
driver->beginScene(true, true, 0);
pthread_mutex_lock(&mutex);

if (rt)
{
// draw scene into render target

// set render target texture
driver->setRenderTarget(rt, true, true, video::SColor(0,0,0,255));
driver->setViewPort(core::rect<s32>(0,0,20,20));
// make cube invisible and set fixed camera as active camera
test->setVisible(false);

// draw whole scene into render buffer
smgr->drawAll();
}
pthread_mutex_unlock(&mutex);
// set back old render target
// The buffer might have been distorted, so clear it
driver->setRenderTarget(0, true, true, 0);

// make the cube visible and set the user controlled camera as active one
test->setVisible(true);
smgr->setActiveCamera(fpsCamera);

// draw scene normally
smgr->drawAll();
env->drawAll();

driver->endScene();

// display frames per second in window title
int fps = driver->getFPS();
if (lastFPS != fps)
{
core::stringw str = L"Irrlicht Engine - Render to Texture and Specular Highlights example";
str += " FPS:";
str += fps;

device->setWindowCaption(str.c_str());
lastFPS = fps;
}
}

device->drop(); // drop device
return 0;
}

[hybrid]

It's not clear if you use the color formats and dimensions (esp. pitch) correctly. Pitch should be 320*4 btw, as opengl (as well as burningsvideo if you didn't change the settings) will return a 32bit texture. I can only advise to read the vlc docs very carefully and see if the structures that Irrlicht provides will fit those of vlc.

[Shomaa]

I applied the modification but the results are the same.
With BurningsVideo everything is OK, but with D3D and OpenGL nothing is displayed on the cube.
the VLC docs says that OpenGl is compatible with VLC.

I dont know what to do ....

[REDDemon]

You need to use irrlicht built-in color convertion functions. This can be a performance hit, but you will see if the problem depend on that.
As I see it seems you need to convert 32 bit RGB to 16 bit RGB. I don't see in your code where the data are setted up to the texure correctly.
If you want to see a video is supposed you feed data to graphics hardware in some way. (maybe I skipped it). If I remeber correctly the lock function for textures is read only..

maybe burnings video works because the buffer instead of staying on the GPU is in the RAM so can be accessed. But lock() functions is not intended to provide write access to GPU texture memory:


Edit:
TIP: remember to disable mipmapping when sendin a new texture to the GPU.

[hybrid]

Well, actually lock provides an indirect write-through to GPU memory. The RAM is either provided separately by Irrlicht (for OpenGL) or by the hw driver (for direct3d). So it's actually possible to use this way. However, vlc assumes certain color formats and dimensions. Just saying "OpenGL compatible" is not enough. Because it could be any (probably integral) color format, with or without alpha channel. The pitch might be strict (i.e. width*byteperpixel) or some other value. The color element order might be different, etc. You must configure vlc properly, or query for the correct color formats upon RTT creation. Also, check the actual texture dimensions. On many systems, the texture will be larger than what you requested.

[Xaryl]

short version:
I don't think passing ITexture into libvlc's function is a good idea.

long version:
when addRenderTargetTexture(...) is called with OpenGL driver,
on default setting, a COpenGLFBOTexture object is created and returned.

COpenGLFBOTexture is derived from COpenGLTexture,
and both of these classes have several virtual functions and variables,
and some of these variables are Irrlicht specific, too.

and then in this line

Code: Select all

libvlc_video_set_callbacks(mp, lock, unlock, display, &rt);

I don't think libvlc knows anything about Irrlicht, so my guess is it is being used as some kind of OpenGL buffer.

Is there some kind of obscure dark magic going on here to make that work?

[hybrid]

No, it's not dark magic, it's just the callback functions which are set up in that call. These are placed in the code shown above quite at the top. They lock the texture and return the void* into vlc. vlc writes the data to that area, and calls unlock afterwards, which copies the memory back to the GPU. This works in theory (well, also for some known configurations already), but requires a careful setup to meet the parameters I mentioned before.

[Xaryl]

Ah right, sorry I totally misread that line.
Anyways, the main problem is that opengl does not like its textures being manipulated by multiple threads. A work around would be to use IImage as the intermediate buffer.
Here's a working sample.

Code: Select all

#include "irrlicht.h"
#include "vlc/vlc.h"
#include "pthread.h"
#include "assert.h"

#define WIDTH 640
#define HEIGHT 480
#define TEXTUREWIDTH 512
#define TEXTUREHEIGHT 512

using namespace irr;
using namespace core;
using namespace scene;
using namespace video;

video::IImage* rt;
pthread_mutex_t mutex;
bool update_flag;

static void *lock(void *data, void **p_pixels){
    pthread_mutex_lock(&mutex);
    update_flag=true;
    *p_pixels = rt->lock();
    return NULL;
}

static void unlock(void *data, void *id, void *const *pixels){
    rt->unlock();
    pthread_mutex_unlock(&mutex);
    assert(id == NULL);
}

static void display(void *data, void *id){
    (void) data;
    assert(id == NULL);
}

int main(){
    update_flag=true;
    rt = NULL;
    mutex=PTHREAD_MUTEX_INITIALIZER;
    libvlc_instance_t *libvlc;
    libvlc_media_t *m;
    libvlc_media_player_t *mp;
    char const *vlc_argv[] ={
        //"--no-audio", /* skip any audio track */
        "--no-xlib",
    };
    int vlc_argc = sizeof(vlc_argv) / sizeof(*vlc_argv);
    libvlc = libvlc_new(vlc_argc, vlc_argv);
    m  = libvlc_media_new_path(libvlc, "test.mp4");
    mp = libvlc_media_player_new_from_media(m);
    libvlc_media_release(m);
    IrrlichtDevice *device = irr::createDevice(video::EDT_OPENGL, core::dimension2d<u32>(WIDTH, HEIGHT),32, false, false);
    if (device == NULL)
        return 1;
    video::IVideoDriver* driver = device->getVideoDriver();
    scene::ISceneManager* smgr = device->getSceneManager();
    gui::IGUIEnvironment* env = device->getGUIEnvironment();

    scene::ICameraSceneNode* fpsCamera = smgr->addCameraSceneNodeFPS();
    //scene::ICameraSceneNode* fpsCamera=smgr->addCameraSceneNode();
    fpsCamera->setPosition(core::vector3df(-50,50,-150));
    fpsCamera->setTarget(core::vector3df(-147,-54,-66));
    device->getCursorControl()->setVisible(false);

    scene::ISceneNode* test = smgr->addCubeSceneNode(60);
    test->setPosition(core::vector3df(-100,0,-100));
    test->setMaterialFlag(video::EMF_LIGHTING, false);

    assert(driver->queryFeature(video::EVDF_RENDER_TO_TARGET));
    rt = driver->createImage(ECF_A8R8G8B8,dimension2du(TEXTUREWIDTH,TEXTUREHEIGHT));

    libvlc_video_set_callbacks(mp, lock, unlock, display, NULL);
    libvlc_video_set_format(mp, "RV32", TEXTUREWIDTH,TEXTUREHEIGHT,rt->getPitch());
    libvlc_media_player_play(mp);

    ITexture* tmpTexture=NULL;
    while(device->run())
        if (device->isWindowActive()){
            pthread_mutex_lock(&mutex);

            if(update_flag){
                driver->removeTexture(tmpTexture);
                ITexture* tmpTexture = driver->addTexture("tmp",rt);
                tmpTexture->lock();
                tmpTexture->unlock();
                test->setMaterialTexture(0,tmpTexture);
                update_flag=false;
            }

            driver->beginScene(true, true, 0);
            smgr->drawAll();
            env->drawAll();
            driver->endScene();
            pthread_mutex_unlock(&mutex);
        }
    device->drop();
    return 0;
}

Cool stuff, I think everyone should try it.

[Shomaa]

well it's works !

it's a bit slow, so I'll reduce the texture to 256.
Now Irrlicht can play Video on Texture !
Congratulations !

[hybrid]

To speed up you should a) remove mipmaps and b) lock the texture instead of removing and adding. The latter is much slower than locking and updating the texture. Why do you lock/unlock the texture anyway, this copies the pixels completely unnecessarily.
But yeah, the threading problem is of course a major reason to break OpenGL compatibility. Due to all this threading setup being invisible (besides the mutex) I simply didn't see it.

[Xaryl]

Originally I added the lock/unlock to solve a problem with texture flickering, but I am unable to reproduce it, so it was probably just a human error.

I also tried the other optimisations suggested, but I won't spoil the fun here; if speed is really an issue, one can always use a profiler to figure out the limiting factors