Irrlicht-FFmpeg Integration

[mahmoudgalal]

Hi,
I created a Video-playback Library for the irrlicht engine that uses the FFmpeg/Libav as a decoding runtime .The Library is open source and has a very simple API.
You can find the Repo here:
https://github.com/mahmoudgalal/IrrlichtVideoPlayerLib

enjoy : ,

[CuteAlien]

Nice. Little question - why remove/create the texture every frame? Just keep using the same one, that will be way faster.

[Ovan]

more advanced one with ffmpeg and sound support (2008)

http://irrlicht-fr.org/viewtopic.php?id=778
https://github.com/Phirxian/irrlicht-addons (same)

[fysez]

So I'm not sure if I should've started a new thread for this or not, as it's semi-related and came up in the search results.
But pretty much I reached out to the ffmpeg mailing list and to their subreddit, but neither has gotten a response in a few days so thought I'd try here.

I started an Irrlicht project so that I could use ffmpeg to record the screen, and then use Irrlicht to display the screen in real-time on a texture.
That's all working, but now I'm trying to save it to a file (like maybe in .mp4 format). Figured someone here might be a bit better with dealing with pixel formats and such than me.

I got the mp4 file to open and start writing. Here's the code that writes the actual screen pixels to that file:

Code: Select all

void OutputVideoPlayer::pushFrame(AVFrame* pFrame) {
    int err;
    if (!videoFrame) {
        videoFrame = av_frame_alloc();
        videoFrame->format = AV_PIX_FMT_YUV420P;
        videoFrame->width = cctx->width;
        videoFrame->height = cctx->height;
        if ((err = av_frame_get_buffer(videoFrame, 32)) < 0) {
            std::cout << "Failed to allocate picture" << err << std::endl;
            return;
        }
    }
    if (!swsCtx) {
        swsCtx = sws_getContext(cctx->width, cctx->height, AV_PIX_FMT_RGB24, cctx->width,
            cctx->height, AV_PIX_FMT_YUV420P, SWS_BICUBIC, 0, 0, 0);
    }
    // From RGB to YUV
    sws_scale(swsCtx, (uint8_t const* const*)pFrame->data, pFrame->linesize, 0, cctx->height,
        videoFrame->data, videoFrame->linesize);
    //videoFrame->data[0] = pFrame->data[0];

    videoFrame->pts = (1.0 / 30.0) * 90000 * (frameCounter++);
    std::cout << videoFrame->pts << " " << cctx->time_base.num << " " <<
        cctx->time_base.den << " " << frameCounter << std::endl;
    if ((err = avcodec_send_frame(cctx, videoFrame)) < 0) {
        std::cout << "Failed to send frame" << err << std::endl;
        return;
    }
    AV_TIME_BASE;
    AVPacket pkt;
    av_init_packet(&pkt);
    pkt.data = NULL;
    pkt.size = 0;
    pkt.flags |= AV_PKT_FLAG_KEY;
    if (avcodec_receive_packet(cctx, &pkt) == 0) {
        uint8_t* size = ((uint8_t*)pkt.data);
        av_interleaved_write_frame(ofctx, &pkt);
        av_packet_unref(&pkt);

        if (--TEST <= 0)
        {
            finish();
            free();
        }
    }
}

And here's how I'm updating an Irrlicht texture using the above function:

Code: Select all

/// Copies data from the decoded Frame to the irrlicht image
void writeVideoTexture(AVFrame *pFrame, IImage* imageRt) {
	
	if (pFrame->data[0] == NULL) {
		return;
	}

	int width = imageRt->getDimension().Width;
	int height = imageRt->getDimension().Height;

	//from big Indian to little indian if needed
	int i, temp;
	uint8_t* frameData = pFrame->data[0];

	unsigned char* data = (unsigned char*)(imageRt->lock());
	//Decoded format is R8G8B8
	memcpy(data, frameData, width * height * 3);
	imageRt->unlock();
}

int VideoPlayer::decodeFrameInternal() {
	int ret = 0;
	if (ret = av_read_frame(pFormatCtx, &packet) >= 0) {
		// Is this a packet from the video stream?
		if (packet.stream_index == videoStream) {
			// Decode video frame
			avcodec_send_packet(pCodecCtx, &packet);
			while (ret >= 0) {
				ret = avcodec_receive_frame(pCodecCtx, pFrame);
				if (ret == AVERROR(EAGAIN))
				{
					ret = 0;
					avcodec_send_packet(pCodecCtx, &packet);
					break;
				}
				
				//pFormatCtx->duration
				// Convert the image from its native format to RGB
				sws_scale(sws_ctx, (uint8_t const* const*)pFrame->data,
					pFrame->linesize, 0, pCodecCtx->height,
					pFrameRGB->data, pFrameRGB->linesize);

				writeVideoTexture(pFrameRGB, imageRt);

				ovp.pushFrame(pFrame); // OR writeFrame(pFrameRGB), not really sure there's a difference other than conversion in pushFrame()
			}
			ret = 0;
		}
		// Free the packet that was allocated by av_read_frame
		av_packet_unref(&packet);
	}
	return ret;
}

The best I can get it to display looks really bad, like this:

Any ideas on why this might be?

The full, compilable code and instructions to build it are here:
https://github.com/treytencarey/IrrlichtVideoPlayerLib/

[CuteAlien]

When you say above "That's all working" - i'm not sure what you are refering to. You mean record and play works unless files are involved? But writing to file and then showing the resulting mp4 fails? Does it show up in other video tools after writing? Or is the above shot what you see in another tool after writing to file? So your problem is with writing the correct format to file?

(and totally unrelated to the problem: it's not little and big indians but endians - aka, from which end you start reading the numbers).

edit: OK, and looking at the code around the little indian comment - I think you might try to copy 24 bit into a 32 bit texture. Basically all textures in Irrlicht wil be 32-bit until requested explictedly to be 24bit - and even then it might be impossible and you get 32 bit as many graphic cards only support that. In which case you can't use memcpy, but you have to copy stuff bytewise row-by-row. Or as a quick workaround - you can create a 24 bit CImage in Irrlicht and copy to that and let Irrlicht show that in GUI (not for final version - just to check if that's the problem).

[fysez]

Sorry I wasn't too clear with what was working vs. not working.
Basically I could get my screen record to display on Irrlicht (both in 2D and on a 3D texture). But I could not get it to output without what I thought was corruption.

But your comment about 24-bit vs. 32-bit is what helped me fix the issue, so thank you.
Basically I was recording in 24-bit, which Irrlicht handled fine. But scaling it up in a video player looked really bad. Because it was 24-bit.
So I simply changed my formats to 32-bit, using A8R8G8B8 for Irrlicht instead.
Now everything is working amazing so thanks!

GitHub repo is updated with the new working code if it's ever useful for anyone. I'll keep the repo up.