Irrlicht i18n (Unicode, FreeType, etc.)

[robmar]

I implemented the simpler CGUITTFonts into Irrlicht and its working perfectly, really fast. The new more complex version can take a long time loading the cache of characters around the last character loaded though.

[chronologicaldot]

The hash class on line 3874 uses "ustring" but the rest of the code uses "ustring16". Names got changed at some point and this was forgotten.

[chronologicaldot]

I tested CGUITTFonts and it does not render the non-English portion of fonts for me. Anyone else have this problem?
The loading is fine...

Code: Select all

	irr::gui::IGUISpriteBank*  spriteBank = loadGUISprites(guiEnvironment, "SlickGUISkinSprites");
	irr::gui::IGUISkin* skin = new irr::gui::SlickGUISkin(guiEnvironment, true, true);
	skin->setSpriteBank(spriteBank);
	irr::gui::IGUIFont* defaultFont = guiEnvironment->getSkin()->getFont();
	defaultFont->grab();
	guiEnvironment->setSkin(skin);
	irr::gui::CGUITTFont* font = irr::gui::CGUITTFont::createTTFont(
			guiEnvironment,
			"rsc/font/TakaoPMincho.ttf", 13
			);

	if ( font ) {
		skin->setFont( font );
		font->drop();
	} else {
		skin->setFont(defaultFont);
	}
	defaultFont->drop();
	skin->drop();

I thought the problem was likely where I pass in a string. I have strings stored in UTF8. I convert them via multibyteToWString and then save them in the GUI elements where they should draw. English text appears fine, but other languages or even English + other languages does not appear at all. Edit: I see I'm not the only one with this problem. ( viewtopic.php?p=304380#p304380 )

[CuteAlien]

You can try if my version on https://github.com/mzeilfelder/irr-playground-micha works for you, it's got some fixes. Thought I might not have checked foreign language strings yet - not sure. Please give me a link to the font and a simple example string which does not work. Could be that the font simply doesn't have those characters.

[chronologicaldot]

The font has the characters. I checked using LibreOffice Writer. The weird thing is that even if the string contains English + foreign characters, the English letters disappear too! Try something like "Hello world こんにちは"
The font is TakaoPMincho.ttf, which comes standard with Ubuntu. You can find it in:
/usr/share/fonts/truetype/takao-mincho

Thanks for offering to help. I've decided just to switch to gtkmm since it handles Anthy input already anyways. Irrlicht can't handle dbus input *sigh*. Several times I tried looking up dbus documentation and found nothing but a mess. I can't figure out how it ever became popular. But I digress.

[chronologicaldot]

Update: I stumbled upon the dbus documentation for anyone interested. I wish it would show up in Google search results.
[link]https://www.freedesktop.org/wiki/Software/dbus/[/link]

[CuteAlien]

Sorry, works for me. I used ttf.cpp in https://github.com/mzeilfelder/irr-playground-micha
To compile copy ttf.cpp to main.cpp and call: make FREETYPE=1
Replaced the "the quick brown fox" string with core::stringw str(L"Hello world こんにちは");
And the Andika-R.ttf with irr::io::path fontName("/usr/share/fonts/truetype/takao-mincho/TakaoPMincho.ttf");

Note that copy-paste does not work - only using string directly. But that's another bug (which I maybe just got a bugfix for yesterday in the forum here, but might take a while until I get to that one).

[chronologicaldot]

Well, you're right. I compiled your example and got it to work, but where I'm using it still doesn't. I store strings in utf8, which then gets converted to utf16 for the GUI, so my first thought was that my conversions were wrong. But my conversion function is simple:

Code: Select all

irr::core::stringw
CuStrToIrrStrW( util::String in ) {
	irr::core::stringw  wcs;
	multibyteToWString(wcs, in.c_str(), in.size());
	return wcs;
}

util::String is basically a char list, so maybe the char data type isn't storing the high bit correctly? Or maybe I don't have the correct understanding of mbstowcs in multibyteToWString().
Upon testing my program with your CGUITTFont, now the English shows up even when combined with non English (which it didn't before), so some processing got fixed.
hm... Does mbstowcs convert only to utf-16 or will it convert to utf-32 on Linux? Edit: Dang, it depends on locale.

[CuteAlien]

Yeah, bit of a mess. Freetype expects utf-32 I think. And Irrlicht just uses wchar_t and doesn't define any encoding, but passes characters just throught. So on Windows already a problem here as wchar_t is only 16-bit (thought not yet a problem for these characters I think). On Linux it might work in theory. Note that freetype itself might again chose a characters set (a font might have more than one), thought I'm not sure right now on what that selection was based.
I suppose the clean solution would be to tell the font which encoding to expect and let it do the conversion to utf-32 based on that? Otherwise we probably have to change it everywhere the font is used. Or alternatively some global setting in Irrlicht which fonts can check?

edit: And yes, seems mbstowcs depends on LC_CTYPE

[chronologicaldot]

I tried utf8ToWchar() in irrString but alas to no avail. And yet that's doing the conversion without mbstowcs() AND taking into account both Windows and Linux. But at least all the English shows up correctly. I don't mind doing the conversion (instead of making the font do it), but I can't get a reliable working conversion function.
Edit: I'm thinking the problem may actually originate in how I load my strings from file, now that I've dug alittle deeper. Let me do some more digging.
Update: Hm, my loading strings appears to be fine, so I'm stuck looking for a middleman problem again.

Update: I ran a test that printed the wchar_t string L"こんにちは" created within C++ and used the irrString wcharToUtf8 conversion function to convert it to utf8 and ended up with just こ. lol. Obviously wrong. I also tested the same string I loaded from file in utf8 and converted using utf8ToWchar() and obviously ended up with a different value than the string created within C++. I'm curious then how C++ interprets the string values stored directly within the C++ file.

[chronologicaldot]

I got my tests to work, so now my conversion via wcharToUtf8 gives こんにちは. I had a buffer size too small. But I now that the strings are identical, I'm stumped as to where my problem is, but I'm more convinced now that it's in my code which is too complicated for its own good.

Update: Now it's working. I guess my buffers were too small so the asian text kept getting chopped off.

[CuteAlien]

Nice. Guess I don't have to write another test-case then (busy with taxreports anyway *sigh*).