Irrlicht i18n (Unicode, FreeType, etc.)

[Nalin]

Edit: Reading from of this thread, I think I will have to patch the XML loader, as the data came from there.

Correct. Because of how Irrlicht's XML loader was programmed, there is no way to extend or replace it without modifying the Irrlicht source code. You could write your own loader, but then you wouldn't be able to use Irrlicht's serialization functions. I suppose I should re-write the XML loading code one of these days and submit it as a patch, but, for now, you either have to patch Irrlicht or forgo unicode XML files.

[christianclavet]

I tried my "trick" that I've done with TinyXML to correct the strings and it seem to work also with IRRXml. My primary interest is to at least have accented characters working (LATIN1 would be fine). So I will have something while Irrlicht support that fully. (By the way, great work on this! I just hope the DEVs will add theses into Irrlicht one day..)

As for my "trick", I will have to check how the string come out on Linux as the Irrlicht string will surely be different on that platform as on Windows. My sources is UTF8 xml files. Thanks Nalin.

If you're curious here what I've done:

Code: Select all

core::stringw winconvert(core::stringw str)
// Convert accents from loaded XML files (irrXML)
// WARNING: Tested only on windows
// might not work on Linux or other platform.
{
        bool debug = false;
        core::stringw textline = L"";
        core::stringw text = L"";
        u32 base = 0;
        
        char test2 = ' ';
 
        for (u32 a=0; a<str.size(); a++)
        {
                // Get the character first
                text = str.subString(a,1);
                
                // Then check this character directly (convert to unsigned 32bit)
                base=(u32)text[0];
 
                if (base<256) // Standard characters
                {
                        textline+=text;
                }
                
                // All characters after 256 are ignored except thoses
                // Character higher are re-aligned from the offset to match LATIN1
 
                // Reference to the table is here:
                // http://www.utf8-chartable.de/unicode-utf8-table.pl
                
                const u32 offset=65216;
 
                core::stringw replace = L" ";
                if ((base>255) && ((base-offset)<255))
                {
                                replace[0]=(base-offset);
                                textline+=replace;
                }
                
        }
        
        return textline;
}

Then when I want to "convert" the irrXML string to a valid Irrlicht stringw:
(Here getting the stringw into "description" and sending it to a listbox for verification):

Code: Select all

description = xml->getAttributeValue("description");                                                    
list->addItem(winconvert(description).c_str());

Please note that not all of the accented characters are "parsed" as this was only used to test. I think there could be an even better way to get this, but I have to know how Irrlicht string table is build (is it matching the Ascii Latin1 table?). If it's matching I could simply put the "resulting number from the offset" and convert it to ascii back into the string. Right now I'm checking PER character and I think it work fine but not hyper efficient.

EDIT:Cleaned the code having found a better way to get the proper LATIN1 characters. The character is then replaced in the string.

[Nalin]

If you aren't relying on Irrlicht's built-in serialization to create entitites, and you are just using the XML loader raw, just pass the data you receive to the ustring class. Irrlicht's XML loader won't mangle the bytes at all, so you can just directly pass the data into the ustring class and it will be able to parse the data. If you call the char* constructor, it will attempt to load the data as UTF-8. You can also use the loadDataStream.

If you are using UTF-8 files, they should be the same on both Windows and Linux. The difference comes with the wchar_t data type. It is 16-bit on Windows and 32-bit on Linux. Irrlicht writes XML files using wchar_t by default, so you have to watch out for that. ustring will load wchar_t data using the platform specified type. EG, on Windows, it tries to read it as UTF-16, while on Linux it tries to read it as UTF-32.

[Nalin]

Just some small updates:

• xml_io.patch: Updated to the latest trunk revision.
• irrUString.h: Changed u32 to size_t.
• CXMLWriterUTF8.zip: Merged CXMLWriter.cpp performance improvements from trunk.

[feelthat]

can it done on ios and android?

[Nadro]

I can confirm that this add-on works on iOS (I use it in my project for true type fonts).

[feelthat]

if could, would u send me some build step and example?

email: fatalfeel@yahoo.com.tw

I can confirm that this add-on works on iOS (I use it in my project for true type fonts).

[vroad]

http://irrlicht.svn.sourceforge.net/vie ... es/ogl-es/

There is OpenGL-ES version of Irrlicht.
If you don't have svn client, you can download as tarball from sourceforge's browser.
It also contains project for Android and iOS.
This addon should works well on those platforms as it works on desktop.

[feelthat]

I use Proton 3D,

After download http://irrlicht.svn.sourceforge.net/vie ... es/ogl-es/

I did not see any utf8 loader?

Did u mean irrlicht/branches/ogl-es/ include i18n ????

https://irrlicht.svn.sourceforge.net/sv ... hes/ogl-es

There is OpenGL-ES version of Irrlicht.
If you don't have svn client, you can download as tarball from sourceforge's browser.
It also contains project for Android and iOS.
This addon should works well on those platforms as it works on desktop.

[feelthat]

For UTF8 on mobile compare

1. IrrFreeType

2. IrrlichtML 1.7.1

3. Irrlicht i18n

which one is good on mobile?

[Nadro]

I use ogl-es branch + add-ons from this topic (Irrlicht i18n), but as I said only TT was important for me, I didn't test UTF8 features from this project but I think that it should works properly.

[feelthat]

in first page of this project

/*I have a keyboard that has non-ASCII characters. When I type into a text box in Irrlicht, it doesn't show the proper character! Why? (ALSO: Why can't a paste unicode text into Irrlicht?)
These patches don't solve this issue. If you want to be able to input unicode characters into Irrlicht, please take a look at MadHyde's patches*/

so Can input i18n unicode words in edit box
like Japanese Chinese

I use ogl-es branch + add-ons from this topic (Irrlicht i18n), but as I said only TT was important for me, I didn't test UTF8 features from this project but I think that it should works properly.

[vroad]

Unfortunately, typing with unicode characters is only supported on Windows/Linux.
This is OS-Specific part. Other futures should work on every platforms.

[feelthat]

Dear All:

Recently I use Freetype2 to impelement UTF8 words in Irrlicht 1.73

Do you have any other idea, Maybe use ios or android system utf8 on
Irrlicht 1.73?

thanks first by jesse stone

[CuteAlien]

I've started working with this ttf implementation. It really is a lot faster than the other implemenation which I used so far (because this one does put characters on pages while the other used one character per texture).

I modified it somewhat to adapt it to my needs (and getting it a little closer to Irrlicht).
Changes I made are:
- No dependency on ustring (I don't think string-conversions should happen inside the font-class itself)
- Reworked that allocation stuff. That had some memory leaks. SGUITTGlyph uses now the rule of three (destructor, operator=, copy constructor are all needed when one is needed) and so I could get rid of the custom allocator. Also set_free_when_destroyed isn't about pointers of elements but about array-memory itself (yeah confusing stuff).
- remove textscenenodes. I get that they are useful, I just don't think they belong in that class.
- Font should probably not grab()/drop() the driver (we have no weak pointers unfortunately). I rewrote that stuff some more to pass the exact types needed.
- Avoid a few memory allocations in draw (not a good place to use a map as allocations are more expensive than linearly going over a small array).
- Minor stuff tidying up I changed on the way while hunting some independent bug (font-textures sometimes had been complete white, but that turned out to be an opengl-texture bug in svn trunk which will be fixed soon).

My version is here:
https://code.google.com/p/irr-playgroun ... TTFont.cpp
https://code.google.com/p/irr-playgroun ... UITTFont.h
https://code.google.com/p/irr-playgroun ... se/ttf.cpp (an example how to use it)

Will still need some work probably until it can be added to Irrlicht directly. Especially have to think about handling the freetype face management (might move that to an own class so we can get the font constructor back into public).

But all in all a very nice implementation, thanks for it!