CXMLWriter bug, wchar_t & encoding issues on !wi

[Gianluca]

It's a long topic, so I tried to organize it in part enclosed by "Problem section" and "Solution section" that, respectly, describes the bug and the solution proposed by me.
At the end there are some conclusion.

--- Problem:
The following seguence of calls produces an erroneuos xml content if the tag is indented (i.e. is not the root node):

Code: Select all

    
    XMLWriter->writeElement( TAG_NAME );
    XMLWriter->writeText( TEXT_CONTENT );
    XMLWriter->writeClosingTag( TAG_NAME );

In details, the writeClosingTag always add "\t" characters when Tabs>0 condition is satisfied, but it's an error because the writeClosingTag should add "\t" chars only when a new line is just started.

--- Solution:
I added a boolean attribute to the class CXMLWriter, needIndent, that it's setted to true only by writeLineBreak method and it's used in conjuction with Tabs>0 condition to solve the problem.

--- Problem:
the datatype wchar_t is not always 2 bytes long. In linux machines (and in others maybe), the wchar_t is 4 bytes long. This creates troubles when the data is written to the file and in calculation of space occupied.

--- Solution:
It's possibile to force gcc to use 2-byte wchar_t instead of 4-bytes by adding the compiler flags '-fshort-wchar'. I modified the Makefile accordlying.

--- Problem:
A litte preface to this problem: "wchar_t != UTF16 (o UCS-2) encoded character"
But, all irrlicht code assumes that a wchar_t string is encoded in UTF16, but it's not true on all machines!! In particular, modern linux machines encodes wchar_t strings in UCS-4 and declare wchar_t as 4-bytes long datatype. For this reason, the wcslen assume that the wchar_t passed is encoded in UCS-4 (even if -fshort-wchar flags is activated). But the wchar_t strings created by irrlicht are UTF16 encoded, so all lengths returned by wcslen are junks and the CXMLWriter always call the File->Write with the wrong numbers of bytes.

--- Solution:
I created a function, UTF16LEN, to calculate the length of a UTF16 encoded string. I added this function into the CXMLWriter code and I putted a preprocessor condition in order to use my created function instead of wcslen on linux machines. This is only a partial solution to the problem, because actually my UTF16LEN function fails for those characters that UTF16 encodes with 4-bytes. In fact, UCS-2 is different from UTF16 because UCS-2 is 2-bytes fixed encoding and UTF16 allow a range of special chars encoded by 4-bytes codes.
But, due to fact that irrlicht code doesn't care about the 4-byte UTF16 character and it assumes that all UTF16 chars fits in 2-bytes... also my UTF16LEN follow the same (wrong) assumption.

--- Problem:
The literal wide strings declared with the code L"String" doesn't means that "String" is encoded in UTF16. In details, on linux machines the L"String" declaration means that "String" is encoded in UCS-4. Obviosuly, this makes a lot of problems for irrlicht that needs strings encoded in UTF16.

--- Solution:
In order to solve this problem, it's only necessary to add the flag -fwide-exec-charset=UTF16 to gcc compiler. In this ways, all strings declared by L"String" will be encoded in UTF16.
Notes that this doesn't solve the previous problems (i.e. wrong length calculated by wcslen) because the standard library is compiled with 4-bytes wchar_t datatype and with UCS-4 string encoding. This means that all standard function regarding wide character strings assume that the encoding is UCS-4 !!
And also, due to fact that some special UTF-16 chars requires 4-byte in conjuction with -fshort-wchar, the following statement arise a warning:

Code: Select all

    wchar_t c = L's';

because exists some 's' that doesn't fits in 2-bytes. But, actually this is not a problem because all literal wide chars used in irrlicht code belong to the range where UTF16 encoding requires only 2-byte (I hope )

--- Problem:
Sometimes, (I didn't understand this strange behaviour), when gcc encodes an L"String" declaration it prepends the "String" with the Byte Order Mark (BOM). This makes trouble when you use the string<wchar_t>::append( string<wchar_t> ) because the method will append the BOM inside another string. For instance, if you append "Mondo" to "Ciao " the results will be "Ciao <FEFF> Mondo".

--- Solution:
I added a condition into string<T>::append(string<T>) method for manage this particular condition. The added code if enclosed into a preprocessor condition in order use it only in linux machines.
I admit that the solution proposed by me is not a good solution because: (i) is inside the template definition, so it will executed also for string<char> class, (ii) assumes that wchar_t is 2-bytes long and encodes UTF16 strings and (iii) assumes that if sizeof(T)==2 then T is a wchar_t datatype.

--- Conclusions:
Due to this problems arises from different wide character dimension and encoding on different machines and in order to mantains irrlicht a cross-platform project (i.e. usable on very different architecture's machines), I think that all wide strings and XML module implementation must be redesigned, keeping in mind that the follow assumption used until now by irrlicht are wrongs:
a) wchar_t is 2-byte long
b) L"String" is UTF16 encoded
c) wcslen calculate the lenght of an UTF16 string (same for all standard functions over wchar_t datatype)
d) L's' is a character that always fits in 2-bytes (even if it's encoded in UTF16)

I didn't attach the modified code in this topic, but of course I'm glade to give these modifies to irrlicht's authors. If interested, please tell me in which format I have to send you it (e.g. how to generate diffs file).

Ciao,
Gianluca

[hybrid]

Hmm, I took a different approach: Make everything work with 2 and 4 byte wchars. I did not test it yet, though, but all versions are supported by the XMLReader so it should work. I'll check my version and maybe check-in that one. Then we'll see if this still needs some fixing.
Fixing to some byte size is definitely not a solution due to porting issues.

[Gianluca]

Fixing to some byte size is definitely not a solution due to porting issues.

Excatly what I mean... but actually the SVN version of irrlicht has CXMLWriter that works only if wchar_t is 2-bytes and encoded in UTF16.

The main issue is that wchar_t doesn't imply a particular encoding, e.g. UCS or UTF or somethingelse.
So, the best approach is to don't rely on standard wide chartacter function and to L"String" macro for constant string due to porting issues!!!

Now, there are a template string class that it's only a container for primitives char and wchar_t datatype... and they doesn't care about encoding... I think that this approach is wrong, I mean... it's not portable!!

The string class should be an unique class that represent a generic string whatever is the encoding choosed... and the string class must have the capability to convert strings among various encoding. In this way, the string class contains the characters in an internal model... and with the appropriate methods user can retrieve the correct representation.
And also, this class doesn't based on primitives datatype wchar_t and char for encode strings.

Ciao,
Gianluca

[hybrid]

Yes, but changing the string class will need some major efforts. I'd like to fix the XMLWriter to platform dependent working first, and later on do the Unicode things.

[Gianluca]

In my topic, I already notice that:

Sometimes, (I didn't understand this strange behaviour), when gcc encodes an L"String" declaration it prepends the "String" with the Byte Order Mark (BOM).

I've just see that this makes trouble also in operator== of string template.
And for patch this, again, I had to put some very specific code for handling this behaviour under linux.
And again, I admit that my solution is very poor because I putted the code directly in the template for manage the bug that happens only in string<wchar_t> ...
But, Again, I didn't find a better solution ...
And, Again, I advice the irrlicht's authors to redesign completly the class string due to porting issue caused by actual template string design.

Ciao,
Gianluca