Subject Saving UTF-8 Chinese characters produced error - Invalid Token. Malformed String.
Author Roberto Smith
Hi,

I am experiencing this problem. Not sure if this is the bug of Firebird. I can't key in the UTF-8 charaters by ISQL, FlameRobin, Upscene Data Workbench. So the best tool to simulate is IBExpert operating in Windows XP. In Vista, the Chinese characters do not show at all in all tools, including IBExpert.

Thanks.

regards,
roberto

----------

Windows XP: SP2
Firebird: Build 2.1.3.18185
DB Charset: UTF-8
Tools: IBExpert Version 2009.08.17 Personal Edition

Scenario:
1. Using IBExpert, create a table with a single column, Varchar (10), Charset UTF-8, Collate UTF-8.

2. At the IBExpert, Open that Table. At the data tab, try input some UTF-8 Chinese characters based on the character script download from http://www.unicode.org/charts (See a sample of the script which states the hexadecimal of a list of chinese characters

3. To input the UTF-8 Chinese characters, e.g. Type 4E00 Alt-X, you should see  this character 一 (Type 4e00, follow by ALT X)

4. The problem is typing the UTF-8 Chinese characters is not a problem, but when saving into the database, most of the characters produced the following Error Message:

Invalid token. Malformed String.

5. The example list of characters not able to save:
4E01 : 丁
4E02: 丂
4E03: 七 etc.

Attached please find the screen shot of the table contain the hexadecimal number if its corresponding Chinese characters. as well as the records of the table consist of 4 records with UTF-8 Chinese characters, tale note that the first 3 records can be saved easily, whereas the 4th record could not be saved.





[Non-text portions of this message have been removed]