Subject Re: [firebird-support] FB 3 issues with String from FB 2.54
Author
Post date 2016-05-25T18:22:40Z

Mark
 
Are you saying that the filter I placed on FB 2.5 with charset ASCII was correct from the theory, but there was a bug that made the filter be ignored by the FB engine? and the ugly characters got inside the DB because of the bug?
 
And for me to be able to move onto FB 3 I had to remove the ASCII filter (charset) because otherwise the restore onto 3.0 would fail, so now I have lots of records with unwanted characters and cannot put the filter back onto 3.0 because if I do so I would end up having a table that does not allow such characters however the characters are already there, so for intance a user trying to change a dot, or a comma on one of those records would get a failed message as the DB cannot write back into the engine the same text that is alraedy there becuase of the offending character. Is that correct?  How do I solve the issue? At the moment I set charset to NONE and it allows garbage in and out with no issues, but I don't want the bad characters there......
 
Cheers,
Fabian
 
----- Original Message -----
Sent: Thursday, May 26, 2016 4:08 AM
Subject: Re: [firebird-support] FB 3 issues with String from FB 2.54

On 23-5-2016 03:45, fabianch@... [firebird-support] wrote:
> I have been trying to migrate from FB2.54 into FB 3 for a few weeks, and
> after hitting a string related error for some time i have got to the
> point where I do understand the issue, but I don't know how to solve it.
> The issue is pretty simple, the FB 2.54 DB contains a few characters
> that are not allowed into the FB 3 database, one example of a character
> causing an error during the restore was "Mcgarrity’s" (note the ’) as it
> appears to be outside the scope of the FB3 string domain, I have trying
> creating a new FB3 DB with many different charsets but none works. The
> other string causing issues is for example "΢ÈíÑźÚ", I have many
> records with this type of strings because the DB contains raw emails
> received by the system, stored into Varchars, and apparently some emails
> contain very weird characters, all were handled by FB2.54 but FB3
> rejects the records. I have been able to isolate all recrods with issues
> using IBExpert's table data comparer function, as it created a script
> with all recrods from all tables from FB2.54 and when running the script
> against FB3.0 it singles out all the offending records.
>
> Can anyone advise what options I have available to force FB3.0 to accept
> any stuff into string fields?

In your other e-mail you indicate you solved this by changing the
character set from ASCII to NONE. The fact it worked before was a bug,
see http://tracker.firebirdsql.org/browse/CORE-3416. ASCII only supports
characters 0-127, characters outside that range are 'extended ascii', eg
one of the other singly by character sets like WIN1252 or ISO8859_1. The
characters shown (΢ÈíÑÅºÚ and ’) are all outside the ASCII range.

The last (’) is particularly nasty, because it should have been a '
(u+0027 Apostrophe, ascii 39), instead u+2019 Right single quotation
mark (character 146 in Windows-1252) was used.

Given the context of e-mails either NONE or OCTETS is the only real
option, as e-mails can have multiple parts with each their own character
set, and can also have binary parts (although usually those are encoded
with something like base64).

Mark
--
Mark Rotteveel


------------------------------------
Posted by: Mark Rotteveel <mark@...>
------------------------------------

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Visit http://www.firebirdsql.org and click the Documentation item
on the main (top) menu.  Try FAQ and other links from the left-side menu there.

Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-support/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-support/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-support-digest@yahoogroups.com
    firebird-support-fullfeatured@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    firebird-support-unsubscribe@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
    https://info.yahoo.com/legal/us/yahoo/utos/terms/