Subject | Re: [firebird-support] Problem with rounding |
---|---|
Author | Jacek Borowski |
Post date | 2009-08-11T19:58:15Z |
Hi,
Thank you for yours reply.
On Windows results are always "correct", even, when we are using double
precision declaration.
I've changed double precision to numeric(18,6) and now results on Linux are
ok.
Finaly, I wrote my own function and it works fine with double precision
declaration.
Jacek
Thank you for yours reply.
> Svein Erling Tysvær wrote:Ok, but why results on other platforms (Linux and Windows) are different ?
>>> select * from zaok(67.865,2) - returns 67.86 - wrong result
>>
>> Why is this wrong? In your procedure, you've defined the input parameter
>> LICZBA as double precision and double precision is never precise. So when
>> you specify 67.865, you're not passing the actual number 67.865 to the
>> stored procedure, but an approximation that may be close to 67.8650000001
>> or 67.8649999999 (well, probably you have to add more digits to see the
>> difference with double precision). Casting the first of these
>> approximations to NUMERIC (18,2) would result in 67.87, the latter 67.86.
>>
On Windows results are always "correct", even, when we are using double
precision declaration.
I've changed double precision to numeric(18,6) and now results on Linux are
ok.
Finaly, I wrote my own function and it works fine with double precision
declaration.
>> The rule is simply never to assume a floating point number to be exact.Many thanks,
>> Don't even think that 67.865 = 67.865 when using a floating point number.
>
> You are right, that could also be the cause of the (perceived) problem.
>
> --
> Mark Rotteveel
Jacek