In the world of real numbers:
34.27 == 34.27???????????
and
34.27 != 34.2700000000000'

Always compare for equal as follows: (34.27-34.27)<aSmallErrorAllowed

If you store a value (from code) as 34.27 and later compare it you will have a binary equivalent
but still not necessarily 34.2600'


the ' at the end of the number indicates 0 repeating forever.


In the world of integers:
3427==3427 :)

----- Original Message -----
From: Gareth Marshall
To: firebird-support@yahoogroups.com
Sent: Monday, November 12, 2007 3:26 AM
Subject: Re: [firebird-support] Re: Rounding error


Hi

I agree with you that
>
> 7,105427357601E-15 = 0.00710542735760100000 != 0
>

Actually, 7,105427357601E-15 is 0,000000000000007105427357601 a much smaller
number, but still not zero.

But the problem is that the result of doing 34.27 - 34.27 should be ZERO and
>
> not 7,105427357601E-15. I still think this is an error (of floating point
> arithmetic or of FireBird).
>
If I do the following:

CREATE TABLE DblTest ( Val DOUBLE PRECISION );
INSERT INTO DblTest VALUES (34.27);
SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;

I get NULL as the value in the SELECT statement. This means that Firebird is
consistently storing and calculating the value 34.27, so the problem use lie
elsewhere.

How did the value of 34.27 get into your table? Is it the result of a
calculation? If it is, that would explain why it isn't 34.27, but instead is
34.270000000000007105427357601. Alternatively, if the value was stored in a
single precision floating point variable in your code at some point, that
would reduce the accuracy of the number and end up making the stored value
different to the double precision representation (approximation) of 34.27.

Regards
Gareth Marshall

[Non-text portions of this message have been removed]






[Non-text portions of this message have been removed]