Subject Re: [firebird-support] Re: Rounding error
Author Gareth Marshall
Hi

I agree with you that
>
> 7,105427357601E-15 = 0.00710542735760100000 != 0
>

Actually, 7,105427357601E-15 is 0,000000000000007105427357601 a much smaller
number, but still not zero.

But the problem is that the result of doing 34.27 - 34.27 should be ZERO and
>
> not 7,105427357601E-15. I still think this is an error (of floating point
> arithmetic or of FireBird).
>
If I do the following:

CREATE TABLE DblTest ( Val DOUBLE PRECISION );
INSERT INTO DblTest VALUES (34.27);
SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;

I get NULL as the value in the SELECT statement. This means that Firebird is
consistently storing and calculating the value 34.27, so the problem use lie
elsewhere.

How did the value of 34.27 get into your table? Is it the result of a
calculation? If it is, that would explain why it isn't 34.27, but instead is
34.270000000000007105427357601. Alternatively, if the value was stored in a
single precision floating point variable in your code at some point, that
would reduce the accuracy of the number and end up making the stored value
different to the double precision representation (approximation) of 34.27.

Regards
Gareth Marshall


[Non-text portions of this message have been removed]