Subject | Re: [firebird-support] Re: Rounding error |
---|---|
Author | Gareth Marshall |
Post date | 2007-11-12T01:26:15Z |
Hi
I agree with you that
number, but still not zero.
But the problem is that the result of doing 34.27 - 34.27 should be ZERO and
CREATE TABLE DblTest ( Val DOUBLE PRECISION );
INSERT INTO DblTest VALUES (34.27);
SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;
I get NULL as the value in the SELECT statement. This means that Firebird is
consistently storing and calculating the value 34.27, so the problem use lie
elsewhere.
How did the value of 34.27 get into your table? Is it the result of a
calculation? If it is, that would explain why it isn't 34.27, but instead is
34.270000000000007105427357601. Alternatively, if the value was stored in a
single precision floating point variable in your code at some point, that
would reduce the accuracy of the number and end up making the stored value
different to the double precision representation (approximation) of 34.27.
Regards
Gareth Marshall
[Non-text portions of this message have been removed]
I agree with you that
>Actually, 7,105427357601E-15 is 0,000000000000007105427357601 a much smaller
> 7,105427357601E-15 = 0.00710542735760100000 != 0
>
number, but still not zero.
But the problem is that the result of doing 34.27 - 34.27 should be ZERO and
>If I do the following:
> not 7,105427357601E-15. I still think this is an error (of floating point
> arithmetic or of FireBird).
>
CREATE TABLE DblTest ( Val DOUBLE PRECISION );
INSERT INTO DblTest VALUES (34.27);
SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;
I get NULL as the value in the SELECT statement. This means that Firebird is
consistently storing and calculating the value 34.27, so the problem use lie
elsewhere.
How did the value of 34.27 get into your table? Is it the result of a
calculation? If it is, that would explain why it isn't 34.27, but instead is
34.270000000000007105427357601. Alternatively, if the value was stored in a
single precision floating point variable in your code at some point, that
would reduce the accuracy of the number and end up making the stored value
different to the double precision representation (approximation) of 34.27.
Regards
Gareth Marshall
[Non-text portions of this message have been removed]