Subject | Re: [firebird-support] Re: Rounding error |
---|---|

Author | Gareth Marshall |

Post date | 2007-11-12T01:26:15Z |

Hi

I agree with you that

number, but still not zero.

But the problem is that the result of doing 34.27 - 34.27 should be ZERO and

CREATE TABLE DblTest ( Val DOUBLE PRECISION );

INSERT INTO DblTest VALUES (34.27);

SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;

I get NULL as the value in the SELECT statement. This means that Firebird is

consistently storing and calculating the value 34.27, so the problem use lie

elsewhere.

How did the value of 34.27 get into your table? Is it the result of a

calculation? If it is, that would explain why it isn't 34.27, but instead is

34.270000000000007105427357601. Alternatively, if the value was stored in a

single precision floating point variable in your code at some point, that

would reduce the accuracy of the number and end up making the stored value

different to the double precision representation (approximation) of 34.27.

Regards

Gareth Marshall

[Non-text portions of this message have been removed]

I agree with you that

>Actually, 7,105427357601E-15 is 0,000000000000007105427357601 a much smaller

> 7,105427357601E-15 = 0.00710542735760100000 != 0

>

number, but still not zero.

But the problem is that the result of doing 34.27 - 34.27 should be ZERO and

>If I do the following:

> not 7,105427357601E-15. I still think this is an error (of floating point

> arithmetic or of FireBird).

>

CREATE TABLE DblTest ( Val DOUBLE PRECISION );

INSERT INTO DblTest VALUES (34.27);

SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;

I get NULL as the value in the SELECT statement. This means that Firebird is

consistently storing and calculating the value 34.27, so the problem use lie

elsewhere.

How did the value of 34.27 get into your table? Is it the result of a

calculation? If it is, that would explain why it isn't 34.27, but instead is

34.270000000000007105427357601. Alternatively, if the value was stored in a

single precision floating point variable in your code at some point, that

would reduce the accuracy of the number and end up making the stored value

different to the double precision representation (approximation) of 34.27.

Regards

Gareth Marshall

[Non-text portions of this message have been removed]