Subject | Re: [firebird-support] Re: Rounding error |
---|---|

Author | Lafras Henning |

Post date | 2007-11-12T06:12:12Z |

In the world of real numbers:

34.27 == 34.27???????????

and

34.27 != 34.2700000000000'

Always compare for equal as follows: (34.27-34.27)<aSmallErrorAllowed

If you store a value (from code) as 34.27 and later compare it you will have a binary equivalent

but still not necessarily 34.2600'

the ' at the end of the number indicates 0 repeating forever.

In the world of integers:

3427==3427 :)

34.27 == 34.27???????????

and

34.27 != 34.2700000000000'

Always compare for equal as follows: (34.27-34.27)<aSmallErrorAllowed

If you store a value (from code) as 34.27 and later compare it you will have a binary equivalent

but still not necessarily 34.2600'

the ' at the end of the number indicates 0 repeating forever.

In the world of integers:

3427==3427 :)

----- Original Message -----

From: Gareth Marshall

To: firebird-support@yahoogroups.com

Sent: Monday, November 12, 2007 3:26 AM

Subject: Re: [firebird-support] Re: Rounding error

Hi

I agree with you that

>

> 7,105427357601E-15 = 0.00710542735760100000 != 0

>

Actually, 7,105427357601E-15 is 0,000000000000007105427357601 a much smaller

number, but still not zero.

But the problem is that the result of doing 34.27 - 34.27 should be ZERO and

>

> not 7,105427357601E-15. I still think this is an error (of floating point

> arithmetic or of FireBird).

>

If I do the following:

CREATE TABLE DblTest ( Val DOUBLE PRECISION );

INSERT INTO DblTest VALUES (34.27);

SELECT NULLIF(COALESCE(Val,0) - 34.27, 0) FROM DblTest;

I get NULL as the value in the SELECT statement. This means that Firebird is

consistently storing and calculating the value 34.27, so the problem use lie

elsewhere.

How did the value of 34.27 get into your table? Is it the result of a

calculation? If it is, that would explain why it isn't 34.27, but instead is

34.270000000000007105427357601. Alternatively, if the value was stored in a

single precision floating point variable in your code at some point, that

would reduce the accuracy of the number and end up making the stored value

different to the double precision representation (approximation) of 34.27.

Regards

Gareth Marshall

[Non-text portions of this message have been removed]

[Non-text portions of this message have been removed]