Subject | Rounding error |
---|---|
Author | Gustavo |
Post date | 2007-11-07T18:26:51Z |
Hello:
I use Delphi 5, FireBird 2.0 and IBX and I have the following problem.
I have a table named PMOVCC with a field MONTOPEND of type DOUBLE PRECISION and sometimes, when after an UPDATE statement it has to be null, it has a value like 7,105427357601E-15.
I will give an example.
1. In one record of the table, the field MONTOPEND has the value 34.27.
2. I execute the following statement:
UPDATE PMOVCC
SET MONTOPEND = NULLIF(COALESCE(MONTOPEND,0) - 34.27,0)
WHERE (CLAVEMOVIM = 'XXX')
3. Now, the field should be NULL, but it has the value 7,105427357601E-15.
It´s a strange thing because this doesn´t happen always. Sometimes it works good!
My questions are:
1. Is there a way to avoid this rounding error?
2. Is there a way to configure FireBird not to use more than N decimal digits in any calculation? If this coud be done, I could configure not to use more than 8 or 9 digits and then, even if there is a rounding error, the result will be right.
Thanks in advance
Gustavo
[Non-text portions of this message have been removed]
I use Delphi 5, FireBird 2.0 and IBX and I have the following problem.
I have a table named PMOVCC with a field MONTOPEND of type DOUBLE PRECISION and sometimes, when after an UPDATE statement it has to be null, it has a value like 7,105427357601E-15.
I will give an example.
1. In one record of the table, the field MONTOPEND has the value 34.27.
2. I execute the following statement:
UPDATE PMOVCC
SET MONTOPEND = NULLIF(COALESCE(MONTOPEND,0) - 34.27,0)
WHERE (CLAVEMOVIM = 'XXX')
3. Now, the field should be NULL, but it has the value 7,105427357601E-15.
It´s a strange thing because this doesn´t happen always. Sometimes it works good!
My questions are:
1. Is there a way to avoid this rounding error?
2. Is there a way to configure FireBird not to use more than N decimal digits in any calculation? If this coud be done, I could configure not to use more than 8 or 9 digits and then, even if there is a rounding error, the result will be right.
Thanks in advance
Gustavo
[Non-text portions of this message have been removed]