Subject | Re: [ib-support] IB6 BUG #221589: numeric fields and mathematical operations |
---|---|
Author | Svein Erling Tysvær |
Post date | 2001-08-23T07:07:46Z |
Sorry if this appears twice, I had problems in my first attempt.
This is due to IB rounding intermediate values, and is actually visible
even if you reduce your statement to
SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.
To get the correct result, you could simply change this statement to
SELECT 10.5/100.00 FROM rdb$database, which will give you the correct
number of decimals (0.105).
HTH,
Set
lobolo2000 wrote:
This closed bug relates partially to the following:
--------------------------------------------------------
in dialect 1 and 3
select field1*(1-field2/100) from mytable
where:
field1 numeric(9,2) value -> 10
field2 numeric(9,2) value -> 10.5
the result is: 9 instead of: 8.95
Possible explanation:
Since values are numeric(9,2), then
field2/100 is 0.10 because they are exact numerics.
Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.
-------------------------------------------------------
Is it true that IB6 (and FB) output 9 as result?
I tried it and got 8.95, and hope that the explanation
given is not true due to its implication that exact numerics
are not suitable for most cases.
Can anybody comment on this pls?
This is due to IB rounding intermediate values, and is actually visible
even if you reduce your statement to
SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.
To get the correct result, you could simply change this statement to
SELECT 10.5/100.00 FROM rdb$database, which will give you the correct
number of decimals (0.105).
HTH,
Set
lobolo2000 wrote:
This closed bug relates partially to the following:
--------------------------------------------------------
in dialect 1 and 3
select field1*(1-field2/100) from mytable
where:
field1 numeric(9,2) value -> 10
field2 numeric(9,2) value -> 10.5
the result is: 9 instead of: 8.95
Possible explanation:
Since values are numeric(9,2), then
field2/100 is 0.10 because they are exact numerics.
Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.
-------------------------------------------------------
Is it true that IB6 (and FB) output 9 as result?
I tried it and got 8.95, and hope that the explanation
given is not true due to its implication that exact numerics
are not suitable for most cases.
Can anybody comment on this pls?