Subject | Re: [ib-support] IB6 BUG #221589: numeric fields and mathematical operations |
---|---|

Author | Svein Erling Tysvær |

Post date | 2001-08-23T07:07:46Z |

Sorry if this appears twice, I had problems in my first attempt.

This is due to IB rounding intermediate values, and is actually visible

even if you reduce your statement to

SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.

To get the correct result, you could simply change this statement to

SELECT 10.5/100.00 FROM rdb$database, which will give you the correct

number of decimals (0.105).

HTH,

Set

lobolo2000 wrote:

This closed bug relates partially to the following:

--------------------------------------------------------

in dialect 1 and 3

select field1*(1-field2/100) from mytable

where:

field1 numeric(9,2) value -> 10

field2 numeric(9,2) value -> 10.5

the result is: 9 instead of: 8.95

Possible explanation:

Since values are numeric(9,2), then

field2/100 is 0.10 because they are exact numerics.

Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.

-------------------------------------------------------

Is it true that IB6 (and FB) output 9 as result?

I tried it and got 8.95, and hope that the explanation

given is not true due to its implication that exact numerics

are not suitable for most cases.

Can anybody comment on this pls?

This is due to IB rounding intermediate values, and is actually visible

even if you reduce your statement to

SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.

To get the correct result, you could simply change this statement to

SELECT 10.5/100.00 FROM rdb$database, which will give you the correct

number of decimals (0.105).

HTH,

Set

lobolo2000 wrote:

This closed bug relates partially to the following:

--------------------------------------------------------

in dialect 1 and 3

select field1*(1-field2/100) from mytable

where:

field1 numeric(9,2) value -> 10

field2 numeric(9,2) value -> 10.5

the result is: 9 instead of: 8.95

Possible explanation:

Since values are numeric(9,2), then

field2/100 is 0.10 because they are exact numerics.

Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.

-------------------------------------------------------

Is it true that IB6 (and FB) output 9 as result?

I tried it and got 8.95, and hope that the explanation

given is not true due to its implication that exact numerics

are not suitable for most cases.

Can anybody comment on this pls?