Subject | Re: [ib-support] IB6 BUG #221589: numeric fields and mathematical operations |
---|---|

Author | Svein Erling Tysvær |

Post date | 2001-08-23T07:02:42Z |

This is due to IB rounding intermediate values, and is actually visible even if you reduce your

statement to

SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.

To get the correct result, you could simply change this statement to

SELECT 10.5/100.00 FROM rdb$database, which will give you the correct number of decimals (0.105).

HTH,

Set

lobolo2000 wrote:

statement to

SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.

To get the correct result, you could simply change this statement to

SELECT 10.5/100.00 FROM rdb$database, which will give you the correct number of decimals (0.105).

HTH,

Set

lobolo2000 wrote:

> This closed bug relates partially to the following:

> --------------------------------------------------------

> in dialect 1 and 3

> select field1*(1-field2/100) from mytable

> where:

> field1 numeric(9,2) value -> 10

> field2 numeric(9,2) value -> 10.5

>

> the result is: 9 instead of: 8.95

> Possible explanation:

> Since values are numeric(9,2), then

> field2/100 is 0.10 because they are exact numerics.

> Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.

> -------------------------------------------------------

>

> Is it true that IB6 (and FB) output 9 as result?

> I tried it and got 8.95, and hope that the explanation

> given is not true due to its implication that exact numerics

> are not suitable for most cases.

>

> Can anybody comment on this pls?

>

> TIA

>

> _________________________________________________________

>

> Do You Yahoo!?

>

> Get your free @... address at http://mail.yahoo.com

>

>

> To unsubscribe from this group, send an email to:

> ib-support-unsubscribe@egroups.com

>

>

>

> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/