Subject | Re: [ib-support] IB6 BUG #221589: numeric fields and mathematical operations |
---|---|
Author | Svein Erling Tysvær |
Post date | 2001-08-23T07:02:42Z |
This is due to IB rounding intermediate values, and is actually visible even if you reduce your
statement to
SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.
To get the correct result, you could simply change this statement to
SELECT 10.5/100.00 FROM rdb$database, which will give you the correct number of decimals (0.105).
HTH,
Set
lobolo2000 wrote:
statement to
SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.
To get the correct result, you could simply change this statement to
SELECT 10.5/100.00 FROM rdb$database, which will give you the correct number of decimals (0.105).
HTH,
Set
lobolo2000 wrote:
> This closed bug relates partially to the following:
> --------------------------------------------------------
> in dialect 1 and 3
> select field1*(1-field2/100) from mytable
> where:
> field1 numeric(9,2) value -> 10
> field2 numeric(9,2) value -> 10.5
>
> the result is: 9 instead of: 8.95
> Possible explanation:
> Since values are numeric(9,2), then
> field2/100 is 0.10 because they are exact numerics.
> Hence, 10*(1-0.10)=10*0.9 = 9 as IB answered.
> -------------------------------------------------------
>
> Is it true that IB6 (and FB) output 9 as result?
> I tried it and got 8.95, and hope that the explanation
> given is not true due to its implication that exact numerics
> are not suitable for most cases.
>
> Can anybody comment on this pls?
>
> TIA
>
> _________________________________________________________
>
> Do You Yahoo!?
>
> Get your free @... address at http://mail.yahoo.com
>
>
> To unsubscribe from this group, send an email to:
> ib-support-unsubscribe@egroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/