Subject | Re: Mathematics problem |
---|---|
Author | Ali Gökçen |
Post date | 2005-10-08T11:08:02Z |
Hi Miki,
select
cast( cast(109.87 as numeric(18,2))-cast(109.87 as numeric(18,2))
*cast(5.0 as double precision)/100 as double precision) from
rdb$database
gives, 104.365 as expected exact result.
There must be a problem with you, check yourself again.
if you want to escape from rounding and cutting problems as
possible, don't mix
different type values in a calculation.
for example 100 is an integer but doesnt effect here luckly.
operations between 18,2 and double precisions may cause some
precision losts.
Convert all off them to double precision or
anough to save numeric(n,p) type before calculation.
+ modify your data browser tool configuration about display of
numeric values.
(for example ibexpert uses (*,3) and rounds value before display)
Ragards.
Ali
select
cast( cast(109.87 as numeric(18,2))-cast(109.87 as numeric(18,2))
*cast(5.0 as double precision)/100 as double precision) from
rdb$database
gives, 104.365 as expected exact result.
There must be a problem with you, check yourself again.
if you want to escape from rounding and cutting problems as
possible, don't mix
different type values in a calculation.
for example 100 is an integer but doesnt effect here luckly.
operations between 18,2 and double precisions may cause some
precision losts.
Convert all off them to double precision or
anough to save numeric(n,p) type before calculation.
+ modify your data browser tool configuration about display of
numeric values.
(for example ibexpert uses (*,3) and rounds value before display)
Ragards.
Ali