Subject | Re: [ib-support] Really strange calculating behaviour of IB |
---|---|
Author | Geoff Worboys |
Post date | 2002-02-14T02:53:20Z |
> The values the database stores don't have to be scaled. You can doThis idea actually demonstrates one work around for the present
> the scaling yourself using your external application or UDFs.
problem. Again using the second demo that was posted, the following
gives the correct response...
CAST( (
( (CAST((money3 * 100000) as NUMERIC(18,0)) * 100000)
/ CAST((1.95583* 100000) as NUMERIC(18,0))
) ) AS NUMERIC(18,5) ) / 100000
as money3c
Of course the above work-around is of only limited value, since it
limits the integral value to 8 digits (99 million) - after which
overflow conditions will occur.
Note: If you change the divisor to 1.95582 as in the example below...
CAST( (
( (CAST((money3 * 100000) as NUMERIC(18,0)) * 100000)
/ CAST((1.95582* 100000) as NUMERIC(18,0))
) ) AS NUMERIC(18,5) ) / 100000
as money3c
and compare the result with...
(money1/1.95582) as money1a,
You will see that the first calculation results in truncation (and not
rounding) at the 5th decimal place. This is strictly correct, since
we are technically manipulating integers, but may not be what is
expected.
--
Geoff Worboys
Telesis Computing