Subject | Re: [ib-support] Wierd math??? |
---|---|
Author | Helen Borrie |
Post date | 2003-02-20T22:29:39Z |
At 12:27 PM 20/02/2003 -0800, you wrote:
decimal places, but a bunch of floating point numbers. By the rules you
are working under, the returned result is zero.
When I see you use the word "exact" I have to wonder why you are using a
floating point type to store exact numbers.
Geoff Worboys did a Tech Info sheet around the subject of precision and the
way IB stores numbers. You can pick it up at www.ibobjects.com/TechInfo.html
heLen
>Ok. Take the following query:The problem is, you are not summing a bunch of numbers with exactly 2
>
>select txamount from laytransactions where layid = 5
>
>The result is:
>
>-7.34
>7.34
>-29.36
>-8.09
>37.49
>-0.04
>
>Now, I run the following query:
>
>select sum(txamount) from laytransactions where layid = 5
>
>The result is:
>
>-8.53483950180589E-16
>
>What the heck? Why isn't it zero???
>
>I am using FB 1.0. The txamount column is a double precision. But,
>shouldn't summing a bunch of numbers with exactly 2 decimal places yield a
>result with two digits, and not some random very small number???
decimal places, but a bunch of floating point numbers. By the rules you
are working under, the returned result is zero.
When I see you use the word "exact" I have to wonder why you are using a
floating point type to store exact numbers.
Geoff Worboys did a Tech Info sheet around the subject of precision and the
way IB stores numbers. You can pick it up at www.ibobjects.com/TechInfo.html
heLen