Subject | Re: [ib-support] Wierd math??? |
---|---|

Author | Helen Borrie |

Post date | 2003-02-20T22:29:39Z |

At 12:27 PM 20/02/2003 -0800, you wrote:

decimal places, but a bunch of floating point numbers. By the rules you

are working under, the returned result is zero.

When I see you use the word "exact" I have to wonder why you are using a

floating point type to store exact numbers.

Geoff Worboys did a Tech Info sheet around the subject of precision and the

way IB stores numbers. You can pick it up at www.ibobjects.com/TechInfo.html

heLen

>Ok. Take the following query:The problem is, you are not summing a bunch of numbers with exactly 2

>

>select txamount from laytransactions where layid = 5

>

>The result is:

>

>-7.34

>7.34

>-29.36

>-8.09

>37.49

>-0.04

>

>Now, I run the following query:

>

>select sum(txamount) from laytransactions where layid = 5

>

>The result is:

>

>-8.53483950180589E-16

>

>What the heck? Why isn't it zero???

>

>I am using FB 1.0. The txamount column is a double precision. But,

>shouldn't summing a bunch of numbers with exactly 2 decimal places yield a

>result with two digits, and not some random very small number???

decimal places, but a bunch of floating point numbers. By the rules you

are working under, the returned result is zero.

When I see you use the word "exact" I have to wonder why you are using a

floating point type to store exact numbers.

Geoff Worboys did a Tech Info sheet around the subject of precision and the

way IB stores numbers. You can pick it up at www.ibobjects.com/TechInfo.html

heLen