Subject | Re: [ib-support] Wierd math??? |
---|---|

Author | Nando Dessena |

Post date | 2003-02-20T20:47:59Z |

Joe,

JM> -7.34

JM> 7.34

JM> -29.36

JM> -8.09

JM> 37.49

JM> -0.04

JM> select sum(txamount) from laytransactions where layid = 5

JM> -8.53483950180589E-16

JM> What the heck? Why isn't it zero???

In a sense, it is. :-)

JM> I am using FB 1.0. The txamount column is a double precision.

That's the source of your troubles. Double precision is not, err,

"precise".

JM> But, shouldn't summing a bunch of numbers with exactly 2 decimal places yield a

JM> result with two digits, and not some random very small number???

It should, but since you are using floating point numbers your numbers

do not have "exactly" 2 decimal places. Actually, their decimal point

floats. Search the archives for thorough discussions about this

problem.

Ciao

--

Nando mailto:nandod@...

JM> -7.34

JM> 7.34

JM> -29.36

JM> -8.09

JM> 37.49

JM> -0.04

JM> select sum(txamount) from laytransactions where layid = 5

JM> -8.53483950180589E-16

JM> What the heck? Why isn't it zero???

In a sense, it is. :-)

JM> I am using FB 1.0. The txamount column is a double precision.

That's the source of your troubles. Double precision is not, err,

"precise".

JM> But, shouldn't summing a bunch of numbers with exactly 2 decimal places yield a

JM> result with two digits, and not some random very small number???

It should, but since you are using floating point numbers your numbers

do not have "exactly" 2 decimal places. Actually, their decimal point

floats. Search the archives for thorough discussions about this

problem.

Ciao

--

Nando mailto:nandod@...