Subject | Re: [ib-support] Wierd math??? |
---|---|
Author | Nando Dessena |
Post date | 2003-02-20T20:47:59Z |
Joe,
JM> -7.34
JM> 7.34
JM> -29.36
JM> -8.09
JM> 37.49
JM> -0.04
JM> select sum(txamount) from laytransactions where layid = 5
JM> -8.53483950180589E-16
JM> What the heck? Why isn't it zero???
In a sense, it is. :-)
JM> I am using FB 1.0. The txamount column is a double precision.
That's the source of your troubles. Double precision is not, err,
"precise".
JM> But, shouldn't summing a bunch of numbers with exactly 2 decimal places yield a
JM> result with two digits, and not some random very small number???
It should, but since you are using floating point numbers your numbers
do not have "exactly" 2 decimal places. Actually, their decimal point
floats. Search the archives for thorough discussions about this
problem.
Ciao
--
Nando mailto:nandod@...
JM> -7.34
JM> 7.34
JM> -29.36
JM> -8.09
JM> 37.49
JM> -0.04
JM> select sum(txamount) from laytransactions where layid = 5
JM> -8.53483950180589E-16
JM> What the heck? Why isn't it zero???
In a sense, it is. :-)
JM> I am using FB 1.0. The txamount column is a double precision.
That's the source of your troubles. Double precision is not, err,
"precise".
JM> But, shouldn't summing a bunch of numbers with exactly 2 decimal places yield a
JM> result with two digits, and not some random very small number???
It should, but since you are using floating point numbers your numbers
do not have "exactly" 2 decimal places. Actually, their decimal point
floats. Search the archives for thorough discussions about this
problem.
Ciao
--
Nando mailto:nandod@...