Subject | Re: [ib-support] Wierd math??? |
---|---|

Author | Martijn Tonies |

Post date | 2003-02-20T20:44:30Z |

Hi,

floating point value) and computers? If not, use Google and read some.

Floats are aproximate values - anything can happen with them. Heck, if I

do: myfloatvar := 1.23 --> it can store anything _close_ to 1.23 ...

If you want precise math, use integers or other fixed point stuff, like

NUMERIC(10, 2) or something (dialect 3 dbs).

With regards,

Martijn Tonies

InterBase Workbench - the developer tool for InterBase & Firebird

Firebird Workbench - the developer tool for Firebird

Upscene Productions

http://www.upscene.com

"This is an object-oriented system.

If we change anything, the users object."

> Ok. Take the following query:Have you ever read some documentation on FLOATs (as a double prec. is a

>

> select txamount from laytransactions where layid = 5

>

> The result is:

>

> -7.34

> 7.34

> -29.36

> -8.09

> 37.49

> -0.04

>

> Now, I run the following query:

>

> select sum(txamount) from laytransactions where layid = 5

>

> The result is:

>

> -8.53483950180589E-16

>

> What the heck? Why isn't it zero???

>

> I am using FB 1.0. The txamount column is a double precision. But,

> shouldn't summing a bunch of numbers with exactly 2 decimal places yield a

> result with two digits, and not some random very small number???

floating point value) and computers? If not, use Google and read some.

Floats are aproximate values - anything can happen with them. Heck, if I

do: myfloatvar := 1.23 --> it can store anything _close_ to 1.23 ...

If you want precise math, use integers or other fixed point stuff, like

NUMERIC(10, 2) or something (dialect 3 dbs).

With regards,

Martijn Tonies

InterBase Workbench - the developer tool for InterBase & Firebird

Firebird Workbench - the developer tool for Firebird

Upscene Productions

http://www.upscene.com

"This is an object-oriented system.

If we change anything, the users object."