Subject Re: [firebird-support] rounding error in computed expression in query
Author Milan Babuskov
vincent_kwinsey wrote:
> I have the following test case (all mentioned fields are defined as
> double precision)

Say no more. That type should be called "double inprecision" since it is
imprecise "as designed". If you want exact arithmetics, use decimal or
numeric.

> (in SP or trigger, e.g.) - should I make non-clean code and put
> rounding in every place or it is possible to choose more appropriate
> datatype to avoid this? Anyway numeric(...,...) is using double
> precision.

No, it does not.

> I have heard about some bug in some processors' FPU, can this be
> attributed to this strange behavior?

No, double is defined by IEEE standard 754. It's design is such that you
can fit into some bits much more that could go there if it were 100%
exact (something like JPEG compression for images), so it is imprecise:

http://en.wikipedia.org/wiki/IEEE_754


--
Milan Babuskov
http://swoes.blogspot.com/
http://www.flamerobin.org