Subject | Re: [firebird-support] rounding error in computed expression in query |
---|---|
Author | Milan Babuskov |
Post date | 2006-10-06T20:44:41Z |
vincent_kwinsey wrote:
imprecise "as designed". If you want exact arithmetics, use decimal or
numeric.
can fit into some bits much more that could go there if it were 100%
exact (something like JPEG compression for images), so it is imprecise:
http://en.wikipedia.org/wiki/IEEE_754
--
Milan Babuskov
http://swoes.blogspot.com/
http://www.flamerobin.org
> I have the following test case (all mentioned fields are defined asSay no more. That type should be called "double inprecision" since it is
> double precision)
imprecise "as designed". If you want exact arithmetics, use decimal or
numeric.
> (in SP or trigger, e.g.) - should I make non-clean code and putNo, it does not.
> rounding in every place or it is possible to choose more appropriate
> datatype to avoid this? Anyway numeric(...,...) is using double
> precision.
> I have heard about some bug in some processors' FPU, can this beNo, double is defined by IEEE standard 754. It's design is such that you
> attributed to this strange behavior?
can fit into some bits much more that could go there if it were 100%
exact (something like JPEG compression for images), so it is imprecise:
http://en.wikipedia.org/wiki/IEEE_754
--
Milan Babuskov
http://swoes.blogspot.com/
http://www.flamerobin.org