Subject Re: [firebird-support] Re: Problem with floating point precission
Author Helen Borrie
At 08:31 PM 29/07/2004 +0000, you wrote:
>--- In firebird-support@yahoogroups.com, kokou Fostin <atahman@y...> wrote:=
>
>
> > --- Alejandro <amosteo@u...> a écrit : > Hi!
> > >
> > > I'm trying to do some floating point operations in a
> > > select, but
> > > there aren't table columns involved (at least not
> > > for certain
> > > results).
> > >
> > > Say, for example that I have:
> > >
> > > select 7.0 / 800.0, x.Other_Data
> > > from A_Table x;
> > >
> > > My first column is returning 0, because only two
> > > decimals are being
> > > used and result is below that. I've checked it with
> > > larger values and
> > > in fact the maximum precision I get is 0.01
> > >
> > > Is there some parameter or configuration option
> > > which allows for
> > > increased precision?
> > >
> > > Thanks in advance!
> > >
> > >
> > When i execute your example, I get 0.00875 as result.
> > Perhaps you need to use CAST(column as Decimal(8,3))
>
>Strange. Either
>
>select cast (7 as decimal (8, 3)) / 800.0 ...
>
>or
>
>select 7.00000 / 800.00000 ...
>
>give more precision. I'm using 1.5.1 in windows.

The decimal type is fixed precision. With fixed precision numbers, the
scale of the result of division or multiplication is the sum of the scales
of the operands. You will run out of precision once you have 18
significant digits in the result - scale > 19 ----> BOOM!

If you don't know what the final scale will be, either correct en route to
the result, or use double precision for your casting, or use a combination.

/heLen