Subject | Re: [firebird-support] Re: Problem with floating point precission |
---|---|
Author | Helen Borrie |
Post date | 2004-07-29T22:53:40Z |
At 08:31 PM 29/07/2004 +0000, you wrote:
scale of the result of division or multiplication is the sum of the scales
of the operands. You will run out of precision once you have 18
significant digits in the result - scale > 19 ----> BOOM!
If you don't know what the final scale will be, either correct en route to
the result, or use double precision for your casting, or use a combination.
/heLen
>--- In firebird-support@yahoogroups.com, kokou Fostin <atahman@y...> wrote:=The decimal type is fixed precision. With fixed precision numbers, the
>
>
> > --- Alejandro <amosteo@u...> a écrit : > Hi!
> > >
> > > I'm trying to do some floating point operations in a
> > > select, but
> > > there aren't table columns involved (at least not
> > > for certain
> > > results).
> > >
> > > Say, for example that I have:
> > >
> > > select 7.0 / 800.0, x.Other_Data
> > > from A_Table x;
> > >
> > > My first column is returning 0, because only two
> > > decimals are being
> > > used and result is below that. I've checked it with
> > > larger values and
> > > in fact the maximum precision I get is 0.01
> > >
> > > Is there some parameter or configuration option
> > > which allows for
> > > increased precision?
> > >
> > > Thanks in advance!
> > >
> > >
> > When i execute your example, I get 0.00875 as result.
> > Perhaps you need to use CAST(column as Decimal(8,3))
>
>Strange. Either
>
>select cast (7 as decimal (8, 3)) / 800.0 ...
>
>or
>
>select 7.00000 / 800.00000 ...
>
>give more precision. I'm using 1.5.1 in windows.
scale of the result of division or multiplication is the sum of the scales
of the operands. You will run out of precision once you have 18
significant digits in the result - scale > 19 ----> BOOM!
If you don't know what the final scale will be, either correct en route to
the result, or use double precision for your casting, or use a combination.
/heLen