Subject | Re: Problem with floating point precission |
---|---|
Author | Alejandro |
Post date | 2004-07-30T07:25:58Z |
--- In firebird-support@yahoogroups.com, Helen Borrie <helebor@t...>
wrote:
in the terms you describe.
wrote:
> At 08:31 PM 29/07/2004 +0000, you wrote:the
> >--- In firebird-support@yahoogroups.com, kokou Fostin <atahman@y...
> wrote:=
> >
> >
> > > --- Alejandro <amosteo@u...> a écrit : > Hi!
> > > >
> > > > I'm trying to do some floating point operations in a
> > > > select, but
> > > > there aren't table columns involved (at least not
> > > > for certain
> > > > results).
> > > >
> > > > Say, for example that I have:
> > > >
> > > > select 7.0 / 800.0, x.Other_Data
> > > > from A_Table x;
> > > >
> > > > My first column is returning 0, because only two
> > > > decimals are being
> > > > used and result is below that. I've checked it with
> > > > larger values and
> > > > in fact the maximum precision I get is 0.01
> > > >
> > > > Is there some parameter or configuration option
> > > > which allows for
> > > > increased precision?
> > > >
> > > > Thanks in advance!
> > > >
> > > >
> > > When i execute your example, I get 0.00875 as result.
> > > Perhaps you need to use CAST(column as Decimal(8,3))
> >
> >Strange. Either
> >
> >select cast (7 as decimal (8, 3)) / 800.0 ...
> >
> >or
> >
> >select 7.00000 / 800.00000 ...
> >
> >give more precision. I'm using 1.5.1 in windows.
>
> The decimal type is fixed precision. With fixed precision numbers,
> scale of the result of division or multiplication is the sum of thescales
> of the operands. You will run out of precision once you have 18route to
> significant digits in the result - scale > 19 ----> BOOM!
>
> If you don't know what the final scale will be, either correct en
> the result, or use double precision for your casting, or use acombination.
>Thanks Helen, is enough for me to know that I'll get fixed precision
> /heLen
in the terms you describe.