Subject | Re: Problem with floating point precission |
---|---|
Author | Alejandro |
Post date | 2004-07-29T20:31:17Z |
--- In firebird-support@yahoogroups.com, kokou Fostin <atahman@y...> wrote:=
select cast (7 as decimal (8, 3)) / 800.0 ...
or
select 7.00000 / 800.00000 ...
give more precision. I'm using 1.5.1 in windows.
?
> --- Alejandro <amosteo@u...> a écrit : > Hi!Strange. Either
> >
> > I'm trying to do some floating point operations in a
> > select, but
> > there aren't table columns involved (at least not
> > for certain
> > results).
> >
> > Say, for example that I have:
> >
> > select 7.0 / 800.0, x.Other_Data
> > from A_Table x;
> >
> > My first column is returning 0, because only two
> > decimals are being
> > used and result is below that. I've checked it with
> > larger values and
> > in fact the maximum precision I get is 0.01
> >
> > Is there some parameter or configuration option
> > which allows for
> > increased precision?
> >
> > Thanks in advance!
> >
> >
> When i execute your example, I get 0.00875 as result.
> Perhaps you need to use CAST(column as Decimal(8,3))
select cast (7 as decimal (8, 3)) / 800.0 ...
or
select 7.00000 / 800.00000 ...
give more precision. I'm using 1.5.1 in windows.
?