Subject | Re: [firebird-support] Re: casting to decimal(9,2) gives different answers if used with a group by clause |
---|---|
Author | Kjell Rilbe |
Post date | 2009-04-17T04:29:25Z |
wibbleian2 wrote:
that precision, then:
select cast(sum(cast(qty as decimal(11,4))) as decimal(9,2)) from tmp;
Kjell
--
--------------------------------------
Kjell Rilbe
DataDIA AB
E-post: kjell@...
Telefon: 08-761 06 55
Mobil: 0733-44 24 64
> --- In firebird-support@yahoogroups.comOK, so how many decimal places do the added terms require? 4? Cast to
> <mailto:firebird-support%40yahoogroups.com>, Kjell Rilbe
> <kjell.rilbe@...> wrote:
> >
> > Why the rounding errors differ between group and no group, I cannot say,
> > but it's not that strange.
> >
> This is the nub of the question!
>
> > What you should do is to cast to decimal BEFORE adding. Like this:
> >
> > select sum(cast(qty as decimal(9,2))) from tmp;
> >
>
> Unfortunately that would not give the correct result. Assume 12.125 +
> 12.125 - applying the rounding before summing would give 24.26 - rather
> than the required 24.25!
that precision, then:
select cast(sum(cast(qty as decimal(11,4))) as decimal(9,2)) from tmp;
Kjell
--
--------------------------------------
Kjell Rilbe
DataDIA AB
E-post: kjell@...
Telefon: 08-761 06 55
Mobil: 0733-44 24 64