Subject Re: [firebird-support] Re: casting to decimal(9,2) gives different answers if used with a group by clause
Author Kjell Rilbe
wibbleian2 wrote:
> --- In firebird-support@yahoogroups.com
> <mailto:firebird-support%40yahoogroups.com>, Kjell Rilbe
> <kjell.rilbe@...> wrote:
> >
> > Why the rounding errors differ between group and no group, I cannot say,
> > but it's not that strange.
> >
> This is the nub of the question!
>
> > What you should do is to cast to decimal BEFORE adding. Like this:
> >
> > select sum(cast(qty as decimal(9,2))) from tmp;
> >
>
> Unfortunately that would not give the correct result. Assume 12.125 +
> 12.125 - applying the rounding before summing would give 24.26 - rather
> than the required 24.25!

OK, so how many decimal places do the added terms require? 4? Cast to
that precision, then:

select cast(sum(cast(qty as decimal(11,4))) as decimal(9,2)) from tmp;

Kjell
--
--------------------------------------
Kjell Rilbe
DataDIA AB
E-post: kjell@...
Telefon: 08-761 06 55
Mobil: 0733-44 24 64