Subject Re: [firebird-support] Numeric(18,4) calculations Geoff Worboys 2006-05-25T09:30:46Z
> Begin
> LTemp = (:OQty * :OUnitPrice);
> LTemp = LTemp * 1.14;
> OExtPriceInclVAT = LTemp;
> End;

> Ie. I do not more than one multiplication per line.

> Is there another simpler way of telling firebird that you
> don't care bout those extra decimal places, and would rather
> make use of the "space" on the left?

(Qty * UnitPrice * 1.14) as ExtValueInclVAT

and

LTemp = (:OQty * :OUnitPrice);
LTemp = LTemp * 1.14;
OExtPriceInclVAT = LTemp;

are actually quite different calculations when using scaled
integers. The first produces a result of probably 6 decimal
places and then rounds to two places. The second rounds each
interim calculation to two places - and the result may not be
the same with scaled integer types.

Example (with scaled integer/numeric types):
1.32 * 1.33 * 1.14 = 2.001384 :: rounded = 2.00

whereas
1.32 * 1.33 = 1.7556 :: rounded = 1.76
1.76 * 1.14 = 2.0064 :: rounded = 2.01

Note that circumstances can alter what is considered
appropriate. My often repeated example of the Australian tax
office - have quite explicit rules to indicate when rounding
is allowed to occur (usually at the end of a calculation).
There are however other situations where interim rounding is
exactly what you need. Interim rounding can, in some cases,
cause quite wide variation from the expected final result.

If you used double precision types the value would be much the
same with either approach (after explicit rounding at the end
of the calculation) depending on whether more than 15 digits of
precision was required to carry any of the interim results.

It is one of the things that makes the double precision type
desirable, its ability to carry precision (up to its maximum)
across multiple calculations - and to perform explicit rounding
based on your specific requirements, at the appropriate time.

--
Geoff Worboys
Telesis Computing