Subject | Re: [firebird-support] Numeric(18,4) calculations |
---|---|

Author | Geoff Worboys |

Post date | 2006-05-25T09:30:46Z |

> BeginPlease understand that

> LTemp = (:OQty * :OUnitPrice);

> LTemp = LTemp * 1.14;

> OExtPriceInclVAT = LTemp;

> End;

> Ie. I do not more than one multiplication per line.

> Is there another simpler way of telling firebird that you

> don't care bout those extra decimal places, and would rather

> make use of the "space" on the left?

(Qty * UnitPrice * 1.14) as ExtValueInclVAT

and

LTemp = (:OQty * :OUnitPrice);

LTemp = LTemp * 1.14;

OExtPriceInclVAT = LTemp;

are actually quite different calculations when using scaled

integers. The first produces a result of probably 6 decimal

places and then rounds to two places. The second rounds each

interim calculation to two places - and the result may not be

the same with scaled integer types.

Example (with scaled integer/numeric types):

1.32 * 1.33 * 1.14 = 2.001384 :: rounded = 2.00

whereas

1.32 * 1.33 = 1.7556 :: rounded = 1.76

1.76 * 1.14 = 2.0064 :: rounded = 2.01

Note that circumstances can alter what is considered

appropriate. My often repeated example of the Australian tax

office - have quite explicit rules to indicate when rounding

is allowed to occur (usually at the end of a calculation).

There are however other situations where interim rounding is

exactly what you need. Interim rounding can, in some cases,

cause quite wide variation from the expected final result.

If you used double precision types the value would be much the

same with either approach (after explicit rounding at the end

of the calculation) depending on whether more than 15 digits of

precision was required to carry any of the interim results.

It is one of the things that makes the double precision type

desirable, its ability to carry precision (up to its maximum)

across multiple calculations - and to perform explicit rounding

based on your specific requirements, at the appropriate time.

--

Geoff Worboys

Telesis Computing