Subject | Re: [firebird-support] Arithmatic Overflow in FB 1.0 |
---|---|
Author | Ivan Prenosil |
Post date | 2004-07-14T17:19:49Z |
Multiplying and dividing produce result with scale equal to
sum of the scales of the operands. In your case
You have to put rounding somewhere ...
Ivan
sum of the scales of the operands. In your case
> dBegPrice = ((dPrice * dPercent) / dConvert);the scale will be 21! which is too much even for int64.
You have to put rounding somewhere ...
Ivan
----- Original Message -----
From: "sgharp" <steve@...>
To: <firebird-support@yahoogroups.com>
Sent: Wednesday, July 14, 2004 6:20 PM
Subject: [firebird-support] Arithmatic Overflow in FB 1.0
> Hi All,
>
> I'm getting an error
>
> "Arithmetic overflow or division by zero has occurred.
> arithmetic exception, numeric overflow, or string truncation."
>
> in a stored procedure. I've been able to duplicate it in the
> following.
>
> create procedure spTest
> AS
> declare variable dPercent Numeric(16,7);
> declare variable dConvert Numeric(16,7);
> declare variable dPrice Numeric(16,7);
> declare variable dBegPrice Numeric(16,7);
> Begin
> dPrice = 4;
> dPercent = .42;
> dConvert = 30;
> dBegPrice = ((dPrice * dPercent) / dConvert);
>
> suspend;
> End
>
> Why is this a problem? When I define my Numerics as (12,3) instead
> of (16,7) the problem goes away but I need a greater precision.
>
> I'm not sure whether this is an FB error or is coming from
> IBExpert. I caused an access violation in IBExpert that made me use
> task manager to end the process.
>
> Thanks,
> Steve