Subject | Re: [firebird-support] Integer Overflow |
---|---|
Author | Helen Borrie |
Post date | 2004-03-19T12:01:44Z |
At 05:53 PM 19/03/2004 +0700, you wrote:
the scale of both operands. Your calculation produces a result with a
scale of 16 - moving from left to right that's 4 + 2 + 2 + 4 + 4 - hence
the overflow.
Two possible solutions: use double precision numbers for the discount
factors and then cast your result; or reduce their scale to 0 (integer),
then round your result, then divide afterwards to move the scale back, and
cast the result.
/heLen
>When I do this SELECT:With fixed precision numbers, each multiplication or division aggregates
>
>SELECT sum( a.discount * b.price * b.quantity * ( 1 - b.discount2 ) * ( 1 -
>b.discount3 )) FROM a, b ......
>
>Discount, discount2, and discount3 - decimal(8,4)
>Price and Quantity - decimal(15,2)
>
>I found this error: "Integer overflow. The result of an integer operation
>caused the most significant bit of the result to carry."
>
>How to solve it?
the scale of both operands. Your calculation produces a result with a
scale of 16 - moving from left to right that's 4 + 2 + 2 + 4 + 4 - hence
the overflow.
Two possible solutions: use double precision numbers for the discount
factors and then cast your result; or reduce their scale to 0 (integer),
then round your result, then divide afterwards to move the scale back, and
cast the result.
/heLen