Subject | RE: [firebird-support] Problem with division |
---|---|
Author | Kevin Stanton |
Post date | 2004-02-05T07:14:34Z |
Thanks again Helen. I think I've got it now.
Kevin
-----Original Message-----
From: Helen Borrie [mailto:helebor@...]
Sent: Wednesday, February 04, 2004 10:54 PM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] Problem with division
At 10:14 PM 4/02/2004 -0800, you wrote:
or divide, the scales of the operands are added together and the powers of
10 stack up. You have 18 digits in total to work with - when you go over
that, you're busted.
So work out whether you need to keep pushing the scale higher and higher.
It doesn't make much sense for "things you count". If these are
high-precision calculations on measurements (rather than bean-counts) you
should be using double precision numbers.
/heLen
[Non-text portions of this message have been removed]
Kevin
-----Original Message-----
From: Helen Borrie [mailto:helebor@...]
Sent: Wednesday, February 04, 2004 10:54 PM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] Problem with division
At 10:14 PM 4/02/2004 -0800, you wrote:
>Thanks Helen.You need to keep control of the scale of these numbers. When you multiply
>Next ?:
>How would I then take that PCT field and multiply it by the total amount
>being prorated and assign it to a money field AMT numeric(18,2)?
>I've tried:
>AMT = Pct * 1700.00
>
>But I get the error:
>"unsuccessful execution caused by system error that does not preclude
>successful execution of subsequent statements.
>Integer overflow. The result of the integer operation caused the most
>significant bit of the result to carry."
>
>and I just tried (FC = 1700.00):
>AMT = Cast((FC * Pct) as Numeric(18,2));
or divide, the scales of the operands are added together and the powers of
10 stack up. You have 18 digits in total to work with - when you go over
that, you're busted.
So work out whether you need to keep pushing the scale higher and higher.
It doesn't make much sense for "things you count". If these are
high-precision calculations on measurements (rather than bean-counts) you
should be using double precision numbers.
/heLen
[Non-text portions of this message have been removed]