Subject RE: [firebird-support] Problem with division
Author Kevin Stanton
Thanks Helen.
Next ?:
How would I then take that PCT field and multiply it by the total amount
being prorated and assign it to a money field AMT numeric(18,2)?
I've tried:
AMT = Pct * 1700.00

But I get the error:
"unsuccessful execution caused by system error that does not preclude
successful execution of subsequent statements.
Integer overflow. The result of the integer operation caused the most
significant bit of the result to carry."

and I just tried (FC = 1700.00):
AMT = Cast((FC * Pct) as Numeric(18,2));

Thanks again,
Kevin


-----Original Message-----
From: Helen Borrie [mailto:helebor@...]
Sent: Wednesday, February 04, 2004 10:01 PM
To: firebird-support@yahoogroups.com
Subject: Re: [firebird-support] Problem with division


At 09:36 PM 4/02/2004 -0800, you wrote:
>Greetings,
>I'm using FB 1.03 and have the following SP:
>
>create procedure sp_test_pct
>returns (Pct Numeric(8,6))
>as
>begin
> Pct = Cast((29920 / 106532) as Numeric(8,6) );
> suspend;
>end
>
>Right now, Pct is returned as all zeros and it should be something like:
>0.280854
>
>I've tried w/out the casting as well with the same zero results.
>
>Any ideas would be GREATLY appreciated.


Integer/integer division returns an integer in ODS 10, so the result is
"correct" by SQL rules. If you want a decimal portion in the result, you
have to include it in one or both of the operands, e.g.
Pct = Cast(( (1.000 * 29920) / (1.000 * 106532)) as Numeric(8,6) );

/hb


Yahoo! Groups Sponsor
ADVERTISEMENT





----------------------------------------------------------------------------
--
Yahoo! Groups Links

a.. To visit your group on the web, go to:
http://groups.yahoo.com/group/firebird-support/

b.. To unsubscribe from this group, send an email to:
firebird-support-unsubscribe@yahoogroups.com

c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.



[Non-text portions of this message have been removed]