Subject | Arithmetically Challenged - need quick answer if possible |
---|---|
Author | Kevin Stanton |
Post date | 2004-04-20T20:54:36Z |
Greetings,
I've got a CFO waiting for me....
Not getting along well with the changes in FB 1.03 regarding simple
arithmetic.
Division is killing me. Whoever thought/thinks dividing one integer into
another gives an integer result has got to be nuts (ok, maybe it's me that's
nuts :)
2 things:
1) trying to divide a simple percetage by 100:
pct = 33.33 / 100
where 33.33 is a varibale defined as: numeric(5,2). 100 is constant.
pct is a varibale defined as: numeric(6,5). (I've tried
different things here).
I cannot get it into a variable as .3333. Comes out as .333299994468689
2) then I want to take that pct (once it's correct) and multiply that by a
currency
391.06. I get an integer overflow on this one!
Complete proc follows:
create procedure sp_krs_test_b
as
declare variable bpct numeric(5,2);
declare variable spct numeric(5,2);
declare variable pct numeric(6,5);
/*declare variable bmargin numeric(18,2); tried this w/ no luck so now I'm
just guessing below */
declare variable bmargin double precision;
begin
/* Procedure Text */
Execute Procedure sp_Get_Tdr_Comp_Pcts_N ('01', '99')
Returning_Values :bpct, :spct;
pct = bpct / cast(100 as numeric(6,2));
bmargin = (391.06 * pct);
suspend;
end
Any help will be GREATLY appreciated.
Kevin
I've got a CFO waiting for me....
Not getting along well with the changes in FB 1.03 regarding simple
arithmetic.
Division is killing me. Whoever thought/thinks dividing one integer into
another gives an integer result has got to be nuts (ok, maybe it's me that's
nuts :)
2 things:
1) trying to divide a simple percetage by 100:
pct = 33.33 / 100
where 33.33 is a varibale defined as: numeric(5,2). 100 is constant.
pct is a varibale defined as: numeric(6,5). (I've tried
different things here).
I cannot get it into a variable as .3333. Comes out as .333299994468689
2) then I want to take that pct (once it's correct) and multiply that by a
currency
391.06. I get an integer overflow on this one!
Complete proc follows:
create procedure sp_krs_test_b
as
declare variable bpct numeric(5,2);
declare variable spct numeric(5,2);
declare variable pct numeric(6,5);
/*declare variable bmargin numeric(18,2); tried this w/ no luck so now I'm
just guessing below */
declare variable bmargin double precision;
begin
/* Procedure Text */
Execute Procedure sp_Get_Tdr_Comp_Pcts_N ('01', '99')
Returning_Values :bpct, :spct;
pct = bpct / cast(100 as numeric(6,2));
bmargin = (391.06 * pct);
suspend;
end
Any help will be GREATLY appreciated.
Kevin