Subject | Re: Store proc calc (rounding) fails |
---|---|
Author | John B. Moore |
Post date | 2005-10-31T19:35:29Z |
--- In firebird-support@yahoogroups.com, Helen Borrie <helebor@t...>
wrote:
should have been 100.00 and not the integer 100
The purpose of the calc is as follows..
input: 1.3333333
x 100 = 133.333
as Intgeger = 133
/100.00 = 1.33
True is is not a perfect rounding, but as far as I know it is all
that is available within a stored procedure.. (aside from creating a
UDF which is not up my alley..<sigh>)
update TMP_FINANCE_CALC
set FINANCE_CHG = (CAST( ((BALANCE * :IN_RATE )*100) AS
INTEGER))/100.00;
..solved that problem.. (thought it still fails in debug..)
Thanks for you assistance and suggestions..
John..
wrote:
> >update TMP_FINANCE_CALCINTEGER))/100;
> > set FINANCE_CHG = (CAST( ((BALANCE * :IN_RATE )*100) AS
> >update TMP_FINANCE_CALCInterBase,
> > set FINANCE_CHG = (CAST( ((BALANCE * 0.018 )*100) AS INTEGER))/100;
> OK, first, I don't believe you got right results with this in
> unless it was IB 5.6. IB 6 and Firebird follow SQL standards reinteger
> division: integer/integer yields an integer result. So, in IB 5.6,10/3
> would give you a double precision (3.333333...) while IB 6 and FirebirdActually I did get the right results.. I did see that the divisor
> will give you 3.
should have been 100.00 and not the integer 100
The purpose of the calc is as follows..
input: 1.3333333
x 100 = 133.333
as Intgeger = 133
/100.00 = 1.33
True is is not a perfect rounding, but as far as I know it is all
that is available within a stored procedure.. (aside from creating a
UDF which is not up my alley..<sigh>)
>Changing the calc to..
> That's probably enough to get you to a solution. If you want decimal
> parts, you need to have them in one or other of the operands of a
> multiplication or division operation. Note also that casting a mixed
> number as integer will also truncate, not round.
update TMP_FINANCE_CALC
set FINANCE_CHG = (CAST( ((BALANCE * :IN_RATE )*100) AS
INTEGER))/100.00;
..solved that problem.. (thought it still fails in debug..)
Thanks for you assistance and suggestions..
John..