Subject Re: Store proc calc (rounding) fails
Author John B. Moore
--- In firebird-support@yahoogroups.com, Helen Borrie <helebor@t...>
wrote:

> >update TMP_FINANCE_CALC
> > set FINANCE_CHG = (CAST( ((BALANCE * :IN_RATE )*100) AS
INTEGER))/100;

> >update TMP_FINANCE_CALC
> > set FINANCE_CHG = (CAST( ((BALANCE * 0.018 )*100) AS INTEGER))/100;


> OK, first, I don't believe you got right results with this in
InterBase,
> unless it was IB 5.6. IB 6 and Firebird follow SQL standards re
integer
> division: integer/integer yields an integer result. So, in IB 5.6,
10/3
> would give you a double precision (3.333333...) while IB 6 and Firebird
> will give you 3.

Actually I did get the right results.. I did see that the divisor
should have been 100.00 and not the integer 100

The purpose of the calc is as follows..

input: 1.3333333

x 100 = 133.333

as Intgeger = 133

/100.00 = 1.33

True is is not a perfect rounding, but as far as I know it is all
that is available within a stored procedure.. (aside from creating a
UDF which is not up my alley..<sigh>)

>
> That's probably enough to get you to a solution. If you want decimal
> parts, you need to have them in one or other of the operands of a
> multiplication or division operation. Note also that casting a mixed
> number as integer will also truncate, not round.

Changing the calc to..

update TMP_FINANCE_CALC
set FINANCE_CHG = (CAST( ((BALANCE * :IN_RATE )*100) AS
INTEGER))/100.00;

..solved that problem.. (thought it still fails in debug..)


Thanks for you assistance and suggestions..

John..