Subject | Truncation and rounding 'feature' in SP |
---|---|
Author | Rick Debay |
Post date | 2005-05-26T18:57:05Z |
DECLARE VARIABLE UNIT_CNT INTEGER;
DECLARE VARIABLE OUTSTANDING DECIMAL(12,3);
DECLARE VARIABLE NDC_QTY DECIMAL(12,3);
OUTSTANDING = 180;
NDC_QTY = 100;
UNIT_CNT = CAST(OUTSTANDING / NDC_QTY AS INTEGER);
UNIT_CNT is returned as 2, so FB is rounding when it should be
truncating.
SELECT CAST(180 / 100 AS INTEGER) from rdb$database
Returns 1, which is correct.
Is there any workaround, besides using select into statements to do all
my math?!
DECLARE VARIABLE OUTSTANDING DECIMAL(12,3);
DECLARE VARIABLE NDC_QTY DECIMAL(12,3);
OUTSTANDING = 180;
NDC_QTY = 100;
UNIT_CNT = CAST(OUTSTANDING / NDC_QTY AS INTEGER);
UNIT_CNT is returned as 2, so FB is rounding when it should be
truncating.
SELECT CAST(180 / 100 AS INTEGER) from rdb$database
Returns 1, which is correct.
Is there any workaround, besides using select into statements to do all
my math?!