Subject | Re: [firebird-support] Truncation and rounding 'feature' in SP |
---|---|
Author | Helen Borrie |
Post date | 2005-05-27T00:32:41Z |
At 02:57 PM 26/05/2005 -0400, you wrote:
depends a lot on the data types of the operands. Your two examples are
not equivalent. In your first example, a decimal was divided by a decimal,
which returns a decimal (180.000 / 100.000 = 1.800000). Casting a decimal
to an integer does a banker's rounding and returns 2.
In the second, an integer is divided by an integer. By SQL rules,
integer/integer division truncates any decimal portion from the result and
returns the integer portion alone. So the results you got were correct in
both cases.
Your solution is to be aware of how the arithmetic works for each data type
and use casting accordingly.
For example, if you *want* to truncate the result of decimal/decimal
division, don't try to cast it as an integer. Use some mechanism, e.g. the
UDF trunc(), to get the result you want.
./heLen
>DECLARE VARIABLE UNIT_CNT INTEGER;Workaround? In SQL, you get what you ask for. What you need to ask for
>DECLARE VARIABLE OUTSTANDING DECIMAL(12,3);
>DECLARE VARIABLE NDC_QTY DECIMAL(12,3);
>
>OUTSTANDING = 180;
>NDC_QTY = 100;
>
>UNIT_CNT = CAST(OUTSTANDING / NDC_QTY AS INTEGER);
>
>UNIT_CNT is returned as 2, so FB is rounding when it should be
>truncating.
>
>SELECT CAST(180 / 100 AS INTEGER) from rdb$database
>
>Returns 1, which is correct.
>
>Is there any workaround, besides using select into statements to do all
>my math?!
depends a lot on the data types of the operands. Your two examples are
not equivalent. In your first example, a decimal was divided by a decimal,
which returns a decimal (180.000 / 100.000 = 1.800000). Casting a decimal
to an integer does a banker's rounding and returns 2.
In the second, an integer is divided by an integer. By SQL rules,
integer/integer division truncates any decimal portion from the result and
returns the integer portion alone. So the results you got were correct in
both cases.
Your solution is to be aware of how the arithmetic works for each data type
and use casting accordingly.
For example, if you *want* to truncate the result of decimal/decimal
division, don't try to cast it as an integer. Use some mechanism, e.g. the
UDF trunc(), to get the result you want.
./heLen