Subject | Re: [firebird-support] Why does FB evaluate numbers this way? |
---|---|

Author | Helen Borrie |

Post date | 2006-06-23T16:25:02Z |

At 12:24 AM 24/06/2006, you wrote:

integer result. There are no "rounding rules": the answer is

categorically only the integer part. All in accordance with the SQL standard.

types, the rule has nothing to do with which is "larger". These are

fixed (or more correctly, scaled) types. the result of division and

multiplication yields a subdecimal part that is the sum of the scales

of all participating numbers. (And INTEGER is a scaled type with a

scale of 0).

So - 4.5/2 gives 2.2, while 4.5/2.0 gives 2.25.

Floating point numbers are different again. If 4.5 is a float or

double precision type, then 4.5/2 gives 2.25. However, float 4.3/2

gives 2.15000009536743, albeit the last 8 digits must be discarded

because the precision of float is good only for the first 7

significant figures.

./heLen

> > select (3/2)The real reason that 3/2 = 1 is that integer/integer must give an

> > from rdb$database

> >

> > The result should be 3/2 = 1.5. Instead I get 1. Why?

>

>3 is an integer, 2 is an integer. You are performing an integer

>division.

integer result. There are no "rounding rules": the answer is

categorically only the integer part. All in accordance with the SQL standard.

>select (3.0/2) will also bring up 1.5It might be true by observation but for the DECIMAL and NUMERIC

>A float or decimal type is a "larger" type than integer and the larger

>type will determine the type of the operation.

types, the rule has nothing to do with which is "larger". These are

fixed (or more correctly, scaled) types. the result of division and

multiplication yields a subdecimal part that is the sum of the scales

of all participating numbers. (And INTEGER is a scaled type with a

scale of 0).

So - 4.5/2 gives 2.2, while 4.5/2.0 gives 2.25.

Floating point numbers are different again. If 4.5 is a float or

double precision type, then 4.5/2 gives 2.25. However, float 4.3/2

gives 2.15000009536743, albeit the last 8 digits must be discarded

because the precision of float is good only for the first 7

significant figures.

./heLen