Subject | Why does FB evaluate numbers this way? |
---|---|

Author | Paul R. Gardner |

Post date | 2006-06-23T14:03:22Z |

select (3/2)

from rdb$database

The result should be 3/2 = 1.5. Instead I get 1. Why?

select (3.0/2.0)

from rdb$database

The result is finally 1.50. Wht not just 1.5?

I understand what it's doing with significant digits, but it don't

understand why. In the original example, I had 2. This is the same as

2.0, 2.00, or 2.00000000000000000000000000. It's not like the tenth

through the billionth decimal place holders are NULL and therefore could

be anything, they are assumed as 0 because they were not specified. 2

is 2 and not 2.01 or 2.0000000000000000000000001. To me, the answer

should have been 1.5.

OK, so let's say I get past my issue on the significant digits. Why was

the answer 1? It should have been rounded to 2 under any rounding

convention. Typically x.5 is rounded to x+1. In bankers rounding you

round towards the even number which is the same result: 2.

Any thoughts?

Paul

[Non-text portions of this message have been removed]

from rdb$database

The result should be 3/2 = 1.5. Instead I get 1. Why?

select (3.0/2.0)

from rdb$database

The result is finally 1.50. Wht not just 1.5?

I understand what it's doing with significant digits, but it don't

understand why. In the original example, I had 2. This is the same as

2.0, 2.00, or 2.00000000000000000000000000. It's not like the tenth

through the billionth decimal place holders are NULL and therefore could

be anything, they are assumed as 0 because they were not specified. 2

is 2 and not 2.01 or 2.0000000000000000000000001. To me, the answer

should have been 1.5.

OK, so let's say I get past my issue on the significant digits. Why was

the answer 1? It should have been rounded to 2 under any rounding

convention. Typically x.5 is rounded to x+1. In bankers rounding you

round towards the even number which is the same result: 2.

Any thoughts?

Paul

[Non-text portions of this message have been removed]