Subject | Re: [firebird-support] Rounding |
---|---|
Author | Helen Borrie |
Post date | 2004-09-16T23:45:32Z |
At 10:53 AM 17/09/2004 +1200, you wrote:
precision of 7 and double precision types with a precision of 15. Float
types can take numbers with up to 9 signficant digits without
overflowing; double precision up to 18. Under the hood, all numbers are
stored as integers. Mixed numbers have extra attributes stored relating to
their data type and precision.
If you try to store 1.126 into a fixed precision type (numeric or decimal)
that is declared with a scale of 2, you will get an overflow exception.
If you cast a floating type as a fixed type, truncation occurs, so .-
CAST (1.126 AS NUMERIC(15,2)) will return 1.12. If you want to round a FP
number before casting it, you can use a UDF.
If possible, it's a rule of thumb to "store things you count as fixed
precision and store things you measure as floating point." However, with
any mixed numbers, there are traps which ever way you go. Geoff Worboys
has done an interesting explication of the issues in this
paper: http://www.ibobjects.com/TI_Numerics.rtf
./heLen
>HiFirebird doesn't do any rounding at all. It stores float types with a
>
>Just wondering how the default rounding works in firebird. If I have a
>field that is to 2 decimal places and I try and store a value like 1.126
>does Firebird truncate (1.12) or Round (1.13) the value?
>If firebird rounds does it use simple (>= 0.005 then round up) or other?
precision of 7 and double precision types with a precision of 15. Float
types can take numbers with up to 9 signficant digits without
overflowing; double precision up to 18. Under the hood, all numbers are
stored as integers. Mixed numbers have extra attributes stored relating to
their data type and precision.
If you try to store 1.126 into a fixed precision type (numeric or decimal)
that is declared with a scale of 2, you will get an overflow exception.
If you cast a floating type as a fixed type, truncation occurs, so .-
CAST (1.126 AS NUMERIC(15,2)) will return 1.12. If you want to round a FP
number before casting it, you can use a UDF.
If possible, it's a rule of thumb to "store things you count as fixed
precision and store things you measure as floating point." However, with
any mixed numbers, there are traps which ever way you go. Geoff Worboys
has done an interesting explication of the issues in this
paper: http://www.ibobjects.com/TI_Numerics.rtf
./heLen