Subject | Re: [firebird-support] SP float results rounded strangely (dbExpress) |
---|---|
Author | Geoff Worboys |
Post date | 2005-09-06T08:51:44Z |
> Via my Delphi+dbExpress program (and also EMS IB/FB Manager)"float" (as per the various documentation available) is a 32bit
> this procedure gives
> 0.666666686534882
> but I think it should rather give
> 0,666666666666667
> isql gives
> 0.66666669
> so it seems to be client issue. How big is float in Firebird?
> Any recommendations how to avoid this incorrect rounding in
> dbExpress?
float value - significant to only 7 digits - which is what you
can expect from C++ "float" data types and "single" in Pascal.
So 0.666666686534882 is accurate to 7 digits.
The fact that EMS or Delphi/dbExpress may be showing you more
digits than are significant is a problem for them.
If you want higher precision cast to DOUBLE PRECISION which is
a 64bit floating value - 15 digits.
For other datatype see the Interbase 6 documents available at:
http://www.firebirdsql.org/index.php?op=doc&id=userdoc
Even better get Helen's book:
http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_firebird_book
--
Geoff Worboys
Telesis Computing