Subject | double precision rounding problem |
---|---|
Author | Sindu |
Post date | 2001-05-25T04:30:10Z |
Hi all,
Sorry if this might be an FAQ, but I thought (the manual says so)
that double precision has 15 digits precision.
I'm using Borland open source IB 6.01 on Win9x.
My question is how do you guys handle this large numerics before IB v 6.0?
I can't use IB 6's large numerics yet, because I have lots of app using
BDE already (and BDE doesn't support it yet :() and I'd like a DB
independence connectivity.
regards,
Sindu
ps. well the sample below is not actually a large numerics yet.
SQL> select 0.1 - 0.01 from rdb$database;
=====================
0.09
SQL> select cast(0.1 as double precision) - cast(0.01 as double precision)
CON> from rdb$database;
=======================
0.09000000000000001
SQL> create table test (f1 double precision, f2 double precision);
SQL> commit;
SQL> insert into test (f1) values (0.1);
SQL> commit;
SQL> select * from test;
F1 F2
======================= =======================
0.1000000000000000 <null>
SQL> update test set f2 = f1 - 0.01;
SQL> commit;
SQL> select * from test;
F1 F2
======================= =======================
0.1000000000000000 0.09000000000000001
Sorry if this might be an FAQ, but I thought (the manual says so)
that double precision has 15 digits precision.
I'm using Borland open source IB 6.01 on Win9x.
My question is how do you guys handle this large numerics before IB v 6.0?
I can't use IB 6's large numerics yet, because I have lots of app using
BDE already (and BDE doesn't support it yet :() and I'd like a DB
independence connectivity.
regards,
Sindu
ps. well the sample below is not actually a large numerics yet.
SQL> select 0.1 - 0.01 from rdb$database;
=====================
0.09
SQL> select cast(0.1 as double precision) - cast(0.01 as double precision)
CON> from rdb$database;
=======================
0.09000000000000001
SQL> create table test (f1 double precision, f2 double precision);
SQL> commit;
SQL> insert into test (f1) values (0.1);
SQL> commit;
SQL> select * from test;
F1 F2
======================= =======================
0.1000000000000000 <null>
SQL> update test set f2 = f1 - 0.01;
SQL> commit;
SQL> select * from test;
F1 F2
======================= =======================
0.1000000000000000 0.09000000000000001