Subject | Re: IB6 BUG #221589: numeric fields and mathematical operations |
---|---|
Author | lobolo2000 |
Post date | 2001-08-23T15:39:29Z |
Thanx Svein.
I had nearly lost hope in getting a reply to my post.
SELECT 10.5/100 FROM RDB$DATABASE gave 0.105 (and not 0.1) as I would
expect. But anyway, this query is a little bit different than the one in my
post as 0.1 is not an intermediate result. Having intermediate results with
a low precision would be awful. I would expect from any decent expression
evaluator to use doubles or better for intermediate results. And my version
of IB (6 open source) is consistent in this respect.
Am I missing something?
Regards
I had nearly lost hope in getting a reply to my post.
SELECT 10.5/100 FROM RDB$DATABASE gave 0.105 (and not 0.1) as I would
expect. But anyway, this query is a little bit different than the one in my
post as 0.1 is not an intermediate result. Having intermediate results with
a low precision would be awful. I would expect from any decent expression
evaluator to use doubles or better for intermediate results. And my version
of IB (6 open source) is consistent in this respect.
Am I missing something?
Regards
----- Original Message -----
Date: Thu, 23 Aug 2001 09:02:42 +0200
From: Svein Erling Tysvær <svein.erling.tysvaer@...>
Subject: Re: IB6 BUG #221589: numeric fields and mathematical operations
This is due to IB rounding intermediate values, and is actually visible even
if you reduce your
statement to
SELECT 10.5/100 FROM rdb$database, which gives 0.1 as the result.
To get the correct result, you could simply change this statement to
SELECT 10.5/100.00 FROM rdb$database, which will give you the correct number
of decimals (0.105).
HTH,
Set
_________________________________________________________
Do You Yahoo!?
Get your free @... address at http://mail.yahoo.com