Subject | Re: [firebird-support] Firebird Roundoff Error |
---|---|
Author | Paul Vinkenoog |
Post date | 2007-01-22T16:05:44Z |
Hi Niegil,
Firebird sees the constant 2536.68 as an integer with scale -2, and
the constant 30 as an integer with scale 0.
The result is an integer with scale -2.
The correct answer is 84.556, but because of the scale and the fact
that integer division results are rounded toward 0 ("truncated"),
Firebird registers the result as 84.55
After that, 84.55 is multiplied by 2.5. The result is an integer with
scale -3: 211.375
This makes the first result an integer with scale -3, so there are no
rounding losses.
Explicit typecasting or selecting (2536.680 / 30) * 2.5
- or (2536.68 / 30.0) * 2.5 -
will also give you the right result.
Greetings,
Paul Vinkenoog
> 2) Now when this same query is run in firebirdIn both cases, Firebird first computes 2536.68/30
>
> select ((2536.68/30)*2.5) as RND from TestTable.Then value that i
> get is 211.375
> i tried the same query changing it as select ((2536.68/30)*2.5)
> as RND from TestTable got the same value.
Firebird sees the constant 2536.68 as an integer with scale -2, and
the constant 30 as an integer with scale 0.
The result is an integer with scale -2.
The correct answer is 84.556, but because of the scale and the fact
that integer division results are rounded toward 0 ("truncated"),
Firebird registers the result as 84.55
After that, 84.55 is multiplied by 2.5. The result is an integer with
scale -3: 211.375
> Now when i change the query asBecause now, the first operation is 2536.68*2.5
> select 2536.68*2.5/30 as RND from TestTable or select
> ((2536.68*2.5)/30) as RND from TestTable,
> i get the same value that paradox returns (211.39)
This makes the first result an integer with scale -3, so there are no
rounding losses.
Explicit typecasting or selecting (2536.680 / 30) * 2.5
- or (2536.68 / 30.0) * 2.5 -
will also give you the right result.
Greetings,
Paul Vinkenoog