Subject Re: [firebird-support] Firebird Roundoff Error
Author Paul Vinkenoog
Hi Niegil,

> 2) Now when this same query is run in firebird
>
> select ((2536.68/30)*2.5) as RND from TestTable.Then value that i
> get is 211.375
> i tried the same query changing it as select ((2536.68/30)*2.5)
> as RND from TestTable got the same value.

In both cases, Firebird first computes 2536.68/30

Firebird sees the constant 2536.68 as an integer with scale -2, and
the constant 30 as an integer with scale 0.

The result is an integer with scale -2.

The correct answer is 84.556, but because of the scale and the fact
that integer division results are rounded toward 0 ("truncated"),
Firebird registers the result as 84.55

After that, 84.55 is multiplied by 2.5. The result is an integer with
scale -3: 211.375

> Now when i change the query as
> select 2536.68*2.5/30 as RND from TestTable or select
> ((2536.68*2.5)/30) as RND from TestTable,
> i get the same value that paradox returns (211.39)

Because now, the first operation is 2536.68*2.5

This makes the first result an integer with scale -3, so there are no
rounding losses.

Explicit typecasting or selecting (2536.680 / 30) * 2.5
- or (2536.68 / 30.0) * 2.5 -
will also give you the right result.


Greetings,
Paul Vinkenoog