Mpu Gondrong wrote:

>
> I've made a simple test:
>
> CREATE PROCEDURE SP_TEST_2 (PAR SMALLINT)
> returns (RES DOUBLE PRECISION)
> AS
> declare variable D Double precision;
> begin
> D = Par * 3.14159 + 5.4321 * 0.318;
> Res = D * 0.5 + 0.12345 * 4.15678;
> end
>
> And I compare the result with 'calc' in Windows, and it's the same.
> Maybe it will help if you give a working example i can work on.

The problem arises when division is carried out on integers.

For intance,

select (3 / 10) * 3.333 from rdb$database;

=====================

0.000

will produce a result of 0 due to internal rounding of the integer
division.

If the values are presented as floats then the expected result will occur:

select (3.0 / 10.0) * 3.333 from rdb$database;

=====================

0.99990


In Andrew Ferguson's case the problem arises because he is doing
calculations on the fly which return integers prior to his doing the
division. If the values were already stored in the database as floats he
wouldn't have come across the problem.


Paul
--

Paul Reeves
http://www.ibphoenix.com
Supporting users of Firebird and InterBase