--- In firebird-support@yahoogroups.com, Sándor Tamás (HostWare Kft.)
<sandortamas@h...> wrote:

>
> I have created a SP:
>
> CREATE PROCEDURE TEST
> RETURNS (
> NUM_DEC DECIMAL(6,2),
> NUM_DBL DOUBLE PRECISION,
> NUM2_DEC DECIMAL(6,2),
> NUM2_DBL DOUBLE PRECISION,
> INT_DEC INTEGER,
> INT_DBL INTEGER,
> RES_DEC DECIMAL(6,2),
> RES_DBL DOUBLE PRECISION)
> AS
> begin
> NUM_DEC = 0.05;
> NUM_DBL = 0.05;
>
>
> while (NUM_DEC < 1000) do
> begin
> NUM2_DEC = (NUM_DEC * 10);
> NUM2_DBL = (NUM_DBL * 10);
>
> INT_DEC = cast(NUM2_DEC as integer);
> INT_DBL = cast(NUM2_DBL as integer);
>
> RES_DEC = INT_DEC / 10;
> RES_DBL = INT_DBL / 10;
>
> suspend;
>
> NUM_DEC = NUM_DEC + 0.1;
> NUM_DBL = NUM_DBL + 0.1;
> end
> end
>
> If I execute this, I get some strange results. (for example after

the first

> 40-50 in double precision, FB misses some decimals).
>
> Can somebody explain how we went wrong, or what can we do to handle

this?

>
> Thanks,
> ST
>

Perhaps you can state a value where it becomes a problem. My guess is
this line here

RES_DBL = INT_DBL / 10;

This is doing an integer division which truncates any decimal, which
may not be what you expect. Change it to

RES_DBL = INT_DBL / 10.0;

and see if that solves it.

Adam