Subject Strange things with double precision
Author Sándor Tamás (HostWare Kft.)
I have created a SP:

CREATE PROCEDURE TEST
RETURNS (
NUM_DEC DECIMAL(6,2),
NUM_DBL DOUBLE PRECISION,
NUM2_DEC DECIMAL(6,2),
NUM2_DBL DOUBLE PRECISION,
INT_DEC INTEGER,
INT_DBL INTEGER,
RES_DEC DECIMAL(6,2),
RES_DBL DOUBLE PRECISION)
AS
begin
NUM_DEC = 0.05;
NUM_DBL = 0.05;


while (NUM_DEC < 1000) do
begin
NUM2_DEC = (NUM_DEC * 10);
NUM2_DBL = (NUM_DBL * 10);

INT_DEC = cast(NUM2_DEC as integer);
INT_DBL = cast(NUM2_DBL as integer);

RES_DEC = INT_DEC / 10;
RES_DBL = INT_DBL / 10;

suspend;

NUM_DEC = NUM_DEC + 0.1;
NUM_DBL = NUM_DBL + 0.1;
end
end

If I execute this, I get some strange results. (for example after the first
40-50 in double precision, FB misses some decimals).

Can somebody explain how we went wrong, or what can we do to handle this?

Thanks,
ST