Subject | Strange things with double precision |
---|---|
Author | Sándor Tamás (HostWare Kft.) |
Post date | 2005-12-23T13:13:02Z |
I have created a SP:
CREATE PROCEDURE TEST
RETURNS (
NUM_DEC DECIMAL(6,2),
NUM_DBL DOUBLE PRECISION,
NUM2_DEC DECIMAL(6,2),
NUM2_DBL DOUBLE PRECISION,
INT_DEC INTEGER,
INT_DBL INTEGER,
RES_DEC DECIMAL(6,2),
RES_DBL DOUBLE PRECISION)
AS
begin
NUM_DEC = 0.05;
NUM_DBL = 0.05;
while (NUM_DEC < 1000) do
begin
NUM2_DEC = (NUM_DEC * 10);
NUM2_DBL = (NUM_DBL * 10);
INT_DEC = cast(NUM2_DEC as integer);
INT_DBL = cast(NUM2_DBL as integer);
RES_DEC = INT_DEC / 10;
RES_DBL = INT_DBL / 10;
suspend;
NUM_DEC = NUM_DEC + 0.1;
NUM_DBL = NUM_DBL + 0.1;
end
end
If I execute this, I get some strange results. (for example after the first
40-50 in double precision, FB misses some decimals).
Can somebody explain how we went wrong, or what can we do to handle this?
Thanks,
ST
CREATE PROCEDURE TEST
RETURNS (
NUM_DEC DECIMAL(6,2),
NUM_DBL DOUBLE PRECISION,
NUM2_DEC DECIMAL(6,2),
NUM2_DBL DOUBLE PRECISION,
INT_DEC INTEGER,
INT_DBL INTEGER,
RES_DEC DECIMAL(6,2),
RES_DBL DOUBLE PRECISION)
AS
begin
NUM_DEC = 0.05;
NUM_DBL = 0.05;
while (NUM_DEC < 1000) do
begin
NUM2_DEC = (NUM_DEC * 10);
NUM2_DBL = (NUM_DBL * 10);
INT_DEC = cast(NUM2_DEC as integer);
INT_DBL = cast(NUM2_DBL as integer);
RES_DEC = INT_DEC / 10;
RES_DBL = INT_DBL / 10;
suspend;
NUM_DEC = NUM_DEC + 0.1;
NUM_DBL = NUM_DBL + 0.1;
end
end
If I execute this, I get some strange results. (for example after the first
40-50 in double precision, FB misses some decimals).
Can somebody explain how we went wrong, or what can we do to handle this?
Thanks,
ST