Subject | Re: Strange things with double precision |
---|---|

Author | Adam |

Post date | 2005-12-23T22:28:13Z |

--- In firebird-support@yahoogroups.com, Sándor Tamás (HostWare Kft.)

<sandortamas@h...> wrote:

this line here

RES_DBL = INT_DBL / 10;

This is doing an integer division which truncates any decimal, which

may not be what you expect. Change it to

RES_DBL = INT_DBL / 10.0;

and see if that solves it.

Adam

<sandortamas@h...> wrote:

>the first

> I have created a SP:

>

> CREATE PROCEDURE TEST

> RETURNS (

> NUM_DEC DECIMAL(6,2),

> NUM_DBL DOUBLE PRECISION,

> NUM2_DEC DECIMAL(6,2),

> NUM2_DBL DOUBLE PRECISION,

> INT_DEC INTEGER,

> INT_DBL INTEGER,

> RES_DEC DECIMAL(6,2),

> RES_DBL DOUBLE PRECISION)

> AS

> begin

> NUM_DEC = 0.05;

> NUM_DBL = 0.05;

>

>

> while (NUM_DEC < 1000) do

> begin

> NUM2_DEC = (NUM_DEC * 10);

> NUM2_DBL = (NUM_DBL * 10);

>

> INT_DEC = cast(NUM2_DEC as integer);

> INT_DBL = cast(NUM2_DBL as integer);

>

> RES_DEC = INT_DEC / 10;

> RES_DBL = INT_DBL / 10;

>

> suspend;

>

> NUM_DEC = NUM_DEC + 0.1;

> NUM_DBL = NUM_DBL + 0.1;

> end

> end

>

> If I execute this, I get some strange results. (for example after

> 40-50 in double precision, FB misses some decimals).this?

>

> Can somebody explain how we went wrong, or what can we do to handle

>Perhaps you can state a value where it becomes a problem. My guess is

> Thanks,

> ST

>

this line here

RES_DBL = INT_DBL / 10;

This is doing an integer division which truncates any decimal, which

may not be what you expect. Change it to

RES_DBL = INT_DBL / 10.0;

and see if that solves it.

Adam