Subject | Re: [firebird-support] Stored Procedure Question |
---|---|
Author | Svein Erling Tysvaer |
Post date | 2007-01-31T07:55:55Z |
DOUBLE PRECISION is a floating point data type, meaning that the numbers
stored are approximations. Hence, :TTime - :ONEMIN is likely to differ
from TickTime, the difference could well be only a few milliseconds, but
your comparison is for equality and e.g. 8:42:00 is not equal to
8:42:00.0003.
Rule of thumb: Avoid equality comparison for floating point data types.
In your case, that means that you should avoid
where TickTime=:TTime-:ONEMIN
and rather use somthing like
where TickTime between :TTime-:ONEMIN-:ONESEC and :TTime-:ONEMIN+ONESEC
(:ONESEC was used as an example, I think you could use some milliseconds
rather than a full second if you want to, but I don't know the precision
of the datatype).
DOUBLE PRECISION is very different from the two fixed decimal data types
DECIMAL and NUMERIC.
HTH,
Set
enseagui wrote:
stored are approximations. Hence, :TTime - :ONEMIN is likely to differ
from TickTime, the difference could well be only a few milliseconds, but
your comparison is for equality and e.g. 8:42:00 is not equal to
8:42:00.0003.
Rule of thumb: Avoid equality comparison for floating point data types.
In your case, that means that you should avoid
where TickTime=:TTime-:ONEMIN
and rather use somthing like
where TickTime between :TTime-:ONEMIN-:ONESEC and :TTime-:ONEMIN+ONESEC
(:ONESEC was used as an example, I think you could use some milliseconds
rather than a full second if you want to, but I don't know the precision
of the datatype).
DOUBLE PRECISION is very different from the two fixed decimal data types
DECIMAL and NUMERIC.
HTH,
Set
enseagui wrote:
> Here is a stored procedure that is not working as I would like.
> This SP is supposed to update the record with a TickTime one minute
> before. But actually it will not update.
>
> CREATE PROCEDURE CALCSPEED (
> SYMB Varchar(10) )
> AS
> DECLARE VARIABLE Price0 DOUBLE PRECISION;
> DECLARE VARIABLE Price1 DOUBLE PRECISION;
> DECLARE VARIABLE Price2 DOUBLE PRECISION;
> DECLARE VARIABLE S DOUBLE PRECISION;
> DECLARE VARIABLE ONEMIN DOUBLE PRECISION;
> DECLARE VARIABLE TTime TIMESTAMP;
> BEGIN
> ONEMIN = CAST(1 as DOUBLE PRECISION)/CAST(24 as DOUBLE
> PRECISION)/CAST(60 as DOUBLE PRECISION);
> FOR SELECT Close,TickTime FROM MINUTEDATA Where Symbol=:Symb Order
> by TickTime
> INTO :Price2,:TTime
> DO
> BEGIN
> IF (Price0 IS NOT NULL) THEN
> BEGIN
> S = Price2-Price0;
> UPDATE MINUTEDATA Set Speed=:S where
> TickTime=:TTime-:ONEMIN and Symbol=:Symb;
> END
> Price0 = Price1;
> Price1 = Price2;
> END
> END^
>
> This is how I solved the problem but still I would like to understand
> why the first version would not work. I am using FB embedded 1.5
>
> CREATE PROCEDURE CALCSPEED (
> SYMB Varchar(10) )
> AS
> DECLARE VARIABLE Price0 DOUBLE PRECISION;
> DECLARE VARIABLE Price1 DOUBLE PRECISION;
> DECLARE VARIABLE Price2 DOUBLE PRECISION;
> DECLARE VARIABLE S DOUBLE PRECISION;
> DECLARE VARIABLE ONEMIN DOUBLE PRECISION;
> DECLARE VARIABLE TTime TIMESTAMP;
> DECLARE VARIABLE TTime0 TIMESTAMP;
> BEGIN
> ONEMIN = CAST(1 as DOUBLE PRECISION)/CAST(24 as DOUBLE
> PRECISION)/CAST(60 as DOUBLE PRECISION);
> FOR SELECT Close,TickTime FROM MINUTEDATA Where Symbol=:Symb Order
> by TickTime
> INTO :Price2,:TTime
> DO
> BEGIN
> IF (Price0 IS NOT NULL) THEN
> BEGIN
> S = Price2-Price0;
> UPDATE MINUTEDATA Set Speed=:S where TickTime=:TTime0 and
> Symbol=:Symb;
> END
> Price0 = Price1;
> Price1 = Price2;
> TTime0 = TTime;
> END
> END^