Subject | RE: [firebird-support] Precision in stored procedures |
---|---|
Author | Rick Debay |
Post date | 2012-03-07T20:16:52Z |
No one? My (probably unjustified) fear is that since the variables are
stored in a four byte primitive instead of a packed-decimal like format,
I may randomly run in to this.
-----Original Message-----
From: firebird-support@yahoogroups.com
[mailto:firebird-support@yahoogroups.com] On Behalf Of Rick Debay
Sent: Tuesday, March 06, 2012 12:08 PM
To: firebird-support@yahoogroups.com
Subject: [firebird-support] Precision in stored procedures
If a stored procedure has three variables
DECLARE VARIABLE A NUMERIC(18,3);
DECLARE VARIABLE B NUMERIC(18,3);
DECLARE VARIABLE C DOUBLE PRECISION;
And a function is evaluated with these values
A = B / C;
A = 7.3 / 0.839080459770115
Then the value stored in A will be 8.7 not 8.70000000000119 (a firebird
stored procedure debugger) or 8.6999999999999994041095890410959 (windows
7 calculator), correct?
Disclaimer: This message (including attachments) is confidential and may be privileged. If you have received it by mistake please notify the sender by return e-mail and delete this message from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. RxStrategies, Inc. shall not be liable for the improper or incomplete transmission of the information contained in this communication or for any delay in its receipt or damage to your system. RxStrategies, Inc. does not guarantee that the integrity of this communication has been maintained nor that this communication is free from viruses, interceptions or interference.
stored in a four byte primitive instead of a packed-decimal like format,
I may randomly run in to this.
-----Original Message-----
From: firebird-support@yahoogroups.com
[mailto:firebird-support@yahoogroups.com] On Behalf Of Rick Debay
Sent: Tuesday, March 06, 2012 12:08 PM
To: firebird-support@yahoogroups.com
Subject: [firebird-support] Precision in stored procedures
If a stored procedure has three variables
DECLARE VARIABLE A NUMERIC(18,3);
DECLARE VARIABLE B NUMERIC(18,3);
DECLARE VARIABLE C DOUBLE PRECISION;
And a function is evaluated with these values
A = B / C;
A = 7.3 / 0.839080459770115
Then the value stored in A will be 8.7 not 8.70000000000119 (a firebird
stored procedure debugger) or 8.6999999999999994041095890410959 (windows
7 calculator), correct?
Disclaimer: This message (including attachments) is confidential and may be privileged. If you have received it by mistake please notify the sender by return e-mail and delete this message from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. RxStrategies, Inc. shall not be liable for the improper or incomplete transmission of the information contained in this communication or for any delay in its receipt or damage to your system. RxStrategies, Inc. does not guarantee that the integrity of this communication has been maintained nor that this communication is free from viruses, interceptions or interference.