Subject | RE: [firebird-support] Operations |
---|---|
Author | Svein Erling Tysvær |
Post date | 2008-09-10T06:59:38Z |
Things are done in different order, and each substep have to fit the correct type definition:
1.00/241.63 will give a result with four numbers after the decimal point - i.e. 0.0041. Multiplied by 838.10 and rounded to fit into NUMERIC(10,2), this gives 3.44.
Doing things the other way and multiplying first, the intermediate result still has four numbers after the decimal point. However 838.1000 is exact where 0.0041 was truncated (or rounded, I sometimes mix these things). Then dividing by 241.63 and rounding to two decimals gives 3.47.
To get 3.47 when starting with division, you need a higher precision - try using 1.0000 rather than 1.00 to get both calculations to return the same result (they may still differ by 0.01 in certain cases, in general I would recommend the second calculation you used).
HTH,
Set
-----Original Message-----
From: firebird-support@yahoogroups.com [mailto:firebird-support@yahoogroups.com] On Behalf Of "Szabó Sándor (Süni)"
Sent: 9. september 2008 19:44
To: Firebird news
Subject: [firebird-support] Operations
Hi All!
I'd like to know why the following two expression give back different
results:
NETTO_EGYSEGAR_DEVIZA=CAST((1.00/:RATIO*NETTO_EGYSEGAR) AS NUMERIC(10,2));
NETTO_EGYSEGAR_DEVIZA=CAST((NETTO_EGYSEGAR*1.00/:RATIO) AS NUMERIC(10,2));
All the variables has the NUMERIC(10,2) type. The RATIO variable has the
value 241.63.
NETTO_EGYSEGAR has the value 838,10.
In the first case the result is 3.44, in the second 3.47.
TIA:Alex :-)
1.00/241.63 will give a result with four numbers after the decimal point - i.e. 0.0041. Multiplied by 838.10 and rounded to fit into NUMERIC(10,2), this gives 3.44.
Doing things the other way and multiplying first, the intermediate result still has four numbers after the decimal point. However 838.1000 is exact where 0.0041 was truncated (or rounded, I sometimes mix these things). Then dividing by 241.63 and rounding to two decimals gives 3.47.
To get 3.47 when starting with division, you need a higher precision - try using 1.0000 rather than 1.00 to get both calculations to return the same result (they may still differ by 0.01 in certain cases, in general I would recommend the second calculation you used).
HTH,
Set
-----Original Message-----
From: firebird-support@yahoogroups.com [mailto:firebird-support@yahoogroups.com] On Behalf Of "Szabó Sándor (Süni)"
Sent: 9. september 2008 19:44
To: Firebird news
Subject: [firebird-support] Operations
Hi All!
I'd like to know why the following two expression give back different
results:
NETTO_EGYSEGAR_DEVIZA=CAST((1.00/:RATIO*NETTO_EGYSEGAR) AS NUMERIC(10,2));
NETTO_EGYSEGAR_DEVIZA=CAST((NETTO_EGYSEGAR*1.00/:RATIO) AS NUMERIC(10,2));
All the variables has the NUMERIC(10,2) type. The RATIO variable has the
value 241.63.
NETTO_EGYSEGAR has the value 838,10.
In the first case the result is 3.44, in the second 3.47.
TIA:Alex :-)