Subject | Rounding problems |
---|---|
Author | Pirtea Calin Iancu |
Post date | 2003-03-13T22:47:02Z |
I've run today into some rounding problems caused by converting
scaled intergers into real types just to do some + and - operations.
I had 500.000-499.995 equal to 0.0050000....00xxxx
comparing 0.005 with 500.000-499.995 was false.
So, considering the fact that I have precise numerics I would like
the int64 to be surfaced.
I'fe created a property called AsInt64Raw that returned the int64
value without using the scale.
This way I have integers 500000-499995=5 and I find this property
very usefull if I have to use the same fields sometimes for real types
math operations and sometimes for substraction or addition.
Opinions?
Best regards,
Application Developer
Calin Iancu, Pirtea
S.C. SoftScape S.R.L.
pcalin@...
scaled intergers into real types just to do some + and - operations.
I had 500.000-499.995 equal to 0.0050000....00xxxx
comparing 0.005 with 500.000-499.995 was false.
So, considering the fact that I have precise numerics I would like
the int64 to be surfaced.
I'fe created a property called AsInt64Raw that returned the int64
value without using the scale.
This way I have integers 500000-499995=5 and I find this property
very usefull if I have to use the same fields sometimes for real types
math operations and sometimes for substraction or addition.
Opinions?
Best regards,
Application Developer
Calin Iancu, Pirtea
S.C. SoftScape S.R.L.
pcalin@...