Subject | 32 bit vs 64 bit arthmetic |
---|---|
Author | Tim Ward |
Post date | 2016-01-18T10:00:37Z |
So I've got this stored procedure which takes DECIMAL(18,6) parameters
and returns DECIMAL(18,12) results, and internally copies stuff into and
out of DOUBLE PRECISION variables which it uses to call a UDF to do the
actual calculations.
And the results are different (in the 8th decimal place or so) between
running on 32 bit and 64 bit systems.
I would expect that Firebird is going to do exactly the same thing on
both, down to the last bit, as the data types are defined to be what
they are and nothing to do with the native word length of the machine?
So I've got to look at the UDF for the difference?
(Which I'd not expect to find there either, actually, *if* the code is
the same, as IEEE arithmetic also shouldn't change just because the
lengths of integers changes. So I suspect that what I'm actually going
to be looking for is a difference in the code in the two different
versions of the library called by the UDF.)
--
Tim Ward
and returns DECIMAL(18,12) results, and internally copies stuff into and
out of DOUBLE PRECISION variables which it uses to call a UDF to do the
actual calculations.
And the results are different (in the 8th decimal place or so) between
running on 32 bit and 64 bit systems.
I would expect that Firebird is going to do exactly the same thing on
both, down to the last bit, as the data types are defined to be what
they are and nothing to do with the native word length of the machine?
So I've got to look at the UDF for the difference?
(Which I'd not expect to find there either, actually, *if* the code is
the same, as IEEE arithmetic also shouldn't change just because the
lengths of integers changes. So I suspect that what I'm actually going
to be looking for is a difference in the code in the two different
versions of the library called by the UDF.)
--
Tim Ward