On Apr 5, 2007, at 20:21, Helen Borrie wrote:

> Hmmm, something is amiss there that you don't know
> about. RDB$FIELD_PRECISION should never be negative for a fixed
> numeric type. A decimal(15,4) in dialect 3 would have precision of
> 18; in D1 it would be a DOUBLE PRECISION, since D1 doesn't support
> fixed numerics with > 9 digits of precision. Ask for a metadata-only
> backup and see what's really going on (including the
> dialect). However, restoring the metadata-only backup won't give the
> answer if it's an ODS9 database that has been restored to ODS 10 or
> higher...you'd need the data, since it's likely that the restore has
> simply written the precision for those doubles as -1, due to their
> being neither one thing nor the other under the higher ODS.

Sadly, the tester cannot share their data - and even if they could, I
can't fix their schema. But I seem to have figured out how to deal
with it...

> Note, too, that the XSQLVAR.sqlscale member returns RDB$FIELD_SCALE,
> not RDB$FIELD_PRECISION...

Yes, and it turns out that even if sqlscale is set, it is only
relevant for integer return types. Floating point ones have already
been scaled. I was assuming that they had not e.g.

if ( x.sqlscale ) {
if ( int type ) scale integer
else if (float type )scale float
}

Removing the float type case fixed the problem.
________________________________________________________
Richard Wesley Senior Software Developer Tableau
Software
Visit: http://www.trytableau.com/now.html