Ann,

> -----Original Message-----
> From: Ann W. Harrison [mailto:aharrison@...]
> Sent: Thursday, November 16, 2006 6:24 PM
>
>
> Is the problem consistently "expected 7, found 5"? Never
> other numbers?

Yes, always "expected 7, found 5".

> The rest of your explanation sounds as if that's the case, since
> rebuilding indexes makes the problem go away. Do you have a copy of
> the corrupted database? Can you get a free copy of IBSurgeon and

run

> it to identify the corrupted index? Then at a minimum, you could
> reduce the cost to rebuilding that index instead of all indexes.

Yes. I tried running the Diagnose option of IBSurgeon last night.
After 3.5 hours, I had to cancel the operation so as not to interfere
with the backup/restore scripts that run at midnight.

I think gfix is faster..

> Is the problem more likely to occur when your appending data or when

> you're storing data in a table where you've just done some massive
> deletes? If you can identify the index where the problem occurs (if
> it is just one index) knowing how deep that index is might also be
> interesting. This seems to be a fairly low probability event, so
> hints help.

It seems that yesterdays problem related to a single table, which has
three indexes. As I said, I don't know which index caused the
problem, as I rebuilt all three. This particular table is the one
that I mentioned in yesterdays email, which had six massive deletes
followed by appending, each in six different transactions.

The strange thing about this is that after we finish loading, we run a
procedure which stores a summary of the information in another table
(as part of the same transaction). Each row of the newly inserted
data is visited during this summary process, so I would assume that
the data was originally stored without problem (the error did not
surface until the over night backup job).

In the past, there have been dozens of indexes affected (as reported
by gfix) on tables, some on tables that whose data is updated, some on
tables which are wiped clean and appended, and some on tables that do
not participate in the ETL at all.


One issue - I do not seem to be able to get the index statistics of
any database on this server. I ran "gstat -index [dbname] >
gstat.txt", and all I get is the header and the irritating
"unavailable database". When I ran this, there were no connections to
the db, and even the FB service was shut down. Any clue what the
problem may be?

thanks,

Bob M..