| Subject | Re: [firebird-support] How can a pointer page vanish? |
|---|---|
| Author | Ann W. Harrison |
| Post date | 2007-10-09T15:19:53Z |
robertgilland wrote:
vanish. The computation that should have produced its sequence
number actually produced a number that doesn't correspond to
a pointer page according to the RDB$PAGES system table.
As everyone knows, records are identified internally by db_key,
which is 8 bytes for base tables and simple views and 8 bytes
per input stream for views. Of those eight bytes, ODS versions
before 11 used 32 bits to identify the table and 32 bits to
identify the record within the table. ODS 11 uses 24 bits for
tables (way more than necessary to identify the 64K tables
that can be defined for a database) and 40 for the record
number. Even 32 bits should be enough, one might think, to
hold a table with 4G of records. But one would be wrong.
The record number is made up of three parts. The first part
is the sequence number of a pointer page for the table. The
system table RDB$PAGES maps sequence numbers to physical page
numbers. A pointer page is a database page that contains an
array of page numbers of data pages. The second part is the
offset on the pointer page of the page number for the data
page that contains the record. The third part of the number
is an index into the "line index" on the data page that
contains the offset and length of the actual records.
The parts are put together like this: the first element is
the index into the line index. Added to it is the offset of
the page number on the pointer page times the maximum number
of records that could fit on a data page (rec-per-dp). Added
to that is the sequence number of the pointer page times the
maximum number of data pages on a pointer page (dp per pp)
times rec-per-dp.
For example, using a 4Kb page, the record that is number 34
on its data page, which is number 21 on its pointer page,
which has a sequence number of 15 would have a record number
of
34
21 * 239 (rec per dp) 5,019
15 * 956 (dp per pp) * 239 3,427,260
----------
3,432,313
The problem with the computation is that most records aren't
zero bytes long, so there are holes in the record number
sequence. The computation overflows a 32 bit record number
when a table contains about 40 billion bytes of data -
regardless of record size. A 40 bit record number work up
to about 15 terabytes of data. Partially full data pages
and data pages that have been released, leaving empty entries
in pointer pages also affect the result, though they're
ordinarily reused before new pages are added at the end.
You probably don't have 15 terabytes of data, so something
else went wrong. As Vlad suggested, IB_AID has tools that
can locate data pages and rebuild pointer pages, and that
may fix the database. But it would be interesting to know
what actually happened.
Regards,
Ann
> On one of our databases using Firebird 2.0.1.12855The error is a bit fanciful. The pointer page didn't actually
> with a database size 250GB we get the below error:
>
> internal gds software consistency check (pointer page vanished from
> DPM_next (249), file: dpm.cpp line: 1609)
>
> How can this happen and how can it be fixed?
>
vanish. The computation that should have produced its sequence
number actually produced a number that doesn't correspond to
a pointer page according to the RDB$PAGES system table.
As everyone knows, records are identified internally by db_key,
which is 8 bytes for base tables and simple views and 8 bytes
per input stream for views. Of those eight bytes, ODS versions
before 11 used 32 bits to identify the table and 32 bits to
identify the record within the table. ODS 11 uses 24 bits for
tables (way more than necessary to identify the 64K tables
that can be defined for a database) and 40 for the record
number. Even 32 bits should be enough, one might think, to
hold a table with 4G of records. But one would be wrong.
The record number is made up of three parts. The first part
is the sequence number of a pointer page for the table. The
system table RDB$PAGES maps sequence numbers to physical page
numbers. A pointer page is a database page that contains an
array of page numbers of data pages. The second part is the
offset on the pointer page of the page number for the data
page that contains the record. The third part of the number
is an index into the "line index" on the data page that
contains the offset and length of the actual records.
The parts are put together like this: the first element is
the index into the line index. Added to it is the offset of
the page number on the pointer page times the maximum number
of records that could fit on a data page (rec-per-dp). Added
to that is the sequence number of the pointer page times the
maximum number of data pages on a pointer page (dp per pp)
times rec-per-dp.
For example, using a 4Kb page, the record that is number 34
on its data page, which is number 21 on its pointer page,
which has a sequence number of 15 would have a record number
of
34
21 * 239 (rec per dp) 5,019
15 * 956 (dp per pp) * 239 3,427,260
----------
3,432,313
The problem with the computation is that most records aren't
zero bytes long, so there are holes in the record number
sequence. The computation overflows a 32 bit record number
when a table contains about 40 billion bytes of data -
regardless of record size. A 40 bit record number work up
to about 15 terabytes of data. Partially full data pages
and data pages that have been released, leaving empty entries
in pointer pages also affect the result, though they're
ordinarily reused before new pages are added at the end.
You probably don't have 15 terabytes of data, so something
else went wrong. As Vlad suggested, IB_AID has tools that
can locate data pages and rebuild pointer pages, and that
may fix the database. But it would be interesting to know
what actually happened.
Regards,
Ann