Ann W. Harrison wrote:

>At 02:25 AM 4/1/2004, Jonathan Neve wrote:
>
>
>>>It is. That's what careful write is about. And we have it.
>>>
>>>
>>>
>>>
>>Sorry, but what exactly is a careful write? Is that what you were
>>describing above.
>>
>>
>
>"Careful write" is an architecture that follows a few reasonably simple
>rules for on disk layout and more much less simple rules that govern the
>order in which pages are written. The major disk layout rule is that no
>structure that can be written on two pages can depend on two-way
>pointers. For example, a record version has a pointer to the previous
>version. The previous version does not have a pointer forward to its
>parent. The page that contains the old version can be written first. If
>the page that contains the parent is never written, the database is still
>consistent, though a small amount of storage will be lost until reclaimed
>with gfix or a backup/restore.
>
>
>

Ok.

>A counter example are index nodes which have pointers both left and right -
>pathetic attempt at a diagram:
>
> 0-first page on level -> <- second page on level -> <-third page on
>level ->
>
>The right hand pointers are reliable. If you follow the right hand pointer
>from the first node on a level to the second, the page returned will be the
>next page in that level of the indeex. The left hand pointers are not
>reliable. If you follow a left hand pointer from the third page on a level
>back to the second, you may get the second page, or you may get some other
>page. There is code to handle the case where a left hand pointer is not valid.
>
>

Yes, I see. The right hand ones are valid due to the order they are
written in. But in that case, if I understand correctly, the left-hand
pointers are to be avoided...?

>Operationally, "careful write" is a question of maintaining a dependency
>graph among pages in cache. When changes to a page could cause a loop in
>the graph, writing some number of pages will break the loop. Those page
>must be written before the original change can be made. When a page must
>be written, all the pages it depends on must be written first.
>
>

I see. So the order of the physical write is important.

>> Besides, FireBird isn't 100% corruption-proof, so...
>>
>>
>
>That's the design intention. Bugs and asynchronous writes are the exception.
>
>
>
>>>The worst that happens is that you get some orphaned back versions, no in
>>>use. Or if the problem happens in some structural area, you may get an
>>>orphaned page. Anything else is a bug. Plain, simple, ugly bug.
>>>
>>>
>>>
>>Well, in that case, what is corruption due to? Why can the database
>>sometimes get corrupted upon power failure or system crash? It may not
>>happen often, and it may not be likely to happen, but it does happen...
>>
>>
>
>The primary causes of corruption in InterBase databases were two bugs in
>5.6. One caused the database to write over the front of its file when the
>file size exceeded 4GB. Messy. The other allowed two connections to
>access the same database file without locking against each other.
>
>Every corrupt Firebird database I've seen was running with forced writes off.
>
>Yes, there are ways to get "logical" corruption, like nulls in indexes that
>disallow nulls and records that violate constraints. That's a different
>issue, and one which we should take up seriously. Some of those problems
>occur because the system doesn't validate existing data when adding a
>constraint. Others are probably bugs.
>

(Just as a side-comment, shouldn't the former case also be considered a
sort of bug, or at least something that needs to be corrected?)

>But at the physical level,
>maintaining consistency is just a question of following the rules about on
>disk pointers and maintaining the dependency graph correctly.
>
>

I see.

>The reason I got upset last night - aside from a bunch of stuff that had
>nothing to do with your message - is that I have written a lot about this
>topic. It's available on IBPhoenix and in the archives of this
>list. Consistency on disk is a major feature of Firebird/InterBase and has
>been for going on 20 years. Having someone posit that it might be possible
>and ask if anyone has considered it ruffles my fur.
>
>

I understand. But please be patient with newcomers! :-) I've been using
Interbase for about 3/4 years, with nearly all our customers. I now know
Interbase rather well (from the user perspective), and, although
corruption is rare, it has happened to me. Therefore, without knowing
the inner workings of the database, I imagined what might have been the
cause. It turns out that most if not all of the corruption I've know was
probably due to other factors or bugs. So actually, if I understand
correctly, _any_ corruption in Interbase should be considered rather a
bug than merely a misfortunate happening, am I right? I always had the
attitude that a certain risk of corruption was inevitable. If by design
Interbase should be incorruptable (so to speak!), then shouldn't all
corruptions be reported and investigated? Even logical corruptions ought
to be made impossible, if we can.

>>I'm not talking about the theory of "careful write", I'm talking about
>>the practice of database corruption. I don't know exactly what it's due
>>to, and I was just imagining what might be the cause. If the cause I
>>suggested is not actually the real cause, that still doesn't mean that
>>the problem doesn't exist.
>>
>>
>
>Have you actually corrupted a Firebird database that had forced writes
>on?
>

Yes, but perhaps only "logical" corruption. I'm not sure whether it was
due to physical incoherence or not.

>Has anyone on the list done so? If so, you may have critical
>information that could lead to finding and correcting a bug.
>
>
>
>>...the only solution is to kill the
>>
>>
>>>>server process (killing the client isn't enough).>>
>>>>
>>>>
>>>Sure. The server dies, the server is reborn, long live the server. And
>>>the database.
>>>
>>>
>
>Sorry. Firebird is designed to allow you to kill the server as brutally as
>you can and still have a consistent database when you reconnect the disk to
>something running Firebird. I've got a significant number of logs that
>show the server dying horrible deaths, over and over. The guardian
>restarts the server and the database comes up undamaged.
>
>

I see now. This is the way it's designed. If ever a corruption is found
in spite of that, it ought to get reported.

>>>>If the SP was
>>>>performing database writes, and not mere reads, it's very likely to
>>>>corrupt the database, for obvious reasons (this is what happened in my
>>>>case).
>>>>
>>>>
>>To me of course. If it's not obvious to you, then my assumptions must
>>have been wrong. But that doesn't make the problem disappear. It just
>>means that we don't know why it happened.
>>
>>
>
>If you've got the database log from the time of the corruption, I'd like to
>see it.
>
>

I'll see if I still have it. Unfortunately, I might not, because it
happened on my development machine, and since that time, I've had some
trouble with it, such that this log was probably lost. But for the
future, where does this file reside? In the Interbase installation
directory?

>>>But if you let the operating system choose
>>>the order of page writes all bets are off.
>>>
>>>
>>>
>>>
>>Well, I guess my understanding of the structure is lacking, because I
>>can't quite follow the above reasoning. I'll look up the documents you
>>refered to, and perhaps I'll see things more clearly.
>>
>>
>
>I hope I've made the problem with random write order more clear.
>
>

Yes, I undertand the problem now. However, there are file systems (such
as log-structured file systems) that garantee the write-order, without
performing synchronous writes. In other words, due to the structure of
the FS, the order of asynchronous writes is garantees, data blocks in
the FS are written in an atomic way. So on such systems, we could
probably turn ForcedWrites off, without affecting stability, and yet
increasing performance. Do you agree?

>>Anyway, if the structure is such that it makes this sort of thing
>>impossible, couldn't be structure be altered? :-)
>>
>>
>
>I don't see any way to avoid inter-page dependencies...
>
>

Yes, I see. What you're saying isn't that there is a certain inherent
and inevitable risk of corruption that we can't do anything about, but
rather, that if writes aren't garanteed to be performed in the right
order (due to a FS cache), there is an inevitable risk.

Thanks a lot!

Jonathan Neve.



[Non-text portions of this message have been removed]