| Subject | Re: [IB-Architect] Faster DB Backups, WAS Super-transactions and Incremental Backup |
|---|---|
| Author | dcalford |
| Post date | 2000-06-21T17:44:55Z |
Hi Sean,
If you have the disk space, you can perform somthing similar to what you
want right now.
Step 1, create a shadow file for the database.
This will create a new database on the disk, but, it will be populated by
IB using a page level copy - so that page 165 on the source database is
physically identical to page 165 on the shadow database. You need to give
the server time to create the shadow, but, once it is fully completed, IB
will continue to keep the two databases matching.
Step 2, shut down IB
Get your users to log out or set up a time out period so that there is a
point when you know all the changes to the database is finished.
Shutdown the database so that noone can connect and then rename the shadow
file. (First file in a multi-file gdb)
Restart IB.
IB will complain in the log that the shadow does not exist. But, as shadows
were designed to handle the loss of the main db or a shadow db, the end
users will not know anything about this.
Step 3, activate the shadow as a independant gdb
look in the manual for gfix and follow the instructions.
This creates a physical snapshot of the database from the moment that the db
is shut down.
If you have enough drive space, you can keep multiple shadows online and
rotate the setting up of the snapshots.
ie, take off a snapshot every 4 hours.
This works regardless of the physical size of the database because, once
created, the shadow is fulling up to date with the main database.
Therefore there is no time taken in creating a backup or restoring it.
As long as you are adding new shadows (and giving them time to get properly
created) you can continously do this.
THIS IS NOT A PERFECT METHOD - one, the fact that you need to shutdown the
server and bring it backup takes at least a minute. Combine this with the
fact that any corruption in the main database is automatically in all the
shadows. Also, the shadow must exist on the local server - you cannot have
it on a remote share.
It is a tool, and can be improved.
I would extend the shadow command with a shadow disconnect vs shadow drop.
The disconnect would then think of itself as a transaction and only transfer
information to the shadow that is commited at the time of the disconnect.
Once all the information that the transaction can see is finished being
applied, then the shadow file is removed from the database definition, but
is not removed from the disk (the drop shadow command removes the files)
Some form of event should fire to let the developer know that the shadow is
fully created and another event to let the developer know that a shadow is
fully disconnected (or values in a table).
This would be a form of physical database copy, under the control of the
server, so that it is both fast, and safe for data.
It would be useless for streaming backup devices like tapes but, drives are
cheap these days.
I would be nice if the shadow was allowed to be created on a remote share,
but, I am sure the technical problems involved in this would make that sort
of enhancement a very long term project.
best regards
Dalton
"Leyne, Sean" wrote:
If you have the disk space, you can perform somthing similar to what you
want right now.
Step 1, create a shadow file for the database.
This will create a new database on the disk, but, it will be populated by
IB using a page level copy - so that page 165 on the source database is
physically identical to page 165 on the shadow database. You need to give
the server time to create the shadow, but, once it is fully completed, IB
will continue to keep the two databases matching.
Step 2, shut down IB
Get your users to log out or set up a time out period so that there is a
point when you know all the changes to the database is finished.
Shutdown the database so that noone can connect and then rename the shadow
file. (First file in a multi-file gdb)
Restart IB.
IB will complain in the log that the shadow does not exist. But, as shadows
were designed to handle the loss of the main db or a shadow db, the end
users will not know anything about this.
Step 3, activate the shadow as a independant gdb
look in the manual for gfix and follow the instructions.
This creates a physical snapshot of the database from the moment that the db
is shut down.
If you have enough drive space, you can keep multiple shadows online and
rotate the setting up of the snapshots.
ie, take off a snapshot every 4 hours.
This works regardless of the physical size of the database because, once
created, the shadow is fulling up to date with the main database.
Therefore there is no time taken in creating a backup or restoring it.
As long as you are adding new shadows (and giving them time to get properly
created) you can continously do this.
THIS IS NOT A PERFECT METHOD - one, the fact that you need to shutdown the
server and bring it backup takes at least a minute. Combine this with the
fact that any corruption in the main database is automatically in all the
shadows. Also, the shadow must exist on the local server - you cannot have
it on a remote share.
It is a tool, and can be improved.
I would extend the shadow command with a shadow disconnect vs shadow drop.
The disconnect would then think of itself as a transaction and only transfer
information to the shadow that is commited at the time of the disconnect.
Once all the information that the transaction can see is finished being
applied, then the shadow file is removed from the database definition, but
is not removed from the disk (the drop shadow command removes the files)
Some form of event should fire to let the developer know that the shadow is
fully created and another event to let the developer know that a shadow is
fully disconnected (or values in a table).
This would be a form of physical database copy, under the control of the
server, so that it is both fast, and safe for data.
It would be useless for streaming backup devices like tapes but, drives are
cheap these days.
I would be nice if the shadow was allowed to be created on a remote share,
but, I am sure the technical problems involved in this would make that sort
of enhancement a very long term project.
best regards
Dalton
"Leyne, Sean" wrote:
> Markus,
>
> Thanks for the comments.
>
> But along with improving the performance of the existing restore
> process, I, quite honestly, want to see if the whole backup/restore
> approach can be re-thought.
>
> What I'm thinking of is something like a DB "image" so that a "snapshot"
> can be taken and then restored without the "need" to rebuild the
> indexes.
>
> Disk space is now cheap so that storing the indexes in the backup should
> not be an "expensive". This approach could significantly increase the
> speed of a restore.
>
> But the, hopefull, goal of the "image" backup and restore (which would
> NOT replace GBAK for ODS changes or transportable backups) would be
> stream the database pages to disk on a block/page by block/page basis
> rather than the current row by row. Each page would be "cleaned" by the
> engine to only contain data in the 'context' of the backup
> transaction/snap shot.
>
> This approach would require adding a completely new set of functions to
> the engine.
>
> It's an idea, not completely baked right now but hopefully something
> worth further discussion / brainstorming.
>
> Sean
>
> > -----Original Message-----
> > From: Markus Kemper [mailto:mkemper@...]
> > Sent: Wednesday, June 21, 2000 11:44 AM
> > To: IB-Architect@egroups.com
> > Subject: Re: [IB-Architect] Faster DB Backups, WAS Super-transactions
> > and Incremental Backup
> >
> >
> >
> > Sean,
> >
> > I am sure that others know more about the internals of
> > GBAK than I do but, one of the things that I believe
> > could greatly increase the performance of a restore
> > would be the ability to create indexes in parallel.
> >
> > Today, index creation on a restore takes a long time
> > if you have many and the database is large. Part of
> > the reason why is that if you have multiple indexes
> > on a table gbak does multiple full table scans ( one
> > per index per table ) to create the index. If gbak
> > restore could be taught to scan the table once and
> > build all indexes for ( a table ) in parallel the time
> > to restore would come down.
> >
> > Also, there are likely performance improvements possible
> > in the remote or loopback protocol layer. As seen by
> > Greg Deatz's post recently, his large database took
> > ~6 hrs to restore on a big Sun machine running 6.0
> > SuperServer and ~3 hrs to restore on a smaller Linux
> > system running 6.0 Classic. Classic has 'true' local
> > access where as SuperServer ( non-Windows ) goes though
> > local loopback to simulate a local connection.
> >
> > My hope is that some day ( and I don't think we're all
> > that far away from it ) an InterBase database will be
> > self maintaining and stable enough to be able to for-go
> > the restore process entirely. Of course a backup will
> > _always_ be a necessary recovery strategy in a production
> > environment.
> >
> > Markus
> >
> > --------------------------------------------------------------
> > ----------
> > SALESFORCE.COM MAKES SOFTWARE OBSOLETE
> > Secure, online sales force automation with 5 users FREE for 1 year!
> > http://click.egroups.com/1/2658/6/_/830676/_/961602784/
> > --------------------------------------------------------------
> > ----------
> >
> > To unsubscribe from this group, send an email to:
> > IB-Architect-unsubscribe@onelist.com
> >
> >
> >
>
> ------------------------------------------------------------------------
> SALESFORCE.COM MAKES SOFTWARE OBSOLETE
> Secure, online sales force automation with 5 users FREE for 1 year!
> http://click.egroups.com/1/2658/6/_/830676/_/961604289/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> IB-Architect-unsubscribe@onelist.com