Subject Re: [IB-Architect] Backups of large database & super transactions
Author dcalford
Hi All,

Sorry, been offline for a few days and have missed the one thread that is right
down my ally.

We have a very large database.
We have a backup restore cycle that (if not properly maintained) can go on for
over three days.
We have time sensitive data and it is of an accounting nature so it must be
accurate.
We have users who may damage the database and we can not afford to loose the
transactions between the last backup and the current time.
We must have full records of any such changes.

The last point is the most important.
Our data can be used in a court of law as evidence and as such, any change to
the database must be logged.

We have found a solution to all the above problems without any changes to
Interbase, and without relying on UDF's.

The system will work on IB 4 databases and beyond.
It is the basis of our replication scheme as well.

I have verbal permission to share the methods, and I am doing up a 'how-to'
between the 15 other projects I have going.

It is our method of replication, role-forward, load balancing and database
trimming.

If I get more people asking for the way we did it vs saying no to it, I will do
a full core dump on the methods we used to do it. (deadline by monday).

We are able to
(1) Backup a database
(2) Restore it in another location
(3) Using the time logs, roll the backup forward to match with the current
database
(4) Keep the second database in sync with the first database via (user
defined) time periods.
(5) Using a program that we wrote, roll-back a users actions due to somthing
they did.

I have many things that I found I wished IB had in order to make the job easier.

The implimentation of those things should be extreamly simple. (so simple that
6.1 or even 6.0 can have them with little trouble)

Best regards

Dalton


Jason Wharton wrote:

> > First the problem, then the solution.
>
> I thought I had made the problem sufficiently clear. I'll recap and extend
> it some to clarify.
>
> Requirements:
> Huge database. (many GB worth of data)
> Database needs to stay on-line 24X7.
> 98% static & generally inserts only.
> In case of failure, downtime needs to be minimized.
> Immediate failover isn't critical.
> Cannot lose over one day's work but up to that is tolerable.
> All input sources are available for two days to be re-entered if necessary.
>
> Problem.
> Obtaining an efficient backup on a regular basis.
> Efficient meaning the total I/O and CPU toasted in the process, among other
> things.
>
> Problem extension #1:
> Same huge database.
> Needing to be sync'd with many remote copies on a daily basis.
> Real-time is not necessary.
> Remote copies are for inquiry only.
> Potentially low and/or expensive bandwidth constraints.
> New base versions are express-mailed via DVD media or DAT.
>
> Problem extension #2:
> Same huge database.
> DBA needs to cancel out an entire days work due to certain problems
> encountered.
> Database needs to stay on-line.
>
> Have at it. That should cover most of it.
>
> Jason Wharton
> CPS - Mesa AZ
> http://www.ibobjects.com
>
> Now, I would like to share a comment or two from my perspective regarding
> the content of this list.
>
> What I would really appreciate is this list not be mandated to operate on
> the strictly professional software engineering practices as I'm being jammed
> into here. You guys that are actually responsible for what goes into the
> source code of InterBase and shipped out on the CD with your reputation on
> the line can do whatever you want in your own private meetings and
> discussions. That's all well and expected. But, if someone like me who is
> curious to know something about the architecture of InterBase in an area
> where they have some ideas, let's explore them without running them through
> the ringer making them dot their I's and cross their T's.
>
> This is a community forum designed to attract people into contributing all
> sorts of crazy ideas and for learning. Jim, your arrogance is really
> distasteful and it will not bind a community together. You constantly
> insinuate that I'm sharing stupid ideas without having first given them some
> real thought and you don't hesitate to belittle and twist the material that
> I am presenting. If you continue to dish out that kind of treatment I don't
> think it is going to do any ultimate good for this open source/open
> community endeavor. I may be sharing some stupid ideas and thank God a lot
> of them will probably never be implemented. But, come on, I don't have to
> feel like I'm an inch tall for doing it. And moreso, I don't want anyone
> else involved with the IBDI to experience this either. I appreciate Chris'
> more empathetic and suggesting tone. That's what will build a community.
>
> My humble $0.02 worth.
>
> Jason Wharton
> InterBase Developer Initiative
> http://www.interbase2000.org
>
> InterBase will be the database of the new millennium
>
> ------------------------------------------------------------------------
> Wrox Wireless Developer Conference, Amsterdam, July 10-12. Choose from
> 40+ technical sessions covering application of WAP, XML, ASP, Java and
> C++ to mobile computing. Get your ticket to the future today!
> http://click.egroups.com/1/5689/4/_/830676/_/961206299/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> IB-Architect-unsubscribe@onelist.com