Subject Re: [Firebird-Architect] One way to scale Firebird put it on memory virtual disk or ramfs
Author Dalton Calford
Jim,

I have been following nuodb since the first beta. I have not looked at it
for the past few releases due to some limitations it had at the time of
initial testing but, that testing showed that you had spread out the
database across a single cluster of machines - if you have a break in the
network, the version of nuodb that I was looking at could not heal.

The response I received from Barry Morris/Greg Burd at Nuodb to my question
is as follows

*Question*

"What are the recovery methods if members of the cloud detach due to
network fragmentation – does the whole database die or does the fragments
of the database continue to work and then reconcile the work processed once
the network is re-established?"

*Answer Given*

"Should a network partition occur only one segment will remain active and
only if that segment meets the minimal requirements defined by the systems
administrator. Other segments will remain offline until the partition is
resolved, then they will rejoin and synchronize data with the master
segment. If your design require a system that can partition, run
independently and then synchronize when reconnected then we suggest that
each logically independent segment of the database in fact be its own
physical database."


Has that answer changed in the newer releases? Can the different segments
separate and reattach, re-applying work without loss of data that was
committed on two separate segments? Can those segments continue to
work independently until the connection reconnects? Does nuodb reconcile
the work without having the various tiers/clients needing any understanding
of the various elements involved? I am truly curious as we spent years on
in house middle tier solutions that do this for us.

It is not uncommon for network switches to fail, for cables to be pulled,
for hardware to reach end of life. When dealing with financial or
telecommunications infrastructure, you design with this in mind - as you
can have your data spread across multiple offices, in multiple cities, in
multiple states/provinces in multiple countries. You set up
batches/logs/authoritative servers etc. so that the work that did not fully
commit can be reapplied and so that the clients know that the data they are
viewing may not be authoritative. You do not try to commit to all servers
at all locations as latency makes this dubious at best. There are ways to
deal with it - some as old as the original double sided accounting
processes, but each process must be followed from the client to the server
or you find problems throughout.

From what was presented to me with nuodb, was a way of spreading out your
processing/risk across a single pop. You would still treat each pop as a
single server and follow other procedures to have the redundancy span
across pops while the versions I looked at did not have the means of
programming this within the database itself. (no procedural language
support in the beta I last looked at).

Now, if you read the whole email, I began by equating the risk of ram drive
vs any other medium and stated that there is risk and loss of ACID unless
you have taken a great deal of precautions. Attacking the use of ram
disks with a generic attack based upon the possible loss of ACID is an
incorrect argument given the number of other factors that can reduce or
remove ACID. A system designer/developer/solution provider must take into
consideration many factors and balance out the cost/benefit ratios of each
and every risk/solution. I showed that there are many such things that
cause risk that the average developer on this list will never encounter.
I did say such a task was possible but that Firebird would need to mirror
the elements in some other databases.

regards

Dalton








On 22 January 2013 10:57, Jim Starkey <jim@...> wrote:

> **
>
>
> Dalton, that is so much nonsense. NuoDB is an existence proof that your
> assertion is false.
>
> There are lots of ways to make distributed transactions ACID without
> queries being run "atomically upon multiple systems." I demonstrated
> one, but stopped looking after that. The inquisitive mind may want to
> keep looking.
>
> Every time some says something like "The only way to ..." innovation is
> trampled. If I had believed that "the only way to implement consistent
> transactions is two phase locking" MVCC, Interbase, and Firebird
> wouldn't exist.
>
> Programmers have learned count "zero, one, two, infinity". There is
> never only one way to do something. There are the ways you know and the
> ways you haven't thought of. If you stop looking, you'll never find them.
>
> There are many people who believe that all of the good ideas have been
> discovered. They're wrong.
>
> On 1/22/13 9:35 AM, Dalton Calford wrote:
> >
> >
> >
> > *The only way to become fully ACID compliant is to house multiple
> > servers,**
> > **in multiple nocs, with queries being run atomically upon multiple
> > systems**
> > **in parallel so that if one system fails while processing work, the**
> > **secondary system can respond as it has already been doing the same
> > work as**
> > **the primary. * The clients and tiers need to be able to determine if
> the
>
> > answer they are getting is authoritative or non-authoritative. They need
> > to be able to deal with multiple responses to the same query and all work
> > is queue'd locally in case the entire local work process needs to be
> > re-applied (either immediately or delayed due to network latency/drops).
> >
> > It is a very complex subject that involves the design of the pop, the
> > power
> > for the pop, the full system redundancy for each cluster, with full drive
> > redundancy in the sans, with the client software and middle tier software
> > designed to handle a complete system failure and switch-over to a totally
> > different pop in a different city/province/country if needed.
> >
> > Ram drives work, but I normally use them for the temp drive. The
> > benefits of ram drives depend upon the version of firebird, the version
> of
> > the OS and the hardware parameters.
> >
> > If firebird is proposing going down that route, they would need to
> > implement some new structures in Firebird, mirroring some of the elements
> > that oracle and other database platforms have created.
> >
> > On 22 January 2013 08:51, Roman Simakov roman.simakov@...
> > > wrote:
> >
> > > **
>
> > >
> > >
> > > 2013/1/22 Sergey Mereutsa serj@... >:
> > >
> > > > Hello Alex,
> > > >
> > > > and without shadow ACID rule has effect too - RAM-drive is the same
> > > block device as usual HDD - until power is not lost :)
> > >
> > > In this case you have ACI but have no D.
> > >
> > >
> > >
> >
> > [Non-text portions of this message have been removed]
> >
> >
>
> [Non-text portions of this message have been removed]
>
>
>


[Non-text portions of this message have been removed]