> P.S. When I was writing this email, I remembered another idea that
> was floating in the air few years ago. It is about Leslie Lamport's
> PAXOS distributed agreement algorithm. It has very nice story/fiction
> behind it, but proved to be not trivial to implement. It should allow
> building systems that would be able to make decisions also when some
> nodes are offline without compromising the overall consistency. The
> idea was to use this algorithm for distributed databases. Just made
> quick search in Google for "paxos database" and found this PDF:
> http://labs.google.com/papers/paxos_made_live.html - they use it in
> the Chubby, another technology used by GFS and BigTable.

In the last weeks I had come across Paxos in this context, Petal:
http://www.thekkath.org/papers/petal.pdf
Petal was a DEC research project, offering a "virtual distributed
disk", it provides a fault tolerant, scalable numbered page service. In
this implementation, Paxos is a 1MB binary too...

The idea of Petal is nice, and it would solve the redundancy and write
speed issues. Doing partioning of raw pages rather than structured data
is a simple, clean way to partition the data nodes.

The trade-off (I guess with all partioning) is that it buys write
scalability, but at the cost of putting a ceiling on read scalability.
With reads vastly outnumbering writes in web apps, even in
Netsuite/Salesforce type environments, that may not be a good choice.

Perhaps it is time for a thought pause, and see if other FB Arch
readers have good ideas or suggestions.

Paul