Hi Jim,

Thanks for your thoughts. It sounds like we have the beginnings of a
good discussion here.

"I have to disagree here. There are two different types of problems
out there and, not surprisingly, they require different solutions.
[..snip..] But most of the remaining 99.999% of Web applications
aren't like Google or Amazon."

We probably agree, but it would be good to be specific and avoid
confusion. In this thread I have a Netsuite or a Salesforce in the
back of my mind. Could you give one or two high volume example web
apps that you would find a natural fit for this architecture
discussion?

"The metric for Web applications is page latency at a given load." I
agree fully. It is almost a trueism.

I'm not sure I understand the "sources of latency" list. It doesn't
seem to be inclusive nor do you use it later in your reasoning. From
that reasoning I derive a high level list of three:
- internet (WAN) latency
- app server latency (say, http + app code)
- database latency

Internet latency has been well studied & reported, eg. by Yahoo. Note
that precompiling boilerplate does not only reduce the load on the
database, but also helps internet latency by optimising request
pipelining and client/proxy caching. Even with a very fast database,
you would still want to do it.

You address app server latency with the Netfrastructure example. Of
course, connecting to a scripting and/or database server introduces
significant latency. It is often countered by running stuff in a
single process (eg. apache mod_php) or through connection pooling
(eg. fcgi). The former scales up but not out, the second seems to
scale out. On a last note, I hope you agree that scaling out the http
servers is not a key issue for the purposes of this discussion.

Remains the core of the current discussion: database latency at a
given load. Whilst I do not fully agree with your reasoning, we are
fully agreed on the fact that all current solutions are cumbersome
for the dba/app desinger and that is what we are trying to solve.

"I believe that the relational model has passed the test of time and
is organization of choice, other things being equal. The hierarchical,
network (CODASYL), and "object oriented" database have all died while
relational systems have thrived. The issue is not the model (though I
prefer semantic extensions) but the implementation."

Agree fully. That is why we have row stores, column stores, memory
stores, etc. All the relational model, but implemented in different
ways. I also like semantic extensions, and with that on offer,
the "complex record" folks can be happy too -- for good or bad
reasons.

"Last I looked, Salesforce used a zillion MySQL servers with each
instance a separate MySQL database" .

Hmmm, that statement totally puzzles me ... why would you think that?
From their last annual report (in which lying is a criminal offence):

"We built our service as a highly scalable, multi-tenant application
written in Java and Oracle PL/SQL. We use commercially available
hardware and a combination of proprietary and commercially available
software, including database software from Oracle Corporation, to
provide our service. The application server is custom-built and runs
on a lightweight Java Servlet and Java Server Pages engine. We have
custom-built core services such as database connection pooling and
user session management tuned to our specific architecture and
environment, allowing us to continue to scale our service. We have
combined a stateless environment, in which a user is not bound to a
single server but can be routed in the most optimal way to any number
of servers, with an advanced data caching layer."

In my mind the key question is whether a memory database with write
once & only (WOO) backfill scales out better than a real application
cluster (RAC) design -- and I mean the design pattern, not just the
Oracle implementation. Better means lower TCO, including hardware &
engineers. RAC or WOO, that's the question.

"Pure in-memory (i.e. non-persistent) relational systems are much
faster, but with the obvious drawback. If each node in a cloud can
execute at the speed of an in-memory database, the cloud would scale
linearly. But to make this work, we need replication, and the cost of
replication grows with the number of nodes in the cloud. There are
lots clever things we can do to minimize the cost of replication, but
we can[not] avoid it."

In one of his recent papers on in-memory databases, Stonebraker
argues that a confirmed replication to one or more other nodes is
sufficient to make a transaction durable (sorry can't find the link
anymore). After all, writing to disk doesn't mean durable either when
there is an explosion in the server room.

"The cost of a relational database engine isn't the SQL engine. Even a
really stupid SQL engine is fast. The problems are synchronizing
memory and the disk to be ACID. [...] Even better is having SQL nodes
never write at all. The network is better redundancy than a disk.
Disks are bigger than memory, though, and more persistent, so things
ought to get written to disk, but there is no reason for a SQL node
to even bother."

I guess you better define what tasks a SQL node does, and what tasks
a storage node does, because I am totally not getting what you are
trying to explain. Sorry to be so dimwhitted.

Paul