The big problem with this type of thinking is that they do not layer their
database design.

We have referential integrity within our databases.

We replicate our data and have it segregated across multiple
servers/databases.

We use domain-key normalization, with recursive data structures and
separated OLTP/OLAP systems and we do all of this with a combination of
custom client applications, custom middle-tier and a series of firebird
servers.

Now, when we have a machine that is time critical (Interactive Voice Servers
or Telephone Switches, both of which are controlled by Firebird) we use a
series of intermediary tables that hold data before it it moved to the
operational systems. In other words, when you are inserting data, you do
not care if it is indexed or cross referenced. You just care that it is in
the database and some other process can move it into the tables that hold
the various levels of data verification/normalization/replication/indexing
etc.

The biggest problem I have seen with many designs/concepts put forward by
people is that they look at putting everything into one database and do not
consider multiple databases/datastores working together.

Right now I am working on replacing many of our middle tier and custom
clients with standard clients and firebird, as firebird can now act as it's
own middle tier. There are problems, but I have been able to work around
most of them to this point.

For example, we have a series of machines that take commands via a custom
ssh command shell. That shell connects to 20 different databases, performs
the instruction, compares the output from all the different databases and
will activate/promote/demote databases depending upon the responses (or lack
thereof) from the various databases. We have other processes that perform
verification routines upon all the databases. Those databases hold all the
customer service/routing information for telco switches and they are a live
realtime system that has been running for years (firebird 1.5).

Those individual databases write out information to a external file that is
linked to a pipe which in turn sends it's information to various other
machines via udp (if a receiver goes down, it does not block the sender) and
to various other local logging tools. The remote machines populate various
OLTP and OLAP databases (also Firebird). This is a system used for
billing, operations, fraud detection, realtime rerouting of calls etc.

All performed in a system that is 5-9's. We have had no downtime since it
was put into place. Of coarse the system was designed so that we could
loose 60% of our hardware without impacting customers but that is part of
the nature of the design to be stateless for almost every node and if state
information is important, it is for a very short period of time.

I have always viewed the building of reliable systems to be a design
methodology and not something that was overly complex once you understood
the data and the use of the data.

On 28 May 2010 08:56, Milan Babuskov <milanb@...> wrote:

>
>
> Hi Paul,
>
> I agree with you that most web companies do not need a scale-out solution.
>
>
> paulruizendaal wrote:
> > There are a few ways to overcome this problem, just to name a few:
> > - convince number 1,001 to 10,000 on the Alexa list that they need a
> scale out database right from the start because they may become a top-1000
> player
> > - find uses for scale out databases other than web properties
>
> I assume you are aiming at Firebird being the database of choice for
> these kind of systems? I love Firebird, but it would really need to be
> tested in these types of scenarios. Do we know of any use cases?
>
> I believe even MSSQL does not stand a chance against MySQL. If you look
> at the article about stackoverflow, even they concluded that foreign
> keys and normalization are not a good idea when "absolute" performance
> is necessary. This means they are dropping the main benefits of
> "enterprise features" and only using the database as a dumb store
> without any business logic - exactly where MySQL shines. In fact, they
> would've probably got it much cheaper and faster with combined
> Microsoft+OpenSource stack. I would never use MySQL for some ERP, but I
> would think twice before switching away from it for Internet-run web apps.
>
> Hm, maybe I really should do some benchmark. I have one website with 5k+
> daily visitors, 400k+ PHP page hits and some 5milion-record tables in
> the database - which are read at least twice on each page load.
> Currently MySQL load peaks at about 60-70 simultaneous connections and
> it's handling it well. All this on VPS hosting with 512MB of RAM. Maybe
> I should try rewriting the backend to use Firebird and let it run to see
> where we're at. I guess SuperServer should be able to handle this load
> nicely.
>
>
> > - don't sell a cloud database but a web solution, including all the
> surrounding engineering in $ 1 mln projects
>
> Now if only there was a company smart enough to start doing that today...
> ;)
>
>
> --
> Milan Babuskov
>
> ==================================
> The easiest way to import XML, CSV
> and textual files into Firebird:
> http://www.guacosoft.com/xmlwizard
> ==================================
>
>
>


[Non-text portions of this message have been removed]