On February 8, 2005 11:04 pm, David Johnson wrote:

> However, high level tuning is a different issue. A well designed
> application is tunable to maximize its use of the hardware facilities
> when performance is really critical. A better designed application is
> self tuning - but that is much harder to implement. Firebird seems to
> implement a little of both approaches - configuration for coarse tuning,
> then internal instrumentation for fine tuning.

For certain applications I agree. The SSI's need to handle multiple SS7
conversations in order to set up a telephone call. This is a very complex
process and involves everything from prompting the user for more telephone
digits, to responding with certain voice messages, to negotiating with the
switches for other companies to nail up a phone call. This all happens in
realtime - and all of our business logic is handled by FB1.0
Those systems are designed so that they are dedicated to a very few
connections and the procedures are designed to be optimized for the
environment, but that data is updated via replication continuously. Our
CSR's can watch a call and see the log.

Now, I know that alot of people do not think of having different files, on
different machines as a single database, but, that is how it has always been
drilled into my head.
If you need a fast responsive system for realtime activity, you add a nice new
machine, and set up the proper data update links between the new system and
the old.
You have a new department being set up and they need connectivity, you give
them servers that will handle their work, and once the reconsiliation
procedures are in place, you can isolate the entire department without a
problem and know that you can bring them back quickly.

I have seen too many designers go the big hardware route. The return on the
investment, combined with the single points of failure, just do not justify
the cash outlay.

> > Our list of design specifications is called the bible, and the number of
> > fights that have been started over a developer trying to cut time by
> > skipping a few steps, just to loose their bonus, and possibly thier jobs.
>
> We have one of those too, but it is a typical committee document :o(
> It is concerned with producing paper that no one reads, not product.

That is somthing we started with, my boss at the time gave me a single
directive, "if the procedures do not work, change them to work, or, find
other employment". It is funny what happens when you put the full
responsibilty of a set of operating procedures on someone. It really
changes how the procedures are viewed.

> If you don't mind, I will submit this simple gem to our performance and
> tuning team.

Sure, that is why I still lurk on the lists, trying to help, just as I was
helped all those years ago.

> > We aim to have enough hardware resources that once the system enters
> > production, that our CPU and drive utilization do not exceed 25% of
> > total load.
>
> In contrast, our CPU utilization target is no less than 90% and our DASD
> utilization is not published.

We find that if the CPU is running high, we have not spec'd it fast enough for
the task, as we look for a balance between data speed and cpu. We find if
the cpu drops below 25%, then that is an indication that the data speed is
too low. We constantly monitor the system usage as it gives us a
performance ratio based upon data volume (alot of the states come from the
replication system)

> Agreed - in one sense. Scalability is an overused term that means many
> things, depending on the context. You are using it in the context of
> the data/application model being responsive to the business needs. I am
> using it in the sense that a well defined model that already meets this
> definition can support the raw processing needs of 1000 to 8000
> concurrent users of real-time data, plus automated real time feeds from
> all of our equipment, plus trade partners, plus ... well, you get the
> idea.

From my earlier comments, you should see that 8000 concurrent users would be a
small number considering the number of phone calls we process (averages about
21 new calls a second, peak times being much higher) combined with our web
site, three major offices, automated services, redundant clients, audits (we
are heavily legislated and it seems like as one audit stops, another starts)
and other requirements. Almost all of the needs are realtime.

> > Now, you need to ensure that the users who depend upon laptops are able
> > to work offline, again, the design should simply scale to the new
> > requirments.
>
> Agreed ... except that our business needs dictate that all transactions
> must be online. You can't get real-time information about equipment X
> in Chicago if you are off-line in Anaheim. Of course, in Anaheim you
> probably would be doing everything possible to avoid getting on-line
> (oops, I dropped my laptop in the Pirates of the Carribean).

Ahoy Matey, our dev lab has its own small fleet, although the MW:Dark Age
forces have overcome most of the playing surfaces lately. But, blood bowl
is retaking the crown as the popular game again......
No you know why the redundancy is so important, no one wants work to interfere
with thier gaming session.

> In my next project, I have to move somewhere between 240,000,000 and
> 9,600,000,000 rows of real-time data from our current system to a very
> different third party system, changing the database schema along the
> way, in a two hour window. (Hows that for a "precise" scope
> definition?) The translation scope will not be fully known until a few
> hours before (or after) we actually go live. Every minute of downtime,
> after the go-live time, costs us tens of thousands of dollars. A
> relatively small number of incremental optimizations makes a huge
> impact.

If you want some help, let me know, we used to import gig's of smdr records
daily into our system. I know a few tricks, both hardware and software that
may help you.

> Moving this application from desktop to single Intel Hyperthreaded
> server, with the same clock speed, more than doubles the throughput of
> this application and leaves cycles to spare. A dual Hyperthreaded CPU
> server doubles the potential performance yet again (if I had sufficient
> test cases ready to really work the box).

please remember, FB is not that good on hyperthreading/smp. You may want to
split the task out to multiple clients working in concert - with a classic
based server and enough clients to spread the work across all processors.

Vulcan is designed specifically to deal with this issue. I am waiting for
Jim to sign off and give it his blessing. Once that happens, I will be
looking at a few months of no sleep while we test it.

> The move from wintel to AIX/PPC4 architectures is empirically tested to
> grant an order of magnitude improvement for this application - so my
> process that, fully optimized, would take 20 hours to run on Wintel
> hardware, will run in 2 hours on the PPC4 platform. If we open up the
> LPAR and give my process all of the CPU's, we can dramatically improve
> the throughput yet more.

Your application may benefit, but, firebird will not, although using classic
would work for you.

> > The basic truth is, thier are very few programmers out there. The
> > schools churn out people who have no natural ability and who are best
> > promoted into management as they are useless as developers. (in case you
> > are wondering, that is what has happened to me)
>
> Sad, but true.
>

Whats sad, the fact that I was promoted or the fact that I am useless as a
developer.......
:)


best regards

Dalton