Hi Jason, Answers found inline to the questions.



Jason Wharton wrote:

> What was the average CPU utilization during these long term data crunching
> jobs?

The MD5SUMING and other such data validation work all happened at the
client. The client would run a stored procedure passing the appropriate
values to the server where the server would reply whether the file is
needed or not (simple fast query).
The server utilization at this time was very small.

After the server returned the result, the client would upload the blob
(if needed). We found the problem described to occur with the BDE as
well (or I would have been on the email telling you about it). It would
occur with a single client or multiple clients. I even sent a question
a few years back to the list and Ann responded saying that she knew of
no reason it would happen.

> Do you think you were I/O bound or CPU bound?

Although I use alot of server side computation (either with SP/Triggers
or with UDF's) I have found that with my systems, it was never CPU power
that was found to be lacking when it came to system design.

I/O was always the bottleneck and I have gone through alot of different
solutions and system redesigns to combat it. When you are querying GB's
of data, it is best to keep the page size as large as possible to cut
down on the number of disk requests.

> My *guess* is that in today's hardware scenario we can afford to expand in
> the computational demands of maintain a database. Ideally this shouldn't
> purchase less I/O but the ability to do other things previously discounted
> as too "expensive" to return significant value.

I have always been a strong advocate of more information/power being
available to the developer. I have always been pestering whoever would
listen at IB for the need for more global variables - such as the
current transaction ID, or the IP address of the connection machine.
Even the process ID of the thread handling the connection. The list is
long, but, should not be very hard to implement.

>
> One example would be to store the TID (transaction ID) of the record when it
> was inserted and maintain it permanently in order to have persistent
> bookmarks without having to hold up garbage collection. This does increase
> the size of the record stored on disk, increasing the I/O to deal with the
> record, but I think there are some ways to make it up.

This I do not agree with. I would like to have the information
available in a trigger, but I would like to have the choice to include
it in my record structure. I want the tool, not be forced to use that
tool.

> With this foundation of a persistent DB_KEY in place, it would be possible
> to add in MANY very useful capabilities to the engine. (I'd rather think of
> them as sockets or modules that are pluggable as we don't want to bloat the
> core engine.)

We could add alot of things by adding a few global variables and adding
a few triggers. - This is simpler than a persistant DB_KEY (although you
could create a persistant DB_KEY using the extra tools and a little DB
design)

I currently do most of the things you are talking about in the
IBEnterprise model with the combination of IB_BOTS (that do the
replication/load balancing).
We could easily add the triggers and global variables, then create a
client that automatically generates all the neccessary linkage code
(hmmm, sounds like the ib_designer project that I have had held up due
to lack of time).

Also sounds like good fodder for how-to books......
And consulting fee's....
And lots of other sources of income....

:)

best regards

Dalton