Geoff Worboys wrote:

>If a new version of FB *requires* multi-Gb of RAM just to be
>able to run then we will alienate many users. ("requires" is
>different from being able to "take advantage of" - which FB
>must be able to do.)
>
>

Nobody has suggested anything like this. We're really talking about the
difference between reading one blob page and decompressing it into two
or three pages rather than reading two three pages of uncompressed
blobs. I've been trying to make the rather simple point that I picked
run length encoding because it didn't tax the relatively slow CPUs of
the mid 1980s, that CPUs are now two or three orders of magnitudes
faster, and maybe, just maybe, it makes sense to reexamine the issue.
The guys who want to run 8MB 386s have morphed the argument into a
requirement for multi-gigabyte memory. Nobody said, nobody suggested
it, nobody even hinted at it. It is an artifact that we have a lot of
computing fundamentalists among us who consider the old ways to be the
best ways that decisions I made over 20 years ago are sacred. I'm
getting more than a little tired of the attitude than any idea less than
a decade old has to be stamped out as heresy.

Guys, this product was started by heretics who called themselves
"renegade systems". You would have hated everything we stood for (and
still stand for). You came into a project after ideas ossified and
orthodoxy rules. Fundamentalism is inherently reactionary, but if you
guys could go back to the beginning of Interbase you would run screaming
back into your time machines, set the throttle back to VISAM or IMS, and
hit the button.

BLOB COMPRESSION IS FOR PERFORMANCE! Get it? It's to make it run
faster, not slower. It faster to decompress a page than to read another
one. Sooner or later you have to understand that cpus are fast and
disks are slow, and the way to go faster is to read fewer pages.

>This is not to say that we should develop for 8086 with just a
>few Kb of RAM. But a level of compromise is needed. The
>targetted platforms will obviously rise with each new version,
>and this is appropriate. The trick is to try and keep the
>target in mind.
>
>

Other than letting Roman keep selected blobs uncompressed, what
compromise did you have in mind? I don't think you expect Firebird to
run on 16 bit machines, so are we talking about 16 MHz 386s? They don't
have a bios to talk to modern disks, and all disk < 1GB are so way, way
out of warranty is to be unusable.

If we restrict ourselves to machines with a bios to boot from a 4 GB
disk, we're talking at least P-IIs. Any damn P-II ever built can
decompress a block faster than it can read one, so blob compression is
an optimization for everyone except Roman.

>
>It is my belief that Firebird needs to be "scalable" and also
>to have a possibly "small footprint". As such it is important
>that it runs on relatively low-resource machines (in todays
>terms, not in the terms of 20 years ago). It must be able to
>scale down as well as up!
>
>

What's the worst machine you can imagine running Firebird? And what has
anyone suggested that would increase the footprint?

>The compromises required to support such scalability may mean
>that we fail to be as completely fast as possible at either end
>of the scale. But as long as it is fast enough (a very loose
>and subjective phrase) the speed is less important that the
>ability to operate effectively over the range of targets.
>
>

What compromises do you have in mind?

>
>To have a version of Firebird that requires blobs be loaded in
>full into memory on the server is a mistake. Such an approach
>is likely to require machines with a level of memory not
>compatible with the _current_ low end targets.
>
>

Suppose we compressed in 64K hunks. Is that a problem?

>To have a version of Firebird that forces sophisticated high
>level compression on all blobs is also likely to impact low
>end machines. When the FB server process is sharing resources
>with a user or other server procecess the excess (and often
>redundant - eg: jpg) CPU cycles could be a potentially fatal
>performance impact.
>
>

If the compression code had the native wit to notice when the compressed
size reached a significant fraction of the original, it could throw it's
hands in the air and store it uncompressed. But depending on the encode
algorithm, the compressor may already have reach the same decision,
making the test redundant. But really, fatal? Isn't that a little
extreme? Just about all compression schemes decompress very rapidly.

>Another idea I saw mentioned was one type of compression for
>storage with a potentially different compression used for
>communication. So what? We are going to have the server do
>decompression from storage and then recompression for
>transport? This was a joke, right?
>
>

I don't know the tradeoffs for wire compression. Things like text, Word
files compress well. Jpegs and the significant innards of PDFs are
already compressed, so little is to be gained. I don't consider it high
priority, but building in the architecture for client side
compression/decompression is a no-brainer. Building in per-item
compression options on the wire is out of the question, I'm afraid.

>**IF** Firebird is to support compression for blobs it needs
>to be configurable per blob column.
>
>

If we decide this now, Roman will drift out of the discussion. We need
to keep the threat real to keep him involved.

>
>I believe that IB/FB has had the right approach so far. Leave
>such things as compression and encryption to external services.
>
>

It a disk optimization that should be transparent to the user. Without
awareness of blob compression, it is impossible to apply blob operators,
which is a terrible sacrifice to make.