Roman Rokytskyy wrote:

>>>Fetch 2 MB blob into memory in order to get 4k block from it? Not
>>>very efficient, isn't it?
>>>
>>If one blob fetch out of 100,000 is for an interior segment, yes it
>>is. Almost nobody does that sort of thing for a number of reasons,
>>the most important of which is that it performs like a pig across a
>>wire.
>
> This structure is an index "segment" - one is full-text, another is
> spatial. Code jumps from one "record" to another in a fashion that can
> be considered random. Funny enough, when I tried the scheme with ~4k
> VARCHARs in each record and accessing it by PK, it performed approx.
> 4-5 times worser than a big BLOB with seek (page size was 4k).

This is getting round to the discussion on full text indexing. If you
have an index that says which sentence a word is in - in the blob - and
you only need THAT sentence from the blob ( and and a large number of
other blobs ) then it would be better to be able to jump directly to
that point in each blob. Compressing or encrypting the blobs MUST slow
that process down if you are looking for just 10k in 100Mb ?

Of cause if there was a compression decode algorithm that could be
'started' at any point ....

The alternative is something that helps the full-text search process, by
compressing common letter sequences into single characters - but that
gets blown out of the water with Unicode and case preservation. It's
probably a different discussion, but in my simple mind it's just another
algorithm that could be used as part of this process?

Bottom line.
There probably needs to be client side compression/encryption ( same
process different 'library'? ) which the server can then either store
'processed' or pull apart again to store - as an application requires?
Security would know what a client should be fed, and would enforce
encryption as required. At least simple compression would always be used
on packets sent to the client, but there is no reason that in an nice
old unsecure system the 'old' format packets could not be set to an old
client? Until the clients all get updated ;)

--
Lester Caine
-----------------------------
L.S.Caine Electronic Services