Hi Jason and Hans,


Jason, can you please check that this code looks acceptable
(the lines marked //??TC are the ones added/modified by me):


procedure TIB_ColumnBlob.LoadFromBlob( const ABlob: TIB_ColumnBlob );
var
tmpStream: TStream;
tmpRead: TIB_BlobStream; //??TC
bn: PIB_BlobNode;
begin
if ( not Assigned( ABlob )) or ( ABlob.IsNull ) then
Clear
else
begin
tmpStream := Statement.CreateBlobStream( Self, bsmWrite );
tmpRead := ABlob.Statement.CreateBlobStream(ABlob, bsmRead); //??TC
try
if not BlankIsNull then
SysSetIsNull( false );
//??TC Old Code: ABlob.SaveToStream( tmpStream );
//??TC Optimised blob copy - read all and then copy buffer entire
//??TC (rather than constant resizing that goes on with stream copy
tmpRead.Seek(0, soEnd); //??TC
bn := tmpRead.BlobNode; //??TC
if Assigned(bn) and Assigned(bn.BlobBuffer) and (bn.BlobSize > 0) then //??TC
tmpStream.WriteBuffer(bn.BlobBuffer^, bn.BlobSize); //??TC
finally
tmpRead.Free; //??TC
tmpStream.Free;
end;
end;
end;


Note that SaveToBlob will need the equivalent optimisation.


The problem with the original SaveToStream call is that it
calls the default TStream.CopyFrom:

function TStream.CopyFrom(Source: TStream; Count: Int64): Int64;
const
MaxBufSize = $F000;

For blobs larger than $F000 we end up calling IB_ReallocMem
for as many times as it takes to grow to the required size in
$F000 increments (very many reallocations and memory copies).

The new code takes advantage of the fact that CreateBlobStream
actually ends up loading the entire blob into a memory buffer
and so we can easily write a copy of that buffer directly with
no need for realloc etc. (It would be tempting to try and
implement a shared buffer here so that no copy was required at
all - until/unless one of the owners attempted to write.)

The current code is still not ideal for high performance
transfer situations (in DBak I wrote specialised blob-copy that
went directly from field to field, never loading the entire
blob) but the change is certainly an improvement and incurs no
down-side (since the TStream.CopyFrom results in exactly the
same blob load, but with the above code we can take advantage
of the more direct access).


The result after this change was:

= = = xfer_ibo = = =
App Init Time: 0ms
Object Prepare Time: 109ms
Records Transfered: 106
Records Transfer Time: 10484ms
Transaction Commit Time: 1188ms
Connection Close Time: 0ms
Application Run Time: 11781ms

(Over several iterations both IBO and FIBPlus come out around
the same sort of average for this table now... would probably
take a longer test to offer a more accurate comparison.)


Jason, additional (less substantial) gains are possible via
more analysis of IB_ReallocMem - as this can be an expensive
call in large buffer situations. For example in
function TIB_SessionBase.SysGetBlobData(
about 13 lines up from the bottom is the line:
if BlobNode.BlobBufLen > TotalSize then
Due to the previous logic this test is almost always true and
so the subsequent IB_ReallocMem is almost always called - and
yet it is rarely necessary (the intentional excess allocated
previously is for an unusual situation rather than usual).
It seems that there should be a better way to deal with this.
(Commenting out this bloc gave me, approximately, an extra
second on the test... not that big but not nothing :-)

--
Geoff Worboys
Telesis Computing