I found the problem.

The final two lines of GetNextChunk are these:

    OutputBuffer[3 + Length] := #0;
    result := iboDecodeA( AnsiString(PAnsiChar(@OutputBuffer[3])), 'UNICODE_FSS' );

What goes wrong here, is that because the buffer now contains binary data instead of text, there are #0 characters all over the place. Regardless of decoding, turning that PAnsiChar into a string terminates it prematurely.

To work around this, I had to add another method (GetNextBinaryChunk) to TIBOControlAndQueryService where the result is TBytes, in which the last part of the buffer is copied to the result in full:

    SetLength(Result, Length);
    Move(OutputBuffer[3], Pointer(Result)^, Length);

Now setting the BackupFile to 'stdout' and retrieving the data using GetNextBinaryChunk results in a valid backup.

I'm not so much worried anymore about errors not being reported either, because I noticed the service_mgr returns error codes even in this scenario, which IBO turns into exceptions.

It would be great if this code could be incorporated into IBO. Perhaps the whole thing could use some refactoring, given the similarities between GetNextLine, GetNextChunk and now GetNextBinaryChunk. Also using strings as a buffer in general is troublesome...