> 1) All data descriptors are sent fully expanded, including NULLs and
> CHAR strings. Could be solved via variable-length encoding of the
> data (protocol internals change).

This can be fixed even now without changing the protocol version:

This is the code that decodes SQL data in Jaybird:

byte[] buffer;
for (int i = 0; i < numCols; i++) {
int len = ioLength[i];
if (len == 0){
len = readInt();
buffer = new byte[len];
readFully(buffer,0,len);
readFully(pad,0,(4 - len) & 3);
}
else if (len < 0){
buffer = new byte[-len];
readFully(buffer,0,-len);
}
else {
// len is incremented to avoid value
// 0 so it must be decremented
len --;
buffer = new byte[len];
readFully(buffer,0,len);
readFully(pad,0,(4 - len) & 3);
}
if (readInt()==-1)
buffer = null;
row[i] = buffer;
}

As you say, for NULL the full length is sent. However we can reduce
the number of bytes sent to 5. For NULL value we should pass ioLength
to be -1 (0xffffffff) which will act as one-byte CHAR/VARCHAR column
(0xff), which will be thrown away anyway and then we send -1 as NULL
indicator (0xffffffff). That's it.

Regarding the CHAR values. Yes, it would make sense to use RLE
encoding for it, and it can be done when we increase the protocol
version. But on the other hand, if somebody declares column CHAR, he
expects to have it padded (otherwise one can use VARCHAR), so I would
ignore this issue for now.

> 2) The server replies with a number of bytes allocated by the client
> regardless of how many actual bytes are being returned. This is okay
> for fetches when we already know size of the record packet, but this
> wastes a lot of traffic for describe/info calls. Could be solved via
> passing the buffer length as a reference to adjust it at the server
> side (API change).

I think we can solve this issue without changing wire protocol version
too. When we loop through the buffer, we look for isc_info_end and
stop there. So, the only important thing is that we terminate our
response with isc_info_end. In this case application will find the end
of the info block, save the parsed data and then release the complete
huge block it allocated before. There is no need to pad anything here
and I do not see how this is incompatible with the previous versions.

Also, in many cases we can simply ignore this issue. Usually the MTU
size is somewhat close to 1500 bytes, in other words, it does not
really matter whether we send 1 byte or 1400 bytes, there will be one
IP packet sent to the client. In all cases when the allocated buffer
is less than 1400 bytes, we can use the current algorithm, we won't
win anything here anyway. And we have to check in how many cases
people allocate 16k buffer when 1k response is expected. Jaybird uses
1024 bytes buffer on the beginning, increasing it twice when
isc_info_truncated is returned.

Roman