SFTP read-ahead problem

[Tilo Eckert <tilo.eckert@xxxxxxx>]

Hi,

I am trying to implement a simple read-ahead mechanism for reading large
files via SFTP using sftp_async_read_begin() and sftp_async_read(). On
the first attempt to read a 4096 bytes block, I request N times 4K, and
read the first 4K block. On successive reads, for each block read
another one is requested to keep N requests on the wire.

The problem is: sftp_async_read() suddenly fails with the error message
"Short sftp packet!" after successfully reading a few hundred packets
when N is large enough and the server is running OpenSSH. For a remote
host over a 5 Mbit internet connection with 60ms latency, the issue
starts when requesting more than about 70 blocks. In my tests on GBit
LAN, the issue started for somewhat larger values of N.

When I attempt to call sftp_async_read() or sftp_close() after the error
occurs, I can see from the libssh log that all requests are actually
fulfilled by the server (i.e. N * 4K are received by the socket). For
certain combinations of N and network connection, calls to these
functions also get stuck and never return. In other cases, closing fails
with "Unknown packet type 0".

Here is a sample program that demonstrates the issue:
http://pastebin.com/gfiFKfDT

I tested this with different OpenSSH versions between 6.0 and 6.8 on
Windows and Linux, and with freeSSHd on Windows. With all OpenSSH
versions, I get the described behavior. With freeSSHd, there are no
issues at all. I successfully tested it with up to N=4096 (i.e. 16 MiB
read-ahead).

It looks to me like libssh starts to misinterpret packets under certain
conditions. Maybe someone with deeper insight can investigate this.

Regards,
Tilo