Re: SFTP read-ahead problem

[Tilo Eckert <tilo.eckert@xxxxxxx>]

Any idea what might cause this strange behavior when doing read-ahead?

Am 08.06.2015 um 13:09 schrieb Tilo Eckert:
> I used libssh 0.7.0 for my tests. The 0.6 branch from git has the same
> issue. Of the 0.6.x source tarballs none of them compiles here (ArchLinux).
> 
> Here is a verbose log where the sftp_close() does not get stuck (remote
> host, run with N=1024):
> https://paste.ee/p/A4N4i
> and here it gets stuck (same host/file, N=128):
> https://paste.ee/p/WoIeY
> 
> Regards,
> Tilo
> 
> Am 08.06.2015 um 12:04 schrieb Aris Adamantiadis:
>> Hi,
>>
>> I agree with your analysis. Which libssh version did you use ? Could you
>> try again with 0.6.0 and tell us if the same happens ?
>> Please attach verbose logs.
>>
>> Aris
>>
>> Le 08/06/15 11:46, Tilo Eckert a écrit :
>>> Hi,
>>>
>>> I am trying to implement a simple read-ahead mechanism for reading large
>>> files via SFTP using sftp_async_read_begin() and sftp_async_read(). On
>>> the first attempt to read a 4096 bytes block, I request N times 4K, and
>>> read the first 4K block. On successive reads, for each block read
>>> another one is requested to keep N requests on the wire.
>>>
>>> The problem is: sftp_async_read() suddenly fails with the error message
>>> "Short sftp packet!" after successfully reading a few hundred packets
>>> when N is large enough and the server is running OpenSSH. For a remote
>>> host over a 5 Mbit internet connection with 60ms latency, the issue
>>> starts when requesting more than about 70 blocks. In my tests on GBit
>>> LAN, the issue started for somewhat larger values of N.
>>>
>>> When I attempt to call sftp_async_read() or sftp_close() after the error
>>> occurs, I can see from the libssh log that all requests are actually
>>> fulfilled by the server (i.e. N * 4K are received by the socket). For
>>> certain combinations of N and network connection, calls to these
>>> functions also get stuck and never return. In other cases, closing fails
>>> with "Unknown packet type 0".
>>>
>>> Here is a sample program that demonstrates the issue:
>>> http://pastebin.com/gfiFKfDT
>>>
>>> I tested this with different OpenSSH versions between 6.0 and 6.8 on
>>> Windows and Linux, and with freeSSHd on Windows. With all OpenSSH
>>> versions, I get the described behavior. With freeSSHd, there are no
>>> issues at all. I successfully tested it with up to N=4096 (i.e. 16 MiB
>>> read-ahead).
>>>
>>> It looks to me like libssh starts to misinterpret packets under certain
>>> conditions. Maybe someone with deeper insight can investigate this.
>>>
>>> Regards,
>>> Tilo
>>>
>>
>>
> 
>