Re: SFTP Write async

[Tilo Eckert <tilo.eckert@xxxxxxx>]

Hi Carlos

I already implemented a basic API some time ago, which is used in 
production. Maybe it can serve you as a starting point. The API is very 
raw, but it works. It allows you to trigger an arbitrary number of 
concurrent async SFTP reads and writes and to fetch the results later 
using the request id. Look for the sftp_async_*() functions:

https://github.com/limes-datentechnik-gmbh/libssh/blob/limes/src/sftp.c

The hard parts have to be implemented by user code which is why I did 
not submit it to upstream. The user code has to manage lists of 
in-transit requests, handle write ACKs, implement a suitable read-ahead 
strategy (depends on read pattern, e.g. lengthy sequential reads vs. 
heavy seeking), proper seeks (you need to track file offsets from user 
perspective for working relative seeks and discard already requested 
blocks). Those parts should be handled by libssh in a final 
implementation because it's easy to produce memory leaks, lose track of 
the correct file offset or have an underperforming read-ahead algorithm.

The most difficult aspect is the read-ahead strategy. There is no one 
size fits all. If you optimize for maximum throughput by using a large 
read-ahead window, you will waste a lot of bandwidth/time waiting for 
data you no longer need if you seek a lot because there is no way in 
SFTP to cancel sent read requests. If you optimize for seek, you will 
have slower throughput.

The current SFTP API could be re-implemented using the async methods and 
a configurable per-file read-ahead strategy which would considerably 
speed up performance. Libssh users that don't write async applications 
but want reasonable throughput and a simple API would no longer be bound 
by network latency as is the case with the current implementation.

Regards,
Tilo

Am 02.06.2020 um 20:32 schrieb Carlos Cabanero:
> I couldn’t find any async_write method, but some thought was put into it
> from this 2014 thread:
> https://www.libssh.org/archive/libssh/2014-06/0000014.html. Is this
> somewhere? Are we still interested to have it and are there any suggestions?
>
> Thanks!