Re: GSOC Project discussion : Async sftp implementation

[Tom Deseyn <tom.deseyn@xxxxxxxxx>]

Hi Eshan,

It should be possible to implement sftp using the existing async
channel write and read apis, without the need for additional threads
or io_uring.
When an operation completes, a callback is emitted.

io_uring sits at the level of where the library does polling.
Unlike poll, send, and recv, io_uring cares about what thread the
operations runs on.
io_uring is also limited to newer Linux kernels.

As with all async things, you need to track on-going state.
Depending on whether you want to allow 1 on-going operation per sftp
channel or several, things become more complex.
If it's 1, you can track the state together with the sftp channel.
If it's multiple, the user will need to hold some variable that
captures the state per on-going operation.

I hope this was useful.

Thanks, Tom

On Sun, Mar 5, 2023 at 4:50 PM Eshan Kelkar <eshankelkar@xxxxxxxxxxxxx> wrote:
>
> Thanks for the feedback.
>
> On Sun, Mar 5, 2023 at 5:30 PM Jakub Jelen <jjelen@xxxxxxxxxx> wrote:
>>
>> On 3/3/23 06:36, Eshan Kelkar wrote:
>> > Hi, I am Eshan Kelkar and would like to create the async sftp for libssh
>> > as the GSOC project. I have gone through liburing to understand what
>> > async i/o is and how it is implemented using io_uring. So in this async
>> > sftp implementation I believe we can place calls to the liburing api
>> > functions from inside of the async sftp api functions so that things
>> > occur asynchronously.
>> >
>> > Another approach that comes to my mind is that on a call to async sftp
>> > api function a seperate thread gets created which does the waiting and
>> > all and on completion places a call to the callback function notifying
>> > that the operation has occurred. The second approach is async
>> > conceptually as the user of api can continue his job after the call as
>> > the waiting occurs on the separate thread but this approach seems a bit
>> > "naive" as for each api call a new thread gets created which is resource
>> > expensive.
>> >
>> > Kindly comment on these two approaches and suggest any other approach
>> > which you have in mind to implement the async sftp api, those
>> > suggestions will help me prepare better before sending in the proposal.
>>
>> Hello Eshan,
>> first of all, sorry for late reply. I saw your message on IRC, but
>> before I got back to reply, you were already away so thank you for
>> patience to reach out to us on other channels.
>>
>> The async SFTP implementation is one of our priorities and one of the
>> more complicated tasks. I would suggest starting from the benchmark code
>> we have in tests/benchmarks/, which has an example of async donwload
>> (but there is only the download part -- the upload part does not exist now).
>>
>> I do not think it is a good idea to spawn more threads as it would
>> require a lot of synchronization. The example in the benchmarks can run
>> several download requests from a single thread, which can help saturate
>> the network connection without the need for threads.
>>
>> I did not read much about io_uring yet, but it sounds like it solves the
>> issues we have with speed of synchronous writes/reads caused by context
>> switching so this would be our preference. There are already some
>> hints/comments in the following issues, so if you will have some more
>> questions, comments, feel free to ask here or in either of the following
>> issues:
>>
>> https://gitlab.com/libssh/libssh-mirror/-/issues/65
>> https://gitlab.com/libssh/libssh-mirror/-/issues/124
>>
>> Regards,
>> --
>> Jakub Jelen
>> Crypto Team, Security Engineering
>> Red Hat, Inc.
>>