Re: GSOC Project discussion : Async sftp implementation

[Eshan Kelkar <eshankelkar@xxxxxxxxxxxxx>]

Thanks for the feedback.

On Sun, Mar 5, 2023 at 5:30 PM Jakub Jelen <jjelen@xxxxxxxxxx> wrote:

> On 3/3/23 06:36, Eshan Kelkar wrote:
> > Hi, I am Eshan Kelkar and would like to create the async sftp for libssh
> > as the GSOC project. I have gone through liburing to understand what
> > async i/o is and how it is implemented using io_uring. So in this async
> > sftp implementation I believe we can place calls to the liburing api
> > functions from inside of the async sftp api functions so that things
> > occur asynchronously.
> >
> > Another approach that comes to my mind is that on a call to async sftp
> > api function a seperate thread gets created which does the waiting and
> > all and on completion places a call to the callback function notifying
> > that the operation has occurred. The second approach is async
> > conceptually as the user of api can continue his job after the call as
> > the waiting occurs on the separate thread but this approach seems a bit
> > "naive" as for each api call a new thread gets created which is resource
> > expensive.
> >
> > Kindly comment on these two approaches and suggest any other approach
> > which you have in mind to implement the async sftp api, those
> > suggestions will help me prepare better before sending in the proposal.
>
> Hello Eshan,
> first of all, sorry for late reply. I saw your message on IRC, but
> before I got back to reply, you were already away so thank you for
> patience to reach out to us on other channels.
>
> The async SFTP implementation is one of our priorities and one of the
> more complicated tasks. I would suggest starting from the benchmark code
> we have in tests/benchmarks/, which has an example of async donwload
> (but there is only the download part -- the upload part does not exist
> now).
>
> I do not think it is a good idea to spawn more threads as it would
> require a lot of synchronization. The example in the benchmarks can run
> several download requests from a single thread, which can help saturate
> the network connection without the need for threads.
>
> I did not read much about io_uring yet, but it sounds like it solves the
> issues we have with speed of synchronous writes/reads caused by context
> switching so this would be our preference. There are already some
> hints/comments in the following issues, so if you will have some more
> questions, comments, feel free to ask here or in either of the following
> issues:
>
> https://gitlab.com/libssh/libssh-mirror/-/issues/65
> https://gitlab.com/libssh/libssh-mirror/-/issues/124
>
> Regards,
> --
> Jakub Jelen
> Crypto Team, Security Engineering
> Red Hat, Inc.
>
>