Re: Proposal for High level sftp api for uploads and downloads

[Andreas Schneider <asn@xxxxxxxxxxxxxx>]

On Wednesday, 31 May 2023 09:38:49 CEST Eshan Kelkar wrote:
> Libssh needs a high level api for uploading and
> downloading files. Functions like sftp_put() and
> sftp_get() need to be introduced which the users
> can call simply to upload and download files instead
> of having to write their own functions to perform uploads
> and downloads using the low level read/write api's.
> 
> This mail suggests approaches that can be followed
> to develop that api.
> 
> Some terminology :
> 1. local_fd - is the file descriptor [obtained using open()
> or creat()] of a file on the local machine of the user.
> 
> 2.  remote_file  - is a sftp file handle [obtained using
> sftp_open()] of a file on the connected server.
> 
> 3. concurrent_requests -  This is the number of requests
> we initially issue before trying to get the responses.
> 
> Say you are downloading and you have 20 concurrent
> requests, this means that first we have issued 20 read
> requests, and then when we are trying to get/wait for the
> response of the first request, the server may be processing
> or may have processed the other 19 requests (This is the
> advantage of asynchronous request-response pattern over
> synchronous request-response where we would have issued
> the next request [and server would have processed it] only
> after getting a response of the first request).
> 
> And after getting the response of the first request, we issue
> another request if needed and then try to get the responses
> of the other outstanding requests and repeat this procedure
> for them too.
> 
> Though you can see this name "concurrent_requests" is not
> the best as the requests are not being issued in a concurrent
> manner here, neither are they being processed concurrently by
> the server. So if a better name comes to your mind, kindly suggest.
> 
> 4. chunk_size -  The number of bytes we usually issue a
> read/write request for, if the number of bytes to read/write
> is less than this chunk_size then the request is issued for
> those many number of bytes and not chunk_size number
> of bytes.
> 
> How the api can look from User's perspective :
> 
> Approach 1 :
> ----------------------------------------------
> The user has to pass each of the 4 required things
> for the transfer to the put, get functions.
> 
> For the default values of concurrent_requests and
> chunk_size we can provide macros (which expand
> to default values suggested by libssh) which the user
> can use if he/she is not interested in setting some
> custom values.
> 
> int sftp_put(int local_fd,
>                    sftp_file remote_file,
>                    int concurrent_requests,
>                    size_t chunk_size);
> 
> int sftp_get(int local_fd,
>                    sftp_file remote_file,
>                    int concurrent_requests,
>                    size_t chunk_size);

Thanks for the write up.

If you use these function which look simple to use you want to also add a 
progress callback passed down and a userdata pointer.

This way a UI (also cmdline) could display some progress.


-- 
Andreas Schneider                 asn@xxxxxxxxxxxxxx
GPG-ID:     8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D