Re: sftp_write failure question

[Eshan Kelkar <eshankelkar@xxxxxxxxxxxxx>]

(While writing this, I assumed that an openssh server was
used for these tests)

OpenSSH server has limits for max sftp data packet ([1])
which is 256KB for read/write packets and allows a max read
of 255 KB for read requests ([2])

The current libssh master's sftp client tries to get these* limits
from the server and then uses these limits to cap read/write
done using sftp_read() and sftp_write().

A user can obtain these limits using sftp_limits() ([3]) to set the
appropriate chunk/buffer sizes for their data and these are the
max values for which user code won't get short reads/writes
(unless the end of file is encountered during reads). Anything
above the limits will lead to libssh sending requests for shorter
number of bytes leading to short read/writes.

As a side note, if sftp_write(to, data, len) returns value
!= len, then it isn't necessarily an error. It may be a short write.
A more appropriate check for applications to check for error
is that the return value is negative. If not, then a return value != len
indicates a short write (not an error, so ssh and sftp error won't be
set in this case). [I admit that the usage examples should be
updated to use this suggested style]

*By "these" limits I didn't mean the sftp data packet size. I meant
the max number of file data bytes that can be present in a
read/write packet as per the server.

[1] Openssh packet limit
https://github.com/openssh/openssh-portable/blob/7717b9e9155209916cc6b4b4b54f4e8fa578e889/sftp-common.h#L29

[2] Openssh read limit
https://github.com/openssh/openssh-portable/blob/7717b9e9155209916cc6b4b4b54f4e8fa578e889/sftp-server.c#L63

[3] Documentation for sftp_limits():
https://gitlab.com/libssh/libssh-mirror/-/blob/master/include/libssh/sftp.h#L1126
https://gitlab.com/libssh/libssh-mirror/-/blob/master/include/libssh/sftp.h#L209

[2] Documentation for usage of sftp_write() :
https://gitlab.com/libssh/libssh-mirror/-/blob/master/include/libssh/sftp.h#L577
)

Hope this helps,
Regards,
Eshan Kelkar



On Tue, Jul 16, 2024 at 5:54 AM <reid.thompson@xxxxxxxxxxxxxxx> wrote:

> with code of the format below ( pulled from the examples ) where source
> is say 75MB in size.
>
> ...snip...
>     while ((len = sftp_read(source, data, 4096)) > 0) {
>         if (sftp_write(to, data, len) != len) {
>             fprintf(stderr, "Error writing %d bytes: %s\n",
>                     len, ssh_get_error(session));
>             sftp_close(to);
>             sftp_close(fichier);
>             goto end;
>         }
>     }
> ...snip...
>
> read/write will successfully transfer the complete file.
>
> if the code is altered to
>
> ...snip...
> BUF_SIZE=xxxxx
> char data[BUF_SIZE] = {0};
> ...
>     while ((len = sftp_read(source, data, BUF_SIZE)) > 0) {
>         if (sftp_write(to, data, len) != len) {
>             fprintf(stderr, "Error writing %d bytes: %s\n",
>                     len, ssh_get_error(session));
>             sftp_close(to);
>             sftp_close(fichier);
>             goto end;
>         }
>     }
> ...snip...
>
> read/write will succeed until the value of BUF_SIZE is set to 262120 or
> greater.
> at which point sftp_write will write BUF_SIZE and sftp_packet_read will
> fail with "Received EOF while reading sftp packet size" and the SFTP
> server will log a bad message error:
>     "Jul 15 17:28:28 endpoint sftp-server[1537206]: error: bad message
> from 100.98.195.26 local user rthompso"
>
> Is this expected/is this an internal limit to the amount that can be
> written per invocation?
> Is there a way to increase the amount of data that can be successfully
> written per invocation?
>
>
> Thanks,
> Reid
>
>
>
>