RE: libssh sftp download performance

["Mark Riordan" <mriordan@xxxxxxxxxxxx>]

Andreas said:
> You know that sftp_read is a blocking implementation 
> and there is a sftp_async_read?

Wow, that made a big difference!
Now libssh is the fastest at downloading (by a very narrow margin),
instead of the slowest.

The latest master from git is slightly faster than 0.4.6 at
downloading in async mode, though much slower in sync mode:

http://60bits.net/sni/libssh2-perf.htm

I have a feeling I might be approaching the filesystem throughput
limits on this machine; I may run a separate benchmark for this.
But that doesn't really matter, because I am now getting the
performance I wanted.

I enhanced my code to look like this:

   int outhand = open(settings.localfile.c_str(), O_WRONLY | O_CREAT |
_O_BINARY | O_TRUNC, 0700);
   if(!outhand) {
      perror("opening output file");
      return 108;
   }
   long bytes_xferred=0;
   int max_observed_len = 0;
   int len=0, idxNext=0;
   if(settings.asynch_io) {
      int ReadIDs[MAX_READ_IDS];
      printf("Using async reads for downloads with %d read-aheads.\n",
settings.read_aheads);
      // Send multiple read requests to the server, so they'll 
      // eventually come streaming in without delay.
      for(int idx=0; idx<settings.read_aheads; idx++) {
         ReadIDs[idx] = sftp_async_read_begin(my_sftp_file, BUFSIZE);
      }
      do {
         len = sftp_async_read(my_sftp_file, data, BUFSIZE,
ReadIDs[idxNext]);
         ReadIDs[idxNext] = sftp_async_read_begin(my_sftp_file, BUFSIZE);
         idxNext = (1+idxNext) % settings.read_aheads;
         if(len > 0) {
            write(outhand, data, len);
            bytes_xferred += len; 
         } else if(0==len) {
            break;
         } else {
            printf("*** Error on read\n");
            break;
         }
      } while(true);
   } else {
      printf("Using sync reads for downloads.\n");
      while((len=sftp_read(my_sftp_file,data,sizeof(data))) > 0){
         write(outhand, data, len);
         bytes_xferred += len; 
      }
   }
   close(outhand);
   sftp_close(my_sftp_file);

Interestingly, it didn't help to have any more than 2 read-aheads.

Thanks for your help.

Mark