Re: Multithreading with libssh

[g4-lisz@xxxxxxxxxxxx]

H Simon,

OK I understand now what you want to do. But I doubt that several
threads in parallel are faster when each thread has to open the session.
This is a lot of communication overhead, when it's to the same destionation.

Last but not least it also depends on how the server side handles these
parallel connections...

Isn't there a way to start a script remotely which executes the 1500
commands and pipes back the results to you in some way?

I wrote a tunneling client which uses only two threeads: One accepting
the local connections and then a worker thead which reads and writes the
packages from the sockets to channels and vice versa. So each channel
has his time slice... But it's quite fast. And you can open as many
channels as you want. From the perspective of the packet's stream it's
synchronous but from the channels perspective it's kind of asynchronous.

Till

On 26.09.19 14:26, Simon Moselewski wrote:
> Hi Till,
>
> thanks for the quick answer.
> I want to collect the data as fast as possible.
> All channels are to the same destination. 
>
> Problem is: You can only run one ssh_channel_request_exec per Channel.
> So I need 1500 channels. I would like to Open them as simultaneously
> as possible to Save time.
>
> Since the Session cannot be Shared between threads I need multiple
> sessions.
>
> I could also start 20 threads with one session each and 75 channels
> each that are used one after the other. But right now threading
> doesn't seem to work at all. 
>
> Is my code correct for the threading approach or am I missing something?
>
>
> Regards
> Simon
>
>
> -------- Ursprüngliche Nachricht --------
> Von: g4-lisz@xxxxxxxxxxxx
> Datum: 26.09.19 14:02 (GMT+01:00)
> An: libssh@xxxxxxxxxx
> Betreff: Re: Multithreading with libssh
>
> Hi Simon.
>
> You want to open the 1500 channels simultaneously? If not, then why you
> need more than one thread? And each channel is to a different
> destination? If not, then why opening a new session each time? You can't
> share the session between threads, but you can re-use it...
>
> IMHO it's simpler to approach what you want than you think ;-)
>
> Till
>
> On 26.09.19 13:09, Simon Moselewski wrote:
> > Hi everyone,
> >
> > I am using libssh to connect to a remote host, run commands on it and
> > read the results. Since I am doing this for 1500 commands I want to
> > use threads to parallelize that.
> >
> > I am using gcc compiler with flags -libssh -pthread and -lssh_threads.
> >
> > My threadless approach was: 
> > open session
> > repeat 1500 times {
> >     open channel
> >     run remote command
> >     close channel
> > }
> > close session
> >
> > I read in the docu, that sessions cannot be shared over multiple
> > threads, so with the threadful approach I do it differently:
> > repeat 1500 times {
> >     open session
> >     open channel
> >     run remote command
> >     close channel
> >     close session
> > }
> >
> > The threadful programm takes longer than my threadless approach so I
> > guess something with the threading is not working correctly.
> >
> > I am very inexperienced with threads and would need some help here.
> >
> > Thanks in advance for any hints!!!
> > Regards
> > Simon
> >
> >
> >
> > My threadful code (shortened):
> >
> > ssh_threads_set_callbacks(ssh_threads_get_pthread());
> > ssh_init();
> > std::mutex mutex;
> > std::vector<std::future<void>> tasks;
> > tasks.reserve(16);
> > for (list<Metric>::iterator it = metrics.begin(); it != metrics.end();
> > it++) {
> >     Metric metric = *it;
> >     tasks.emplace_back(std::async(std::launch::async, [&ip, &user,
> > &pw, metric, i, &mutex]() {
> >         std::lock_guard<std::mutex> lock(mutex);
> >         try {
> >             ssh_session session = ssh_new();
> >             ssh_options_set(session, SSH_OPTIONS_HOST, IP);
> >             rc = ssh_userauth_password(session, USERNAME, PASSWORD);
> >             ssh_channel channel = ssh_channel_new(session);
> >             rc = ssh_channel_open_session(channel);
> >             rc = ssh_channel_request_exec(channel, COMMAND);
> >             // get output
> >             char buffer[32] = {0};
> >             int nbytes;
> >             nbytes = ssh_channel_read(channel, buffer,
> > sizeof(buffer)-1, 0);
> >             ssh_channel_send_eof(channel);
> >             //get exit code
> >             int exitCode = -1;
> >             exitCode = ssh_channel_get_exit_status(channel);
> >             ssh_channel_close(channel);
> >             ssh_channel_free(channel);
> >             ssh_disconnect(session);
> >             ssh_free(session);
> >          }
> >          catch(...) {
> >               //something
> >          }
> >      }));
> > }
> > std::for_each(std::begin(tasks), std::end(tasks), [](auto& task) {
> >      task.get();
> > });
> > std::cout << "Done.\n";
> > ssh_finalize();
>