Re: multithreading for a single session

[Gearoid Murphy <gearoid.murphy@xxxxxx>]

It is possible to use a single session in a multi-threaded context as 
long as you protect access to the session (and associated channels) 
using a session mutex. It's very inefficient however, as every channel 
read operation must timeout in order to release the mutex when the 
associated channel is inactive so other channels can be processed. This 
timeout is the source of the performance penalty, it decimates 
bandwidth. I've seen single threaded applications capable of 30-40 mbs 
being reduced to 256kps using the multi-threaded approach.

My favored solution these days is to use a boost io_service to peek at 
the session socket, whenever data needs to be processed, an asynchronous 
handler is invoked which processes the active channels.

On 14/03/12 13:38, Eddy Valdes wrote:
> Jeetu,
>
> I seem to recall somebody telling me that the if you are going to use 
> threads, then you must use a separate session for each thread. The 
> application that used libssh that I wrote and used to maintain had a 
> separate session for each thread and it worked well.
>
> Eddy
>
> On 03/14/2012 09:31 AM, jeetu.golani@xxxxxxxxx wrote:
> > Hi guys,
> >
> > Just wanted to know if anyone has had a chance to look into the
> > possibility of multithreading over a single session which I had
> > discussed in any earlier mail here (pasted below).
> >
> > For now for my open source eBrainPool (ebrain.in) project I have gone
> > ahead with piped openssh server and daemon however this is not ideal
> > for variety of reasons. I would love to use libssh however the issues
> > I highlighted earlier prevent me from doing that.
> >
> > Please let me know how I can help with this :)....thanks again for all
> > the help everyone here has provided me with in the past :)
> >
> > Regards,
> > Jeetu
> > ebrain.in | Beehive Computing
> > Share your software and computing resources - discover and run
> > software from any device around you.
> >
> > On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
> > <jeetu.golani@xxxxxxxxx>  wrote:
> > > oops forgot to mention the git link where this code can be seen
> > > ;)....git://ebrain.in:/libssh-server.git :)
> > >
> > > sorry
> > >
> > >
> > > On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
> > > <jeetu.golani@xxxxxxxxx>  wrote:
> > > > Hi Aris,
> > > >
> > > > I've been continuing work with my libssh server based on 
> > > > suggestions you
> > > > had made with regards multithreading when we were debugging issues 
> > > > where I
> > > > was seeing bad packet length, HMAC errors et al.
> > > >
> > > > I refactored my code as you had suggested (below) such that there 
> > > > would be
> > > > only one thread per session, although there could be multiple 
> > > > connects i.e.
> > > > sessions. This worked fantastic for me and I was able to solve the 
> > > > packet
> > > > corruption issues caused due to a race condition with multiple ssh 
> > > > loops
> > > > running. I could therefore have different ssh clients connect to 
> > > > the same
> > > > instance of the server and X forward simple x clients like xcalc, 
> > > > xclock,
> > > > xeyes, xgc.
> > > >
> > > > Unfortunately as soon as I tried to forward more complex 
> > > > applications such
> > > > as libreoffice, chromium etc things weren't working. Further 
> > > > analysis and a
> > > > look at the openssh code showed that these applications spawn multiple
> > > > connects to the same $DISPLAY. Therefore as I understand it (could 
> > > > be wrong)
> > > > the server needs to accept and process different event connections 
> > > > on the
> > > > same session.
> > > >
> > > > I have refactored the code again to account for this but basically 
> > > > this
> > > > reintroduces the same race condition with multiple event handling 
> > > > we had
> > > > seen earlier and while programs like libreoffice, chromium etc more 
> > > > further
> > > > along they eventually crash due to bad packet length, mac errors,etc.
> > > >
> > > > I tried to protect the event handling function ssh_event_dopoll with a
> > > > mutex hoping this will resolve the issue, however it didn't make any
> > > > difference.
> > > >
> > > > I'm at my wits end here and could appreciate advice here. One 
> > > > obvious way
> > > > is for me to somehow refactor my code in a way that eludes me at 
> > > > the moment
> > > > to get around this issue. The other could be if libssh itself would 
> > > > handle
> > > > multithreaded event handling for the same session differently or if 
> > > > I am
> > > > somehow using it in a wrong way.
> > > >
> > > > I have created a git repo of my code so anyone can study it. I have 
> > > > not
> > > > yet put a GPL license header to the code (just realized that ;) ) 
> > > > however I
> > > > don't care and in fact would be honoured if people would find some 
> > > > use for
> > > > this code. The code will eventually be part of the eBrainPool 
> > > > (ebrain.in)
> > > > project which I am working on which itself is GPLv3 - so feel free 
> > > > to use it
> > > > in any way you want.
> > > >
> > > > For now I could really use some ideas on how I could either 
> > > > refactor my
> > > > code to avoid the above issues or somehow if libssh could be 
> > > > modified to
> > > > handle multithreading for the same session differently.
> > > >
> > > > Thanks,
> > > >
> > > > Bye for now
> > > > Jeetu
> > > > ebrain.in | Beehive Computing
> > > > Share your software and computing resources - discover and run 
> > > > software
> > > > from any device around you.
> > > >
> > > >
> > > >
> > > > On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
> > > > <jeetu.golani@xxxxxxxxx>  wrote:
> > > > > Hi Aris,
> > > > >
> > > > > > took me a lot of time and head scratching, but I narrowed it down 
> > > > > > to a
> > > > > > threading problem. Two of your threads are running ssh loops at the
> > > > > > same
> > > > > I can completely appreciate the effort you must have gone through to
> > > > > debug this :)...threading issues can be pesky and since I didn't
> > > > > understand the internals of libssh it was proving quite difficult for
> > > > > me to trace this out.....thank you so much Aris :)
> > > > >
> > > > > > I hacked your example so the server_thread() functions leaves 
> > > > > > early. It
> > > > > > doesn't remove the race but reduces its window. now it works for me,
> > > > > > and I'm
> > > > > > able to run an xterm with it.
> > > > > > I strongly recommend that you refactor your code in a singlethread
> > > > > > design,
> > > > > > and that you move the x11 callbacks back in the server loop.
> > > > > I just did a cursory test of your code and it definitely runs much
> > > > > better from what I could see :)....I will take your advice to heed 
> > > > > and
> > > > > try to refactor the code.
> > > > >
> > > > > > Let's discuss what libssh can do better to help you write better 
> > > > > > code.
> > > > > I'm a little concerned, I would ultimately need to handle multiple
> > > > > sessions.....the code you saw dealt with a single session....do you
> > > > > think this would also trigger similar issues?....what precautions do
> > > > > you think I need to keep in mind while I attempt this?
> > > > >
> > > > > Thanks so much Aris, I'm sooooooo glad to finally have the answer to
> > > > > my question and know that this was a problem in my code....now I can
> > > > > dedicate my energies in this direction :)
> > > > >
> > > > > Bye for now
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
> > > > > wrote:
> > > > > > Hi Jeetu,
> > > > > >
> > > > > > took me a lot of time and head scratching, but I narrowed it down 
> > > > > > to a
> > > > > > threading problem. Two of your threads are running ssh loops at the
> > > > > > same
> > > > > > time, and this cause races. The probability that the race causes
> > > > > > problem can
> > > > > > be high, because there's a trigger (data on socket) and both threads
> > > > > > run
> > > > > > almost the same code at the same time.
> > > > > >
> > > > > > Thread 1:      message = ssh_message_get(session);
> > > > > > in server_thread()
> > > > > > Thread 2:                 ssh_event_dopoll(event, 1000);
> > > > > > in wait_for_something()
> > > > > >
> > > > > > I hacked your example so the server_thread() functions leaves 
> > > > > > early. It
> > > > > > doesn't remove the race but reduces its window. now it works for me,
> > > > > > and I'm
> > > > > > able to run an xterm with it.
> > > > > > I strongly recommend that you refactor your code in a singlethread
> > > > > > design,
> > > > > > and that you move the x11 callbacks back in the server loop.
> > > > > >
> > > > > > Let's discuss what libssh can do better to help you write better 
> > > > > > code.
> > > > > >
> > > > > > Kr,
> > > > > >
> > > > > > Aris
> > > > > > Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > > Hi Aris,
> > > > > > >
> > > > > > > No probs and I completely understand and appreciate your 
> > > > > > > interest in
> > > > > > > looking into this....please let me know how I can help further :)
> > > > > > >
> > > > > > > Bye for now
> > > > > > >
> > > > > > > On Wed, Nov 9, 2011 at 3:32 PM, Aris 
> > > > > > > Adamantiadis<aris@xxxxxxxxxxxx>
> > > > > > >   wrote:
> > > > > > > > Hi Jeetu,
> > > > > > > >
> > > > > > > > I apologize, still had no time to check it. I'm pretty busy and 
> > > > > > > > all
> > > > > > > > my
> > > > > > > > free time is swallowed by another project. I hope I can free 
> > > > > > > > myself
> > > > > > > > an
> > > > > > > > hour or two next week to debug this.
> > > > > > > >
> > > > > > > > kr,
> > > > > > > >
> > > > > > > > Aris
> > > > > > > >
> > > > > > > > Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > > > > Hi Aris,
> > > > > > > > >
> > > > > > > > > Just wanted to check in if you've had a chance to try out the 
> > > > > > > > > libssh
> > > > > > > > > server code I've sent and reproduce the errors I've been seeing?
> > > > > > > > >
> > > > > > > > > Thanks so much again for looking into this.
> > > > > > > > >
> > > > > > > > > Bye for now
> > > > > > > > >
> > > > > > > > > On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
> > > > > > > > > <jeetu.golani@xxxxxxxxx>    wrote:
> > > > > > > > > > Hi Aris,
> > > > > > > > > >
> > > > > > > > > > I'm attaching my proof of concept server code
> > > > > > > > > > as.....ebpsshd-singlesession.c has compile instructions at the
> > > > > > > > > > beginning of the code. You will also need to generate a key.h 
> > > > > > > > > > file
> > > > > > > > > > that holds the public key of the user who will be connecting to
> > > > > > > > > > this
> > > > > > > > > > server - this is tempoarary since as of now I'm not reading this
> > > > > > > > > > info
> > > > > > > > > > from an authorized_keys or something similar.
> > > > > > > > > >
> > > > > > > > > > Just create a key.h file in the same directory and put something
> > > > > > > > > > like :
> > > > > > > > > >
> > > > > > > > > > #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
> > > > > > > > > >
> > > > > > > > > > Also as of now ebpsshd-singlesession listens in on port 2000. So
> > > > > > > > > > ssh
> > > > > > > > > > should connect to that port.
> > > > > > > > > >
> > > > > > > > > > I also have a libssh-project-wrapper script that allows me to 
> > > > > > > > > > try
> > > > > > > > > > this
> > > > > > > > > > out without needing to install the libssh i've built. It 
> > > > > > > > > > basically
> > > > > > > > > > has
> > > > > > > > > > the content:
> > > > > > > > > >
> > > > > > > > > > #!/bin/sh
> > > > > > > > > >
> > > > > > > > > > export
> > > > > > > > > >
> > > > > > > > > > LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads 
> > > > > > > > > >
> > > > > > > > > > ./$1
> > > > > > > > > >
> > > > > > > > > > I have been testing this code with simple examples like xeyes 
> > > > > > > > > > and
> > > > > > > > > > xcalc. For some reason xcalc shows the problem much sooner than
> > > > > > > > > > with
> > > > > > > > > > xeyes, maybe because of the volume of data being transmitted 
> > > > > > > > > > to and
> > > > > > > > > > fro?
> > > > > > > > > >
> > > > > > > > > > This is proof of concept code with a lot of fiddling with buffer
> > > > > > > > > > sizes
> > > > > > > > > > as I have been trying to study if any of that makes an impact
> > > > > > > > > > however
> > > > > > > > > > please point out any way you think this can be improved on :)
> > > > > > > > > >
> > > > > > > > > > I'm sorry to drop this in your lap especially if it turns out it
> > > > > > > > > > was
> > > > > > > > > > some server side code issue, however I completely appreciate 
> > > > > > > > > > your
> > > > > > > > > > help
> > > > > > > > > > on this. I would like to squash this bug regardless of where it
> > > > > > > > > > lies
> > > > > > > > > > i.e. in my code or libssh, unfortunately my understanding of 
> > > > > > > > > > libssh
> > > > > > > > > > and the ssh protocol is a little limited. However I do not 
> > > > > > > > > > want to
> > > > > > > > > > put
> > > > > > > > > > all of this load completely in your lap and if you share your
> > > > > > > > > > thoughts
> > > > > > > > > > and there's something you would like me to look into then please
> > > > > > > > > > let
> > > > > > > > > > me know.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Jeetu
> > > > > > > > > > ebrain.in | Beehive Computing
> > > > > > > > > > Discover and run software from any device around you - an open
> > > > > > > > > > source
> > > > > > > > > > (GPL) project.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
> > > > > > > > > > <jeetu.golani@xxxxxxxxx>    wrote:
> > > > > > > > > > > Hi Aris,
> > > > > > > > > > >
> > > > > > > > > > > > I think I'll need a proof-of-concept code to debug this. Would
> > > > > > > > > > > > you
> > > > > > > > > > > > mind
> > > > > > > > > > > > sharing your code, or it's not possible (too much integration
> > > > > > > > > > > > with
> > > > > > > > > > > > existing code).
> > > > > > > > > > > No problem at all :) The code is an independent unit as of now
> > > > > > > > > > > since I
> > > > > > > > > > > wanted to make it work before I integrate it within my open 
> > > > > > > > > > > source
> > > > > > > > > > > project (eBrainPool)
> > > > > > > > > > >
> > > > > > > > > > > I'll mail this out to you tomm (not on the machine with the 
> > > > > > > > > > > code
> > > > > > > > > > > right
> > > > > > > > > > > now :) )
> > > > > > > > > > >
> > > > > > > > > > > Thanks so much for looking into this.....truly appreciate it :)
> > > > > > > > > > >
> > > > > > > > > > > Bye for now
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 21, 2011 at 7:26 PM, Aris
> > > > > > > > > > > Adamantiadis<aris@xxxxxxxxxxxx>
> > > > > > > > > > >   wrote:
> > > > > > > > > > > > Hi Jeetu,
> > > > > > > > > > > >
> > > > > > > > > > > > By seeing your logs, I understand this:
> > > > > > > > > > > > Both side have a hmac error. The openssh client sees it first,
> > > > > > > > > > > > sends
> > > > > > > > > > > > a
> > > > > > > > > > > > disconnect (that works), then there's the error in the libssh
> > > > > > > > > > > > log.
> > > > > > > > > > > >
> > > > > > > > > > > > I think I'll need a proof-of-concept code to debug this. Would
> > > > > > > > > > > > you
> > > > > > > > > > > > mind
> > > > > > > > > > > > sharing your code, or it's not possible (too much integration
> > > > > > > > > > > > with
> > > > > > > > > > > > existing code).
> > > > > > > > > > > >
> > > > > > > > > > > > If so, do you think I can reproduce the problem by "fixing"
> > > > > > > > > > > > samplesshd
> > > > > > > > > > > > to make new X11 channels connection connect to the local 
> > > > > > > > > > > > X11 unix
> > > > > > > > > > > > socket ?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > Aris
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > > > > > > > > > This is not a problem and shouldn't cause invalid MAC 
> > > > > > > > > > > > > > errors.
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > travel so I'll look at the log when i'm back.
> > > > > > > > > > > > > Thanks so much Aris :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Oct 18, 2011 at 5:52 PM, Aris
> > > > > > > > > > > > > Adamantiadis<aris@xxxxxxxxxxxx>    wrote:
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is not a problem and shouldn't cause invalid MAC 
> > > > > > > > > > > > > > errors.
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > travel so I'll look at the log when i'm back.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Aris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > debug3: Incorrect RSA1 identifier
> > > > > > > > > > > > > > > debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
> > > > > > > > > > > > > > > public
> > > > > > > > > > > > > > > key
> > > > > > > > > > > > > > > debug2: key_type_from_name: unknown key type '-----BEGIN'
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Oct 18, 2011 at 03:22:08PM +0500,
> > > > > > > > > > > > > > > jeetu.golani@xxxxxxxxx
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > Hi Aris,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've attached a log of the libssh server
> > > > > > > > > > > > > > > > (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
> > > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > (log-ssh-1-18102011.txt).
> > > > > > > > > > > > > > > Greetings
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Stefan Kuttler ==*== nc.netbeisser.de