Re: multithreading for a single session

[Eddy Valdes <evaldes@xxxxxxxxxxxxxxxxxxxxxx>]

Jeetu,

I seem to recall somebody telling me that the if you are going to use 
threads, then you must use a separate session for each thread. The 
application that used libssh that I wrote and used to maintain had a 
separate session for each thread and it worked well.

Eddy

On 03/14/2012 09:31 AM, jeetu.golani@xxxxxxxxx wrote:
> Hi guys,
>
> Just wanted to know if anyone has had a chance to look into the
> possibility of multithreading over a single session which I had
> discussed in any earlier mail here (pasted below).
>
> For now for my open source eBrainPool (ebrain.in) project I have gone
> ahead with piped openssh server and daemon however this is not ideal
> for variety of reasons. I would love to use libssh however the issues
> I highlighted earlier prevent me from doing that.
>
> Please let me know how I can help with this :)....thanks again for all
> the help everyone here has provided me with in the past :)
>
> Regards,
> Jeetu
> ebrain.in | Beehive Computing
> Share your software and computing resources - discover and run
> software from any device around you.
>
> On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
> <jeetu.golani@xxxxxxxxx>  wrote:
> > oops forgot to mention the git link where this code can be seen
> > ;)....git://ebrain.in:/libssh-server.git :)
> >
> > sorry
> >
> >
> > On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
> > <jeetu.golani@xxxxxxxxx>  wrote:
> > > Hi Aris,
> > >
> > > I've been continuing work with my libssh server based on suggestions you
> > > had made with regards multithreading when we were debugging issues where I
> > > was seeing bad packet length, HMAC errors et al.
> > >
> > > I refactored my code as you had suggested (below) such that there would be
> > > only one thread per session, although there could be multiple connects i.e.
> > > sessions. This worked fantastic for me and I was able to solve the packet
> > > corruption issues caused due to a race condition with multiple ssh loops
> > > running. I could therefore have different ssh clients connect to the same
> > > instance of the server and X forward simple x clients like xcalc, xclock,
> > > xeyes, xgc.
> > >
> > > Unfortunately as soon as I tried to forward more complex applications such
> > > as libreoffice, chromium etc things weren't working. Further analysis and a
> > > look at the openssh code showed that these applications spawn multiple
> > > connects to the same $DISPLAY. Therefore as I understand it (could be wrong)
> > > the server needs to accept and process different event connections on the
> > > same session.
> > >
> > > I have refactored the code again to account for this but basically this
> > > reintroduces the same race condition with multiple event handling we had
> > > seen earlier and while programs like libreoffice, chromium etc more further
> > > along they eventually crash due to bad packet length, mac errors,etc.
> > >
> > > I tried to protect the event handling function ssh_event_dopoll with a
> > > mutex hoping this will resolve the issue, however it didn't make any
> > > difference.
> > >
> > > I'm at my wits end here and could appreciate advice here. One obvious way
> > > is for me to somehow refactor my code in a way that eludes me at the moment
> > > to get around this issue. The other could be if libssh itself would handle
> > > multithreaded event handling for the same session differently or if I am
> > > somehow using it in a wrong way.
> > >
> > > I have created a git repo of my code so anyone can study it. I have not
> > > yet put a GPL license header to the code (just realized that ;) ) however I
> > > don't care and in fact would be honoured if people would find some use for
> > > this code. The code will eventually be part of the eBrainPool (ebrain.in)
> > > project which I am working on which itself is GPLv3 - so feel free to use it
> > > in any way you want.
> > >
> > > For now I could really use some ideas on how I could either refactor my
> > > code to avoid the above issues or somehow if libssh could be modified to
> > > handle multithreading for the same session differently.
> > >
> > > Thanks,
> > >
> > > Bye for now
> > > Jeetu
> > > ebrain.in | Beehive Computing
> > > Share your software and computing resources - discover and run software
> > > from any device around you.
> > >
> > >
> > >
> > > On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
> > > <jeetu.golani@xxxxxxxxx>  wrote:
> > > > Hi Aris,
> > > >
> > > > > took me a lot of time and head scratching, but I narrowed it down to a
> > > > > threading problem. Two of your threads are running ssh loops at the
> > > > > same
> > > > I can completely appreciate the effort you must have gone through to
> > > > debug this :)...threading issues can be pesky and since I didn't
> > > > understand the internals of libssh it was proving quite difficult for
> > > > me to trace this out.....thank you so much Aris :)
> > > >
> > > > > I hacked your example so the server_thread() functions leaves early. It
> > > > > doesn't remove the race but reduces its window. now it works for me,
> > > > > and I'm
> > > > > able to run an xterm with it.
> > > > > I strongly recommend that you refactor your code in a singlethread
> > > > > design,
> > > > > and that you move the x11 callbacks back in the server loop.
> > > > I just did a cursory test of your code and it definitely runs much
> > > > better from what I could see :)....I will take your advice to heed and
> > > > try to refactor the code.
> > > >
> > > > > Let's discuss what libssh can do better to help you write better code.
> > > > I'm a little concerned, I would ultimately need to handle multiple
> > > > sessions.....the code you saw dealt with a single session....do you
> > > > think this would also trigger similar issues?....what precautions do
> > > > you think I need to keep in mind while I attempt this?
> > > >
> > > > Thanks so much Aris, I'm sooooooo glad to finally have the answer to
> > > > my question and know that this was a problem in my code....now I can
> > > > dedicate my energies in this direction :)
> > > >
> > > > Bye for now
> > > >
> > > >
> > > >
> > > > On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
> > > > wrote:
> > > > > Hi Jeetu,
> > > > >
> > > > > took me a lot of time and head scratching, but I narrowed it down to a
> > > > > threading problem. Two of your threads are running ssh loops at the
> > > > > same
> > > > > time, and this cause races. The probability that the race causes
> > > > > problem can
> > > > > be high, because there's a trigger (data on socket) and both threads
> > > > > run
> > > > > almost the same code at the same time.
> > > > >
> > > > > Thread 1:      message = ssh_message_get(session);
> > > > > in server_thread()
> > > > > Thread 2:                 ssh_event_dopoll(event, 1000);
> > > > > in wait_for_something()
> > > > >
> > > > > I hacked your example so the server_thread() functions leaves early. It
> > > > > doesn't remove the race but reduces its window. now it works for me,
> > > > > and I'm
> > > > > able to run an xterm with it.
> > > > > I strongly recommend that you refactor your code in a singlethread
> > > > > design,
> > > > > and that you move the x11 callbacks back in the server loop.
> > > > >
> > > > > Let's discuss what libssh can do better to help you write better code.
> > > > >
> > > > > Kr,
> > > > >
> > > > > Aris
> > > > > Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > Hi Aris,
> > > > > >
> > > > > > No probs and I completely understand and appreciate your interest in
> > > > > > looking into this....please let me know how I can help further :)
> > > > > >
> > > > > > Bye for now
> > > > > >
> > > > > > On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
> > > > > >   wrote:
> > > > > > > Hi Jeetu,
> > > > > > >
> > > > > > > I apologize, still had no time to check it. I'm pretty busy and all
> > > > > > > my
> > > > > > > free time is swallowed by another project. I hope I can free myself
> > > > > > > an
> > > > > > > hour or two next week to debug this.
> > > > > > >
> > > > > > > kr,
> > > > > > >
> > > > > > > Aris
> > > > > > >
> > > > > > > Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > > > Hi Aris,
> > > > > > > >
> > > > > > > > Just wanted to check in if you've had a chance to try out the libssh
> > > > > > > > server code I've sent and reproduce the errors I've been seeing?
> > > > > > > >
> > > > > > > > Thanks so much again for looking into this.
> > > > > > > >
> > > > > > > > Bye for now
> > > > > > > >
> > > > > > > > On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
> > > > > > > > <jeetu.golani@xxxxxxxxx>    wrote:
> > > > > > > > > Hi Aris,
> > > > > > > > >
> > > > > > > > > I'm attaching my proof of concept server code
> > > > > > > > > as.....ebpsshd-singlesession.c has compile instructions at the
> > > > > > > > > beginning of the code. You will also need to generate a key.h file
> > > > > > > > > that holds the public key of the user who will be connecting to
> > > > > > > > > this
> > > > > > > > > server - this is tempoarary since as of now I'm not reading this
> > > > > > > > > info
> > > > > > > > > from an authorized_keys or something similar.
> > > > > > > > >
> > > > > > > > > Just create a key.h file in the same directory and put something
> > > > > > > > > like :
> > > > > > > > >
> > > > > > > > > #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
> > > > > > > > >
> > > > > > > > > Also as of now ebpsshd-singlesession listens in on port 2000. So
> > > > > > > > > ssh
> > > > > > > > > should connect to that port.
> > > > > > > > >
> > > > > > > > > I also have a libssh-project-wrapper script that allows me to try
> > > > > > > > > this
> > > > > > > > > out without needing to install the libssh i've built. It basically
> > > > > > > > > has
> > > > > > > > > the content:
> > > > > > > > >
> > > > > > > > > #!/bin/sh
> > > > > > > > >
> > > > > > > > > export
> > > > > > > > >
> > > > > > > > > LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
> > > > > > > > > ./$1
> > > > > > > > >
> > > > > > > > > I have been testing this code with simple examples like xeyes and
> > > > > > > > > xcalc. For some reason xcalc shows the problem much sooner than
> > > > > > > > > with
> > > > > > > > > xeyes, maybe because of the volume of data being transmitted to and
> > > > > > > > > fro?
> > > > > > > > >
> > > > > > > > > This is proof of concept code with a lot of fiddling with buffer
> > > > > > > > > sizes
> > > > > > > > > as I have been trying to study if any of that makes an impact
> > > > > > > > > however
> > > > > > > > > please point out any way you think this can be improved on :)
> > > > > > > > >
> > > > > > > > > I'm sorry to drop this in your lap especially if it turns out it
> > > > > > > > > was
> > > > > > > > > some server side code issue, however I completely appreciate your
> > > > > > > > > help
> > > > > > > > > on this. I would like to squash this bug regardless of where it
> > > > > > > > > lies
> > > > > > > > > i.e. in my code or libssh, unfortunately my understanding of libssh
> > > > > > > > > and the ssh protocol is a little limited. However I do not want to
> > > > > > > > > put
> > > > > > > > > all of this load completely in your lap and if you share your
> > > > > > > > > thoughts
> > > > > > > > > and there's something you would like me to look into then please
> > > > > > > > > let
> > > > > > > > > me know.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jeetu
> > > > > > > > > ebrain.in | Beehive Computing
> > > > > > > > > Discover and run software from any device around you - an open
> > > > > > > > > source
> > > > > > > > > (GPL) project.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
> > > > > > > > > <jeetu.golani@xxxxxxxxx>    wrote:
> > > > > > > > > > Hi Aris,
> > > > > > > > > >
> > > > > > > > > > > I think I'll need a proof-of-concept code to debug this. Would
> > > > > > > > > > > you
> > > > > > > > > > > mind
> > > > > > > > > > > sharing your code, or it's not possible (too much integration
> > > > > > > > > > > with
> > > > > > > > > > > existing code).
> > > > > > > > > > No problem at all :) The code is an independent unit as of now
> > > > > > > > > > since I
> > > > > > > > > > wanted to make it work before I integrate it within my open source
> > > > > > > > > > project (eBrainPool)
> > > > > > > > > >
> > > > > > > > > > I'll mail this out to you tomm (not on the machine with the code
> > > > > > > > > > right
> > > > > > > > > > now :) )
> > > > > > > > > >
> > > > > > > > > > Thanks so much for looking into this.....truly appreciate it :)
> > > > > > > > > >
> > > > > > > > > > Bye for now
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 21, 2011 at 7:26 PM, Aris
> > > > > > > > > > Adamantiadis<aris@xxxxxxxxxxxx>
> > > > > > > > > >   wrote:
> > > > > > > > > > > Hi Jeetu,
> > > > > > > > > > >
> > > > > > > > > > > By seeing your logs, I understand this:
> > > > > > > > > > > Both side have a hmac error. The openssh client sees it first,
> > > > > > > > > > > sends
> > > > > > > > > > > a
> > > > > > > > > > > disconnect (that works), then there's the error in the libssh
> > > > > > > > > > > log.
> > > > > > > > > > >
> > > > > > > > > > > I think I'll need a proof-of-concept code to debug this. Would
> > > > > > > > > > > you
> > > > > > > > > > > mind
> > > > > > > > > > > sharing your code, or it's not possible (too much integration
> > > > > > > > > > > with
> > > > > > > > > > > existing code).
> > > > > > > > > > >
> > > > > > > > > > > If so, do you think I can reproduce the problem by "fixing"
> > > > > > > > > > > samplesshd
> > > > > > > > > > > to make new X11 channels connection connect to the local X11 unix
> > > > > > > > > > > socket ?
> > > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > >
> > > > > > > > > > > Aris
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
> > > > > > > > > > > > > This is not a problem and shouldn't cause invalid MAC errors.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > on
> > > > > > > > > > > > > travel so I'll look at the log when i'm back.
> > > > > > > > > > > > Thanks so much Aris :)
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Oct 18, 2011 at 5:52 PM, Aris
> > > > > > > > > > > > Adamantiadis<aris@xxxxxxxxxxxx>    wrote:
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is not a problem and shouldn't cause invalid MAC errors.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > on
> > > > > > > > > > > > > travel so I'll look at the log when i'm back.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Aris
> > > > > > > > > > > > >
> > > > > > > > > > > > > Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > debug3: Incorrect RSA1 identifier
> > > > > > > > > > > > > > debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
> > > > > > > > > > > > > > public
> > > > > > > > > > > > > > key
> > > > > > > > > > > > > > debug2: key_type_from_name: unknown key type '-----BEGIN'
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Oct 18, 2011 at 03:22:08PM +0500,
> > > > > > > > > > > > > > jeetu.golani@xxxxxxxxx
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > Hi Aris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've attached a log of the libssh server
> > > > > > > > > > > > > > > (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
> > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > (log-ssh-1-18102011.txt).
> > > > > > > > > > > > > > Greetings
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Stefan Kuttler ==*== nc.netbeisser.de


--
Eddy Valdes
Project Manager
Atronix Engineering
evaldes@xxxxxxxxxxxxxxxxxxxxxx
770-840-4045 x235