multithreading for a single session

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

Hi Aris,

I've been continuing work with my libssh server based on suggestions you
had made with regards multithreading when we were debugging issues where I
was seeing bad packet length, HMAC errors et al.

I refactored my code as you had suggested (below) such that there would be
only one thread per session, although there could be multiple connects i.e.
sessions. This worked fantastic for me and I was able to solve the packet
corruption issues caused due to a race condition with multiple ssh loops
running. I could therefore have different ssh clients connect to the same
instance of the server and X forward simple x clients like xcalc, xclock,
xeyes, xgc.

Unfortunately as soon as I tried to forward more complex applications such
as libreoffice, chromium etc things weren't working. Further analysis and a
look at the openssh code showed that these applications spawn multiple
connects to the same $DISPLAY. Therefore as I understand it (could be
wrong) the server needs to accept and process different event connections
on the same session.

I have refactored the code again to account for this but basically this
reintroduces the same race condition with multiple event handling we had
seen earlier and while programs like libreoffice, chromium etc more further
along they eventually crash due to bad packet length, mac errors,etc.

I tried to protect the event handling function ssh_event_dopoll with a
mutex hoping this will resolve the issue, however it didn't make any
difference.

I'm at my wits end here and could appreciate advice here. One obvious way
is for me to somehow refactor my code in a way that eludes me at the moment
to get around this issue. The other could be if libssh itself would handle
multithreaded event handling for the same session differently or if I am
somehow using it in a wrong way.

I have created a git repo of my code so anyone can study it. I have not yet
put a GPL license header to the code (just realized that ;) ) however I
don't care and in fact would be honoured if people would find some use for
this code. The code will eventually be part of the eBrainPool (ebrain.in)
project which I am working on which itself is GPLv3 - so feel free to use
it in any way you want.

For now I could really use some ideas on how I could either refactor my
code to avoid the above issues or somehow if libssh could be modified to
handle multithreading for the same session differently.

Thanks,

Bye for now
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run software
from any device around you.



On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx <
jeetu.golani@xxxxxxxxx> wrote:

> Hi Aris,
>
> >
> > took me a lot of time and head scratching, but I narrowed it down to a
> > threading problem. Two of your threads are running ssh loops at the same
>
> I can completely appreciate the effort you must have gone through to
> debug this :)...threading issues can be pesky and since I didn't
> understand the internals of libssh it was proving quite difficult for
> me to trace this out.....thank you so much Aris :)
>
> > I hacked your example so the server_thread() functions leaves early. It
> > doesn't remove the race but reduces its window. now it works for me, and
> I'm
> > able to run an xterm with it.
> > I strongly recommend that you refactor your code in a singlethread
> design,
> > and that you move the x11 callbacks back in the server loop.
>
> I just did a cursory test of your code and it definitely runs much
> better from what I could see :)....I will take your advice to heed and
> try to refactor the code.
>
> >
> > Let's discuss what libssh can do better to help you write better code.
> >
>
> I'm a little concerned, I would ultimately need to handle multiple
> sessions.....the code you saw dealt with a single session....do you
> think this would also trigger similar issues?....what precautions do
> you think I need to keep in mind while I attempt this?
>
> Thanks so much Aris, I'm sooooooo glad to finally have the answer to
> my question and know that this was a problem in my code....now I can
> dedicate my energies in this direction :)
>
> Bye for now
>
>
>
> On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis <aris@xxxxxxxxxxxx>
> wrote:
> > Hi Jeetu,
> >
> > took me a lot of time and head scratching, but I narrowed it down to a
> > threading problem. Two of your threads are running ssh loops at the same
> > time, and this cause races. The probability that the race causes problem
> can
> > be high, because there's a trigger (data on socket) and both threads run
> > almost the same code at the same time.
> >
> > Thread 1:      message = ssh_message_get(session);
> > in server_thread()
> > Thread 2:                 ssh_event_dopoll(event, 1000);
> > in wait_for_something()
> >
> > I hacked your example so the server_thread() functions leaves early. It
> > doesn't remove the race but reduces its window. now it works for me, and
> I'm
> > able to run an xterm with it.
> > I strongly recommend that you refactor your code in a singlethread
> design,
> > and that you move the x11 callbacks back in the server loop.
> >
> > Let's discuss what libssh can do better to help you write better code.
> >
> > Kr,
> >
> > Aris
> > Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
> >>
> >> Hi Aris,
> >>
> >> No probs and I completely understand and appreciate your interest in
> >> looking into this....please let me know how I can help further :)
> >>
> >> Bye for now
> >>
> >> On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
> >>  wrote:
> >>>
> >>> Hi Jeetu,
> >>>
> >>> I apologize, still had no time to check it. I'm pretty busy and all my
> >>> free time is swallowed by another project. I hope I can free myself an
> >>> hour or two next week to debug this.
> >>>
> >>> kr,
> >>>
> >>> Aris
> >>>
> >>> Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
> >>>>
> >>>> Hi Aris,
> >>>>
> >>>> Just wanted to check in if you've had a chance to try out the libssh
> >>>> server code I've sent and reproduce the errors I've been seeing?
> >>>>
> >>>> Thanks so much again for looking into this.
> >>>>
> >>>> Bye for now
> >>>>
> >>>> On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
> >>>> <jeetu.golani@xxxxxxxxx>  wrote:
> >>>>>
> >>>>> Hi Aris,
> >>>>>
> >>>>> I'm attaching my proof of concept server code
> >>>>> as.....ebpsshd-singlesession.c has compile instructions at the
> >>>>> beginning of the code. You will also need to generate a key.h file
> >>>>> that holds the public key of the user who will be connecting to this
> >>>>> server - this is tempoarary since as of now I'm not reading this info
> >>>>> from an authorized_keys or something similar.
> >>>>>
> >>>>> Just create a key.h file in the same directory and put something
> like :
> >>>>>
> >>>>> #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
> >>>>>
> >>>>> Also as of now ebpsshd-singlesession listens in on port 2000. So ssh
> >>>>> should connect to that port.
> >>>>>
> >>>>> I also have a libssh-project-wrapper script that allows me to try
> this
> >>>>> out without needing to install the libssh i've built. It basically
> has
> >>>>> the content:
> >>>>>
> >>>>> #!/bin/sh
> >>>>>
> >>>>> export
> >>>>>
> LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
> >>>>> ./$1
> >>>>>
> >>>>> I have been testing this code with simple examples like xeyes and
> >>>>> xcalc. For some reason xcalc shows the problem much sooner than with
> >>>>> xeyes, maybe because of the volume of data being transmitted to and
> >>>>> fro?
> >>>>>
> >>>>> This is proof of concept code with a lot of fiddling with buffer
> sizes
> >>>>> as I have been trying to study if any of that makes an impact however
> >>>>> please point out any way you think this can be improved on :)
> >>>>>
> >>>>> I'm sorry to drop this in your lap especially if it turns out it was
> >>>>> some server side code issue, however I completely appreciate your
> help
> >>>>> on this. I would like to squash this bug regardless of where it lies
> >>>>> i.e. in my code or libssh, unfortunately my understanding of libssh
> >>>>> and the ssh protocol is a little limited. However I do not want to
> put
> >>>>> all of this load completely in your lap and if you share your
> thoughts
> >>>>> and there's something you would like me to look into then please let
> >>>>> me know.
> >>>>>
> >>>>> Thanks,
> >>>>> Jeetu
> >>>>> ebrain.in | Beehive Computing
> >>>>> Discover and run software from any device around you - an open source
> >>>>> (GPL) project.
> >>>>>
> >>>>>
> >>>>> On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
> >>>>> <jeetu.golani@xxxxxxxxx>  wrote:
> >>>>>>
> >>>>>> Hi Aris,
> >>>>>>
> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
> >>>>>>> mind
> >>>>>>> sharing your code, or it's not possible (too much integration with
> >>>>>>> existing code).
> >>>>>>
> >>>>>> No problem at all :) The code is an independent unit as of now
> since I
> >>>>>> wanted to make it work before I integrate it within my open source
> >>>>>> project (eBrainPool)
> >>>>>>
> >>>>>> I'll mail this out to you tomm (not on the machine with the code
> right
> >>>>>> now :) )
> >>>>>>
> >>>>>> Thanks so much for looking into this.....truly appreciate it :)
> >>>>>>
> >>>>>> Bye for now
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Oct 21, 2011 at 7:26 PM, Aris Adamantiadis<
> aris@xxxxxxxxxxxx>
> >>>>>>  wrote:
> >>>>>>>
> >>>>>>> Hi Jeetu,
> >>>>>>>
> >>>>>>> By seeing your logs, I understand this:
> >>>>>>> Both side have a hmac error. The openssh client sees it first,
> sends
> >>>>>>> a
> >>>>>>> disconnect (that works), then there's the error in the libssh log.
> >>>>>>>
> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
> >>>>>>> mind
> >>>>>>> sharing your code, or it's not possible (too much integration with
> >>>>>>> existing code).
> >>>>>>>
> >>>>>>> If so, do you think I can reproduce the problem by "fixing"
> >>>>>>> samplesshd
> >>>>>>> to make new X11 channels connection connect to the local X11 unix
> >>>>>>> socket ?
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> Aris
> >>>>>>>
> >>>>>>>
> >>>>>>> Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
> >>>>>>>>>
> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors. I'm
> >>>>>>>>> on
> >>>>>>>>> travel so I'll look at the log when i'm back.
> >>>>>>>>
> >>>>>>>> Thanks so much Aris :)
> >>>>>>>>
> >>>>>>>> On Tue, Oct 18, 2011 at 5:52 PM, Aris
> >>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>  wrote:
> >>>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors. I'm
> >>>>>>>>> on
> >>>>>>>>> travel so I'll look at the log when i'm back.
> >>>>>>>>>
> >>>>>>>>> Aris
> >>>>>>>>>
> >>>>>>>>> Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
> >>>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> debug3: Incorrect RSA1 identifier
> >>>>>>>>>> debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
> public
> >>>>>>>>>> key
> >>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Oct 18, 2011 at 03:22:08PM +0500,
> jeetu.golani@xxxxxxxxx
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Aris,
> >>>>>>>>>>>
> >>>>>>>>>>> I've attached a log of the libssh server
> >>>>>>>>>>> (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
> client
> >>>>>>>>>>> (log-ssh-1-18102011.txt).
> >>>>>>>>>>
> >>>>>>>>>> Greetings
> >>>>>>>>>> --
> >>>>>>>>>> Stefan Kuttler ==*== nc.netbeisser.de
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>