Re: multithreading for a single session

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

Hi guys,

Just wanted to know if anyone has had a chance to look into the
possibility of multithreading over a single session which I had
discussed in any earlier mail here (pasted below).

For now for my open source eBrainPool (ebrain.in) project I have gone
ahead with piped openssh server and daemon however this is not ideal
for variety of reasons. I would love to use libssh however the issues
I highlighted earlier prevent me from doing that.

Please let me know how I can help with this :)....thanks again for all
the help everyone here has provided me with in the past :)

Regards,
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run
software from any device around you.

On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx> wrote:
> oops forgot to mention the git link where this code can be seen
> ;)....git://ebrain.in:/libssh-server.git :)
>
> sorry
>
>
> On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
> <jeetu.golani@xxxxxxxxx> wrote:
>>
>> Hi Aris,
>>
>> I've been continuing work with my libssh server based on suggestions you
>> had made with regards multithreading when we were debugging issues where I
>> was seeing bad packet length, HMAC errors et al.
>>
>> I refactored my code as you had suggested (below) such that there would be
>> only one thread per session, although there could be multiple connects i.e.
>> sessions. This worked fantastic for me and I was able to solve the packet
>> corruption issues caused due to a race condition with multiple ssh loops
>> running. I could therefore have different ssh clients connect to the same
>> instance of the server and X forward simple x clients like xcalc, xclock,
>> xeyes, xgc.
>>
>> Unfortunately as soon as I tried to forward more complex applications such
>> as libreoffice, chromium etc things weren't working. Further analysis and a
>> look at the openssh code showed that these applications spawn multiple
>> connects to the same $DISPLAY. Therefore as I understand it (could be wrong)
>> the server needs to accept and process different event connections on the
>> same session.
>>
>> I have refactored the code again to account for this but basically this
>> reintroduces the same race condition with multiple event handling we had
>> seen earlier and while programs like libreoffice, chromium etc more further
>> along they eventually crash due to bad packet length, mac errors,etc.
>>
>> I tried to protect the event handling function ssh_event_dopoll with a
>> mutex hoping this will resolve the issue, however it didn't make any
>> difference.
>>
>> I'm at my wits end here and could appreciate advice here. One obvious way
>> is for me to somehow refactor my code in a way that eludes me at the moment
>> to get around this issue. The other could be if libssh itself would handle
>> multithreaded event handling for the same session differently or if I am
>> somehow using it in a wrong way.
>>
>> I have created a git repo of my code so anyone can study it. I have not
>> yet put a GPL license header to the code (just realized that ;) ) however I
>> don't care and in fact would be honoured if people would find some use for
>> this code. The code will eventually be part of the eBrainPool (ebrain.in)
>> project which I am working on which itself is GPLv3 - so feel free to use it
>> in any way you want.
>>
>> For now I could really use some ideas on how I could either refactor my
>> code to avoid the above issues or somehow if libssh could be modified to
>> handle multithreading for the same session differently.
>>
>> Thanks,
>>
>> Bye for now
>> Jeetu
>> ebrain.in | Beehive Computing
>> Share your software and computing resources - discover and run software
>> from any device around you.
>>
>>
>>
>> On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
>> <jeetu.golani@xxxxxxxxx> wrote:
>>>
>>> Hi Aris,
>>>
>>> >
>>> > took me a lot of time and head scratching, but I narrowed it down to a
>>> > threading problem. Two of your threads are running ssh loops at the
>>> > same
>>>
>>> I can completely appreciate the effort you must have gone through to
>>> debug this :)...threading issues can be pesky and since I didn't
>>> understand the internals of libssh it was proving quite difficult for
>>> me to trace this out.....thank you so much Aris :)
>>>
>>> > I hacked your example so the server_thread() functions leaves early. It
>>> > doesn't remove the race but reduces its window. now it works for me,
>>> > and I'm
>>> > able to run an xterm with it.
>>> > I strongly recommend that you refactor your code in a singlethread
>>> > design,
>>> > and that you move the x11 callbacks back in the server loop.
>>>
>>> I just did a cursory test of your code and it definitely runs much
>>> better from what I could see :)....I will take your advice to heed and
>>> try to refactor the code.
>>>
>>> >
>>> > Let's discuss what libssh can do better to help you write better code.
>>> >
>>>
>>> I'm a little concerned, I would ultimately need to handle multiple
>>> sessions.....the code you saw dealt with a single session....do you
>>> think this would also trigger similar issues?....what precautions do
>>> you think I need to keep in mind while I attempt this?
>>>
>>> Thanks so much Aris, I'm sooooooo glad to finally have the answer to
>>> my question and know that this was a problem in my code....now I can
>>> dedicate my energies in this direction :)
>>>
>>> Bye for now
>>>
>>>
>>>
>>> On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis <aris@xxxxxxxxxxxx>
>>> wrote:
>>> > Hi Jeetu,
>>> >
>>> > took me a lot of time and head scratching, but I narrowed it down to a
>>> > threading problem. Two of your threads are running ssh loops at the
>>> > same
>>> > time, and this cause races. The probability that the race causes
>>> > problem can
>>> > be high, because there's a trigger (data on socket) and both threads
>>> > run
>>> > almost the same code at the same time.
>>> >
>>> > Thread 1:      message = ssh_message_get(session);
>>> > in server_thread()
>>> > Thread 2:                 ssh_event_dopoll(event, 1000);
>>> > in wait_for_something()
>>> >
>>> > I hacked your example so the server_thread() functions leaves early. It
>>> > doesn't remove the race but reduces its window. now it works for me,
>>> > and I'm
>>> > able to run an xterm with it.
>>> > I strongly recommend that you refactor your code in a singlethread
>>> > design,
>>> > and that you move the x11 callbacks back in the server loop.
>>> >
>>> > Let's discuss what libssh can do better to help you write better code.
>>> >
>>> > Kr,
>>> >
>>> > Aris
>>> > Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
>>> >>
>>> >> Hi Aris,
>>> >>
>>> >> No probs and I completely understand and appreciate your interest in
>>> >> looking into this....please let me know how I can help further :)
>>> >>
>>> >> Bye for now
>>> >>
>>> >> On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
>>> >>  wrote:
>>> >>>
>>> >>> Hi Jeetu,
>>> >>>
>>> >>> I apologize, still had no time to check it. I'm pretty busy and all
>>> >>> my
>>> >>> free time is swallowed by another project. I hope I can free myself
>>> >>> an
>>> >>> hour or two next week to debug this.
>>> >>>
>>> >>> kr,
>>> >>>
>>> >>> Aris
>>> >>>
>>> >>> Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
>>> >>>>
>>> >>>> Hi Aris,
>>> >>>>
>>> >>>> Just wanted to check in if you've had a chance to try out the libssh
>>> >>>> server code I've sent and reproduce the errors I've been seeing?
>>> >>>>
>>> >>>> Thanks so much again for looking into this.
>>> >>>>
>>> >>>> Bye for now
>>> >>>>
>>> >>>> On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
>>> >>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>> >>>>>
>>> >>>>> Hi Aris,
>>> >>>>>
>>> >>>>> I'm attaching my proof of concept server code
>>> >>>>> as.....ebpsshd-singlesession.c has compile instructions at the
>>> >>>>> beginning of the code. You will also need to generate a key.h file
>>> >>>>> that holds the public key of the user who will be connecting to
>>> >>>>> this
>>> >>>>> server - this is tempoarary since as of now I'm not reading this
>>> >>>>> info
>>> >>>>> from an authorized_keys or something similar.
>>> >>>>>
>>> >>>>> Just create a key.h file in the same directory and put something
>>> >>>>> like :
>>> >>>>>
>>> >>>>> #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
>>> >>>>>
>>> >>>>> Also as of now ebpsshd-singlesession listens in on port 2000. So
>>> >>>>> ssh
>>> >>>>> should connect to that port.
>>> >>>>>
>>> >>>>> I also have a libssh-project-wrapper script that allows me to try
>>> >>>>> this
>>> >>>>> out without needing to install the libssh i've built. It basically
>>> >>>>> has
>>> >>>>> the content:
>>> >>>>>
>>> >>>>> #!/bin/sh
>>> >>>>>
>>> >>>>> export
>>> >>>>>
>>> >>>>> LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
>>> >>>>> ./$1
>>> >>>>>
>>> >>>>> I have been testing this code with simple examples like xeyes and
>>> >>>>> xcalc. For some reason xcalc shows the problem much sooner than
>>> >>>>> with
>>> >>>>> xeyes, maybe because of the volume of data being transmitted to and
>>> >>>>> fro?
>>> >>>>>
>>> >>>>> This is proof of concept code with a lot of fiddling with buffer
>>> >>>>> sizes
>>> >>>>> as I have been trying to study if any of that makes an impact
>>> >>>>> however
>>> >>>>> please point out any way you think this can be improved on :)
>>> >>>>>
>>> >>>>> I'm sorry to drop this in your lap especially if it turns out it
>>> >>>>> was
>>> >>>>> some server side code issue, however I completely appreciate your
>>> >>>>> help
>>> >>>>> on this. I would like to squash this bug regardless of where it
>>> >>>>> lies
>>> >>>>> i.e. in my code or libssh, unfortunately my understanding of libssh
>>> >>>>> and the ssh protocol is a little limited. However I do not want to
>>> >>>>> put
>>> >>>>> all of this load completely in your lap and if you share your
>>> >>>>> thoughts
>>> >>>>> and there's something you would like me to look into then please
>>> >>>>> let
>>> >>>>> me know.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Jeetu
>>> >>>>> ebrain.in | Beehive Computing
>>> >>>>> Discover and run software from any device around you - an open
>>> >>>>> source
>>> >>>>> (GPL) project.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
>>> >>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>> >>>>>>
>>> >>>>>> Hi Aris,
>>> >>>>>>
>>> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would
>>> >>>>>>> you
>>> >>>>>>> mind
>>> >>>>>>> sharing your code, or it's not possible (too much integration
>>> >>>>>>> with
>>> >>>>>>> existing code).
>>> >>>>>>
>>> >>>>>> No problem at all :) The code is an independent unit as of now
>>> >>>>>> since I
>>> >>>>>> wanted to make it work before I integrate it within my open source
>>> >>>>>> project (eBrainPool)
>>> >>>>>>
>>> >>>>>> I'll mail this out to you tomm (not on the machine with the code
>>> >>>>>> right
>>> >>>>>> now :) )
>>> >>>>>>
>>> >>>>>> Thanks so much for looking into this.....truly appreciate it :)
>>> >>>>>>
>>> >>>>>> Bye for now
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Fri, Oct 21, 2011 at 7:26 PM, Aris
>>> >>>>>> Adamantiadis<aris@xxxxxxxxxxxx>
>>> >>>>>>  wrote:
>>> >>>>>>>
>>> >>>>>>> Hi Jeetu,
>>> >>>>>>>
>>> >>>>>>> By seeing your logs, I understand this:
>>> >>>>>>> Both side have a hmac error. The openssh client sees it first,
>>> >>>>>>> sends
>>> >>>>>>> a
>>> >>>>>>> disconnect (that works), then there's the error in the libssh
>>> >>>>>>> log.
>>> >>>>>>>
>>> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would
>>> >>>>>>> you
>>> >>>>>>> mind
>>> >>>>>>> sharing your code, or it's not possible (too much integration
>>> >>>>>>> with
>>> >>>>>>> existing code).
>>> >>>>>>>
>>> >>>>>>> If so, do you think I can reproduce the problem by "fixing"
>>> >>>>>>> samplesshd
>>> >>>>>>> to make new X11 channels connection connect to the local X11 unix
>>> >>>>>>> socket ?
>>> >>>>>>>
>>> >>>>>>> Thanks.
>>> >>>>>>>
>>> >>>>>>> Aris
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
>>> >>>>>>>>>
>>> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors.
>>> >>>>>>>>> I'm
>>> >>>>>>>>> on
>>> >>>>>>>>> travel so I'll look at the log when i'm back.
>>> >>>>>>>>
>>> >>>>>>>> Thanks so much Aris :)
>>> >>>>>>>>
>>> >>>>>>>> On Tue, Oct 18, 2011 at 5:52 PM, Aris
>>> >>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>  wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> Hi,
>>> >>>>>>>>>
>>> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors.
>>> >>>>>>>>> I'm
>>> >>>>>>>>> on
>>> >>>>>>>>> travel so I'll look at the log when i'm back.
>>> >>>>>>>>>
>>> >>>>>>>>> Aris
>>> >>>>>>>>>
>>> >>>>>>>>> Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
>>> >>>>>>>>>>
>>> >>>>>>>>>> Hi all,
>>> >>>>>>>>>>
>>> >>>>>>>>>> debug3: Incorrect RSA1 identifier
>>> >>>>>>>>>> debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
>>> >>>>>>>>>> public
>>> >>>>>>>>>> key
>>> >>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Tue, Oct 18, 2011 at 03:22:08PM +0500,
>>> >>>>>>>>>> jeetu.golani@xxxxxxxxx
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Hi Aris,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I've attached a log of the libssh server
>>> >>>>>>>>>>> (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
>>> >>>>>>>>>>> client
>>> >>>>>>>>>>> (log-ssh-1-18102011.txt).
>>> >>>>>>>>>>
>>> >>>>>>>>>> Greetings
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Stefan Kuttler ==*== nc.netbeisser.de
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>
>>> >
>>
>>
>