Re: multithreading for a single session

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

oops forgot to mention the git link where this code can be seen
;)....git://ebrain.in:/libssh-server.git :)

sorry

On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx <
jeetu.golani@xxxxxxxxx> wrote:

> Hi Aris,
>
> I've been continuing work with my libssh server based on suggestions you
> had made with regards multithreading when we were debugging issues where I
> was seeing bad packet length, HMAC errors et al.
>
> I refactored my code as you had suggested (below) such that there would be
> only one thread per session, although there could be multiple connects i.e.
> sessions. This worked fantastic for me and I was able to solve the packet
> corruption issues caused due to a race condition with multiple ssh loops
> running. I could therefore have different ssh clients connect to the same
> instance of the server and X forward simple x clients like xcalc, xclock,
> xeyes, xgc.
>
> Unfortunately as soon as I tried to forward more complex applications such
> as libreoffice, chromium etc things weren't working. Further analysis and a
> look at the openssh code showed that these applications spawn multiple
> connects to the same $DISPLAY. Therefore as I understand it (could be
> wrong) the server needs to accept and process different event connections
> on the same session.
>
> I have refactored the code again to account for this but basically this
> reintroduces the same race condition with multiple event handling we had
> seen earlier and while programs like libreoffice, chromium etc more further
> along they eventually crash due to bad packet length, mac errors,etc.
>
> I tried to protect the event handling function ssh_event_dopoll with a
> mutex hoping this will resolve the issue, however it didn't make any
> difference.
>
> I'm at my wits end here and could appreciate advice here. One obvious way
> is for me to somehow refactor my code in a way that eludes me at the moment
> to get around this issue. The other could be if libssh itself would handle
> multithreaded event handling for the same session differently or if I am
> somehow using it in a wrong way.
>
> I have created a git repo of my code so anyone can study it. I have not
> yet put a GPL license header to the code (just realized that ;) ) however I
> don't care and in fact would be honoured if people would find some use for
> this code. The code will eventually be part of the eBrainPool (ebrain.in)
> project which I am working on which itself is GPLv3 - so feel free to use
> it in any way you want.
>
> For now I could really use some ideas on how I could either refactor my
> code to avoid the above issues or somehow if libssh could be modified to
> handle multithreading for the same session differently.
>
> Thanks,
>
> Bye for now
> Jeetu
> ebrain.in | Beehive Computing
> Share your software and computing resources - discover and run software
> from any device around you.
>
>
>
> On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx <
> jeetu.golani@xxxxxxxxx> wrote:
>
>> Hi Aris,
>>
>> >
>> > took me a lot of time and head scratching, but I narrowed it down to a
>> > threading problem. Two of your threads are running ssh loops at the same
>>
>> I can completely appreciate the effort you must have gone through to
>> debug this :)...threading issues can be pesky and since I didn't
>> understand the internals of libssh it was proving quite difficult for
>> me to trace this out.....thank you so much Aris :)
>>
>> > I hacked your example so the server_thread() functions leaves early. It
>> > doesn't remove the race but reduces its window. now it works for me,
>> and I'm
>> > able to run an xterm with it.
>> > I strongly recommend that you refactor your code in a singlethread
>> design,
>> > and that you move the x11 callbacks back in the server loop.
>>
>> I just did a cursory test of your code and it definitely runs much
>> better from what I could see :)....I will take your advice to heed and
>> try to refactor the code.
>>
>> >
>> > Let's discuss what libssh can do better to help you write better code.
>> >
>>
>> I'm a little concerned, I would ultimately need to handle multiple
>> sessions.....the code you saw dealt with a single session....do you
>> think this would also trigger similar issues?....what precautions do
>> you think I need to keep in mind while I attempt this?
>>
>> Thanks so much Aris, I'm sooooooo glad to finally have the answer to
>> my question and know that this was a problem in my code....now I can
>> dedicate my energies in this direction :)
>>
>> Bye for now
>>
>>
>>
>> On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis <aris@xxxxxxxxxxxx>
>> wrote:
>> > Hi Jeetu,
>> >
>> > took me a lot of time and head scratching, but I narrowed it down to a
>> > threading problem. Two of your threads are running ssh loops at the same
>> > time, and this cause races. The probability that the race causes
>> problem can
>> > be high, because there's a trigger (data on socket) and both threads run
>> > almost the same code at the same time.
>> >
>> > Thread 1:      message = ssh_message_get(session);
>> > in server_thread()
>> > Thread 2:                 ssh_event_dopoll(event, 1000);
>> > in wait_for_something()
>> >
>> > I hacked your example so the server_thread() functions leaves early. It
>> > doesn't remove the race but reduces its window. now it works for me,
>> and I'm
>> > able to run an xterm with it.
>> > I strongly recommend that you refactor your code in a singlethread
>> design,
>> > and that you move the x11 callbacks back in the server loop.
>> >
>> > Let's discuss what libssh can do better to help you write better code.
>> >
>> > Kr,
>> >
>> > Aris
>> > Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
>> >>
>> >> Hi Aris,
>> >>
>> >> No probs and I completely understand and appreciate your interest in
>> >> looking into this....please let me know how I can help further :)
>> >>
>> >> Bye for now
>> >>
>> >> On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
>> >>  wrote:
>> >>>
>> >>> Hi Jeetu,
>> >>>
>> >>> I apologize, still had no time to check it. I'm pretty busy and all my
>> >>> free time is swallowed by another project. I hope I can free myself an
>> >>> hour or two next week to debug this.
>> >>>
>> >>> kr,
>> >>>
>> >>> Aris
>> >>>
>> >>> Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
>> >>>>
>> >>>> Hi Aris,
>> >>>>
>> >>>> Just wanted to check in if you've had a chance to try out the libssh
>> >>>> server code I've sent and reproduce the errors I've been seeing?
>> >>>>
>> >>>> Thanks so much again for looking into this.
>> >>>>
>> >>>> Bye for now
>> >>>>
>> >>>> On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
>> >>>> <jeetu.golani@xxxxxxxxx>  wrote:
>> >>>>>
>> >>>>> Hi Aris,
>> >>>>>
>> >>>>> I'm attaching my proof of concept server code
>> >>>>> as.....ebpsshd-singlesession.c has compile instructions at the
>> >>>>> beginning of the code. You will also need to generate a key.h file
>> >>>>> that holds the public key of the user who will be connecting to this
>> >>>>> server - this is tempoarary since as of now I'm not reading this
>> info
>> >>>>> from an authorized_keys or something similar.
>> >>>>>
>> >>>>> Just create a key.h file in the same directory and put something
>> like :
>> >>>>>
>> >>>>> #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
>> >>>>>
>> >>>>> Also as of now ebpsshd-singlesession listens in on port 2000. So ssh
>> >>>>> should connect to that port.
>> >>>>>
>> >>>>> I also have a libssh-project-wrapper script that allows me to try
>> this
>> >>>>> out without needing to install the libssh i've built. It basically
>> has
>> >>>>> the content:
>> >>>>>
>> >>>>> #!/bin/sh
>> >>>>>
>> >>>>> export
>> >>>>>
>> LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
>> >>>>> ./$1
>> >>>>>
>> >>>>> I have been testing this code with simple examples like xeyes and
>> >>>>> xcalc. For some reason xcalc shows the problem much sooner than with
>> >>>>> xeyes, maybe because of the volume of data being transmitted to and
>> >>>>> fro?
>> >>>>>
>> >>>>> This is proof of concept code with a lot of fiddling with buffer
>> sizes
>> >>>>> as I have been trying to study if any of that makes an impact
>> however
>> >>>>> please point out any way you think this can be improved on :)
>> >>>>>
>> >>>>> I'm sorry to drop this in your lap especially if it turns out it was
>> >>>>> some server side code issue, however I completely appreciate your
>> help
>> >>>>> on this. I would like to squash this bug regardless of where it lies
>> >>>>> i.e. in my code or libssh, unfortunately my understanding of libssh
>> >>>>> and the ssh protocol is a little limited. However I do not want to
>> put
>> >>>>> all of this load completely in your lap and if you share your
>> thoughts
>> >>>>> and there's something you would like me to look into then please let
>> >>>>> me know.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Jeetu
>> >>>>> ebrain.in | Beehive Computing
>> >>>>> Discover and run software from any device around you - an open
>> source
>> >>>>> (GPL) project.
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
>> >>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>> >>>>>>
>> >>>>>> Hi Aris,
>> >>>>>>
>> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
>> >>>>>>> mind
>> >>>>>>> sharing your code, or it's not possible (too much integration with
>> >>>>>>> existing code).
>> >>>>>>
>> >>>>>> No problem at all :) The code is an independent unit as of now
>> since I
>> >>>>>> wanted to make it work before I integrate it within my open source
>> >>>>>> project (eBrainPool)
>> >>>>>>
>> >>>>>> I'll mail this out to you tomm (not on the machine with the code
>> right
>> >>>>>> now :) )
>> >>>>>>
>> >>>>>> Thanks so much for looking into this.....truly appreciate it :)
>> >>>>>>
>> >>>>>> Bye for now
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Oct 21, 2011 at 7:26 PM, Aris Adamantiadis<
>> aris@xxxxxxxxxxxx>
>> >>>>>>  wrote:
>> >>>>>>>
>> >>>>>>> Hi Jeetu,
>> >>>>>>>
>> >>>>>>> By seeing your logs, I understand this:
>> >>>>>>> Both side have a hmac error. The openssh client sees it first,
>> sends
>> >>>>>>> a
>> >>>>>>> disconnect (that works), then there's the error in the libssh log.
>> >>>>>>>
>> >>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
>> >>>>>>> mind
>> >>>>>>> sharing your code, or it's not possible (too much integration with
>> >>>>>>> existing code).
>> >>>>>>>
>> >>>>>>> If so, do you think I can reproduce the problem by "fixing"
>> >>>>>>> samplesshd
>> >>>>>>> to make new X11 channels connection connect to the local X11 unix
>> >>>>>>> socket ?
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> Aris
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
>> >>>>>>>>>
>> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors.
>> I'm
>> >>>>>>>>> on
>> >>>>>>>>> travel so I'll look at the log when i'm back.
>> >>>>>>>>
>> >>>>>>>> Thanks so much Aris :)
>> >>>>>>>>
>> >>>>>>>> On Tue, Oct 18, 2011 at 5:52 PM, Aris
>> >>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>  wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi,
>> >>>>>>>>>
>> >>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors.
>> I'm
>> >>>>>>>>> on
>> >>>>>>>>> travel so I'll look at the log when i'm back.
>> >>>>>>>>>
>> >>>>>>>>> Aris
>> >>>>>>>>>
>> >>>>>>>>> Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
>> >>>>>>>>>>
>> >>>>>>>>>> Hi all,
>> >>>>>>>>>>
>> >>>>>>>>>> debug3: Incorrect RSA1 identifier
>> >>>>>>>>>> debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
>> public
>> >>>>>>>>>> key
>> >>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Oct 18, 2011 at 03:22:08PM +0500,
>> jeetu.golani@xxxxxxxxx
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Aris,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I've attached a log of the libssh server
>> >>>>>>>>>>> (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH
>> client
>> >>>>>>>>>>> (log-ssh-1-18102011.txt).
>> >>>>>>>>>>
>> >>>>>>>>>> Greetings
>> >>>>>>>>>> --
>> >>>>>>>>>> Stefan Kuttler ==*== nc.netbeisser.de
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>
>