Re: multithreading for a single session

[Aris Adamantiadis <aris@xxxxxxxxxxxx>]

Do you guys think we should include inside libssh a support for
multi-threaded single sessions ? I see a possibility but I also see
many things that can go wrong.

Aris

Le 14/03/12 14:58, Gearoid Murphy a écrit :
> 
> It is possible to use a single session in a multi-threaded context as
> long as you protect access to the session (and associated channels)
> using a session mutex. It's very inefficient however, as every channel
> read operation must timeout in order to release the mutex when the
> associated channel is inactive so other channels can be processed. This
> timeout is the source of the performance penalty, it decimates
> bandwidth. I've seen single threaded applications capable of 30-40 mbs
> being reduced to 256kps using the multi-threaded approach.
> 
> My favored solution these days is to use a boost io_service to peek at
> the session socket, whenever data needs to be processed, an
> asynchronous handler is invoked which processes the active channels.
> 
> On 14/03/12 13:38, Eddy Valdes wrote:
>> Jeetu,
>>
>> I seem to recall somebody telling me that the if you are going to use
>> threads, then you must use a separate session for each thread. The
>> application that used libssh that I wrote and used to maintain had a
>> separate session for each thread and it worked well.
>>
>> Eddy
>>
>> On 03/14/2012 09:31 AM, jeetu.golani@xxxxxxxxx wrote:
>>> Hi guys,
>>>
>>> Just wanted to know if anyone has had a chance to look into the
>>> possibility of multithreading over a single session which I had
>>> discussed in any earlier mail here (pasted below).
>>>
>>> For now for my open source eBrainPool (ebrain.in) project I have gone
>>> ahead with piped openssh server and daemon however this is not ideal
>>> for variety of reasons. I would love to use libssh however the issues
>>> I highlighted earlier prevent me from doing that.
>>>
>>> Please let me know how I can help with this :)....thanks again for all
>>> the help everyone here has provided me with in the past :)
>>>
>>> Regards,
>>> Jeetu
>>> ebrain.in | Beehive Computing
>>> Share your software and computing resources - discover and run
>>> software from any device around you.
>>>
>>> On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>>> oops forgot to mention the git link where this code can be seen
>>>> ;)....git://ebrain.in:/libssh-server.git :)
>>>>
>>>> sorry
>>>>
>>>>
>>>> On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>>>> Hi Aris,
>>>>>
>>>>> I've been continuing work with my libssh server based on
>>>>> suggestions you
>>>>> had made with regards multithreading when we were debugging issues
>>>>> where I
>>>>> was seeing bad packet length, HMAC errors et al.
>>>>>
>>>>> I refactored my code as you had suggested (below) such that there
>>>>> would be
>>>>> only one thread per session, although there could be multiple
>>>>> connects i.e.
>>>>> sessions. This worked fantastic for me and I was able to solve the
>>>>> packet
>>>>> corruption issues caused due to a race condition with multiple ssh
>>>>> loops
>>>>> running. I could therefore have different ssh clients connect to
>>>>> the same
>>>>> instance of the server and X forward simple x clients like xcalc,
>>>>> xclock,
>>>>> xeyes, xgc.
>>>>>
>>>>> Unfortunately as soon as I tried to forward more complex
>>>>> applications such
>>>>> as libreoffice, chromium etc things weren't working. Further
>>>>> analysis and a
>>>>> look at the openssh code showed that these applications spawn
>>>>> multiple
>>>>> connects to the same $DISPLAY. Therefore as I understand it (could
>>>>> be wrong)
>>>>> the server needs to accept and process different event connections
>>>>> on the
>>>>> same session.
>>>>>
>>>>> I have refactored the code again to account for this but basically
>>>>> this
>>>>> reintroduces the same race condition with multiple event handling
>>>>> we had
>>>>> seen earlier and while programs like libreoffice, chromium etc
>>>>> more further
>>>>> along they eventually crash due to bad packet length, mac errors,etc.
>>>>>
>>>>> I tried to protect the event handling function ssh_event_dopoll
>>>>> with a
>>>>> mutex hoping this will resolve the issue, however it didn't make any
>>>>> difference.
>>>>>
>>>>> I'm at my wits end here and could appreciate advice here. One
>>>>> obvious way
>>>>> is for me to somehow refactor my code in a way that eludes me at
>>>>> the moment
>>>>> to get around this issue. The other could be if libssh itself
>>>>> would handle
>>>>> multithreaded event handling for the same session differently or
>>>>> if I am
>>>>> somehow using it in a wrong way.
>>>>>
>>>>> I have created a git repo of my code so anyone can study it. I
>>>>> have not
>>>>> yet put a GPL license header to the code (just realized that ;) )
>>>>> however I
>>>>> don't care and in fact would be honoured if people would find some
>>>>> use for
>>>>> this code. The code will eventually be part of the eBrainPool
>>>>> (ebrain.in)
>>>>> project which I am working on which itself is GPLv3 - so feel free
>>>>> to use it
>>>>> in any way you want.
>>>>>
>>>>> For now I could really use some ideas on how I could either
>>>>> refactor my
>>>>> code to avoid the above issues or somehow if libssh could be
>>>>> modified to
>>>>> handle multithreading for the same session differently.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Bye for now
>>>>> Jeetu
>>>>> ebrain.in | Beehive Computing
>>>>> Share your software and computing resources - discover and run
>>>>> software
>>>>> from any device around you.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
>>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>>>>> Hi Aris,
>>>>>>
>>>>>>> took me a lot of time and head scratching, but I narrowed it
>>>>>>> down to a
>>>>>>> threading problem. Two of your threads are running ssh loops at the
>>>>>>> same
>>>>>> I can completely appreciate the effort you must have gone through to
>>>>>> debug this :)...threading issues can be pesky and since I didn't
>>>>>> understand the internals of libssh it was proving quite difficult
>>>>>> for
>>>>>> me to trace this out.....thank you so much Aris :)
>>>>>>
>>>>>>> I hacked your example so the server_thread() functions leaves
>>>>>>> early. It
>>>>>>> doesn't remove the race but reduces its window. now it works for
>>>>>>> me,
>>>>>>> and I'm
>>>>>>> able to run an xterm with it.
>>>>>>> I strongly recommend that you refactor your code in a singlethread
>>>>>>> design,
>>>>>>> and that you move the x11 callbacks back in the server loop.
>>>>>> I just did a cursory test of your code and it definitely runs much
>>>>>> better from what I could see :)....I will take your advice to
>>>>>> heed and
>>>>>> try to refactor the code.
>>>>>>
>>>>>>> Let's discuss what libssh can do better to help you write better
>>>>>>> code.
>>>>>>>
>>>>>> I'm a little concerned, I would ultimately need to handle multiple
>>>>>> sessions.....the code you saw dealt with a single session....do you
>>>>>> think this would also trigger similar issues?....what precautions do
>>>>>> you think I need to keep in mind while I attempt this?
>>>>>>
>>>>>> Thanks so much Aris, I'm sooooooo glad to finally have the answer to
>>>>>> my question and know that this was a problem in my code....now I can
>>>>>> dedicate my energies in this direction :)
>>>>>>
>>>>>> Bye for now
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 21, 2011 at 3:11 AM, Aris
>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>> Hi Jeetu,
>>>>>>>
>>>>>>> took me a lot of time and head scratching, but I narrowed it
>>>>>>> down to a
>>>>>>> threading problem. Two of your threads are running ssh loops at the
>>>>>>> same
>>>>>>> time, and this cause races. The probability that the race causes
>>>>>>> problem can
>>>>>>> be high, because there's a trigger (data on socket) and both
>>>>>>> threads
>>>>>>> run
>>>>>>> almost the same code at the same time.
>>>>>>>
>>>>>>> Thread 1:      message = ssh_message_get(session);
>>>>>>> in server_thread()
>>>>>>> Thread 2:                 ssh_event_dopoll(event, 1000);
>>>>>>> in wait_for_something()
>>>>>>>
>>>>>>> I hacked your example so the server_thread() functions leaves
>>>>>>> early. It
>>>>>>> doesn't remove the race but reduces its window. now it works for
>>>>>>> me,
>>>>>>> and I'm
>>>>>>> able to run an xterm with it.
>>>>>>> I strongly recommend that you refactor your code in a singlethread
>>>>>>> design,
>>>>>>> and that you move the x11 callbacks back in the server loop.
>>>>>>>
>>>>>>> Let's discuss what libssh can do better to help you write better
>>>>>>> code.
>>>>>>>
>>>>>>> Kr,
>>>>>>>
>>>>>>> Aris
>>>>>>> Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
>>>>>>>> Hi Aris,
>>>>>>>>
>>>>>>>> No probs and I completely understand and appreciate your
>>>>>>>> interest in
>>>>>>>> looking into this....please let me know how I can help further :)
>>>>>>>>
>>>>>>>> Bye for now
>>>>>>>>
>>>>>>>> On Wed, Nov 9, 2011 at 3:32 PM, Aris
>>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>
>>>>>>>>   wrote:
>>>>>>>>> Hi Jeetu,
>>>>>>>>>
>>>>>>>>> I apologize, still had no time to check it. I'm pretty busy
>>>>>>>>> and all
>>>>>>>>> my
>>>>>>>>> free time is swallowed by another project. I hope I can free
>>>>>>>>> myself
>>>>>>>>> an
>>>>>>>>> hour or two next week to debug this.
>>>>>>>>>
>>>>>>>>> kr,
>>>>>>>>>
>>>>>>>>> Aris
>>>>>>>>>
>>>>>>>>> Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
>>>>>>>>>> Hi Aris,
>>>>>>>>>>
>>>>>>>>>> Just wanted to check in if you've had a chance to try out the
>>>>>>>>>> libssh
>>>>>>>>>> server code I've sent and reproduce the errors I've been seeing?
>>>>>>>>>>
>>>>>>>>>> Thanks so much again for looking into this.
>>>>>>>>>>
>>>>>>>>>> Bye for now
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
>>>>>>>>>> <jeetu.golani@xxxxxxxxx>    wrote:
>>>>>>>>>>> Hi Aris,
>>>>>>>>>>>
>>>>>>>>>>> I'm attaching my proof of concept server code
>>>>>>>>>>> as.....ebpsshd-singlesession.c has compile instructions at the
>>>>>>>>>>> beginning of the code. You will also need to generate a
>>>>>>>>>>> key.h file
>>>>>>>>>>> that holds the public key of the user who will be connecting to
>>>>>>>>>>> this
>>>>>>>>>>> server - this is tempoarary since as of now I'm not reading
>>>>>>>>>>> this
>>>>>>>>>>> info
>>>>>>>>>>> from an authorized_keys or something similar.
>>>>>>>>>>>
>>>>>>>>>>> Just create a key.h file in the same directory and put
>>>>>>>>>>> something
>>>>>>>>>>> like :
>>>>>>>>>>>
>>>>>>>>>>> #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
>>>>>>>>>>>
>>>>>>>>>>> Also as of now ebpsshd-singlesession listens in on port
>>>>>>>>>>> 2000. So
>>>>>>>>>>> ssh
>>>>>>>>>>> should connect to that port.
>>>>>>>>>>>
>>>>>>>>>>> I also have a libssh-project-wrapper script that allows me
>>>>>>>>>>> to try
>>>>>>>>>>> this
>>>>>>>>>>> out without needing to install the libssh i've built. It
>>>>>>>>>>> basically
>>>>>>>>>>> has
>>>>>>>>>>> the content:
>>>>>>>>>>>
>>>>>>>>>>> #!/bin/sh
>>>>>>>>>>>
>>>>>>>>>>> export
>>>>>>>>>>>
>>>>>>>>>>> LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
>>>>>>>>>>>
>>>>>>>>>>> ./$1
>>>>>>>>>>>
>>>>>>>>>>> I have been testing this code with simple examples like
>>>>>>>>>>> xeyes and
>>>>>>>>>>> xcalc. For some reason xcalc shows the problem much sooner than
>>>>>>>>>>> with
>>>>>>>>>>> xeyes, maybe because of the volume of data being transmitted
>>>>>>>>>>> to and
>>>>>>>>>>> fro?
>>>>>>>>>>>
>>>>>>>>>>> This is proof of concept code with a lot of fiddling with
>>>>>>>>>>> buffer
>>>>>>>>>>> sizes
>>>>>>>>>>> as I have been trying to study if any of that makes an impact
>>>>>>>>>>> however
>>>>>>>>>>> please point out any way you think this can be improved on :)
>>>>>>>>>>>
>>>>>>>>>>> I'm sorry to drop this in your lap especially if it turns
>>>>>>>>>>> out it
>>>>>>>>>>> was
>>>>>>>>>>> some server side code issue, however I completely appreciate
>>>>>>>>>>> your
>>>>>>>>>>> help
>>>>>>>>>>> on this. I would like to squash this bug regardless of where it
>>>>>>>>>>> lies
>>>>>>>>>>> i.e. in my code or libssh, unfortunately my understanding of
>>>>>>>>>>> libssh
>>>>>>>>>>> and the ssh protocol is a little limited. However I do not
>>>>>>>>>>> want to
>>>>>>>>>>> put
>>>>>>>>>>> all of this load completely in your lap and if you share your
>>>>>>>>>>> thoughts
>>>>>>>>>>> and there's something you would like me to look into then
>>>>>>>>>>> please
>>>>>>>>>>> let
>>>>>>>>>>> me know.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jeetu
>>>>>>>>>>> ebrain.in | Beehive Computing
>>>>>>>>>>> Discover and run software from any device around you - an open
>>>>>>>>>>> source
>>>>>>>>>>> (GPL) project.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
>>>>>>>>>>> <jeetu.golani@xxxxxxxxx>    wrote:
>>>>>>>>>>>> Hi Aris,
>>>>>>>>>>>>
>>>>>>>>>>>>> I think I'll need a proof-of-concept code to debug this.
>>>>>>>>>>>>> Would
>>>>>>>>>>>>> you
>>>>>>>>>>>>> mind
>>>>>>>>>>>>> sharing your code, or it's not possible (too much integration
>>>>>>>>>>>>> with
>>>>>>>>>>>>> existing code).
>>>>>>>>>>>> No problem at all :) The code is an independent unit as of now
>>>>>>>>>>>> since I
>>>>>>>>>>>> wanted to make it work before I integrate it within my open
>>>>>>>>>>>> source
>>>>>>>>>>>> project (eBrainPool)
>>>>>>>>>>>>
>>>>>>>>>>>> I'll mail this out to you tomm (not on the machine with the
>>>>>>>>>>>> code
>>>>>>>>>>>> right
>>>>>>>>>>>> now :) )
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks so much for looking into this.....truly appreciate
>>>>>>>>>>>> it :)
>>>>>>>>>>>>
>>>>>>>>>>>> Bye for now
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Oct 21, 2011 at 7:26 PM, Aris
>>>>>>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>
>>>>>>>>>>>>   wrote:
>>>>>>>>>>>>> Hi Jeetu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> By seeing your logs, I understand this:
>>>>>>>>>>>>> Both side have a hmac error. The openssh client sees it
>>>>>>>>>>>>> first,
>>>>>>>>>>>>> sends
>>>>>>>>>>>>> a
>>>>>>>>>>>>> disconnect (that works), then there's the error in the libssh
>>>>>>>>>>>>> log.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I'll need a proof-of-concept code to debug this.
>>>>>>>>>>>>> Would
>>>>>>>>>>>>> you
>>>>>>>>>>>>> mind
>>>>>>>>>>>>> sharing your code, or it's not possible (too much integration
>>>>>>>>>>>>> with
>>>>>>>>>>>>> existing code).
>>>>>>>>>>>>>
>>>>>>>>>>>>> If so, do you think I can reproduce the problem by "fixing"
>>>>>>>>>>>>> samplesshd
>>>>>>>>>>>>> to make new X11 channels connection connect to the local
>>>>>>>>>>>>> X11 unix
>>>>>>>>>>>>> socket ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Aris
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
>>>>>>>>>>>>>>> This is not a problem and shouldn't cause invalid MAC
>>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> travel so I'll look at the log when i'm back.
>>>>>>>>>>>>>> Thanks so much Aris :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Oct 18, 2011 at 5:52 PM, Aris
>>>>>>>>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>    wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is not a problem and shouldn't cause invalid MAC
>>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> travel so I'll look at the log when i'm back.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Aris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> debug3: Incorrect RSA1 identifier
>>>>>>>>>>>>>>>> debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> key
>>>>>>>>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Oct 18, 2011 at 03:22:08PM +0500,
>>>>>>>>>>>>>>>> jeetu.golani@xxxxxxxxx
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hi Aris,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've attached a log of the libssh server
>>>>>>>>>>>>>>>>> (log-1-ebpsshd-singlesession-18102011.txt) and the
>>>>>>>>>>>>>>>>> OpenSSH
>>>>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>>>> (log-ssh-1-18102011.txt).
>>>>>>>>>>>>>>>> Greetings
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Stefan Kuttler ==*== nc.netbeisser.de
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>
>>
>>
> 
>