[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: multithreading for a single session

Agreed.

On 03/14/2012 10:47 AM, Gearoid Murphy wrote:
My personal feeling would be to remain single-threaded. Writing asynchronous single-threaded code is initially more challenging than writing synchronous multi-threaded code but when problems occur, the single-threaded context is much easier to diagnose. 

On 14/03/12 14:36, Aris Adamantiadis wrote:

Do you guys think we should include inside libssh a support for
multi-threaded single sessions ? I see a possibility but I also see
many things that can go wrong.

Aris

Le 14/03/12 14:58, Gearoid Murphy a écrit :

It is possible to use a single session in a multi-threaded context as
long as you protect access to the session (and associated channels)
using a session mutex. It's very inefficient however, as every channel
read operation must timeout in order to release the mutex when the
associated channel is inactive so other channels can be processed. This
timeout is the source of the performance penalty, it decimates
bandwidth. I've seen single threaded applications capable of 30-40 mbs
being reduced to 256kps using the multi-threaded approach.

My favored solution these days is to use a boost io_service to peek at
the session socket, whenever data needs to be processed, an
asynchronous handler is invoked which processes the active channels.

On 14/03/12 13:38, Eddy Valdes wrote:

Jeetu,

I seem to recall somebody telling me that the if you are going to use
threads, then you must use a separate session for each thread. The
application that used libssh that I wrote and used to maintain had a
separate session for each thread and it worked well.

Eddy

On 03/14/2012 09:31 AM, jeetu.golani@xxxxxxxxx wrote:

Hi guys,

Just wanted to know if anyone has had a chance to look into the
possibility of multithreading over a single session which I had
discussed in any earlier mail here (pasted below).

For now for my open source eBrainPool (ebrain.in) project I have gone
ahead with piped openssh server and daemon however this is not ideal
for variety of reasons. I would love to use libssh however the issues
I highlighted earlier prevent me from doing that.
Please let me know how I can help with this :)....thanks again for all 
the help everyone here has provided me with in the past :)

Regards,
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run
software from any device around you.

On Wed, Feb 15, 2012 at 2:09 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>   wrote:

oops forgot to mention the git link where this code can be seen
;)....git://ebrain.in:/libssh-server.git :)

sorry


On Wed, Feb 15, 2012 at 1:59 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>   wrote:

Hi Aris,

I've been continuing work with my libssh server based on
suggestions you
had made with regards multithreading when we were debugging issues
where I
was seeing bad packet length, HMAC errors et al.

I refactored my code as you had suggested (below) such that there
would be
only one thread per session, although there could be multiple
connects i.e.
sessions. This worked fantastic for me and I was able to solve the
packet
corruption issues caused due to a race condition with multiple ssh
loops
running. I could therefore have different ssh clients connect to
the same
instance of the server and X forward simple x clients like xcalc,
xclock,
xeyes, xgc.

Unfortunately as soon as I tried to forward more complex
applications such
as libreoffice, chromium etc things weren't working. Further
analysis and a
look at the openssh code showed that these applications spawn
multiple
connects to the same $DISPLAY. Therefore as I understand it (could
be wrong)
the server needs to accept and process different event connections
on the
same session.

I have refactored the code again to account for this but basically
this
reintroduces the same race condition with multiple event handling
we had
seen earlier and while programs like libreoffice, chromium etc
more further
along they eventually crash due to bad packet length, mac errors,etc. 

I tried to protect the event handling function ssh_event_dopoll
with a
mutex hoping this will resolve the issue, however it didn't make any 
difference.

I'm at my wits end here and could appreciate advice here. One
obvious way
is for me to somehow refactor my code in a way that eludes me at
the moment
to get around this issue. The other could be if libssh itself
would handle
multithreaded event handling for the same session differently or
if I am
somehow using it in a wrong way.

I have created a git repo of my code so anyone can study it. I
have not
yet put a GPL license header to the code (just realized that ;) )
however I
don't care and in fact would be honoured if people would find some
use for
this code. The code will eventually be part of the eBrainPool
(ebrain.in)
project which I am working on which itself is GPLv3 - so feel free
to use it
in any way you want.

For now I could really use some ideas on how I could either
refactor my
code to avoid the above issues or somehow if libssh could be
modified to
handle multithreading for the same session differently.

Thanks,

Bye for now
Jeetu
ebrain.in | Beehive Computing
Share your software and computing resources - discover and run
software
from any device around you.



On Mon, Nov 21, 2011 at 5:42 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>   wrote:

Hi Aris,


took me a lot of time and head scratching, but I narrowed it
down to a
threading problem. Two of your threads are running ssh loops at the 
same
I can completely appreciate the effort you must have gone through to 
debug this :)...threading issues can be pesky and since I didn't
understand the internals of libssh it was proving quite difficult
for
me to trace this out.....thank you so much Aris :)


I hacked your example so the server_thread() functions leaves
early. It
doesn't remove the race but reduces its window. now it works for
me,
and I'm
able to run an xterm with it.
I strongly recommend that you refactor your code in a singlethread 
design,
and that you move the x11 callbacks back in the server loop.

I just did a cursory test of your code and it definitely runs much
better from what I could see :)....I will take your advice to
heed and
try to refactor the code.


Let's discuss what libssh can do better to help you write better
code.


I'm a little concerned, I would ultimately need to handle multiple
sessions.....the code you saw dealt with a single session....do you think this would also trigger similar issues?....what precautions do 
you think I need to keep in mind while I attempt this?
Thanks so much Aris, I'm sooooooo glad to finally have the answer to my question and know that this was a problem in my code....now I can 
dedicate my energies in this direction :)

Bye for now



On Mon, Nov 21, 2011 at 3:11 AM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>
wrote:

Hi Jeetu,

took me a lot of time and head scratching, but I narrowed it
down to a
threading problem. Two of your threads are running ssh loops at the 
same
time, and this cause races. The probability that the race causes
problem can
be high, because there's a trigger (data on socket) and both
threads
run
almost the same code at the same time.

Thread 1:      message = ssh_message_get(session);
in server_thread()
Thread 2:                 ssh_event_dopoll(event, 1000);
in wait_for_something()

I hacked your example so the server_thread() functions leaves
early. It
doesn't remove the race but reduces its window. now it works for
me,
and I'm
able to run an xterm with it.
I strongly recommend that you refactor your code in a singlethread 
design,
and that you move the x11 callbacks back in the server loop.

Let's discuss what libssh can do better to help you write better
code.

Kr,

Aris
Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :

Hi Aris,

No probs and I completely understand and appreciate your
interest in
looking into this....please let me know how I can help further :) 

Bye for now

On Wed, Nov 9, 2011 at 3:32 PM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>
   wrote:

Hi Jeetu,

I apologize, still had no time to check it. I'm pretty busy
and all
my
free time is swallowed by another project. I hope I can free
myself
an
hour or two next week to debug this.

kr,

Aris

Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :

Hi Aris,

Just wanted to check in if you've had a chance to try out the
libssh
server code I've sent and reproduce the errors I've been seeing? 

Thanks so much again for looking into this.

Bye for now

On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>     wrote:

Hi Aris,

I'm attaching my proof of concept server code
as.....ebpsshd-singlesession.c has compile instructions at the 
beginning of the code. You will also need to generate a
key.h file
that holds the public key of the user who will be connecting to 
this
server - this is tempoarary since as of now I'm not reading
this
info
from an authorized_keys or something similar.

Just create a key.h file in the same directory and put
something
like :

#define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"

Also as of now ebpsshd-singlesession listens in on port
2000. So
ssh
should connect to that port.

I also have a libssh-project-wrapper script that allows me
to try
this
out without needing to install the libssh i've built. It
basically
has
the content:

#!/bin/sh

export
LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads 

./$1

I have been testing this code with simple examples like
xeyes and
xcalc. For some reason xcalc shows the problem much sooner than 
with
xeyes, maybe because of the volume of data being transmitted
to and
fro?

This is proof of concept code with a lot of fiddling with
buffer
sizes
as I have been trying to study if any of that makes an impact
however
please point out any way you think this can be improved on :)

I'm sorry to drop this in your lap especially if it turns
out it
was
some server side code issue, however I completely appreciate
your
help
on this. I would like to squash this bug regardless of where it 
lies
i.e. in my code or libssh, unfortunately my understanding of
libssh
and the ssh protocol is a little limited. However I do not
want to
put
all of this load completely in your lap and if you share your
thoughts
and there's something you would like me to look into then
please
let
me know.

Thanks,
Jeetu
ebrain.in | Beehive Computing
Discover and run software from any device around you - an open 
source
(GPL) project.


On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx>     wrote:

Hi Aris,


I think I'll need a proof-of-concept code to debug this.
Would
you
mind
sharing your code, or it's not possible (too much integration 
with
existing code).
No problem at all :) The code is an independent unit as of now 
since I
wanted to make it work before I integrate it within my open
source
project (eBrainPool)

I'll mail this out to you tomm (not on the machine with the
code
right
now :) )

Thanks so much for looking into this.....truly appreciate
it :)

Bye for now


On Fri, Oct 21, 2011 at 7:26 PM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>
   wrote:

Hi Jeetu,

By seeing your logs, I understand this:
Both side have a hmac error. The openssh client sees it
first,
sends
a
disconnect (that works), then there's the error in the libssh 
log.

I think I'll need a proof-of-concept code to debug this.
Would
you
mind
sharing your code, or it's not possible (too much integration 
with
existing code).

If so, do you think I can reproduce the problem by "fixing"
samplesshd
to make new X11 channels connection connect to the local
X11 unix
socket ?

Thanks.

Aris


Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :

This is not a problem and shouldn't cause invalid MAC
errors.
I'm
on
travel so I'll look at the log when i'm back.

Thanks so much Aris :)

On Tue, Oct 18, 2011 at 5:52 PM, Aris
Adamantiadis<aris@xxxxxxxxxxxx>     wrote:

Hi,

This is not a problem and shouldn't cause invalid MAC
errors.
I'm
on
travel so I'll look at the log when i'm back.

Aris

Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :

Hi all,

debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1 
public
key
debug2: key_type_from_name: unknown key type '-----BEGIN' 


On Tue, Oct 18, 2011 at 03:22:08PM +0500,
jeetu.golani@xxxxxxxxx
wrote:

Hi Aris,

I've attached a log of the libssh server
(log-1-ebpsshd-singlesession-18102011.txt) and the
OpenSSH
client
(log-ssh-1-18102011.txt).

Greetings
--
Stefan Kuttler ==*== nc.netbeisser.de











--
Eddy Valdes
Project Manager
Atronix Engineering
evaldes@xxxxxxxxxxxxxxxxxxxxxx
770-840-4045 x235
Follow-Ups:
Re: multithreading for a single session"jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>
References:
Re: multithreading for a single session"jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>
Re: multithreading for a single sessionEddy Valdes <evaldes@xxxxxxxxxxxxxxxxxxxxxx>
Re: multithreading for a single sessionGearoid Murphy <gearoid.murphy@xxxxxx>
Re: multithreading for a single sessionAris Adamantiadis <aris@xxxxxxxxxxxx>
Re: multithreading for a single sessionGearoid Murphy <gearoid.murphy@xxxxxx>
Archive administrator: postmaster@lists.cynapses.org