Re: Server woes - Bad packet length, corrupted MAC on input

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

Hi Aris,

>
> took me a lot of time and head scratching, but I narrowed it down to a
> threading problem. Two of your threads are running ssh loops at the same

I can completely appreciate the effort you must have gone through to
debug this :)...threading issues can be pesky and since I didn't
understand the internals of libssh it was proving quite difficult for
me to trace this out.....thank you so much Aris :)

> I hacked your example so the server_thread() functions leaves early. It
> doesn't remove the race but reduces its window. now it works for me, and I'm
> able to run an xterm with it.
> I strongly recommend that you refactor your code in a singlethread design,
> and that you move the x11 callbacks back in the server loop.

I just did a cursory test of your code and it definitely runs much
better from what I could see :)....I will take your advice to heed and
try to refactor the code.

>
> Let's discuss what libssh can do better to help you write better code.
>

I'm a little concerned, I would ultimately need to handle multiple
sessions.....the code you saw dealt with a single session....do you
think this would also trigger similar issues?....what precautions do
you think I need to keep in mind while I attempt this?

Thanks so much Aris, I'm sooooooo glad to finally have the answer to
my question and know that this was a problem in my code....now I can
dedicate my energies in this direction :)

Bye for now



On Mon, Nov 21, 2011 at 3:11 AM, Aris Adamantiadis <aris@xxxxxxxxxxxx> wrote:
> Hi Jeetu,
>
> took me a lot of time and head scratching, but I narrowed it down to a
> threading problem. Two of your threads are running ssh loops at the same
> time, and this cause races. The probability that the race causes problem can
> be high, because there's a trigger (data on socket) and both threads run
> almost the same code at the same time.
>
> Thread 1:      message = ssh_message_get(session);
> in server_thread()
> Thread 2:                 ssh_event_dopoll(event, 1000);
> in wait_for_something()
>
> I hacked your example so the server_thread() functions leaves early. It
> doesn't remove the race but reduces its window. now it works for me, and I'm
> able to run an xterm with it.
> I strongly recommend that you refactor your code in a singlethread design,
> and that you move the x11 callbacks back in the server loop.
>
> Let's discuss what libssh can do better to help you write better code.
>
> Kr,
>
> Aris
> Le 09/11/11 12:38, jeetu.golani@xxxxxxxxx a écrit :
>>
>> Hi Aris,
>>
>> No probs and I completely understand and appreciate your interest in
>> looking into this....please let me know how I can help further :)
>>
>> Bye for now
>>
>> On Wed, Nov 9, 2011 at 3:32 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
>>  wrote:
>>>
>>> Hi Jeetu,
>>>
>>> I apologize, still had no time to check it. I'm pretty busy and all my
>>> free time is swallowed by another project. I hope I can free myself an
>>> hour or two next week to debug this.
>>>
>>> kr,
>>>
>>> Aris
>>>
>>> Le 3/11/11 19:13, jeetu.golani@xxxxxxxxx a écrit :
>>>>
>>>> Hi Aris,
>>>>
>>>> Just wanted to check in if you've had a chance to try out the libssh
>>>> server code I've sent and reproduce the errors I've been seeing?
>>>>
>>>> Thanks so much again for looking into this.
>>>>
>>>> Bye for now
>>>>
>>>> On Sat, Oct 22, 2011 at 2:20 PM, jeetu.golani@xxxxxxxxx
>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>>>>
>>>>> Hi Aris,
>>>>>
>>>>> I'm attaching my proof of concept server code
>>>>> as.....ebpsshd-singlesession.c has compile instructions at the
>>>>> beginning of the code. You will also need to generate a key.h file
>>>>> that holds the public key of the user who will be connecting to this
>>>>> server - this is tempoarary since as of now I'm not reading this info
>>>>> from an authorized_keys or something similar.
>>>>>
>>>>> Just create a key.h file in the same directory and put something like :
>>>>>
>>>>> #define MY_PUB_KEY "[YOUR PUBLIC KEY WITHIN THESE QUOTES]"
>>>>>
>>>>> Also as of now ebpsshd-singlesession listens in on port 2000. So ssh
>>>>> should connect to that port.
>>>>>
>>>>> I also have a libssh-project-wrapper script that allows me to try this
>>>>> out without needing to install the libssh i've built. It basically has
>>>>> the content:
>>>>>
>>>>> #!/bin/sh
>>>>>
>>>>> export
>>>>> LD_LIBRARY_PATH=/home/jeetu/utils/libssh/libssh-project/build/src:/home/jeetu/utils/libssh/libssh-project/build/src/threads
>>>>> ./$1
>>>>>
>>>>> I have been testing this code with simple examples like xeyes and
>>>>> xcalc. For some reason xcalc shows the problem much sooner than with
>>>>> xeyes, maybe because of the volume of data being transmitted to and
>>>>> fro?
>>>>>
>>>>> This is proof of concept code with a lot of fiddling with buffer sizes
>>>>> as I have been trying to study if any of that makes an impact however
>>>>> please point out any way you think this can be improved on :)
>>>>>
>>>>> I'm sorry to drop this in your lap especially if it turns out it was
>>>>> some server side code issue, however I completely appreciate your help
>>>>> on this. I would like to squash this bug regardless of where it lies
>>>>> i.e. in my code or libssh, unfortunately my understanding of libssh
>>>>> and the ssh protocol is a little limited. However I do not want to put
>>>>> all of this load completely in your lap and if you share your thoughts
>>>>> and there's something you would like me to look into then please let
>>>>> me know.
>>>>>
>>>>> Thanks,
>>>>> Jeetu
>>>>> ebrain.in | Beehive Computing
>>>>> Discover and run software from any device around you - an open source
>>>>> (GPL) project.
>>>>>
>>>>>
>>>>> On Fri, Oct 21, 2011 at 11:22 PM, jeetu.golani@xxxxxxxxx
>>>>> <jeetu.golani@xxxxxxxxx>  wrote:
>>>>>>
>>>>>> Hi Aris,
>>>>>>
>>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
>>>>>>> mind
>>>>>>> sharing your code, or it's not possible (too much integration with
>>>>>>> existing code).
>>>>>>
>>>>>> No problem at all :) The code is an independent unit as of now since I
>>>>>> wanted to make it work before I integrate it within my open source
>>>>>> project (eBrainPool)
>>>>>>
>>>>>> I'll mail this out to you tomm (not on the machine with the code right
>>>>>> now :) )
>>>>>>
>>>>>> Thanks so much for looking into this.....truly appreciate it :)
>>>>>>
>>>>>> Bye for now
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 21, 2011 at 7:26 PM, Aris Adamantiadis<aris@xxxxxxxxxxxx>
>>>>>>  wrote:
>>>>>>>
>>>>>>> Hi Jeetu,
>>>>>>>
>>>>>>> By seeing your logs, I understand this:
>>>>>>> Both side have a hmac error. The openssh client sees it first, sends
>>>>>>> a
>>>>>>> disconnect (that works), then there's the error in the libssh log.
>>>>>>>
>>>>>>> I think I'll need a proof-of-concept code to debug this. Would you
>>>>>>> mind
>>>>>>> sharing your code, or it's not possible (too much integration with
>>>>>>> existing code).
>>>>>>>
>>>>>>> If so, do you think I can reproduce the problem by "fixing"
>>>>>>> samplesshd
>>>>>>> to make new X11 channels connection connect to the local X11 unix
>>>>>>> socket ?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Aris
>>>>>>>
>>>>>>>
>>>>>>> Le 18/10/11 20:34, jeetu.golani@xxxxxxxxx a écrit :
>>>>>>>>>
>>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors. I'm
>>>>>>>>> on
>>>>>>>>> travel so I'll look at the log when i'm back.
>>>>>>>>
>>>>>>>> Thanks so much Aris :)
>>>>>>>>
>>>>>>>> On Tue, Oct 18, 2011 at 5:52 PM, Aris
>>>>>>>> Adamantiadis<aris@xxxxxxxxxxxx>  wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> This is not a problem and shouldn't cause invalid MAC errors. I'm
>>>>>>>>> on
>>>>>>>>> travel so I'll look at the log when i'm back.
>>>>>>>>>
>>>>>>>>> Aris
>>>>>>>>>
>>>>>>>>> Le 18/10/11 12:30, u@xxxxxxxxxxxxx a écrit :
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> debug3: Incorrect RSA1 identifier
>>>>>>>>>> debug3: Could not load "/home/jeetu/.ssh/id_rsa" as a RSA1 public
>>>>>>>>>> key
>>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 18, 2011 at 03:22:08PM +0500, jeetu.golani@xxxxxxxxx
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Aris,
>>>>>>>>>>>
>>>>>>>>>>> I've attached a log of the libssh server
>>>>>>>>>>> (log-1-ebpsshd-singlesession-18102011.txt) and the OpenSSH client
>>>>>>>>>>> (log-ssh-1-18102011.txt).
>>>>>>>>>>
>>>>>>>>>> Greetings
>>>>>>>>>> --
>>>>>>>>>> Stefan Kuttler ==*== nc.netbeisser.de
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>