Re: multithreaded server issues

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

Hi again,

I've made further progress in my attempts to debug my server code.

I've increased the buffer size in copy_fd_to_chan from 4000 to 40000
and am reading slightly less than that. I also noticed that I had not
initialized the buffer.

Result:
The apps launched seem to be far more stable, even with 3 concurrent
connections :) I did have a crash after a lot of use - will study this
some more.

However this gives credence to my thought that it is something wrong
in my code (and/or my understanding so far) than libssh itself.

A question running though my mind yet though is if I should have
mutexes before reading from fd and then again while writing to the
channel or since each connection is a separate session and thread it
doesn't matter.

Thanks again for everything, any thoughts on any of the above and
general pointers on things I should be keeping in mind based on your
own experience would be appreciated.

Bye for now

On Sat, Jun 4, 2011 at 2:32 PM, jeetu.golani@xxxxxxxxx
<jeetu.golani@xxxxxxxxx> wrote:
> Hi guys,
>
> I'm facing some issues with the libssh server that I'm putting
> together. I'm using the latest libssh from the master. My code can be
> found on pastebin at the link below. Also pasted below is the crash
> report and what gdb has to about this.
>
> The problem and how to reproduce:
>
> If I make multiple X forwarding connections, and say x forward xeyes
> and xcalc I get both the applications. Now if I simply just take one
> of these windows and keep moving them around, I get a crash with the
> following output.
>
> This crash happens under other circumstances too but the above is a
> sureshot way to trigger it.
>
> To go through the server code:
>
> Main thread :
>
> - I am using gthreads for the threading and initialize gthread, call
> ssh_threads_set_callbacks, ssh_init()
>
> - I have a global ssh_session* (ssh_session is pointer to a pointer)
> and I initiate this with a ssh_new (assuming this would allocate
> memory).
>
> - my ssh_bind and ssh_bind_listen is common for all ssh_sessions.
>
> - I then enter a loop where I allocate a new
> ssh_session[session_count] = ssh_new i.e. a new session, and move to
> ssh_bind_accept (sshbind is common for all sessions, is this a
> problem?)
>
> - then public key authentication
>
> - then i start a new thread and pass along session[session_count],
> increment session_count.
>
> - The server_thread sees which requests are coming along and calls
> appropriate functions to open new channels or exec stuff.
>
> - If SSH_CHANNEL_REQUEST_X11 is received it goes about setting up x11
> forwarding, setting the x auth cookie, creating a port e.g. 6010 for
> xclients to connect to, and listening for these connections on the
> sockets.
>
> - when a client connects to this socket, then I start a new thread
> (wait_for_something function) and there is a select and if select goes
> through then I accept() and get a client_socket.
>
> - I then go about opening an x11 channel with ssh_channel_open_x11
>
> *****these are steps I'm thinking maybe causing the crash*****
>
> - wait for something thread has as a local variable the following
> callback struct, since this is local and in a separate thread I'm
> hoping this ensures that each thread / connection basically has it's
> own callback and access to functions like copy_chan_to_fd etc and this
> alone ensures that no threads or apps are overwriting/corrupting data.
> is that correct?
>
> struct ssh_channel_callbacks_struct cb =
>        {
>        .channel_data_function = copy_chan_to_fd,
>        .channel_eof_function = chan_close,
>        .channel_close_function = chan_close,
>        .userdata = NULL
>        };
>
> - I then do a :
>
> ssh_callbacks_init(&cb);
> ssh_set_channel_callbacks(chan_x11, &cb);
> events = POLLIN | POLLPRI | POLLERR | POLLHUP | POLLNVAL;
> event = ssh_event_new();
>
> - and after adding events, fd, etc as shown in samplesshd-tty.c i go
> into a ssh_event_dopoll loop.
>
> ....am not confident about my call back structures and if there is
> some kind of thread issues being caused there.....am not using mutexes
> here and I've experimented with them but not sure where in the code I
> need them (mutexes I've experimented with are in comments).
>
> Could definitely use any help.
>
> Sorry for the large descript and offloading a rather arduous task to
> you guys......am just not sure if this is an issue with my code (most
> likely) or libssh threading issue?.....or if someone could help me
> decipher what the gdb bt below is trying to tell me would appreciate
> it :)...any pointers would be great pls.
>
> Bye for now
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Server code:
>
> http://pastebin.com/F8Mbgzh2
>
> compile with:
>
> gcc -g -Wall -Wstrict-prototypes -O0 -o ebpsshd
> -I/home/jeetu/utils/libssh/libssh-project/include
> -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include/ ebpsshd.c
> -L/home/jeetu/utils/libssh/libssh-project/build/src -lssh
> -L/home/jeetu/utils/libssh/libssh-project/build/src/threads
> -lssh_threads -lglib-2.0 -lgthread-2.0
>
>
> -----------------------------------------------------------------------------------------------------------------------------------
>
> *** glibc detected *** /home/jeetu/code/misc/ebpsshd: double free or
> corruption (!prev): 0x000000000061a8b0 ***
> ======= Backtrace: =========
> /lib/libc.so.6(+0x71ad6)[0x7ffff71acad6]
> /lib/libc.so.6(+0x77563)[0x7ffff71b2563]
> /lib/libc.so.6(realloc+0xf0)[0x7ffff71b2a00]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x11591)[0x7ffff7b9c591]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x116ff)[0x7ffff7b9c6ff]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x3676e)[0x7ffff7bc176e]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x2feff)[0x7ffff7bbaeff]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x30300)[0x7ffff7bbb300]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x303cf)[0x7ffff7bbb3cf]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x155a6)[0x7ffff7ba05a6]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(ssh_channel_write+0x2f)[0x7ffff7ba0755]
> /home/jeetu/code/misc/ebpsshd[0x403286]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x321c7)[0x7ffff7bbd1c7]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x32029)[0x7ffff7bbd029]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x334cd)[0x7ffff7bbe4cd]
> /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(ssh_message_get+0x105)[0x7ffff7bb3c66]
> /home/jeetu/code/misc/ebpsshd[0x4023a5]
> /lib/libglib-2.0.so.0(+0x6ce64)[0x7ffff770ce64]
> /lib/libpthread.so.0(+0x68ba)[0x7ffff65108ba]
> /lib/libc.so.6(clone+0x6d)[0x7ffff720a02d]
> ======= Memory map: ========
> 00400000-00404000 r-xp 00000000 08:03 492746
>  /home/jeetu/code/misc/ebpsshd
> 00604000-00605000 rw-p 00004000 08:03 492746
>  /home/jeetu/code/misc/ebpsshd
> 00605000-00626000 rw-p 00000000 00:00 0                                  [heap]
> 7fffefdea000-7fffefe00000 r-xp 00000000 08:03 3481639
>  /lib/libgcc_s.so.1
> 7fffefe00000-7fffeffff000 ---p 00016000 08:03 3481639
>  /lib/libgcc_s.so.1
> 7fffeffff000-7ffff0000000 rw-p 00015000 08:03 3481639
>  /lib/libgcc_s.so.1
> 7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
> 7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
> 7ffff40c7000-7ffff40c8000 ---p 00000000 00:00 0
> 7ffff40c8000-7ffff48c8000 rw-p 00000000 00:00 0
> 7ffff48c8000-7ffff48c9000 ---p 00000000 00:00 0
> 7ffff48c9000-7ffff50c9000 rw-p 00000000 00:00 0
> 7ffff50c9000-7ffff50ca000 ---p 00000000 00:00 0
> 7ffff50ca000-7ffff58ca000 rw-p 00000000 00:00 0
> 7ffff58ca000-7ffff58cb000 ---p 00000000 00:00 0
> 7ffff58cb000-7ffff60cb000 rw-p 00000000 00:00 0
> 7ffff60cb000-7ffff60cd000 r-xp 00000000 08:03 3500833
>  /lib/libdl-2.11.2.so
> 7ffff60cd000-7ffff62cd000 ---p 00002000 08:03 3500833
>  /lib/libdl-2.11.2.so
> 7ffff62cd000-7ffff62ce000 r--p 00002000 08:03 3500833
>  /lib/libdl-2.11.2.so
> 7ffff62ce000-7ffff62cf000 rw-p 00003000 08:03 3500833
>  /lib/libdl-2.11.2.so
> 7ffff62cf000-7ffff630a000 r-xp 00000000 08:03 2695171
>  /lib/libpcre.so.3.12.1
> 7ffff630a000-7ffff6509000 ---p 0003b000 08:03 2695171
>  /lib/libpcre.so.3.12.1
> 7ffff6509000-7ffff650a000 rw-p 0003a000 08:03 2695171
>  /lib/libpcre.so.3.12.1
> 7ffff650a000-7ffff6521000 r-xp 00000000 08:03 3500844
>  /lib/libpthread-2.11.2.so
> 7ffff6521000-7ffff6720000 ---p 00017000 08:03 3500844
>  /lib/libpthread-2.11.2.so
> 7ffff6720000-7ffff6721000 r--p 00016000 08:03 3500844
>  /lib/libpthread-2.11.2.so
> 7ffff6721000-7ffff6722000 rw-p 00017000 08:03 3500844
>  /lib/libpthread-2.11.2.so
> 7ffff6722000-7ffff6726000 rw-p 00000000 00:00 0
> 7ffff6726000-7ffff689b000 r-xp 00000000 08:03 4613790
>  /usr/lib/libcrypto.so.0.9.8
> 7ffff689b000-7ffff6a9b000 ---p 00175000 08:03 4613790
>  /usr/lib/libcrypto.so.0.9.8
> 7ffff6a9b000-7ffff6ac3000 rw-p 00175000 08:03 4613790
>  /usr/lib/libcrypto.so.0.9.8
> 7ffff6ac3000-7ffff6ac7000 rw-p 00000000 00:00 0
> 7ffff6ac7000-7ffff6b16000 r-xp 00000000 08:03 4613791
>  /usr/lib/libssl.so.0.9.8
> 7ffff6b16000-7ffff6d15000 ---p 0004f000 08:03 4613791
>  /usr/lib/libssl.so.0.9.8
> 7ffff6d15000-7ffff6d1c000 rw-p 0004e000 08:03 4613791
>  /usr/lib/libssl.so.0.9.8
> 7ffff6d1c000-7ffff6d33000 r-xp 00000000 08:03 771057
>  /usr/lib/libz.so.1.2.3.4
> 7ffff6d33000-7ffff6f32000 ---p 00017000 08:03 771057
>  /usr/lib/libz.so.1.2.3.4
> 7ffff6f32000-7ffff6f33000 rw-p 00016000 08:03 771057
>  /usr/lib/libz.so.1.2.3.4
> 7ffff6f33000-7ffff6f3a000 r-xp 00000000 08:03 3500846
>  /lib/librt-2.11.2.so
> 7ffff6f3a000-7ffff7139000 ---p 00007000 08:03 3500846
>  /lib/librt-2.11.2.so
> 7ffff7139000-7ffff713a000 r--p 00006000 08:03 3500846
>  /lib/librt-2.11.2.so
> 7ffff713a000-7ffff713b000 rw-p 00007000 08:03 3500846
>  /lib/librt-2.11.2.so
> 7ffff713b000-7ffff7293000 r-xp 00000000 08:03 3500830
>  /lib/libc-2.11.2.so
> 7ffff7293000-7ffff7492000 ---p 00158000 08:03 3500830
>  /lib/libc-2.11.2.so
> 7ffff7492000-7ffff7496000 r--p 00157000 08:03 3500830
>  /lib/libc-2.11.2.so
> 7ffff7496000-7ffff7497000 rw-p 0015b000 08:03 3500830
>  /lib/libc-2.11.2.so
> 7ffff7497000-7ffff749c000 rw-p 00000000 00:00 0
> 7ffff749c000-7ffff74a0000 r-xp 00000000 08:03 4612307
>  /usr/lib/libgthread-2.0.so.0.2800.1
> 7ffff74a0000-7ffff769f000 ---p 00004000 08:03 4612307
>  /usr/lib/libgthread-2.0.so.0.2800.1
> 7ffff769f000-7ffff76a0000 rw-p 00003000 08:03 4612307
>  /usr/lib/libgthread-2.0.so.0.2800.1
> 7ffff76a0000-7ffff7789000 r-xp 00000000 08:03 2695183
>  /lib/libglib-2.0.so.0.2800.1
> 7ffff7789000-7ffff7988000 ---p 000e9000 08:03 2695183
>  /lib/libglib-2.0.so.0.2800.1
> 7ffff7988000-7ffff7989000 rw-p 000e8000 08:03 2695183
>  /lib/libglib-2.0.so.0.2800.1
> 7ffff7989000-7ffff798a000 rw-p 00000000 00:00 0
> 7ffff798a000-7ffff798b000 r-xp 00000000 08:03 3663774
>  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
> 7ffff798b000-7ffff7b8a000 ---p 00001000 08:03 3663774
>  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
> 7ffff7b8a000-7ffff7b8b000 rw-p 00000000 08:03 3663774
>  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
> 7ffff7b8b000-7ffff7bde000 r-xp 00000000 08:03 3663761
>  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
> 7ffff7bde000-7ffff7ddd000 ---p 00053000 08:03 3663761
>  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
> 7ffff7ddd000-7ffff7ddf000 rw-p 00052000 08:03 3663761
>  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
> 7ffff7ddf000-7ffff7dfd000 r-xp 00000000 08:03 3500827
>  /lib/ld-2.11.2.so
> 7ffff7fd4000-7ffff7fdb000 rw-p 00000000 00:00 0
> 7ffff7ff8000-7ffff7ffb000 rw-p 00000000 00:00 0
> 7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
> 7ffff7ffc000-7ffff7ffd000 r--p 0001d000 08:03 3500827
>  /lib/ld-2.11.2.so
> 7ffff7ffd000-7ffff7ffe000 rw-p 0001e000 08:03 3500827
>  /lib/ld-2.11.2.so
> 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
> 7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>  [vsyscall]
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7ffff60ca700 (LWP 3433)]
> 0x00007ffff716d165 in raise () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007ffff716d165 in raise () from /lib/libc.so.6
> #1  0x00007ffff716ff70 in abort () from /lib/libc.so.6
> #2  0x00007ffff71a327b in ?? () from /lib/libc.so.6
> #3  0x00007ffff71acad6 in ?? () from /lib/libc.so.6
> #4  0x00007ffff71b2563 in ?? () from /lib/libc.so.6
> #5  0x00007ffff71b2a00 in realloc () from /lib/libc.so.6
> #6  0x00007ffff7b9c591 in realloc_buffer (buffer=0x60dd70, needed=8192)
>    at /home/jeetu/utils/libssh/libssh-project/src/buffer.c:121
> #7  0x00007ffff7b9c6ff in buffer_add_data (buffer=0x60dd70, data=0x61b8c0,
>    len=8055) at /home/jeetu/utils/libssh/libssh-project/src/buffer.c:186
> #8  0x00007ffff7bc176e in ssh_socket_write (s=0x60dd10, buffer=0x61b8c0,
>    len=8055) at /home/jeetu/utils/libssh/libssh-project/src/socket.c:586
> #9  0x00007ffff7bbaeff in ssh_packet_write (session=0x60d4f0)
>    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:437
> #10 0x00007ffff7bbb300 in packet_send2 (session=0x60d4f0)
>    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:508
> #11 0x00007ffff7bbb3cf in packet_send (session=0x60d4f0)
>    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:526
> #12 0x00007ffff7ba05a6 in channel_write_common (channel=0x60fe50,
>    data=0x7ffff60c8d80, len=4000, is_stderr=0)
>    at /home/jeetu/utils/libssh/libssh-project/src/channels.c:1208
> #13 0x00007ffff7ba0755 in ssh_channel_write (channel=0x60fe50,
>    data=0x7ffff60c8d80, len=4000)
> ---Type <return> to continue, or q <return> to quit---
>    at /home/jeetu/utils/libssh/libssh-project/src/channels.c:1248
> #14 0x0000000000403286 in copy_fd_to_chan (fd=10, revents=1, userdata=0x60fe50)
>    at ebpsshd.c:612
> #15 0x00007ffff7bbd1c7 in ssh_event_fd_wrapper_callback (p=0x60e1e0, fd=10,
>    revents=1, userdata=0x610570)
>    at /home/jeetu/utils/libssh/libssh-project/src/poll.c:705
> #16 0x00007ffff7bbd029 in ssh_poll_ctx_dopoll (ctx=0x60e870, timeout=-1)
>    at /home/jeetu/utils/libssh/libssh-project/src/poll.c:619
> #17 0x00007ffff7bbe4cd in ssh_handle_packets (session=0x60d4f0, timeout=-1)
>    at /home/jeetu/utils/libssh/libssh-project/src/session.c:442
> #18 0x00007ffff7bb3c66 in ssh_message_get (session=0x60d4f0)
>    at /home/jeetu/utils/libssh/libssh-project/src/messages.c:186
> #19 0x00000000004023a5 in server_thread (session_data=0x60d4f0)
>    at ebpsshd.c:258
> #20 0x00007ffff770ce64 in ?? () from /lib/libglib-2.0.so.0
> #21 0x00007ffff65108ba in start_thread () from /lib/libpthread.so.0
> #22 0x00007ffff720a02d in clone () from /lib/libc.so.6
> #23 0x0000000000000000 in ?? ()
> (gdb)
> (gdb) x 0x60dd70
> 0x60dd70:       0x0061d8d0
> (gdb)
>