multithreaded server issues

["jeetu.golani@xxxxxxxxx" <jeetu.golani@xxxxxxxxx>]

Hi guys,

I'm facing some issues with the libssh server that I'm putting
together. I'm using the latest libssh from the master. My code can be
found on pastebin at the link below. Also pasted below is the crash
report and what gdb has to about this.

The problem and how to reproduce:

If I make multiple X forwarding connections, and say x forward xeyes
and xcalc I get both the applications. Now if I simply just take one
of these windows and keep moving them around, I get a crash with the
following output.

This crash happens under other circumstances too but the above is a
sureshot way to trigger it.

To go through the server code:

Main thread :

- I am using gthreads for the threading and initialize gthread, call
ssh_threads_set_callbacks, ssh_init()

- I have a global ssh_session* (ssh_session is pointer to a pointer)
and I initiate this with a ssh_new (assuming this would allocate
memory).

- my ssh_bind and ssh_bind_listen is common for all ssh_sessions.

- I then enter a loop where I allocate a new
ssh_session[session_count] = ssh_new i.e. a new session, and move to
ssh_bind_accept (sshbind is common for all sessions, is this a
problem?)

- then public key authentication

- then i start a new thread and pass along session[session_count],
increment session_count.

- The server_thread sees which requests are coming along and calls
appropriate functions to open new channels or exec stuff.

- If SSH_CHANNEL_REQUEST_X11 is received it goes about setting up x11
forwarding, setting the x auth cookie, creating a port e.g. 6010 for
xclients to connect to, and listening for these connections on the
sockets.

- when a client connects to this socket, then I start a new thread
(wait_for_something function) and there is a select and if select goes
through then I accept() and get a client_socket.

- I then go about opening an x11 channel with ssh_channel_open_x11

*****these are steps I'm thinking maybe causing the crash*****

- wait for something thread has as a local variable the following
callback struct, since this is local and in a separate thread I'm
hoping this ensures that each thread / connection basically has it's
own callback and access to functions like copy_chan_to_fd etc and this
alone ensures that no threads or apps are overwriting/corrupting data.
is that correct?

struct ssh_channel_callbacks_struct cb =
	{
    	.channel_data_function = copy_chan_to_fd,
    	.channel_eof_function = chan_close,
    	.channel_close_function = chan_close,
	.userdata = NULL
	};

- I then do a :

ssh_callbacks_init(&cb);
ssh_set_channel_callbacks(chan_x11, &cb);
events = POLLIN | POLLPRI | POLLERR | POLLHUP | POLLNVAL;
event = ssh_event_new();

- and after adding events, fd, etc as shown in samplesshd-tty.c i go
into a ssh_event_dopoll loop.

....am not confident about my call back structures and if there is
some kind of thread issues being caused there.....am not using mutexes
here and I've experimented with them but not sure where in the code I
need them (mutexes I've experimented with are in comments).

Could definitely use any help.

Sorry for the large descript and offloading a rather arduous task to
you guys......am just not sure if this is an issue with my code (most
likely) or libssh threading issue?.....or if someone could help me
decipher what the gdb bt below is trying to tell me would appreciate
it :)...any pointers would be great pls.

Bye for now

----------------------------------------------------------------------------------------------------------------------------------

Server code:

http://pastebin.com/F8Mbgzh2

compile with:

gcc -g -Wall -Wstrict-prototypes -O0 -o ebpsshd
-I/home/jeetu/utils/libssh/libssh-project/include
-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include/ ebpsshd.c
-L/home/jeetu/utils/libssh/libssh-project/build/src -lssh
-L/home/jeetu/utils/libssh/libssh-project/build/src/threads
-lssh_threads -lglib-2.0 -lgthread-2.0


-----------------------------------------------------------------------------------------------------------------------------------

*** glibc detected *** /home/jeetu/code/misc/ebpsshd: double free or
corruption (!prev): 0x000000000061a8b0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71ad6)[0x7ffff71acad6]
/lib/libc.so.6(+0x77563)[0x7ffff71b2563]
/lib/libc.so.6(realloc+0xf0)[0x7ffff71b2a00]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x11591)[0x7ffff7b9c591]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x116ff)[0x7ffff7b9c6ff]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x3676e)[0x7ffff7bc176e]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x2feff)[0x7ffff7bbaeff]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x30300)[0x7ffff7bbb300]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x303cf)[0x7ffff7bbb3cf]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x155a6)[0x7ffff7ba05a6]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(ssh_channel_write+0x2f)[0x7ffff7ba0755]
/home/jeetu/code/misc/ebpsshd[0x403286]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x321c7)[0x7ffff7bbd1c7]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x32029)[0x7ffff7bbd029]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(+0x334cd)[0x7ffff7bbe4cd]
/home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4(ssh_message_get+0x105)[0x7ffff7bb3c66]
/home/jeetu/code/misc/ebpsshd[0x4023a5]
/lib/libglib-2.0.so.0(+0x6ce64)[0x7ffff770ce64]
/lib/libpthread.so.0(+0x68ba)[0x7ffff65108ba]
/lib/libc.so.6(clone+0x6d)[0x7ffff720a02d]
======= Memory map: ========
00400000-00404000 r-xp 00000000 08:03 492746
  /home/jeetu/code/misc/ebpsshd
00604000-00605000 rw-p 00004000 08:03 492746
  /home/jeetu/code/misc/ebpsshd
00605000-00626000 rw-p 00000000 00:00 0                                  [heap]
7fffefdea000-7fffefe00000 r-xp 00000000 08:03 3481639
  /lib/libgcc_s.so.1
7fffefe00000-7fffeffff000 ---p 00016000 08:03 3481639
  /lib/libgcc_s.so.1
7fffeffff000-7ffff0000000 rw-p 00015000 08:03 3481639
  /lib/libgcc_s.so.1
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff40c7000-7ffff40c8000 ---p 00000000 00:00 0
7ffff40c8000-7ffff48c8000 rw-p 00000000 00:00 0
7ffff48c8000-7ffff48c9000 ---p 00000000 00:00 0
7ffff48c9000-7ffff50c9000 rw-p 00000000 00:00 0
7ffff50c9000-7ffff50ca000 ---p 00000000 00:00 0
7ffff50ca000-7ffff58ca000 rw-p 00000000 00:00 0
7ffff58ca000-7ffff58cb000 ---p 00000000 00:00 0
7ffff58cb000-7ffff60cb000 rw-p 00000000 00:00 0
7ffff60cb000-7ffff60cd000 r-xp 00000000 08:03 3500833
  /lib/libdl-2.11.2.so
7ffff60cd000-7ffff62cd000 ---p 00002000 08:03 3500833
  /lib/libdl-2.11.2.so
7ffff62cd000-7ffff62ce000 r--p 00002000 08:03 3500833
  /lib/libdl-2.11.2.so
7ffff62ce000-7ffff62cf000 rw-p 00003000 08:03 3500833
  /lib/libdl-2.11.2.so
7ffff62cf000-7ffff630a000 r-xp 00000000 08:03 2695171
  /lib/libpcre.so.3.12.1
7ffff630a000-7ffff6509000 ---p 0003b000 08:03 2695171
  /lib/libpcre.so.3.12.1
7ffff6509000-7ffff650a000 rw-p 0003a000 08:03 2695171
  /lib/libpcre.so.3.12.1
7ffff650a000-7ffff6521000 r-xp 00000000 08:03 3500844
  /lib/libpthread-2.11.2.so
7ffff6521000-7ffff6720000 ---p 00017000 08:03 3500844
  /lib/libpthread-2.11.2.so
7ffff6720000-7ffff6721000 r--p 00016000 08:03 3500844
  /lib/libpthread-2.11.2.so
7ffff6721000-7ffff6722000 rw-p 00017000 08:03 3500844
  /lib/libpthread-2.11.2.so
7ffff6722000-7ffff6726000 rw-p 00000000 00:00 0
7ffff6726000-7ffff689b000 r-xp 00000000 08:03 4613790
  /usr/lib/libcrypto.so.0.9.8
7ffff689b000-7ffff6a9b000 ---p 00175000 08:03 4613790
  /usr/lib/libcrypto.so.0.9.8
7ffff6a9b000-7ffff6ac3000 rw-p 00175000 08:03 4613790
  /usr/lib/libcrypto.so.0.9.8
7ffff6ac3000-7ffff6ac7000 rw-p 00000000 00:00 0
7ffff6ac7000-7ffff6b16000 r-xp 00000000 08:03 4613791
  /usr/lib/libssl.so.0.9.8
7ffff6b16000-7ffff6d15000 ---p 0004f000 08:03 4613791
  /usr/lib/libssl.so.0.9.8
7ffff6d15000-7ffff6d1c000 rw-p 0004e000 08:03 4613791
  /usr/lib/libssl.so.0.9.8
7ffff6d1c000-7ffff6d33000 r-xp 00000000 08:03 771057
  /usr/lib/libz.so.1.2.3.4
7ffff6d33000-7ffff6f32000 ---p 00017000 08:03 771057
  /usr/lib/libz.so.1.2.3.4
7ffff6f32000-7ffff6f33000 rw-p 00016000 08:03 771057
  /usr/lib/libz.so.1.2.3.4
7ffff6f33000-7ffff6f3a000 r-xp 00000000 08:03 3500846
  /lib/librt-2.11.2.so
7ffff6f3a000-7ffff7139000 ---p 00007000 08:03 3500846
  /lib/librt-2.11.2.so
7ffff7139000-7ffff713a000 r--p 00006000 08:03 3500846
  /lib/librt-2.11.2.so
7ffff713a000-7ffff713b000 rw-p 00007000 08:03 3500846
  /lib/librt-2.11.2.so
7ffff713b000-7ffff7293000 r-xp 00000000 08:03 3500830
  /lib/libc-2.11.2.so
7ffff7293000-7ffff7492000 ---p 00158000 08:03 3500830
  /lib/libc-2.11.2.so
7ffff7492000-7ffff7496000 r--p 00157000 08:03 3500830
  /lib/libc-2.11.2.so
7ffff7496000-7ffff7497000 rw-p 0015b000 08:03 3500830
  /lib/libc-2.11.2.so
7ffff7497000-7ffff749c000 rw-p 00000000 00:00 0
7ffff749c000-7ffff74a0000 r-xp 00000000 08:03 4612307
  /usr/lib/libgthread-2.0.so.0.2800.1
7ffff74a0000-7ffff769f000 ---p 00004000 08:03 4612307
  /usr/lib/libgthread-2.0.so.0.2800.1
7ffff769f000-7ffff76a0000 rw-p 00003000 08:03 4612307
  /usr/lib/libgthread-2.0.so.0.2800.1
7ffff76a0000-7ffff7789000 r-xp 00000000 08:03 2695183
  /lib/libglib-2.0.so.0.2800.1
7ffff7789000-7ffff7988000 ---p 000e9000 08:03 2695183
  /lib/libglib-2.0.so.0.2800.1
7ffff7988000-7ffff7989000 rw-p 000e8000 08:03 2695183
  /lib/libglib-2.0.so.0.2800.1
7ffff7989000-7ffff798a000 rw-p 00000000 00:00 0
7ffff798a000-7ffff798b000 r-xp 00000000 08:03 3663774
  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
7ffff798b000-7ffff7b8a000 ---p 00001000 08:03 3663774
  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
7ffff7b8a000-7ffff7b8b000 rw-p 00000000 08:03 3663774
  /home/jeetu/utils/libssh/libssh-project/build/src/threads/libssh_threads.so.4.3.0
7ffff7b8b000-7ffff7bde000 r-xp 00000000 08:03 3663761
  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
7ffff7bde000-7ffff7ddd000 ---p 00053000 08:03 3663761
  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
7ffff7ddd000-7ffff7ddf000 rw-p 00052000 08:03 3663761
  /home/jeetu/utils/libssh/libssh-project/build/src/libssh.so.4.3.0
7ffff7ddf000-7ffff7dfd000 r-xp 00000000 08:03 3500827
  /lib/ld-2.11.2.so
7ffff7fd4000-7ffff7fdb000 rw-p 00000000 00:00 0
7ffff7ff8000-7ffff7ffb000 rw-p 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 0001d000 08:03 3500827
  /lib/ld-2.11.2.so
7ffff7ffd000-7ffff7ffe000 rw-p 0001e000 08:03 3500827
  /lib/ld-2.11.2.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
  [vsyscall]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff60ca700 (LWP 3433)]
0x00007ffff716d165 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff716d165 in raise () from /lib/libc.so.6
#1  0x00007ffff716ff70 in abort () from /lib/libc.so.6
#2  0x00007ffff71a327b in ?? () from /lib/libc.so.6
#3  0x00007ffff71acad6 in ?? () from /lib/libc.so.6
#4  0x00007ffff71b2563 in ?? () from /lib/libc.so.6
#5  0x00007ffff71b2a00 in realloc () from /lib/libc.so.6
#6  0x00007ffff7b9c591 in realloc_buffer (buffer=0x60dd70, needed=8192)
    at /home/jeetu/utils/libssh/libssh-project/src/buffer.c:121
#7  0x00007ffff7b9c6ff in buffer_add_data (buffer=0x60dd70, data=0x61b8c0,
    len=8055) at /home/jeetu/utils/libssh/libssh-project/src/buffer.c:186
#8  0x00007ffff7bc176e in ssh_socket_write (s=0x60dd10, buffer=0x61b8c0,
    len=8055) at /home/jeetu/utils/libssh/libssh-project/src/socket.c:586
#9  0x00007ffff7bbaeff in ssh_packet_write (session=0x60d4f0)
    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:437
#10 0x00007ffff7bbb300 in packet_send2 (session=0x60d4f0)
    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:508
#11 0x00007ffff7bbb3cf in packet_send (session=0x60d4f0)
    at /home/jeetu/utils/libssh/libssh-project/src/packet.c:526
#12 0x00007ffff7ba05a6 in channel_write_common (channel=0x60fe50,
    data=0x7ffff60c8d80, len=4000, is_stderr=0)
    at /home/jeetu/utils/libssh/libssh-project/src/channels.c:1208
#13 0x00007ffff7ba0755 in ssh_channel_write (channel=0x60fe50,
    data=0x7ffff60c8d80, len=4000)
---Type <return> to continue, or q <return> to quit---
    at /home/jeetu/utils/libssh/libssh-project/src/channels.c:1248
#14 0x0000000000403286 in copy_fd_to_chan (fd=10, revents=1, userdata=0x60fe50)
    at ebpsshd.c:612
#15 0x00007ffff7bbd1c7 in ssh_event_fd_wrapper_callback (p=0x60e1e0, fd=10,
    revents=1, userdata=0x610570)
    at /home/jeetu/utils/libssh/libssh-project/src/poll.c:705
#16 0x00007ffff7bbd029 in ssh_poll_ctx_dopoll (ctx=0x60e870, timeout=-1)
    at /home/jeetu/utils/libssh/libssh-project/src/poll.c:619
#17 0x00007ffff7bbe4cd in ssh_handle_packets (session=0x60d4f0, timeout=-1)
    at /home/jeetu/utils/libssh/libssh-project/src/session.c:442
#18 0x00007ffff7bb3c66 in ssh_message_get (session=0x60d4f0)
    at /home/jeetu/utils/libssh/libssh-project/src/messages.c:186
#19 0x00000000004023a5 in server_thread (session_data=0x60d4f0)
    at ebpsshd.c:258
#20 0x00007ffff770ce64 in ?? () from /lib/libglib-2.0.so.0
#21 0x00007ffff65108ba in start_thread () from /lib/libpthread.so.0
#22 0x00007ffff720a02d in clone () from /lib/libc.so.6
#23 0x0000000000000000 in ?? ()
(gdb)
(gdb) x 0x60dd70
0x60dd70:	0x0061d8d0
(gdb)