Trying to debug segfault

[Orion Poplawski <orion@xxxxxxxx>]

I'm trying to debug a segfault in libssh used by x2goclient.


I'm starting with this valgrind report:

x2go-DEBUG-../src/sshmasterconnection.cpp:2199> New exec channel created.

[2018/01/25 10:05:34.194015, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT
for socket
==22928== Invalid read of size 2
==22928==    at 0x586CF30: ssh_poll_get_events (poll.c:336)
==22928==    by 0x58705DB: ssh_socket_unbuffered_write (socket.c:574)
==22928==    by 0x58705DB: ssh_socket_nonblocking_flush (socket.c:667)
==22928==    by 0x58706D3: ssh_socket_write (socket.c:628)
==22928==    by 0x586755E: ssh_packet_write (packet.c:532)
==22928==    by 0x586755E: packet_send2 (packet.c:602)
==22928==    by 0x5855689: channel_write_common (channels.c:1336)
==22928==    by 0x4D64F0: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2322)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Address 0x11785478 is 24 bytes inside a block of size 48 free'd
==22928==    at 0x4C2ACDD: free (vg_replace_malloc.c:530)
==22928==    by 0x586D232: ssh_poll_ctx_free (poll.c:462)
==22928==    by 0x586D886: ssh_event_free (poll.c:1023)
==22928==    by 0x585BB4F: ssh_select (connect.c:516)
==22928==    by 0x4D5D85: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2210)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Block was alloc'd at
==22928==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==22928==    by 0x586CEDE: ssh_poll_new (poll.c:288)
==22928==    by 0x5870053: ssh_socket_get_poll_handle_out (socket.c:374)
==22928==    by 0x5870E97: ssh_socket_connect_proxycommand (socket.c:878)
==22928==    by 0x5859F5B: ssh_connect (client.c:585)
==22928==    by 0x4CA6BA: SshMasterConnection::sshConnect()
(sshmasterconnection.cpp:847)
==22928==    by 0x4D8203: SshMasterConnection::run() (sshmasterconnection.cpp:637)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==
==22928== Invalid read of size 8
==22928==    at 0x586CF40: ssh_poll_set_events (poll.c:348)
==22928==    by 0x58705EA: ssh_socket_unbuffered_write (socket.c:574)
==22928==    by 0x58705EA: ssh_socket_nonblocking_flush (socket.c:667)
==22928==    by 0x58706D3: ssh_socket_write (socket.c:628)
==22928==    by 0x586755E: ssh_packet_write (packet.c:532)
==22928==    by 0x586755E: packet_send2 (packet.c:602)
==22928==    by 0x5855689: channel_write_common (channels.c:1336)
==22928==    by 0x4D64F0: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2322)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Address 0x11785460 is 0 bytes inside a block of size 48 free'd
==22928==    at 0x4C2ACDD: free (vg_replace_malloc.c:530)
==22928==    by 0x586D232: ssh_poll_ctx_free (poll.c:462)
==22928==    by 0x586D886: ssh_event_free (poll.c:1023)
==22928==    by 0x585BB4F: ssh_select (connect.c:516)
==22928==    by 0x4D5D85: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2210)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Block was alloc'd at
==22928==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==22928==    by 0x586CEDE: ssh_poll_new (poll.c:288)
==22928==    by 0x5870053: ssh_socket_get_poll_handle_out (socket.c:374)
==22928==    by 0x5870E97: ssh_socket_connect_proxycommand (socket.c:878)
==22928==    by 0x5859F5B: ssh_connect (client.c:585)
==22928==    by 0x4CA6BA: SshMasterConnection::sshConnect()
(sshmasterconnection.cpp:847)
==22928==    by 0x4D8203: SshMasterConnection::run() (sshmasterconnection.cpp:637)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==
==22928== Invalid write of size 2
==22928==    at 0x586CF43: ssh_poll_set_events (poll.c:347)
==22928==    by 0x58705EA: ssh_socket_unbuffered_write (socket.c:574)
==22928==    by 0x58705EA: ssh_socket_nonblocking_flush (socket.c:667)
==22928==    by 0x58706D3: ssh_socket_write (socket.c:628)
==22928==    by 0x586755E: ssh_packet_write (packet.c:532)
==22928==    by 0x586755E: packet_send2 (packet.c:602)
==22928==    by 0x5855689: channel_write_common (channels.c:1336)
==22928==    by 0x4D64F0: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2322)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Address 0x11785478 is 24 bytes inside a block of size 48 free'd
==22928==    at 0x4C2ACDD: free (vg_replace_malloc.c:530)
==22928==    by 0x586D232: ssh_poll_ctx_free (poll.c:462)
==22928==    by 0x586D886: ssh_event_free (poll.c:1023)
==22928==    by 0x585BB4F: ssh_select (connect.c:516)
==22928==    by 0x4D5D85: SshMasterConnection::channelLoop()
(sshmasterconnection.cpp:2210)
==22928==    by 0x4D9839: SshMasterConnection::run() (sshmasterconnection.cpp:785)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==  Block was alloc'd at
==22928==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==22928==    by 0x586CEDE: ssh_poll_new (poll.c:288)
==22928==    by 0x5870053: ssh_socket_get_poll_handle_out (socket.c:374)
==22928==    by 0x5870E97: ssh_socket_connect_proxycommand (socket.c:878)
==22928==    by 0x5859F5B: ssh_connect (client.c:585)
==22928==    by 0x4CA6BA: SshMasterConnection::sshConnect()
(sshmasterconnection.cpp:847)
==22928==    by 0x4D8203: SshMasterConnection::run() (sshmasterconnection.cpp:637)
==22928==    by 0x71B511E: QThreadPrivate::start(void*) (qthread_unix.cpp:338)
==22928==    by 0x762BE24: start_thread (pthread_create.c:308)
==22928==    by 0x815834C: clone (clone.S:113)
==22928==
[2018/01/25 10:05:34.209205, 3] packet_send2:  packet: wrote
[len=7820,padding=6,comp=7813,payload=7813]

So it looks like we're trying to access a poll_handle (poll_out) attached to a
socket that has been freed elsewhere.

ssh_socket_unbuffered_write()
  /* Reactive the POLLOUT detector in the poll multiplexer system */
  if(s->poll_out){
      SSH_LOG(SSH_LOG_PACKET, "Enabling POLLOUT for socket");
-->   ssh_poll_set_events(s->poll_out,ssh_poll_get_events(s->poll_out) | POLLOUT);
  }

But at near as I can tell this poll_out object connected to this session was
freed when freeing an event:

void ssh_poll_ctx_free(ssh_poll_ctx ctx) {
  if (ctx->polls_allocated > 0) {
    while (ctx->polls_used > 0){
      ssh_poll_handle p = ctx->pollptrs[0];
      /*
       * The free function calls ssh_poll_ctx_remove() and decrements
       * ctx->polls_used
       */
-->   ssh_poll_free(p);


But not being more familiar with libssh, I'm getting lost.  There seems to be
some strange circular linking between poll_handles and poll_contexts.


-- 
Orion Poplawski
Manager of NWRA Technical Systems          720-772-5637
NWRA, Boulder/CoRA Office             FAX: 303-415-9702
3380 Mitchell Lane                       orion@xxxxxxxx
Boulder, CO 80301                 https://www.nwra.com/