Re: Deadlock on key re-exchange handling when a forward tunnel is open

[Jakub Jelen <jjelen@xxxxxxxxxx>]

On 4/30/22 19:47, PFLANZER Moritz wrote:
> Hello,
>
> The attached example opens a forwarding tunnel, waits for a client to 
> connect to a local socket and joins the two together using the connector 
> API. This is working fine until the remote server requests a key 
> re-exchange after a gigabyte of data has been transferred via the 
> channel. Then the poll loop deadlocks.
>
> The problem seems to be that the handling of the key re-exchange is 
> initiated from the poll callback. Before the callback the events of the 
> file descriptor are cleared to prevent race conditions on the poll. Then 
> with the events still cleared the key re-exchange handler waits for the 
> remote server to transmit its new keys. But since the events for the 
> file descriptor are still empty poll will never signal that a new 
> message can be read from the file descriptor.
>
> Please find a small example program attached that reproduces the 
> problem. A test setup could look like the following:
>
> # Listen on port 10000
> $ nc -kl 10000 | pv -b > /dev/null
>
> # Run the example program
> $ ./examples/tunnel
>
> # Send data to port 55555
> $ dd if=/dev/zero bs=10M | pv -b | nc 127.0.0.1 55555
>
> I also attached (the tail of) an strace capture of what is happening and 
> the patch I applied to libssh to get a few more prints along the way.
>
> I started looking into how to potentially fix the problem but so far 
> it's not clear to me how the interaction between the key re-exchange 
> handlers and the poll callbacks was designed to work. Would be great to 
> get some pointers how a fix should look like.

Hello,
thank you for the report. Can you open a new issue on gitlab to keep 
this tracked?

https://gitlab.com/libssh/libssh-mirror

This sounds like something more people can hit.

I wrote the key re-exchange support in libssh couple of years back so I 
hope I would be able to clarify if you have some particular questions. 
It was highly inspired by the code used in openssh, but I might have 
missed some corner cases.

Thanks,
--
Jakub Jelen
Crypto Team, Security Engineering
Red Hat, Inc.