socket_dist_controller: sends can block the controller and hide a dead peer #2424

socket_dist_controller (libs/estdlib/src/socket_dist_controller.erl) sends with socket:send/2 from the tick handler and from send_data_loop.

Since #2345, socket:send/2 waits for the socket to be writable again when the send buffer is full. If the peer stops reading (a hung node, a dead link not yet detected by TCP), the controller gen_server blocks in socket:send/2. It also stops handling tick, which is what should detect a dead peer.

OTP's distribution uses nowait sends for this reason: the controller keeps running, and the sender is notified when the socket can be written.

Possible direction: send with a nowait variant (or socket:nif_send/2 and nif_select_write/2 directly) from the controller, keep the unsent rest in the state, resume from the write notification, and treat a blocked send that lasts longer than the tick timeout as a failed connection.

Raised by @bettio in the review of #2345, where a follow-up issue was agreed.