db-synthesizer never exits when the KES agent socket is unreachable

[jasagredo]

Reproduction

Credentials, taken from the first entry of the tools-test bulk file and split
into envelopes:

python3 - <<'EOF'
import json
d = json.load(open('ouroboros-consensus-cardano/test/tools-test/disk/config/bulk-creds-k2.json'))
oc, vrf, kes = d[0]
json.dump(oc,  open('opcert.json', 'w'))
json.dump(vrf, open('vrf.skey',   'w'))
json.dump(kes, open('kes.skey',   'w'))
EOF
cp -r ouroboros-consensus-cardano/test/tools-test/disk/config .

Run from that directory. A unix socket address holds 108 characters, so the
path to the socket must stay short.

One note on reading the output. When stdout is not a terminal, a GHC program
buffers it in blocks. GHC does not use the stdio of libc, so stdbuf changes
nothing. If timeout kills such a run, the buffer is lost and the log is
empty. Use a pseudo terminal instead, as below.

What happens

script -qefc "timeout 120 db-synthesizer --config config/config.json --db db \
  --shelley-operational-certificate opcert.json --shelley-vrf-key vrf.skey \
  --shelley-kes-agent-socket dead.sock -s 500 -f" out.log

with no agent listening on dead.sock:

--> forger count: 1
--> KES agent: KESAgentClientTrace (ServiceClientAttemptReconnect 10 100000 "Network.Socket.connect: <socket: 137>: does not exist (No such file or directory)" "dead.sock")
--> forged and adopted 0 blocks; reached SlotNo 500
--> KES agent: KESAgentClientTrace ServiceClientSocketClosed

The process then stays alive. It was killed at 120 seconds. The same run with
--shelley-kes-key kes.skey forges 12 blocks and exits 0.

The last line printed is ServiceClientSocketClosed, so the slot loop already
ended and the process sits in its shutdown path. synthesize releases the
forgers, finalize on the hot key runs cancel keyThreadAsync, and that call
never returns. While the process is stuck, no thread is on the CPU and no
syscall is issued. It is blocked, not spinning.

Where it comes from

runKESAgentClient
(ouroboros-consensus-protocol/src/ouroboros-consensus-protocol/Ouroboros/Consensus/Protocol/Praos/AgentClient.hs:162)
swallows AsyncCancelled:

      Agent.runServiceClient ...
        `catch` (\(_e :: AsyncCancelled) -> return ())
        `catch` (\(e :: SomeException) -> traceWith tracer (KESAgentClientException e))
      threadDelay 10000000

The whole body sits inside forever. A cancellation that arrives while
runServiceClient runs is caught and discarded, and the loop goes round again.
cancel then waits for a thread that never dies.

Neither handler covers threadDelay. A cancellation that arrives during the
delay does kill the thread. That is why the outcome depends on the timing: a
socket path longer than 108 characters fails so early that the retry loop sits
in its delay, and that run exits 0.

Whichever handler is meant to stay, an async exception must be rethrown.

This is not specific to db-synthesizer. Any consumer of
PraosCredentialsAgent that cancels the key-producer thread can block on it.