Every place the router replies to a client is a bare send on a bounded channel — no select, no default, no liveness check. If nothing is reading that channel and its 1000 slots are full, the sender blocks forever.
The channel is per-client. The senders are not: responses come from each stream's sendRequests/readResponses goroutines and from the single configSubmitter goroutine. So one client that stops draining retires a goroutine every other client depends on. There are 16 such sends, across stream.go, shard_router.go, config_submitter.go and router.go.
Three ways the reader goes away:
- Disconnect.
Broadcast returns and its feedback goroutine stops, but that client's already-submitted requests are queued in the shared stream buffers, and the shared sender reaches them later and tries to reply.
- Connected but not reading.
stream.Send blocks on HTTP/2 flow control until the client sends a WINDOW_UPDATE; a client that never reads never sends one. Flow control is per-direction, so it can keep submitting throughout. No disconnect needed, repeatable at will.
- Soft stop.
SoftStop closes drainChan and the feedback goroutines exit while Broadcast is still receiving. Both throttle sites already carry // TODO deal with drain signal leaving this channel with no reader.
Why it matters:
- Monotonic capacity loss. Verification runs one goroutine per stream (
conns × streams × shards), so each wedged stream permanently costs parallelism, and it accumulates over the process's lifetime.
- Silent. Nothing is logged, no metric moves, and
IsAllStreamsOKinSR reports healthy — the stream's context was never cancelled, so it is not faulty, it is wedged. A degraded router is indistinguishable from a fresh one from outside.
Two aggravating cases: sendResponseToAllClientsOnError sends while holding s.lock, so it also stops the registration map and the metrics reader. forwardResponseToClient blocks the readResponses goroutine, which is the only thing that marks a stream faulty — so that stream stays registered and nominally healthy while forwarding nothing.
Every place the router replies to a client is a bare send on a bounded channel — no
select, nodefault, no liveness check. If nothing is reading that channel and its 1000 slots are full, the sender blocks forever.The channel is per-client. The senders are not: responses come from each stream's
sendRequests/readResponsesgoroutines and from the singleconfigSubmittergoroutine. So one client that stops draining retires a goroutine every other client depends on. There are 16 such sends, acrossstream.go,shard_router.go,config_submitter.goandrouter.go.Three ways the reader goes away:
Broadcastreturns and its feedback goroutine stops, but that client's already-submitted requests are queued in the shared stream buffers, and the shared sender reaches them later and tries to reply.stream.Sendblocks on HTTP/2 flow control until the client sends aWINDOW_UPDATE; a client that never reads never sends one. Flow control is per-direction, so it can keep submitting throughout. No disconnect needed, repeatable at will.SoftStopclosesdrainChanand the feedback goroutines exit whileBroadcastis still receiving. Both throttle sites already carry// TODO deal with drain signal leaving this channel with no reader.Why it matters:
conns × streams × shards), so each wedged stream permanently costs parallelism, and it accumulates over the process's lifetime.IsAllStreamsOKinSRreports healthy — the stream's context was never cancelled, so it is not faulty, it is wedged. A degraded router is indistinguishable from a fresh one from outside.Two aggravating cases:
sendResponseToAllClientsOnErrorsends while holdings.lock, so it also stops the registration map and the metrics reader.forwardResponseToClientblocks thereadResponsesgoroutine, which is the only thing that marks a stream faulty — so that stream stays registered and nominally healthy while forwarding nothing.