Repository navigation
feat(remote-hosts): keep retrying remote hosts that were unreachable at startup - #5628
Conversation
…at startup A remote host that did not answer when the hub started was dropped for good. One that went away later already came back, because its event stream reconnects, so startup was the only time a restart was needed. Keep those hosts, list them as unavailable, and build their client again in the same 30s retry the agents use. The client is built from scratch each time because it reads the engine's id and specs once. The first contact with a remote host is now bounded by the timeout. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Claude finished @amir20's task in 1m 11s —— View job Review of PR #56280 Important, 1 Nit, 0 Pre-existing.
I read the whole diff (Go plus all five docs files) and the code around it:
On top of the limitations the PR already lists: I couldn't run |
| ctx, cancel := context.WithTimeout(ctx, args.Timeout) | ||
| service, err := connectRemote(ctx, host) | ||
| cancel() | ||
| if err != nil { |
There was a problem hiding this comment.
Nit: every error from connectRemote now lands here, including the ones that can never fix themselves. NewRemoteClient returns invalid scheme: %s for a non-tcp URL (internal/container/docker/client.go:214), and client.New fails the same way on every call for a broken TLS cert path. Before this PR those were logged as "Could not create remote client" and the host was dropped. Now a typo like ssh://nas shows as an unavailable host forever, and gets rebuilt every 30s with only a Debug log after the first warning. Checking host.URL.Scheme != "tcp" before the loop and treating it as a config error (or skipping RetryLater for it) would keep config errors out of the retry list.
There was a problem hiding this comment.
fixed. a remote host whose scheme isn't tcp is now skipped at startup with a warning and never goes on the retry list, same as before this PR. left the TLS case alone, a cert path that gets fixed on disk does recover on a later retry.
…f retrying it Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Claude finished @amir20's task in 30s —— View job Review of PR #5628 (after a531f60)0 findings. Nothing to fix. The new commit adds a scheme check before the connect loop ( One part of the earlier Nit is still open, and it doesn't need a new thread. A broken TLS cert path still gets retried. When What I re-read on this push (the whole Go diff against
I didn't run |
|
Try this PR without waiting for a release: Also on |
Stacked on #5626. Asked for in a comment on #5624.
A remote host (
DOZZLE_REMOTE_HOST) that didn't answer when the hub started was dropped for good. One that went away later already came back, because its event stream reconnects, so startup was the only case that needed a hub restart.Changes
update-hostwithreplacesIdso the placeholder is dropped. No frontend change.--timeout. It had no deadline before.tcpis skipped with a warning and never retried, since retrying can't fix a config mistake. Checked by starting a hub with--remote-host ssh://nas: it logged the skip and the host was not listed.RetryFailedAgentsis renamedRetryFailedHosts.guide/agent.mdand its four translations, since it said remote hosts need a restart.Known limitation
Alerts don't start for a remote host that joins late. The notification listeners take their list of Docker services once at boot (
StartNotificationManagerininternal/hostservice/multi_host.go), so a late remote host shows up with logs and stats but no log, metric or event rule looks at its containers until the hub restarts. Fixing it means letting the three listeners accept a client after start, which is a new mechanism, so it's left out here. Accepted as is for this PR.Not in this PR
Upgrade note
A
DOZZLE_REMOTE_HOSTthat is down now appears in the UI as unavailable. It used to be absent.Tests
make test(-race),staticcheck, tygo diff: passbun docs/scripts/check-translations.mjs: in syncinternal/hostservice/agents_test.go, all pass under-race:RetryLaterConnectsPendingHost: listed as down, connects on a later tick, published withReplacesID, not built again afterwardsRetryLaterDropsDuplicateRetryAndListLeavesPendingAlone: a container listing makes 0 connect calls--remote-host tcp://127.0.0.1:32375|dindand nothing listening, then adocker:dindengine started on that port:connected to host that was unreachableabout 30s later, an events stream that was already open receivedupdate-hostwithavailable: trueandreplacesId: tcp:127.0.0.1:32375, and a container started on the second engine showed up incontainers-changedInfo/ServerVersionafter the ping still have no deadline), 1 Pre-existing (exit when every host is down).🤖 Generated with Claude Code