Skip to content

feat(remote-hosts): keep retrying remote hosts that were unreachable at startup - #5628

Merged
amir20 merged 3 commits into
fix/retry-failed-agentsfrom
feat/retry-remote-hosts
Nov 11, 2026
Merged

amir20 merged 3 commits into
fix/retry-failed-agentsfrom
feat/retry-remote-hosts

Conversation

@amir20

@amir20 amir20 commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #5626. Asked for in a comment on #5624.

A remote host (DOZZLE_REMOTE_HOST) that didn't answer when the hub started was dropped for good. One that went away later already came back, because its event stream reconnects, so startup was the only case that needed a hub restart.

Changes

  • A remote host that fails at startup is kept and listed as unavailable, under the id its URL gives it.
  • The 30s loop from fix(agent): keep retrying agents that were unreachable at startup #5626 builds its client again on every tick. The client is rebuilt from scratch each time because it reads the engine's id, CPU count and memory once, when it is made.
  • When it connects it is served under the engine's real id, and open tabs get an update-host with replacesId so the placeholder is dropped. No frontend change.
  • Only the background loop retries remote hosts. A container listing (every page load) never does, so a host that drops packets can't hold up the UI.
  • The first ping to a remote host is bounded by --timeout. It had no deadline before.
  • A remote host whose scheme isn't tcp is skipped with a warning and never retried, since retrying can't fix a config mistake. Checked by starting a hub with --remote-host ssh://nas: it logged the skip and the host was not listed.
  • RetryFailedAgents is renamed RetryFailedHosts.
  • Docs: removed the "Reconnect" row from the agent vs remote comparison in guide/agent.md and its four translations, since it said remote hosts need a restart.

Known limitation

Alerts don't start for a remote host that joins late. The notification listeners take their list of Docker services once at boot (StartNotificationManager in internal/hostservice/multi_host.go), so a late remote host shows up with logs and stats but no log, metric or event rule looks at its containers until the hub restarts. Fixing it means letting the three listeners accept a client after start, which is a new mechanism, so it's left out here. Accepted as is for this PR.

Not in this PR

  • Alerts on a late remote host (above).
  • A hub whose only hosts are all down at startup still exits with "Could not connect to any Docker Engine".
  • Swarm mode.
  • A late remote host that turns out to be a duplicate of a served engine is dropped, but the store started for it keeps running. The startup path already behaves the same way.

Upgrade note

A DOZZLE_REMOTE_HOST that is down now appears in the UI as unavailable. It used to be absent.

Tests

  • Go: make test (-race), staticcheck, tygo diff: pass
  • Docs: bun docs/scripts/check-translations.mjs: in sync
  • Behaviour, new tests in internal/hostservice/agents_test.go, all pass under -race:
    • RetryLaterConnectsPendingHost: listed as down, connects on a later tick, published with ReplacesID, not built again afterwards
    • RetryLaterDropsDuplicate
    • RetryAndListLeavesPendingAlone: a container listing makes 0 connect calls
  • Real run, server mode, hub started with --remote-host tcp://127.0.0.1:32375|dind and nothing listening, then a docker:dind engine started on that port:
    • this branch: hub logged connected to host that was unreachable about 30s later, an events stream that was already open received update-host with available: true and replacesId: tcp:127.0.0.1:32375, and a container started on the second engine showed up in containers-changed
    • base branch (fix(agent): keep retrying agents that were unreachable at startup #5626): the second engine was up for 75s and the hub never connected
  • Fresh-context review, run twice: first pass found page loads waiting on a pending host, fixed by keeping remote retries off the listing path. Second pass: 1 Important (the alerts gap above, accepted as a known limitation), 2 Nit (duplicate store, and Info/ServerVersion after the ping still have no deadline), 1 Pre-existing (exit when every host is down).
  • Not run: frontend suite (no frontend changes), a browser showing the unavailable remote host card, TLS remote hosts, a socket proxy, swarm and k8s modes, Playwright e2e

🤖 Generated with Claude Code

…at startup

A remote host that did not answer when the hub started was dropped for
good. One that went away later already came back, because its event
stream reconnects, so startup was the only time a restart was needed.

Keep those hosts, list them as unavailable, and build their client again
in the same 30s retry the agents use. The client is built from scratch
each time because it reads the engine's id and specs once.

The first contact with a remote host is now bounded by the timeout.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@claude

claude Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Claude finished @amir20's task in 1m 11s —— View job


Review of PR #5628

0 Important, 1 Nit, 0 Pre-existing.

  • Nit (inline, internal/cli/clients.go:58): config errors that won't fix themselves are now retried forever. That covers invalid scheme from internal/container/docker/client.go:214 and a broken TLS path in client.New. Before this PR they were dropped with a warning.

I read the whole diff (Go plus all five docs files) and the code around it:

  • Placeholder to real id. ParseConnection gives a tcp:host:port id (internal/container/host.go:179). NewClient falls back to that id only when the engine gives none (internal/container/docker/client.go:131). retryPending sets ReplacesID only when the two ids differ (retriable_client_manager.go:408-412), and assets/stores/hosts.ts:65 deletes the placeholder. No frontend change is needed.
  • A late host gets its streams. multi_host.go:270-284 looks up the published host with Find. The client isn't in seen yet, so it subscribes with late=true, the same path a late agent takes.
  • Locking. Only the RetryFailedHosts goroutine removes entries from pending. retryPending clones the slice under RLock, connects without the lock, and removes by placeholder id under Lock. Hosts clones pending under the same RLock as failedAgents. Nothing returns a backing slice.
  • No retry on the listing path. RetryAndList doesn't touch pending, and TestRetriableClientManager_RetryAndListLeavesPendingAlone covers that.
  • Throwaway clients are closed. A failed ping closes cli (client.go:239). A failed listing closes the client (clients.go:40). NewService only runs after both succeed, so a host that stays down never starts a store.
  • Startup ping. It now runs on context.WithTimeout(ctx, args.Timeout) (clients.go:55) instead of context.Background().
  • Docs. The "Reconnect" row is removed from the English page and all four translations. sourceHash is re-stamped in each.

On top of the limitations the PR already lists: Info/ServerVersion inside NewClient (client.go:114, :258) still have no deadline, and retryPending runs in the same loop as the agent retries. An engine that answers the ping and then hangs on Info would therefore stall agent retries too, not just its own. It takes an odd engine to get there, so I'm not raising it as a finding.

I couldn't run go test -race here because the command needs approval in this environment. The PR body reports it passing.

Comment thread internal/cli/clients.go
ctx, cancel := context.WithTimeout(ctx, args.Timeout)
service, err := connectRemote(ctx, host)
cancel()
if err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: every error from connectRemote now lands here, including the ones that can never fix themselves. NewRemoteClient returns invalid scheme: %s for a non-tcp URL (internal/container/docker/client.go:214), and client.New fails the same way on every call for a broken TLS cert path. Before this PR those were logged as "Could not create remote client" and the host was dropped. Now a typo like ssh://nas shows as an unavailable host forever, and gets rebuilt every 30s with only a Debug log after the first warning. Checking host.URL.Scheme != "tcp" before the loop and treating it as a config error (or skipping RetryLater for it) would keep config errors out of the retry list.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed. a remote host whose scheme isn't tcp is now skipped at startup with a warning and never goes on the retry list, same as before this PR. left the TLS case alone, a cert path that gets fixed on disk does recover on a later retry.

…f retrying it

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@claude

claude Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Claude finished @amir20's task in 30s —— View job


Review of PR #5628 (after a531f60)

0 findings. Nothing to fix.

The new commit adds a scheme check before the connect loop (internal/cli/clients.go:54-58). A non-tcp remote host is now logged and skipped, so it no longer ends up in downRemotes. That restores the base branch's behaviour for that case: the host is dropped with a warning and never shown as unavailable. NewRemoteClient keeps its own scheme guard (internal/container/docker/client.go:214), so a non-tcp host fails in the same place whichever path reaches it.

One part of the earlier Nit is still open, and it doesn't need a new thread. A broken TLS cert path still gets retried. When ca.pem exists, ValidCerts is true (internal/container/host.go:172-176). If cert.pem or key.pem is then missing, client.New fails on every call (client.go:222-235), so that host shows as unavailable and is rebuilt every 30s. It's rarer than a scheme typo, and the first warning names the error, so it's fine to leave.

What I re-read on this push (the whole Go diff against origin/fix/retry-failed-agents):

  • Startup path. connectRemote only runs docker.NewService once both the ping and ListContainers succeed. A failed listing closes the client (clients.go:40). A failed ping closes cli (client.go:239).
  • Retry loop. retryPending clones pending under RLock and connects with no lock held. It removes entries by placeholder id under Lock. It sets ReplacesID only when the real id differs from the placeholder (retriable_client_manager.go:405-412). If the same URL is configured twice, both pending entries are removed on the first success, and the second result goes down the duplicate branch.
  • Listing path. RetryAndList doesn't touch pending. Hosts clones pending under the same RLock as failedAgents (retriable_client_manager.go:612), so no backing slice escapes the lock.
  • Docs. The "Reconnect" row is removed from the English page and all four translations.

I didn't run go test -race, because commands need approval in this environment.

@github-actions

Copy link
Copy Markdown
Contributor

Try this PR without waiting for a release:

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8080:8080 amir20/dozzle:pr-5628

Also on ghcr.io/amir20/dozzle:pr-5628 and as amir20/dozzle:pr-5628-alpine. Rebuilt on every push, currently a531f60.

@amir20
amir20 merged commit a43a54a into main Nov 11, 2026
19 checks passed
@amir20
amir20 deleted the feat/retry-remote-hosts branch November 11, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant