[oeway]

The #63 race explains zero confirmed production incidents

kth-k8s:fast-gilt pulled the ReplicaSet history. Every recent prod restart carries a kubectl.kubernetes.io/restartedAt annotation — all kubectl rollout restart, no bare pod delete, no OOMKill; helm history shows no upgrade since 2026-08-20.

rev169  2026-09-23 11:54:39   restartedAt
rev170  2026-09-23 12:17:42   restartedAt
rev171  2026-09-24 14:44:07   restartedAt
rev172  2026-09-24 14:54:27   restartedAt
rev173  2026-09-29 15:04:13   restartedAt   <- the restart that left login 404 for 21h
rev174  2026-09-30 12:45:57   restartedAt   <- manual restore

Under maxSurge=0 every one of those was no-overlap, so the handoff race cannot have caused any of them — including 09-23, which the previous revision treated as the one confirmed instance.

The guard is still right. It is justified as a cause-agnostic invariant keeper, not as the fix for a diagnosed outage. Claiming otherwise sends the next investigator down a path the deployment strategy already rules out. Corrected in CLAUDE.md, the store.py docstrings, and the test module docstring.

The remaining puzzle, resolved: the skip line does not prove a peer

That left something genuinely odd — the 09-23 pod logged already registered and reachable (owner=...) with no live peer available, which looked like a liveness-check false positive in #1049.

It isn't. _ensure_login_service_registered runs after startup functions (deliberately, so a startup-registered custom login is not clobbered). So anything that registers hypha-login earlier in this same pod's boot makes the check resolve it and ping its owner — itself, trivially alive.

The skip line means "a login resolved and its owner answered", not "a previous generation was still live".

Reproduced and pinned in test_skip_line_can_be_a_self_match: the line appears with owner equal to the pod's own public client id, with no second server in existence.

Field discriminator (needs only the boot log)

Compare owner=public/<id> against the pod's own server_id from Server info: {...}:

	meaning
same	self-match — the skip was legitimate, and the outage came from a later runtime deregistration
different	a genuine foreign owner existed, which under maxSurge=0 means something else shares the Redis and is worth hunting

The self-match case is also exactly where overwrite=True is load-bearing: the same pod's RPC peer already holds hypha-login in RPC._services.

The other candidate did not reproduce

fast-gilt also proposed: a stale login key from a SIGKILLed pod (so _clear_all_server_services never ran), with the liveness ping reporting it reachable anyway.

I tested it — with the owning store torn down and its service keys restored into Redis, built-in resolution failed and all three fallback resolutions failed, so ping_client correctly returned Failed to ping client .... Not the cause.

The fallback does contain a real latent hazard, though: elif hasattr(svc, "_config"): return "pong" reports a client alive without any round-trip, making pong a proof of service-entry resolvability rather than liveness for any service lacking ping/echo. That underpins both #1049's login idempotency and the #15 orphan reaper. Filed separately rather than fixed here, since I could not demonstrate it firing.

Tests

9 passed (test_login_handoff_race + test_login_registration_stale_marker). Docs and one new test only — no behavior change.

🤖 Generated with Claude Code