docs(login): #63 race explains zero confirmed prod incidents; pin the self-match reading

CLAUDE.md

@@ -692,10 +692,20 @@ invalidated moments later and **nothing re-evaluates it**.
   start while the outgoing pod is still in graceful shutdown. **Do not claim a
   rolling update as the trigger** — an earlier revision of this section did, and it
   was wrong.
-- **Incident (#63, prod kth-k8s 09-23, pod wheat-accordion-70572137; cleared only
-  by manual rollout restart):** a **recurring total login outage**. Two generations
-  overlapped: the NEW pod booted while the
-  login-owning OLD pod was still **LIVE**; the new pod's boot check resolves the old
+- **This race explains ZERO confirmed production incidents — do not cite one.**
+  Cluster history (2026-09-30) shows **every** recent kth-k8s restart carried a
+  `kubectl.kubernetes.io/restartedAt` annotation — all `kubectl rollout restart`,
+  no bare pod delete or OOMKill, and no helm upgrade since 2026-08-20. Under
+  `maxSurge=0` every one of those was no-overlap, so the race cannot have caused
+  them, including 09-23. The guard is justified as a **cause-agnostic invariant
+  keeper**, not as the fix for a diagnosed outage; claiming otherwise sends the
+  next investigator down a path the deployment strategy already rules out.
+  **Still unexplained:** the 09-23 pod logged the skip line with *no* live peer
+  available. The mechanism that fits is a **self-match** — see the sub-bullet
+  under the mechanism below.
+- **Mechanism (as it would occur):** two generations overlap; the NEW pod boots
+  while the
+  login-owning OLD pod is still **LIVE**; the new pod's boot check resolves the old
   login, pings it, gets `pong`, logs `Login service already registered and
   reachable (owner=<oldserver>)`, and correctly **defers** (skips registering). The
   old pod then terminates and its graceful shutdown `_clear_all_server_services`
@@ -745,6 +755,24 @@ invalidated moments later and **nothing re-evaluates it**.
   deregistration test (`test_single_server_runtime_deregistration_self_heals`) deletes
   the key straight out of Redis, deliberately **mechanism-blind**, which is the whole
   point of a reconcile loop: it must heal without knowing what broke it.
+- **The skip line does NOT prove a peer existed — `owner=` can be THIS pod (SELF-MATCH).**
+  `_ensure_login_service_registered` runs *after* startup functions, deliberately, so a
+  custom login registered by a startup function is not clobbered. But that means anything
+  which registers `hypha-login` earlier in **this same pod's** boot makes the check resolve
+  it, ping its owner — **itself, obviously alive** — and log
+  `login is registered and reachable (owner=public/<id>)`. Reproduced: the line appears with
+  `owner` equal to the pod's *own* public client id, with no second server anywhere. So the
+  skip line is evidence that *a* login resolved and its owner answered, **not** that a
+  previous generation was still live.
+  **Diagnostic:** compare the `owner=public/<id>` in the line against that pod's own
+  server/client id (`Server info: {'server_id': ...}` at boot). **Same → self-match**, the
+  skip was legitimate and the outage must come from a *later* runtime deregistration.
+  **Different → a genuine foreign owner existed**, which under `maxSurge=0` means something
+  else shares the Redis and is worth hunting. This is the cheapest available discriminator
+  and it needs only the boot log.
+  Note the self-match case is exactly the one where **`overwrite=True` is load-bearing**:
+  the same pod's RPC peer already holds `hypha-login` in `RPC._services`, so a guard
+  re-registration without it raises `Service already exists` forever.
 - **Key Lesson:** a boot-time idempotency/skip decision about a **shared singleton**
   (login, queue, any register-if-absent public service) is a TOCTOU — it can be
   invalidated by a peer's later shutdown and there is no retry. Pair the

hypha/core/store.py

@@ -1018,9 +1018,11 @@ async def _ensure_login_service_registered(self, source="boot"):
         generations overlap, a NEW pod can boot while the OLD login owner is
         still LIVE, correctly defer to it — and then the old pod's graceful
         shutdown clears the login key, leaving login permanently orphaned because
-        nothing re-evaluates the boot decision (#63, prod 09-23). Re-running this
-        check on an interval converts that permanent outage into a bounded
-        self-heal.
+        nothing re-evaluates the boot decision (#63). Re-running this check on an
+        interval converts that permanent outage into a bounded self-heal — and,
+        because it is cause-agnostic, covers every other way the key can go
+        missing too (which matters, since no confirmed prod incident has been
+        traced to the overlap race itself).
 
         ``source`` labels the caller ("boot" or "guard") in every decision log
         line. This is deliberately logged on EVERY pass, not just on a state
@@ -1118,15 +1120,21 @@ async def _login_guard_loop(self):
         overlap, a new pod boots while the old login owner is still LIVE, defers
         to it, and then the old pod's graceful shutdown clears the login key —
         leaving login permanently orphaned (``/public/services/hypha-login`` 404
-        while the server is fully healthy). This was a recurring prod outage
-        (09-23) that only a manual restart cleared.
-
-        Note the overlap does NOT come from helm rollouts: prod runs
-        ``replicas=1`` with ``RollingUpdate maxSurge=0, maxUnavailable=1``
-        precisely so the old pod is fully gone before the new one starts. It
-        comes from restarts ``maxSurge`` does not govern — a bare pod delete, a
-        node eviction, an OOMKill, or an auto-heal restart — where the
-        replacement can start while the outgoing pod is still shutting down.
+        while the server is fully healthy).
+
+        This race explains **zero confirmed production incidents** — do not cite
+        one. kth-k8s runs ``replicas=1`` with ``RollingUpdate maxSurge=0,
+        maxUnavailable=1``, so a rollout cannot create two live generations, and
+        cluster history shows **every** recent prod restart carried a
+        ``kubectl.kubernetes.io/restartedAt`` annotation, i.e. all were rollout
+        restarts (no bare pod delete / OOMKill, and no helm upgrade since
+        2026-08-20). The guard is justified as a **cause-agnostic** invariant
+        keeper, not as the fix for a diagnosed outage.
+
+        The race is still reachable in principle — through restarts ``maxSurge``
+        does not govern (bare pod delete, node eviction, OOMKill, auto-heal),
+        where a replacement can start while the outgoing pod is still shutting
+        down — it just has not been the observed trigger anywhere yet.
 
         Re-running the boot check on an interval heals it: after any orphaning
         event the leader re-registers the default login within one interval

tests/test_login_handoff_race.py

@@ -18,8 +18,17 @@
 outgoing pod is still in graceful shutdown. The race below is reachable in prod,
 just NOT via helm upgrades.
 
-Production symptom (kth-k8s, 09-23, pod wheat-accordion-70572137; cured only by a
-manual rollout restart):
+THIS RACE EXPLAINS ZERO CONFIRMED PRODUCTION INCIDENTS — do not cite one. Cluster
+history (2026-09-30) shows every recent kth-k8s restart carried a
+``kubectl.kubernetes.io/restartedAt`` annotation (all ``kubectl rollout restart``;
+no bare pod delete, no OOMKill, no helm upgrade since 2026-08-20), and under
+``maxSurge=0`` every one of those was no-overlap. The guard is justified as a
+cause-agnostic invariant keeper, not as the fix for a diagnosed outage.
+
+Note in particular that the 09-23 skip line does NOT prove a peer existed — see
+``test_skip_line_can_be_a_self_match`` below.
+
+Mechanism, as it would occur:
 
 1. Two generations overlap: the NEW pod boots WHILE the login-owning OLD pod is
    still reachable.
@@ -56,6 +65,7 @@
 """
 
 import asyncio
+import logging
 
 import pytest
 
@@ -355,3 +365,63 @@ async def test_clean_boot_still_registers_login(monkeypatch):
         )
     finally:
         await store.teardown()
+
+
+async def test_skip_line_can_be_a_self_match(monkeypatch, caplog):
+    """The 'already registered and reachable' skip line does NOT prove a peer.
+
+    Forensic pin, added after kth-k8s established (2026-09-30, from ReplicaSet
+    history) that EVERY recent prod restart was a ``kubectl rollout restart``.
+    Under ``maxSurge=0`` those are all no-overlap, so on 09-23 there was no live
+    peer to defer to — yet that pod logged the skip line. That looked like a
+    liveness-check false positive.
+
+    It is not. ``_ensure_login_service_registered`` runs AFTER startup functions
+    (deliberately, so a custom login from a startup function is not clobbered),
+    so anything that registers ``hypha-login`` earlier in THIS pod's own boot
+    makes the check resolve it and ping its owner — itself, trivially alive.
+
+    The line therefore means "a login resolved and its owner answered", not "a
+    previous generation was still live". The discriminator is whether ``owner``
+    equals this pod's own client id, which is what this test pins.
+
+    This is also precisely the case where ``overwrite=True`` is load-bearing: the
+    same pod's RPC peer already holds ``hypha-login`` in ``RPC._services``.
+    """
+    monkeypatch.setenv("HYPHA_ORPHAN_REAP_INITIAL_DELAY", "60")
+    monkeypatch.setenv("HYPHA_LOGIN_GUARD_INTERVAL", "0")
+    monkeypatch.setenv("HYPHA_LOGIN_PING_TIMEOUT", "2")
+
+    store = RedisStore(None, redis_uri=None)
+    await store.init(reset_redis=True)
+    try:
+        api = await store.get_public_api()
+        own_client = api.rpc.get_client_info()["id"]
+
+        # Exactly one server exists; init() already registered login on it.
+        assert await _login_owners(store) == {store._server_id}
+
+        # Re-run the boot check, as a fresh pod would after its startup
+        # functions had already registered a login.
+        with caplog.at_level(logging.INFO, logger="redis-store"):
+            live = await store._ensure_login_service_registered(source="boot")
+
+        assert live is True, "the check should report a live login"
+
+        skip_lines = [
+            r.getMessage()
+            for r in caplog.records
+            if "is registered and reachable" in r.getMessage()
+        ]
+        assert skip_lines, f"expected the skip line, got: {caplog.messages}"
+
+        # THE POINT: the owner is THIS pod, with no peer in existence.
+        assert f"owner=public/{own_client}" in skip_lines[0], (
+            "the skip line should name this pod's own client as the owner — "
+            f"got {skip_lines[0]!r}"
+        )
+
+        # And the login is still registered exactly once (no duplicate).
+        assert await _login_owners(store) == {store._server_id}
+    finally:
+        await store.teardown()