Retention has never run in a cluster drill

[HectorIFC]

Part of #185.
Blocked by #190, #197, #194, #198, #199, #200.

Context

Retention is the one subsystem that deletes acknowledged data on purpose, and it has never run against a real cluster under faults.

• No drill turns it on. A search for MALACHI_RETENTION in docker-compose.cluster.yml and scripts/*.sh matches nothing; the compose file passes MALACHI_LOG_ROLL_MAX_AGE_MS through (line 88) and none of the three retention variables, which exist only in config/runtime.exs:273-275.
• The tests are in-process: test/malachi/cluster/retention_test.exs (the pure rule), test/malachi/cluster/retention_coordinator_test.exs (which passes its OWN copy of the expire function at lines 104-108, so the production Application.expire_segment/1 runs in no test), and the policy cases in test/malachi/metadata_test.exs.
• The harness exists and has earned its keep: scripts/chaos_lib.sh, scripts/chaos_checker.exs (modes produce and verify, main/1 clauses at lines 539 and 549), and the node, storage, config-deploy and reshard-restart drills.

The defects of #190 were found by reading code. The dependency list is long because this drill certifies what those issues build; the point is to find the next such defect by running the system.

Plan

A. A dedicated drill and a checker mode. scripts/docker-retention-chaos.sh on top of scripts/chaos_lib.sh, with the retention variables and a small :segment_max_bytes passed through docker-compose.cluster.yml so segments seal and expire within seconds (MALACHI_RETENTION_INTERVAL_MS already exists, so the sweep can run every second with no code change). A new verify-retention mode in scripts/chaos_checker.exs that knows the policy and each record's acknowledgment time. Invariants:

1. Nothing younger than the policy (plus a stated grace window) is unreadable.
2. A leader killed between the control-plane delete and the replica delete converges with zero orphan directories.
3. A slow group under Retention deletes data a consumer group has not read, and there is no backlog quota #200 is never skipped below its commit while under the pin caps.
4. A node with a skewed clock does not cause expiry with holes.
5. Expiry racing a vnode split, a purge (There is no way to purge a topic up to a point or a time #198) and a time roll (A low-traffic topic never expires by age, because segments seal only at 64MB #197) ends with every listed segment readable and no wedged consumer.

It runs as its own CI job on the cadence of the other drills. Cost: 1.5 to 2.5 weeks.

B. Multinode ExUnit only. Peer BEAM nodes under the :multinode tag. Cost: about 1 week. Faster and less flaky, but no real disks, no kill -9, no containers: it cannot express invariants 2 and 4 faithfully.

C. Turn retention on inside the storage drill. No new script. That drill's core invariant is "every acknowledged record is readable", which retention falsifies by design; bending its checker would weaken a certification that currently means something.

Do nothing. Retention stays certified by unit tests and code review.

Recommendation: A. Invariants 2 and 4 are the ones most likely to hide bugs, and only a real cluster can state them.

Risks and open questions

• CI duration and flakiness. A flaky drill is a P0 bug in this project. Every invariant needs an explicit grace window (policy, plus sweep interval, plus seal time, plus injected skew) derived from the configuration and printed in the report, never a bare sleep.
• The kill window of invariant 2 is microseconds wide. Between BrokerServer.delete_segment/2 and the first replica delete there is no natural pause. Options: an env-gated delay seam in the expire function (production code that exists only for the drill), or a statistical approach of many kills under a one-second sweep. Recommendation: the seam, documented, off by default, and rejected at boot outside a drill profile. Needs a decision.
• Clock skew inside Docker. Containers share the host kernel clock. RetentionCoordinator has a :clock option (lib/malachi/cluster/retention_coordinator.ex:46), but sealed_at is stamped by whichever node seals. Open question: a test-only clock offset setting honored by both, or libfaketime in the image?
• Whatever it finds. Findings become their own P0 bug issues with the evidence attached; they are not fixed inside this PR, so the drill may land with a known-failing invariant marked as such.
• Measurements count on Linux only. Local macOS runs are smoke runs (Docker Desktop shares only /Users, Colima needs explicit cpusets); the certification result is the CI run.
• Checked, not a risk: the existing checker already separates producing from verifying and keeps an acknowledgment file, so the new mode extends its input (add the acknowledgment time) and does not need a second checker.

Verification

• Each invariant is demonstrated to FAIL first: invariants 1 and 2 against a build with the fix of Retention deletes data the control plane refused to delete, and leaks replica directories forever #190 reverted, invariant 3 with the pin disabled, invariant 5 with the zero-length guard of A low-traffic topic never expires by age, because segments seal only at 64MB #197 removed. A drill that cannot fail certifies nothing.
• Ten consecutive green runs on Linux CI before the job becomes required, with the duration recorded.
• The checker's new mode has unit coverage for the grace-window arithmetic and for the orphan-directory scan.
• The report artifact lists, per run: segments expired, orphans found and removed, the largest observed gap between policy age and actual deletion, and consumer skips.
• Full suite including mix test --only multinode, mix format --check-formatted, mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, coverage on touched files.
• The four existing chaos drills still pass with the compose changes, retention off by default.

PR

Branch

test/retention-chaos-drill

Description

Adds a retention chaos drill and a checker mode that certify retention in a real cluster: nothing young is lost, a leader kill mid-expiry leaves no orphan directories, slow groups are not skipped below their commit, and clock skew or a vnode split does not create holes.