A single-copy segment whose active tail rotted blocks its range instead of rolling

[HectorIFC]

Part of #184. Follow-up of #189.

Context

Since #189, a copy whose ACTIVE segment recovers with a preserved tail (rot, or a frame in a format the release does not know) refuses appends with {:error, :damaged_tail} instead of writing over the damage. The replication server takes that as a storage failure, and with a majority of intact copies (RF 3 or more) the heal pass seals the segment on them and writing rolls to a new segment, which is NorthGuard's rule: "If you're producing to a segment and there's a broker that fails on that segment, we just seal it, make a new one, move the producers over to that new segment. And yeah, so produce availability just stays good." (meetup transcript, line 636).

At RF 1 the damaged copy is the only one, so no majority of intact copies can answer and the range stops taking writes until an operator acts (procedure in docs/guides/operations.md, section Integrity scrub). The transcript does not cover this case: every example uses three replicas (lines 456 and 479). Its two principles still apply: produce availability (line 636) and a safe seal point chosen by the coordinators (lines 630 and 631).

The safe seal point is the hard part. Sealing at the valid prefix reissues offsets that may already have been acknowledged, which is exactly what Malachi.Cluster.Failover exists to prevent (its moduledoc, lines 6 to 14). A seal point ABOVE every assigned offset is safe for offsets, but leaves a hole the read path does not handle: the consume cursor stops on :eof rather than advancing (lib/malachi/broker.ex around line 1270).

Plan

A. Seal above every assigned offset and move producers:

• Resync past the damage: every frame carries its own offset, so scanning forward for the next frame whose magic and CRC verify gives the highest offset still on disk.
• Bound what cannot be decoded: no frame is shorter than the header plus the fixed payload fields (35 bytes), so the unreadable region holds at most ceil(unreadable_bytes / 35) records.
• Seal at the larger of the two, run the heal pass in single-node mode too (Malachi.Application.coordinator_children/2 only starts it when clustered), and teach the read path to step over a hole in a sealed segment.

B. Keep RF 1 blocked and documented, as today. Cost: zero. It contradicts produce availability.

C. Seal at the valid prefix. Violates the never-reissue-an-offset invariant. Not viable.

Recommendation: A, as the option closest to NorthGuard's principles.

Risks and open questions

• The read-path change is in lib/malachi/broker.ex; coordinate with whatever touches it at the time.
• RF 2 stays blocked by the failover majority rule (decision recorded in An older binary starts on a data directory written by a newer one #189). NorthGuard fsyncs on all replicas before the ack (transcript line 646), which would make one intact copy enough; Malachi acks on a majority. That is a separate question.

Verification

• Unit tests for the resync scan and the bound, including rot in the very last frame.
• A single-node test: rot in the active segment, produce continues on a new segment, a consumer reads past the hole.
• The storage drill, on Linux.

PR

Branch

feat/rf1-rotted-tail-seal-and-roll

Description

Seal a single-copy segment whose active tail rotted at a point above every assigned offset and move producers to a new segment, with the read path stepping over the hole, so RF 1 keeps taking writes the way NorthGuard does when a replica fails.