There is no log compaction

[HectorIFC]

Part of #185.
Blocked by #188, #193, #194, #203, #202.

Context

Malachi has no log compaction. A keyed topic used as a changelog (latest state per key) either grows without bound or, under time and size retention, loses the latest value of any key that has not changed recently. Kafka and Redpanda offer cleanup.policy=compact, Pulsar has topic compaction, NATS has per-subject limits for its key-value store. It is the largest missing retention mode, and several assumptions in the storage layer stand in its way:

• Offsets are dense by construction. Segment.end_offset/1 is base_offset + record_count (lib/malachi/log/segment.ex:83-84); Log.append/2 (lib/malachi/log.ex:127-129) and ReplicationServer.follow/4 (lib/malachi/cluster/replication_server.ex:284, handler at 561) assign offsets themselves. No API writes a record at a given offset, so a compacted segment cannot even be copied to a new replica today.
• Placement is per SEGMENT. No replica holds a whole range, so "each replica compacts what it has" cannot decide which record is the latest for a key. A central compactor per range is forced.
• Sealed segments keep a cached open handle. state.logs (replication_server.ex:103 and 456) loses entries only on delete, on fail_segment/3 and at terminate. Swapping a rewritten file needs a short critical section inside that server.
• A tombstone cannot be expressed. Wire.encode_record/1 writes the value as a bare length plus bytes and the wire record has NO flags byte (lib/malachi/wire.ex:370-372). The STORAGE frame does: flags bit 0 is "key present" (@key_present 0x01, lib/malachi/log/record.ex:16-20 and 47), seven bits are free, and decode masks only that bit (line 240), so unknown bits are ignored.
• Split lineage. A child range reads its ancestors' records filtered to its key slice (filter_records/2 over history_sources/2, lib/malachi/broker.ex:1331-1334 and 1371-1381). A tombstone in the child may be dropped only once no ancestor holds a record for that key; otherwise a consumer reading from the start sees the old value come back.

Plan

A. Stable offsets with holes, a central compactor, deterministic per-replica rewrite.

1. Storage: append_at with explicit offsets, an explicit end_offset decoupled from the record count, and catch-up and follow that preserve offsets.
2. A compactor per range, off the hot loop, on the vnode leader's node, reads the range's dirty sealed segments, builds the latest-offset-per-key map and ships a survivor list per dirty segment.
3. Each replica rewrites ITS copy deterministically from its own bytes plus the list and swaps it atomically, reusing the manifest, generation and atomic swap of The disk stores one frame per record, with a per-record CRC and no compression #202, and targeting the final storage format so nothing is rewritten twice.
4. The control plane records {:segment_compacted, id, generation, records, bytes, hash}; the scrubber and self-healing compare copies against that generation, no longer against seal-time values.
5. Tombstones: a spare flags bit, set through the new produce key of The wire protocol cannot carry a compressed batch #195 behind the capability gate of Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193, with a bounded lifetime, and retained across split lineage until the ancestors are clean.
6. The policy of Per-topic retention exists in the control plane and no API reaches it #194 gains the cleanup mode. Any log line goes through I18n.

Cost: 7 to 11 weeks.

B. Copy compaction with re-assigned dense offsets. Rewrite survivors into new segments numbered densely and keep an offset translation map. Storage stays untouched. It breaks every committed cursor ({source_index, source_offset}), needs the map consulted on each fetch forever, and makes a group's position mean different things before and after a pass. Cost: similar, with worse failure modes. Rejected.

C. Read-time "latest per key" only. Keep an index of the latest offset per key and filter reads; reclaim disk only when a whole segment is dead. Cost: 3 to 4 weeks. No storage format change and no rewrite, but a mixed segment (one live key among thousands of dead ones) is never reclaimed, which is the normal shape of a changelog.

Do nothing. Users snapshot state externally and recreate topics.

Recommendation: A. Only stable offsets keep cursors, lineage and replication semantics intact, and doing it after #202 means the rewrite machinery is built once. C remains a fallback if the storage changes of step 1 prove too invasive.

Risks and open questions

• Tombstone resurrection across split lineage. Dropping a child's tombstone while an ancestor still holds the key brings the old value back for every new consumer. The rule "retain until the ancestors are clean" needs the compactor to reason over the whole lineage and not over one range; it is the first property to test.
• Generation divergence under failures. A replica that crashed mid-swap, or that never received the survivor list, holds an older generation. If the scrubber still validated against seal-time counts it would "repair" a compacted copy from an uncompacted peer and resurrect deleted keys. Healing must only ever move a copy FORWARD in generation.
• Dense-offset assumptions outside storage. sealed_end/1 is start + length (broker.ex:1298), and last_offset/2 falls back to counting from the requested offset (broker.ex:1311-1316). length must split into offset span and surviving record count (Retention cannot limit a topic by record count #199 reads the count), and the skip count of Retention has no metrics, and a consumer is moved past expired data without being told #191 must not read holes as expiry.
• Compactor memory and placement. A latest-offset-per-key map for a range is bounded by distinct keys, not by bytes, and it lives on the vnode leader. Open question: a fixed budget with several passes (as Kafka's dedupe buffer does), or a disk-backed map?
• An old node would serve a tombstone as an empty value. Decode ignoring unknown bits avoids a crash, not a wrong answer. No producer may set the bit until the gate of Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193 reports every node capable.
• Compact together with delete? Whether a topic may combine compaction with age or size retention, and what the active segment means for dirtiness once A low-traffic topic never expires by age, because segments seal only at 64MB #197 rolls it by time, needs a decision.
• Checked, not a risk: the {:delete, id} handler already closes the cached handle before removing files (replication_server.ex:615-619), so the critical section the swap needs has a precedent in the same server and is no new concurrency pattern.

Verification

• Pure compactor tests and properties (stream_data): the latest-per-key view of the compacted log equals that of the original; compaction is idempotent; the survivor list is a function of the input alone, so two replicas produce byte-identical files.
• Lineage property: for any tree of splits and merges and any interleaving of writes, tombstones and passes, a read from the start never yields a value for a key whose latest record is a tombstone.
• Storage tests: append_at refuses a non-monotonic offset; a segment with holes recovers after a crash to the same end_offset; catch-up of a compacted segment to a fresh replica preserves every offset.
• Failure tests: a crash before, during and after the swap leaves one valid generation; a replica one generation behind is healed forward and never backward; the scrubber accepts a compacted copy and rejects a copy that matches seal-time values only.
• Mixed-version test: with one node below the capability, producing a tombstone is refused.
• Full suite including mix test --only multinode, mix format --check-formatted, mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, coverage on touched files.
• All chaos drills, plus the retention drill of Retention has never run in a cluster drill #203 extended with compaction invariants (no resurrection after a leader kill mid-pass; generations converge), and paired produce and consume benchmarks on Linux to show that a running pass does not regress the hot path.

PR

Branch

feat/log-compaction

Description

Adds log compaction with stable offsets: a central per-range compactor ships survivor lists, each replica rewrites its own copy deterministically and swaps it atomically, and the control plane tracks a generation per segment. Tombstones use a spare storage flag bit and are retained across split lineage until ancestors are clean.