Replication ships record structs uncompressed, and every replica re-encodes them

[HectorIFC]

Part of #186.
Blocked by #187, #193, #202.

Context

Replication ships record structs, uncompressed, and every replica encodes them again. On the primary, park_and_push/8 (lib/malachi/cluster/replication_server.ex:989) builds {:replica_append, segment_id, base_offset, first, records, committed, ref} (:1000) with the ORIGINAL record struct list. On the follower, the cast handler (:762) opens the segment and follow_push/3 (:794 to :845) checks the fence (Log.sealed?/1), then log.next_offset == expected_first, and calls write_records/5 (:1336), which encodes every frame and computes every CRC a second and a third time. The failure latch is checked on the way in through open_segment/3 (:1319).

The same record lists travel on the two repair paths: follow/4 (:284, handler at :561, which requires next_offset: ^expected_first) and the background catch-up started by trigger_catchup/4 (:842, :1116) when a follower answers :out_of_sync.

All of this happens in one process per node, and crosses distributed Erlang, which offers no compression and gives each node pair ONE TCP connection shared with ra and gossip. A list of structs is term-encoded field by field for each follower; a single large binary goes into the port by reference. A full distribution buffer (+zdbbl, default 1MB, not set in rel/vm.args.eex, the Dockerfile or compose) suspends the sender, which here is the replication loop itself.

Replicas must stay byte-identical: the scrubber, self-healing and the storage chaos drill compare copies. Followers that write the primary's bytes verbatim satisfy that by construction.

Plan

A. The primary ships the opaque batch bytes and followers write them verbatim. The push carries the batch binary plus the few numbers the follower needs without opening it (first offset, last offset, record count, max timestamp). The follower keeps today's checks exactly (failure latch, fence, expected_first against next_offset), verifies the batch CRC, and appends the bytes. The first step inside this branch needs no format change: ship the already-encoded record frames as one binary, which removes the follower-side encoding and the per-field term encoding and is measurable by itself. follow/4, catch-up and the self-healing backfill move to the same verbatim copy. The new message shape sits behind the capability gate of #193 and relies on #187 so that an old member survives seeing it. 1 to 2 weeks.

B. Only the first step, and stop. 3 to 4 days. Removes the redundant encoding and most of the distribution cost, in the old format. Replication traffic stays uncompressed, which is one of the three places the umbrella set out not to lose to Kafka.

Do nothing. Free. With #202 active, each replica would compress by itself: three times the codec CPU inside single-process loops, and byte identity resting on zstd being deterministic across nodes.

Recommendation: A, with B as its first commits. B alone leaves the replication link at full size; A costs about one more week and is the only option under which a compressed format keeps replicas identical without an assumption about the codec.

Risks and open questions

• Follower-side validation must stay exact. The next_offset > expected_first branch (:826) acks a batch the follower already holds. With opaque batches a follower may only ever end on a batch boundary; a next_offset that falls INSIDE the pushed batch must be treated as out of sync, never as "already have it". This needs its own test, because record lists made the partial case harmless.
• Every copy path has to be verbatim, not only the push. If catch-up (:1116), follow/4 (:561) or the backfill still moved records and re-encoded them, a healed replica would differ byte for byte from its peers and the comparisons of The disk stores one frame per record, with a per-record CRC and no compression #202 would be the only thing standing between that and a heal loop.
• A follower that does not verify the CRC replicates corruption silently. Verifying costs what computing costs today (:erlang.crc32/1 measured at 1.3 to 4.8 GB/s), and it is the only check between the primary's memory and the follower's disk. A failed check answers an error ack, as a fenced or failed copy does today (:764 to :769), so the quorum simply does not count that replica.
• Large binaries share one connection with ra and gossip. A batch near 1MB fills the default distribution buffer and suspends the replication loop. Compressed batches make this better than today, not worse, but whether +zdbbl should be tuned in rel/vm.args.eex is an open question for The CPU cost of compression and the gain against Kafka are unmeasured #208 to measure.
• This touches the same files as The disk stores one frame per record, with a per-record CRC and no compression #202. The two are serial; working them in parallel guarantees conflicts in replication_server.ex and elixir_store.ex.
• Checked, not a risk: pushes from one primary arrive in offset order (per-pair FIFO, stated at :760), so opaque batches need no reordering logic, and the follower already ignores the committed field (:762), so the message can change shape without touching commit tracking.

Verification

• A multinode test that produces at replication factor 3 and asserts that followers performed no record encoding (counted with a tracer on Malachi.Log.Record.encode/1) and that each replica's files have the same digest. It fails before the change and again with the verbatim path reverted.
• Follower tests: a batch with a corrupted CRC is refused with an error ack and nothing is written; a push whose first offset falls inside the follower's last batch answers :out_of_sync; fenced and failed segments answer exactly as before.
• Repair tests: a follower several batches behind catches up and ends byte-identical; a backfilled replica is byte-identical to its source.
• Mixed-version test: a member without the capability never receives the new shape (Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193).
• Full suite including multinode, mix format --check-formatted, mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, coverage on touched files, and the node-fault and storage chaos drills. Every new Logger call goes through I18n.
• Performance: benchmark/docker-cluster.sh at replication factor 3, first step alone and then full A, payloads from Both load generators send constant bytes, so no compression measurement means anything #192, on Linux, a fresh server per case, with an A-A control, the repetitions and the noise floor stated before running. No difference is a valid result.

PR

Branch

feat/replicate-opaque-batches

Description

Replicate the primary's encoded batch bytes instead of record structs: followers validate the latch, the fence, expected_first and the batch CRC, then write the bytes verbatim, on the push, catch-up and backfill paths alike.