There is no controlled activation of a new storage format, and no rollback floor

[HectorIFC]

Part of #186.
Blocked by #196, #202, #204.

Context

There is no controlled way to turn the new storage format on, and no floor below which a rollback is refused. Three properties of the code make that dangerous rather than merely untidy.

The record frame has no version. lib/malachi/log/record.ex:160 accepts exactly one magic (0x4D51, :42); anything else is reported as damage. A binary that predates #202 would therefore read a batch frame as rot, and the scrubber would demote the copy and fetch another one, which is equally unreadable to it.

Replicas must be byte-identical, so the format of a segment is ONE decision for all its replicas. A per-node setting (an environment variable, an application config) is the wrong shape: three members with different values would write divergent copies of the same segment, and the comparisons in lib/malachi/cluster/scrubber.ex:316 and lib/malachi/cluster/self_healing.ex:176 to :186 would read that as lost data. The natural single decision point already exists: a segment is born through the Raft command register_segment (lib/malachi/metadata.ex:131, :351), issued by the one broker that rolls it (lib/malachi/broker_server.ex:912, :948).

The cluster cannot version its own protocol yet. No ra machine declares version/0, register_segment is a fixed 5-tuple, and the replication push is a fixed 7-tuple (lib/malachi/cluster/replication_server.ex:762). The pieces that close this are in #184: the capability gate and cluster flag (#193), the data-directory format marker (#189) and the rolling upgrade and rollback drill (#196).

Plan

A. Reader first, then an operator-flipped, Raft-recorded cluster flag. Release N reads both formats and writes the old one. In a later step the operator flips a cluster flag, recorded in Raft, which the control plane accepts only while EVERY member advertises the capability (#193). From then on newly registered segments use the batch format; existing and active segments are untouched, and formats never mix inside a segment. Rolling back to release N is always safe; booting anything older is refused by the marker (#189), which is raised when the flag flips. Flipping the flag back is allowed and only affects segments registered afterwards. The count of segments per format is exposed as a metric and on the dashboard, and the whole sequence is certified by the drill of #196. About 1 week (an estimate the plan did not break down).

B. A per-topic setting instead of a cluster flag. About 1 week more. It allows one topic to be a canary, but it multiplies the test matrix (format by topic by member capability) and still needs the cluster-wide capability check underneath.

C. Activate automatically once every member advertises the capability. About 3 days. No operator step, and therefore no operator decision at the moment the rollback floor moves, which is the one moment that deserves one.

Do nothing. Free. The format could only be enabled by per-node configuration, which the second paragraph above shows to be a way to manufacture divergent replicas.

Recommendation: A. It is the only option where the rollback floor moves by a deliberate, recorded act. B can be layered on later without changing A; C saves days and removes the control the operator needs.

Risks and open questions

• The flag is a one-way door in practice. Turning it off stops new batch-format segments, but the ones already written stay until retention removes them, so the floor remains release N for that long. The operator documentation and the flip command's confirmation text must say this in plain words.
• Which event raises the marker is a real choice. Raising it at the flip is conservative (the floor moves even if no segment was written yet) and explicit. Raising it at the first batch-format segment is tighter and adds a second moving part. Recommended: at the flip. Open for the owner to confirm.
• A member that lacks the capability can appear after the flip. A node restarted onto an older binary is stopped by the marker; a NEW node joining with an older binary has no marker yet and must be refused at join by Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193. The drill has to cover both.
• Activation completes only as segments roll. An idle topic keeps its active segment in the old format indefinitely. That is correct, but without the per-format metric an operator cannot tell "not activated" from "nothing rolled yet".
• The order of the blockers is load-bearing. If the flag could be flipped before Replication ships record structs uncompressed, and every replica re-encodes them #204, followers would compress by themselves and byte identity would depend on the codec being deterministic across nodes. The capability advertised to Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193 must therefore mean "reads the format AND replicates verbatim", not only the first.
• Checked, not a risk: formats cannot mix inside a segment by accident, because only the broker that rolls a range registers its next segment (broker_server.ex:912, :948) and the format is fixed by that one command; a flip during a produce affects the next segment, never the one in flight.

Verification

• A multinode test on a mixed cluster (one member without the capability): the flip is refused, and no batch-format file exists anywhere afterwards. With all members capable the flip succeeds and the next rolled segment is in the batch format. It fails before the change (no flag exists) and again with the gate reverted.
• The drill of No drill certifies a rolling upgrade followed by a rollback #196 run through the full sequence: upgrade to the reader release, flip, produce and roll, roll back to the reader release and read everything, then attempt to boot the pre-reader release and assert that it refuses with the marker's error.
• State-machine tests for the flag command, in the style of the existing property tests of the deciders: idempotent flips, flip refused on a missing capability, a member joining after the flip.
• A test that a node restarted with the flag on and a stale local view never writes the old format into a segment registered as batch format, or the reverse.
• Full suite including multinode, mix format --check-formatted, mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, coverage on touched files, and the config-deploy, node-fault and storage chaos drills with the flag on. Every new Logger call goes through I18n.
• Performance is not the subject here; the flag's cost on the produce path is one lookup at segment registration, asserted by an A-A run of the existing ceiling harness on Linux with the repetitions and the noise floor stated before running. No difference is the expected and valid result.

PR

Branch

feat/storage-format-activation

Description

Activate the batch storage format through a Raft-recorded cluster flag that the operator flips only when every member advertises the capability. New segments only, never mixed; rollback to the reader release is always safe and refused below it by the data-directory marker.