There is no per-message TTL

[HectorIFC]

Part of #185.
Blocked by #188, #193, #194.

Context

Every record of a topic lives exactly as long as its segment: there is no way to say "this message is worthless after 30 seconds" (a command, a price tick, a one-time code). Pulsar has a message TTL per namespace and topic, and NATS has a per-message TTL; Kafka and Redpanda have none. With max_age_ms the closest approximation is segment-granular and, until #197, unbounded for quiet topics.

What exists to build on:

• The wire record already carries headers and a timestamp (Wire.encode_record/1, lib/malachi/wire.ex:370-372), so a TTL can travel in a header with no new api key.
• The timestamp is client supplied: Wire.decode_record/1 takes ts::64 straight from the producer (wire.ex:374-379). Retention deliberately never trusts it and uses the control plane's sealed_at instead (lib/malachi/cluster/retention.ex:61), which is why Malachi has no equivalent of Kafka's future-timestamp problem (KIP-937). A TTL must not reintroduce it.
• There is one read-side filter today, filter_records/2 (lib/malachi/broker.ex:1371-1381), applied in consume_page/8 and read_history_page/6.

Plan

A. Broker-stamped expiry, read-time filter, segment-level reclaim. The TTL comes from a reserved header or from a default_ttl_ms in the topic policy of #194. At produce the broker computes the absolute expiry from ITS clock and writes it into a reserved header, so the stored frame format does not change. Reads drop expired records beside filter_records/2. For disk, each replica reports the segment's largest expiry with its fence answer, the seal records it as max_expiry, and Retention.expired/3 gains "a sealed segment whose max_expiry has passed is expirable". Per-record physical removal is not built here; it rides on the rewriter of #206. Any log line goes through I18n. Cost: 1.5 to 2.5 weeks.

B. Reclaim only, no read-time filter. Record max_expiry and expire whole segments; never filter reads. Cost: about 1 week. A record outlives its TTL by up to a segment's lifetime, which for a 30 second TTL means the feature does not do what its name says.

C. Topic-level TTL only. No header; one default_ttl_ms per topic, enforced at read time. Cost: about 1 week. It covers the common case and avoids a reserved header, but producers cannot mix lifetimes in one topic.

Do nothing. Use max_age_ms with #197 and accept segment granularity.

Recommendation: A, implemented as C first (policy default) and the header second, inside the same branch. The read-time filter is what makes the TTL true for a consumer; the segment-level reclaim keeps the disk honest without a rewriter.

Risks and open questions

• Broker time versus producer time. Expiry computed from the client timestamp lets a producer with a bad clock publish records that are born expired or that never expire. Recommendation: broker receive time only, with the client timestamp ignored for TTL. The cost is that a record replayed by a bridge gets a fresh lifetime. Needs a decision.
• A read-time filter defeats verbatim batch serving. History is re-encoded record by record for every consumer, inside two single processes #205 serves stored batches without decoding them. Filtering per record forces a decode on every TTL topic. Options: make expiry a BATCH property (the batch expires when its last record does), or exempt TTL topics from the verbatim path. Must be settled with History is re-encoded record by record for every consumer, inside two single processes #205 before either ships.
• Recording max_expiry changes the seal. A new field in seal_segment and in the fence answer is a new metadata command shape and a new inter-node reply shape during a rolling upgrade. That is the territory of A metadata command an old member does not know silently diverges the Raft group #188 and Nodes advertise no capabilities, so nothing can wait for the whole cluster to support a change #193, which this issue does not list as blockers; either add them or deliver reclaim in a follow-up.
• Clocks differ across nodes. The filter runs on whichever node serves the read, so after a failover a record that had just expired can reappear for the length of the skew. Tolerable for a TTL; it must be documented, and the drill of Retention has never run in a cluster drill #203 already skews a clock.
• Filtered is not skipped. Records dropped by TTL must not feed the expired-data count of Retention has no metrics, and a consumer is moved past expired data without being told #191, or every TTL topic looks like data loss. Separate series.
• Checked, not a risk: a page whose records are ALL filtered still advances the cursor, because consume_page/8 computes the next offset from the unfiltered records (broker.ex:1402-1409). The "empty page read as end of source" trap documented at broker.ex:1360-1366 is not triggered by this filter.

Verification

• Broker tests with an injected clock: a record is served before its expiry and dropped after; a page that is entirely expired advances the cursor and the next page is served; a record with no TTL on a topic with no default is never dropped. They fail today.
• A producer-supplied timestamp far in the past or in the future changes nothing about the expiry.
• Retention test: a sealed segment whose max_expiry has passed is returned by expired/3 even under max_age_ms; one holding a single record without TTL is not.
• Split test: the TTL filter composes with the key-slice filter on ancestor reads.
• Wire test: the reserved header round-trips through the Elixir and Node clients, and a client cannot forge the broker-written expiry header.
• Full suite including mix test --only multinode, mix format --check-formatted, mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, coverage on touched files.
• Chaos drills (node, storage), since the seal and the read path change; paired consume latency before and after on Linux.

PR

Branch

feat/per-message-ttl

Description

Adds per-message TTL from a reserved header or a topic default, with the expiry stamped by the broker clock and enforced at read time. Sealed segments record their largest expiry so retention can reclaim them whole.