[fulmicoton]

Follow-up to #3179. Adds a benchmark for plain blocks, and uses it to make opening an OnPair16 block about 1.9x faster. Also makes the block index check added in #3179 vectorizable.

Benchmark

columnar/benches/bench_plain_block.rs (binggan), on the messages of the loghub Apache_2k.log sample (2,000 values, 95,095 raw bytes, one 14,556-byte OnPair16 block). The data is in columnar/benches/data/, with a README crediting loghub.

It measures:

• open_onpair16_block: opening a block, with a reused buffer (what PlainValueReader does);
• open_onpair16_block_fresh_buffer: opening a block with a new buffer (the first block of a reader);
• read_all_values: reading the 2,000 values of an open block;
• open_and_read_one_value: a point lookup.

PlainBlock, DictOffsetsBuffer and encode_plain_block are exported as #[doc(hidden)] for the benchmark: they are not part of the stable API.

Faster block open

Opening a block dominates point lookups: on main it takes ~416 ns, while reading one value of an open block takes ~15 ns. On the benchmark block (main measured with the same benchmark, adapted to its API):

	main (#3179)	This PR
Open a block (open_onpair16_block)	416 ns	220 ns
Open a block with a new buffer (open_onpair16_block_fresh_buffer)	416 ns	379 ns
Point lookup (open_and_read_one_value)	477 ns	260 ns
Read the 2,000 values (read_all_values)	29.6 µs	28.7 µs
Block size	14,520 bytes	14,556 bytes

On main, every block open allocates its dict offsets, so both open rows are the same.

The changes:

• Token ends with BitPacker4x (format change). The dictionary offsets were rebuilt with a scalar loop over 4-bit token lengths. They are now stored as token ends, compressed with BitPacker4x::compress_strictly_sorted on 4 bits, in chunks of 128, and decoded with SIMD unpacking and prefix sum. A 4-bit delta always decodes to a token length in 1..=16, even on corrupted data. It takes 64 bytes per 128 tokens, as before, but the last chunk is padded with zero deltas: up to 63 more bytes per block when the dictionary is not a multiple of 128 tokens (36 bytes on the benchmark block).
• Explicit padding check. Of the invariants checked by CompactDictionaryView::validate_safety, only the read padding of the last token is not guaranteed by construction (at most 2^12 tokens, dict_offsets[0] == 0, token lengths in 1..=16). open now checks only that, and runs validate_safety in a debug_assert.
• Reusable dict offsets buffer. The offsets are decoded into a DictOffsetsBuffer (Box<[u32; 2^12 + 1]>, 16 KiB), sized for the largest dictionary (MAX_CODE_BITS = 12). PlainBlock::open takes it, PlainBlock::recycle gives it back, and PlainValueReader keeps it from block to block, so a reader only allocates it for its first block.
• The reader rejects dictionaries of more than 2^12 tokens, which the buffer could not hold.

Reading the values of an open block is unchanged.

The format change is safe: the plain format of #3179 is not released yet, and the columnar writer does not produce plain columns yet.

Faster block index check

#3179 checks at column open that the value starts and byte starts of the block index are strictly increasing. The check looped over the entries with an early exit and a short-circuit ||, which prevents vectorization. It is now one branch-free fold per array (is_strictly_increasing, over a cloneable iterator), optimized for valid indexes:

Blocks in the column	Check before	Check after
100	~70 ns	~13 ns
1,000	~650 ns	~121 ns
10,000	~6.8 µs	~1.2 µs

The check only runs when a column is opened, so this does not show in bench_plain_block. An invalid index is still rejected, with the same error, after the full pass.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-26T16:17:25.000112Z	7a9e9e0	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Mallets]

nit: from an API point of view it's a bit weird to write &mut None.

I didn't check the feasibility but an alternative could be:

fn open(bytes: OwnedBytes,  num_values: u32,  dict_offsets_buffer: &mut Option<DictOffsetsBuffer>)

and

fn recycle(self) -> Option<DictOffsetsBuffer>