Skip to content

[common] Reject out-of-range range-bitmap chunk-size option - #9765

Draft
LuciferYang wants to merge 5 commits into
apache:masterfrom
LuciferYang:fix/rangebitmap-chunk-size-truncation
Draft

[common] Reject out-of-range range-bitmap chunk-size option#9765
LuciferYang wants to merge 5 commits into
apache:masterfrom
LuciferYang:fix/rangebitmap-chunk-size-truncation

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

close #9764

(int) MemorySize.parse(chunkSize).getBytes() narrowed the chunk-size option to an int, and what the narrowing produces depends on the value:

chunk-size narrows to result
2g -2147483648 crashes on the writer's first ByteBuffer.allocate
4g, 8g 0 no failure at all: every key gets its own chunk and the index is silently bloated
5g 1g no failure at all: silently builds the index at a quarter of the requested size

The crashing case is the one you notice. The other two are the reason to validate, because they produce a working-looking index from a configuration that was never honoured. The chunk size becomes an eagerly allocated per-chunk buffer, so nothing above int range can work in the first place.

This PR rejects anything above Integer.MAX_VALUE at writer creation with a message naming the option and the offending value. chunk-size is read as a free-form string with no typed ConfigOption, and MemorySize.parseBytes rejects a negative or overflowing value itself, so the accepted set after this guard is exactly [0, Integer.MAX_VALUE], which is exactly the set that narrows to itself.

Tests

RangeBitmapFileIndexTest#testChunkSizeBeyondIntRangeRejected pins the boundary rather than a sample either side of it: 2147483648 bytes is rejected with a message containing chunk-size, 2147483647 bytes is accepted, and 16mb still writes and serializes. 2g and 4g were dropped from it because they reach the guard through the same comparison as 2147483648 bytes, and their narrowing consequences cannot be asserted once the guard prevents them.

Verified on JDK 11 by mutating the comparison rather than reasoning about it. 18 tests pass as they stand. With the guard removed the test fails with Expecting code to raise a throwable, which also shows that on master createWriter() returns normally and the damage happens later during the write. With > changed to >= it fails on the accepted case with The 'chunk-size' option must not exceed 2147483647 bytes, but was '2147483647 bytes'.

API and Format

A new validation on an existing user option. Values that previously worked are unaffected; values above 2GB either crashed mid-write or silently built an index that ignored the setting.

Documentation

None needed. The behaviour on out-of-range values was a crash or silent misconfiguration, not a documented feature.

RangeBitmapFileIndex parsed the user-facing chunk-size option with
(int) MemorySize.parse(...).getBytes(), so a legitimate value such
as "2g" silently narrowed to a negative int and the index writer
crashed with an unrelated failure deep inside the first chunk. The
chunk size also becomes an eagerly allocated per-chunk buffer, so
values beyond int range can never work.

Validate the option instead: reject anything above Integer.MAX_VALUE
with a message naming the option and the offending value.

Assisted-by: GLM-5.3
@LuciferYang
LuciferYang marked this pull request as draft September 13, 2026 03:06
LuciferYang and others added 4 commits September 13, 2026 12:28
"2g" narrows to a negative int and crashes on the first chunk allocation, but
"4g" narrows to 0 and "5g" to 1g: those build a silently wrong index without
failing anywhere, which is the case the guard is really there for. Add the
"4g" case and say so in the comment.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The "4g" case re-entered the same comparison as "2g", so it pinned nothing;
mutating the guard from > to >= went undetected. Assert the boundary itself
instead.

The comment also claimed more than the code does. Chunk size 0 is a supported
mode (ChunkedDictionaryTest builds one deliberately) and 1g is a value the
guard accepts, so neither "4g" nor "5g" produces a wrong index: narrowing
substitutes a different size than the one configured, and only the negative
case fails.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Range-bitmap chunk-size option above 2GB silently truncates to a negative int

1 participant