[common] Widen hilbert index bytes beyond 8 dimensions

[LuciferYang]

Purpose

close #9776

HilbertIndexer.hilbertCurvePosBytes padded the index to a fixed 63 bytes. ConvertBinaryUtil.paddingToNByte keeps the first n bytes when the array is longer, so it drops low-order bytes off a big-endian value. An N-dimensional 63-bit index needs up to 63*N/8 + 1 bytes, the extra byte covering BigInteger's sign byte when the top bit is set, so 63 was too narrow from 8 dimensions upward and the sort key silently lost resolution.

At 8 dimensions this is worse than lost resolution. The index fills exactly 63 bytes, so a value in the top half of the space serializes to 64 bytes with a leading 0x00. Truncating that back to 63 leaves the zero in front, and the key then sorts below the untruncated key of a numerically smaller index. Clustering does not just get coarser, it gets the order backwards for half the space.

The width is now 63*N/8 + 1 at every dimension count. Nothing needs the old shape: all three consumers (HibertSorter in paimon-core, HilbertSorter in paimon-flink, SparkHilbertUDF) use the key as a transient sort key and strip it before writing, so no data file, manifest or index file carries it, and two keys are only ever compared within one sort where the dimension count is fixed. Ordering is unaffected for 1 to 7 dimensions, where 63 bytes never truncated: those keys just stop carrying dozens of leading zero bytes, so a 2-dimension key goes from 63 bytes to 16.

Tests

HilbertIndexerTest.testHighDimensionIndexKeepsAllBits asserts the widths (16 for 2 dimensions, 64 for 8, 71 for 9, 127 for 16) and that points differing only in low-order bits stay distinct at 9 and 16 dimensions.

testEightDimensionKeysOrderLikeTheirIndex is the one that pins the ordering bug: it spreads 24 points across the space, recomputes each index with HilbertCurve directly, and asserts that comparing the encoded keys unsigned agrees with comparing the indexes for every pair.

Verified on JDK 11: all 3 tests pass with the change; restoring the 63-byte width for 8 dimensions and nothing else makes testEightDimensionKeysOrderLikeTheirIndex fail on a concrete pair, alongside the width assertion. paimon-common builds clean with checkstyle and spotless.

[JingsongLi]

Reviewed a55f428. Requirement fit: SUPPORTED. Implementation: CLEAN.

This repairs the clustering key rather than merely changing its size: at eight dimensions the BigInteger sign byte can invert bytewise ordering, and higher dimensions lose low bits. The computed width covers the full nonnegative index. I traced the core/Flink/Spark consumers; the key is temporary within a sort with a fixed dimension count and is removed before table data is written.

All four HilbertIndexerTest cases passed with the exact changed classes on JDK 8, including order equivalence and full-index round trips. Current CI is successful. No blocking issue found; I have not independently measured query-pruning or clustering throughput gains.