[python] Align native scan planning with Rust

[JingsongLi]

Purpose

Use the planning APIs from apache/paimon-rust#831 (merged) to support native incremental and distributed scans in PyPaimon. Timestamp windows combine APPEND deltas into one plan so primary-key versions across commits are merged together; the ending snapshot supplies metadata and deletion vectors even when it contributes no APPEND files.

• Add native append slices/shards, primary-key bucket shards, and Data Evolution row-position selection. Assign DE positions before explicit/global-index range pruning, and apply limits after distribution.
• Preserve snapshot IDs for empty plans and explain output, validate branch resolution, and fall back to Python when the installed binding cannot support the scan. Positional append distribution requires Rust 0.4's stable file order.
• Honor index-file-in-data-file-dir for deletion-vector writes, reads, and abort cleanup. Preserve explicit paths and legacy Python index locations, handle partition-directory differences, and retain multiple DV index files in one bucket.
• Resolve Java bucket-local DV paths using partition field types and partition.legacy-name, including timestamp precision and JVM-dependent floating spellings. Explicit DV paths remain authoritative; Java locations take precedence over older Python locations.
• Normalize LTZ partition values before formatting: legacy names use UTC fields; non-legacy names use the process default timezone, with declared fractional precision and no timezone suffix. Cover both LTZ type spellings and UTC/non-UTC processes.
• Preserve signed-zero ordering when grouping overlapping composite key ranges.
• Run real native planner tests only in the dedicated Rust Plan CI job, which builds the latest apache/paimon-rust main. The regular Python version matrix runs Python planning and mocked adapter tests; it still installs the released binding for SQL tests. Restore the shard upper-bound validation message and initialize distribution fields in the global-index test fixture.

The native planner remains optional and disabled by default. The README documents supported scans, binding requirements, and fallback behavior.

The fork-runtime fix in apache/paimon-rust#832, Java timestamp formatting fix in apache/paimon-rust#833, and LTZ alias parser fix in apache/paimon-rust#834 are merged. Rust Plan CI builds the latest upstream main and now additionally requires apache/paimon-rust#838: it preserves the first occurrence of each (partition, bucket) group across interleaved partitions, aligning append shard/slice positions with Python.

Tests

Latest CI-fix validation with Python 3.11 and a wheel built from the Rust #838 branch:

• 345 related Python tests passed, including both original CI failures, append/PK shard tests, native integration, Data Evolution, DV paths, external paths, and adapter tests. The forced-native lane recorded 146 successful native plans; explicit integration comparisons additionally prohibit fallback.
• Added an interleaved partition/bucket regression comparing all three shards, four slices, and shard/slice combinations with limits. Reads are serial so limits select a deterministic subset.
• The multimodal cache regression now checks the public TableScan.plan entry point: eight batches reuse four plans with either planner. The focused test passed in both lanes.
• Flake8, Python 3.6 grammar checks, and git diff --check passed.

Review-fix validation with Python 3.11:

• Reproduced the review's FLOAT 0.1 and TIMESTAMP(3) whole-second failures before the fix.
• Regular Python lane: 248 passed, 51 native cases deselected.
• With the rebuilt Rust wheel from fix(core): preserve local timezone semantics for TIMESTAMP_LTZ schemas paimon-rust#834: 299 related tests passed (84 tracked native plans), including all 131 DV-path cases and isolated subprocess native LTZ scans.
• DV tests cover Java/JVM floating spellings, overflow and subnormal boundaries, timestamp precisions, legacy formats, reordered typed partition keys, canonical-path precedence, repeated deletes, and historical reads.
• New LTZ regressions reproduced 36 failures before the Python fix; the remaining two non-UTC native failures exposed the Rust alias parser bug and pass with [core] Introduce SecurityContext #834. Cases cover both type spellings, naive/UTC/+08:00 inputs, millisecond/microsecond precision, UTC and Asia/Shanghai processes, and day rollover.
• Flake8, Python 3.6 grammar checks, and git diff --check passed.

Earlier validation on this branch also passed 183 Python reader/Data Evolution tests including Lance, 25 native contiguous-window dataset tests, and 28 CI selection tests. The full Python version matrix and Java integration suite were not run locally; CI must confirm the final commits.

[JingsongLi]

Re-open to trigger tests.

[leaves12138]

Reviewed the production changes against the Java and Rust implementations, including incremental snapshot selection, append/PK/DE distribution, range intersections, limit placement, fallback gates, and DV read/write/abort paths.

Validation at eef6215 with a locally built pypaimon-rust 0.4.0 from 7fd9bab99f63a3860a72cd93f31b047f9f770f89: 162 focused tests passed; the expanded native-planning regression run passed 309 tests and exercised 233 native plans; the default Python-planning regression run passed 224 tests. Seven Vortex tests were skipped because the local interpreter is Python 3.10. These are targeted runs, not the full CI suite.

One reproducible interoperability issue remains in canonical bucket-local DV path resolution; details and reproduction are inline.

[leaves12138]

Re-reviewed the changes since eef6215, using Java as the authoritative format. The previously reported FLOAT and ordinary TIMESTAMP DV-path cases are fixed. I checked typed partition ordering, legacy formatting, floating-point directory fallback, explicit-path precedence and the new regressions.

At ead0e87, 214 focused tests passed with the Rust binding built from apache/paimon-rust#833. The expanded native-planning regression run passed 395 tests and exercised 233 native plans; seven Vortex tests were skipped because the local interpreter is Python 3.10. These are targeted runs rather than the full CI suite.

One remaining Java-interoperability issue is reproducible for TIMESTAMP_LTZ partition keys: both legacy modes retain a Python timezone suffix in the lookup path. I verified the Java paths with InternalRowPartitionComputer/PartitionPathUtils and reproduced FileNotFoundError in four independent whole-second/fractional DV-read cases. Details are inline. This finding concerns Java-produced paths, not compatibility with incorrect older Rust paths.

[leaves12138]

LGTM. Re-reviewed 10b5984866bf3e7e23fc2f47fc2dfae1554be7f2, including the changes since my previous review, with Java as the authoritative implementation.

The remaining LTZ issue is fixed: values are normalized to UTC fields for legacy names or to the process default timezone for non-legacy names, without an offset suffix, and the declared timestamp precision is retained. Both LTZ spellings are handled. The earlier FLOAT and ordinary TIMESTAMP fixes remain covered, and apache/paimon-rust#834 is now merged.

Independent validation using Python 3.10 and a Rust wheel built from apache/paimon-rust#834 (2cdab049):

• All four original LTZ failure reproductions now pass; all 131 DV-path tests pass.
• Expanded native-planning regression: 331 passed, with 158 tracked native plans. The corresponding Python-planning run passed 226 tests, with native-only cases deselected.
• 216 path comparisons against Java InternalRowPartitionComputer passed across UTC, Asia/Shanghai, and America/New_York, covering alias/canonical types, naive/UTC/+08:00 values, both legacy modes, millisecond/microsecond precision, whole/fractional seconds, day rollover, and pre-epoch values.
• 12 additional pre-epoch cases passed with both Python and native planners, including canonical bucket-DV precedence over stale copies, repeated deletion, and historical reads.

Seven Vortex tests were skipped in each expanded run because they require Python >= 3.11. These are targeted local runs, not the full CI suite; the Rust Plan CI job is still running at review time. No remaining blocking issues found.