Skip to content

[WIP][python] Route reads to io-cache targets per file type - #10351

Draft
sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:pypaimon-io-cache-routing
Draft

sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:pypaimon-io-cache-routing

Conversation

@sundapeng

@sundapeng sundapeng commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Purpose

The pypaimon side of #10350: when a REST table token vends OSS cache endpoints, RESTTokenFileIO wraps the OSS FileIO so that immutable files are read through a cache endpoint chosen by file type, and every other request stays on the OSS endpoint. The options and rules are the same as in #10350:

  • io-cache.endpoint names one cache endpoint; io-cache.targets with io-cache.target.<name>.endpoint (and optional .region, .path-style-access) names several, and io-cache.routes maps file types to them. io-cache.policy, io-cache.whitelist and io-cache.origin.endpoint work as in Java, and io-cache.enabled set in the catalog options overrides the token.
  • Only reads and file status use a cache. Writes, exists, listings, deletes, renames, copies and non-oss:// paths stay on OSS; a client-side dlf.oss-endpoint turns routing off.
  • Files rewritten in place (FileType.is_mutable) and files named by a sequence id never use a cache.
  • Data files are recognized by name: a data or changelog file prefix, including a custom one, followed by {uuid}-{count}.{extension}. Manifests, indexes and statistics have no extension, and the manifest sidecar is excluded explicitly. Names Paimon does not write stay on OSS.
  • Errors from a cache endpoint are raised as they are, without retrying on OSS.
  • The PyArrow, Lance and Vortex readers use the endpoint, scheme and addressing style of the chosen target (fs.oss.second.level.domain.enable for path-style).

The test cases in tests/resources/io_cache are the same files as in #10350 and apache/paimon-rust#1014.

Tests

  • io_cache_routing_test.py runs the shared cases (82 request cases, 64 file-name cases) and the option parsing.
  • io_cache_file_io_test.py: which operations use a target, per-target file systems, errors from a target.
  • rest_token_file_io_test.py, file_io_test.py and lance_utils_test.py cover the token merge, the PyArrow endpoint and addressing, and the Lance and Vortex storage options.

@sundapeng sundapeng changed the title [python] Route reads to io-cache targets per file type 【WIP】[python] Route reads to io-cache targets per file type Oct 3, 2026
@sundapeng sundapeng changed the title 【WIP】[python] Route reads to io-cache targets per file type [WIP][python] Route reads to io-cache targets per file type Oct 3, 2026
@sundapeng
sundapeng force-pushed the pypaimon-io-cache-routing branch 4 times, most recently from ca21bcc to 72068f9 Compare October 4, 2026 08:01
@JingsongLi
JingsongLi marked this pull request as draft October 4, 2026 08:23
@sundapeng sundapeng closed this Oct 5, 2026
@sundapeng sundapeng reopened this Oct 5, 2026
When a REST table token vends io-cache endpoints, RESTTokenFileIO now
wraps the OSS FileIO so that immutable files are read through a cache
target chosen by file type, and every other request stays on the OSS
endpoint. The rules are the same as in the Java Jindo FileIO.

- io-cache.endpoint names one target. io-cache.targets with
  io-cache.target.<name>.endpoint (and optional .region and
  .path-style-access) names several, and io-cache.routes maps file types
  to them. io-cache.whitelist and io-cache.policy keep their meaning, and
  io-cache.origin.endpoint names the OSS endpoint when fs.oss.endpoint
  points at a cache for older clients. io-cache.enabled set in the
  catalog options overrides the token.
- Only read and meta requests use a target. Writes, exists, listings,
  deletes, renames, copies and non-oss paths stay on OSS, and a
  client-side dlf.oss-endpoint turns routing off.
- Files rewritten in place (FileType.is_mutable) and metadata named by a
  sequence id never use a target.
- Data files are recognized by name: a data or changelog file prefix,
  including a custom one, followed by {uuid}-{count}.{extension}.
  Manifests, indexes and statistics share the {uuid}-{count} shape but
  have no extension, so a custom prefix such as manifest- cannot mix them
  up; the manifest sidecar is excluded explicitly. Other names keep their
  FileType, and names Paimon does not write stay on OSS.
- Errors from a target are raised as they are, without retrying on OSS.
- PyArrow, Lance and Vortex readers use the target's endpoint, scheme and
  addressing style (fs.oss.second.level.domain.enable for path-style).

The routing cases in tests/resources/io_cache are shared with the Java
Jindo FileIO and paimon-rust.
@sundapeng
sundapeng force-pushed the pypaimon-io-cache-routing branch from 0dc1efe to 0af7167 Compare October 5, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant