Conversation
This was referenced Oct 3, 2026
sundapeng
force-pushed
the
pypaimon-io-cache-routing
branch
4 times, most recently
from
October 4, 2026 08:01
ca21bcc to
72068f9
Compare
JingsongLi
marked this pull request as draft
October 4, 2026 08:23
When a REST table token vends io-cache endpoints, RESTTokenFileIO now
wraps the OSS FileIO so that immutable files are read through a cache
target chosen by file type, and every other request stays on the OSS
endpoint. The rules are the same as in the Java Jindo FileIO.
- io-cache.endpoint names one target. io-cache.targets with
io-cache.target.<name>.endpoint (and optional .region and
.path-style-access) names several, and io-cache.routes maps file types
to them. io-cache.whitelist and io-cache.policy keep their meaning, and
io-cache.origin.endpoint names the OSS endpoint when fs.oss.endpoint
points at a cache for older clients. io-cache.enabled set in the
catalog options overrides the token.
- Only read and meta requests use a target. Writes, exists, listings,
deletes, renames, copies and non-oss paths stay on OSS, and a
client-side dlf.oss-endpoint turns routing off.
- Files rewritten in place (FileType.is_mutable) and metadata named by a
sequence id never use a target.
- Data files are recognized by name: a data or changelog file prefix,
including a custom one, followed by {uuid}-{count}.{extension}.
Manifests, indexes and statistics share the {uuid}-{count} shape but
have no extension, so a custom prefix such as manifest- cannot mix them
up; the manifest sidecar is excluded explicitly. Other names keep their
FileType, and names Paimon does not write stay on OSS.
- Errors from a target are raised as they are, without retrying on OSS.
- PyArrow, Lance and Vortex readers use the target's endpoint, scheme and
addressing style (fs.oss.second.level.domain.enable for path-style).
The routing cases in tests/resources/io_cache are shared with the Java
Jindo FileIO and paimon-rust.
sundapeng
force-pushed
the
pypaimon-io-cache-routing
branch
from
October 5, 2026 05:33
0dc1efe to
0af7167
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The pypaimon side of #10350: when a REST table token vends OSS cache endpoints,
RESTTokenFileIOwraps the OSS FileIO so that immutable files are read through a cache endpoint chosen by file type, and every other request stays on the OSS endpoint. The options and rules are the same as in #10350:io-cache.endpointnames one cache endpoint;io-cache.targetswithio-cache.target.<name>.endpoint(and optional.region,.path-style-access) names several, andio-cache.routesmaps file types to them.io-cache.policy,io-cache.whitelistandio-cache.origin.endpointwork as in Java, andio-cache.enabledset in the catalog options overrides the token.exists, listings, deletes, renames, copies and non-oss://paths stay on OSS; a client-sidedlf.oss-endpointturns routing off.FileType.is_mutable) and files named by a sequence id never use a cache.{uuid}-{count}.{extension}. Manifests, indexes and statistics have no extension, and the manifest sidecar is excluded explicitly. Names Paimon does not write stay on OSS.fs.oss.second.level.domain.enablefor path-style).The test cases in
tests/resources/io_cacheare the same files as in #10350 and apache/paimon-rust#1014.Tests
io_cache_routing_test.pyruns the shared cases (82 request cases, 64 file-name cases) and the option parsing.io_cache_file_io_test.py: which operations use a target, per-target file systems, errors from a target.rest_token_file_io_test.py,file_io_test.pyandlance_utils_test.pycover the token merge, the PyArrow endpoint and addressing, and the Lance and Vortex storage options.