[python] Strip the scheme from the Vortex OSS endpoint

[jackylee-ch]

Purpose

Reading a Vortex file on OSS, to_vortex_specified builds the virtual-hosted
endpoint as https://<bucket>.<fs.oss.endpoint>. If fs.oss.endpoint was
configured with a scheme — e.g. https://oss-cn-hangzhou.aliyuncs.com, a
common form — the result is the malformed
https://<bucket>.https://oss-cn-hangzhou.aliyuncs.com and the read fails.
lance_utils already strips the scheme for exactly this reason;
vortex_utils did not.

Change

• Strip a leading http:// / https:// from fs.oss.endpoint before
  composing the endpoint, mirroring lance_utils. A scheme-less endpoint is
  unchanged.

Tests

• A scheme-carrying endpoint no longer yields the doubled scheme; a plain
  endpoint, the oss://→s3:// path rewrite and the security token are
  unchanged. Verified non-vacuous (without the fix the scheme case produces
  https://<bucket>.https://<host>).

Written with Claude Code; verification is mine.

[JingsongLi]

The endpoint normalization has a real benefit and I did not find an unintended regression in this diff. Beyond the two new tests, I verified the generated options using the actual pinned Vortex 0.70.0 SDK against a local HTTPS range-serving fixture: https/http/plain endpoint inputs each read 500 rows and nulls correctly; object key, virtual-hosted endpoint, SigV4 header and session token remain correct. The old doubled-scheme endpoint fails. This fixture does not certify live OSS service behavior.

There is a separate, pre-existing blocker to the PR's stated end-to-end read workflow at this head. dev/requirements-dev.txt pins vortex-data==0.70.0, but FormatVortexReader calls vortex_store.open(). A 0.70.0 S3Store has no such method. Invoking the actual FormatVortexReader with the fixed OSS endpoint raises:

AttributeError: 'builtins.S3Store' object has no attribute 'open'

The supported SDK entry point is vortex.open(path, store=...); I used that API in the transport verification above. This reader/API mismatch is already present on the base and is not a regression introduced by the endpoint change. However, the current production read cannot complete until it is resolved, preferably with a real SDK-backed reader test. Please account for that alongside this fix so the intended OSS Vortex read workflow is actually usable.

[jackylee-ch]

Thanks — you're right that the reader was the real blocker. Fixed it alongside the endpoint change so the OSS read path is actually usable.

FormatVortexReader built the store with store.from_url(...) and then called vortex_store.open(), which 0.70.0 object stores don't expose. Switched to the supported entry point, vortex.open(path, store=...), passing the store so it carries the endpoint/credentials while vortex resolves the object; the local-file branch is unchanged.

Added a reader test (format_vortex_reader_test.py) that drives the remote branch with the real OSS store kwargs from to_vortex_specified and a stand-in vortex module, asserting the reader opens through vortex.open(path, store=...) and never store.open() — reverting the fix makes it fail. The SDK boundary is faked so it runs on the Python lane without the native package; the end-to-end OSS read over the wire is validated against the pinned SDK in the native lane (as you did with the HTTPS fixture). Pushed in ef41fa4.

[JingsongLi]

Requirement fit: SUPPORTED. Endpoint normalization and correcting the SDK entry point have real OSS read value. Implementation: FINDINGS: the new entry-point call still cannot complete the intended read because its path is not relative to the explicit store. The three committed tests and configured flake8 pass, but the actual pinned Vortex 0.70.0 reader fails against a local HTTPS object fixture. A control changing only the open path succeeds for https/http/plain endpoint inputs, 500 rows with nulls, filtering, discrete indices, shard ranges and missing-column projection; request object key, SigV4 and session token were checked. This is not a live OSS service certification. The previous store.open failure was pre-existing; the finding below is the concrete remaining failure of the reader workflow this revision explicitly attempts to fix.

[JingsongLi]

[P1] Pass a path relative to the explicitly configured store

In Vortex 0.70.0, store.from_url(full_file_uri) sets the object's key as the store prefix, and vortex.open(path, store=...) interprets path relative to that store; it does not strip the full URI here. This call therefore appends the URI to the file prefix. With the actual SDK and this reader, oss://127/fixture.vortex produced a signed HEAD request for /fixture.vortex/s3%3A/127/fixture.vortex rather than /fixture.vortex, and failed with 404. The usual oss://bucket/db/table/file.vortex input has the same duplicated-path problem, so the OSS read workflow remains unusable despite replacing the nonexistent store.open() API. A control retaining the same file-prefixed store and calling vortex.open('', store=...) reads all 500 fixture rows/nulls and passes filtering, row-index, shard and projection checks. Please use an empty relative path with the current file-prefixed store, or construct a bucket-root store and pass only the relative object key, and cover this with a real pinned-SDK reader test; the fake vortex.open assertion currently accepts the incorrect full URI.

[jackylee-ch]

You're right — the store is virtual-hosted (the bucket is in the endpoint host), so passing the full s3://bucket/key URL to vortex.open resolved the bucket twice. It now passes the object key relative to the store (e.g. db.db/t/bucket-0/data.vortex), with store.from_url unchanged. The reader test asserts the store-relative key is used and that no s3:// prefix leaks — reverting only the reader reproduces the full-URL path. The faked-SDK unit runs on the Python lane; the real pinned-0.70.0 read over the wire is exercised in the native lane. Pushed in 569e230.

[JingsongLi]

Requirement fit: SUPPORTED. Re-reviewed updated head 569e230656 with the actual pinned Vortex 0.70.0 SDK. Implementation: FINDINGS: the previous P1 remains, now with the object key duplicated instead of the full URL. The three committed tests and configured flake8 pass, but the actual reader returns 404 on the HTTPS fixture. This checks real SDK/store path resolution; it does not certify live OSS service behavior.

[JingsongLi]

[P1] Use an empty path with the existing file-prefixed store

Removing the URI scheme/bucket does not make this path relative to the store constructed on line 51. store.from_url(full_file_uri) already prefixes the store with the full object key, so vortex.open(object_key, store=...) appends that key a second time. On this exact head, the real 0.70.0 SDK and actual reader request /fixture.vortex/fixture.vortex for oss://127/fixture.vortex and fail with 404; ordinary table paths similarly become db/table/file.vortex/db/table/file.vortex. The previous review described this prefix behavior. Keeping the same store and passing an empty string reads 500 rows/nulls correctly, with filtering, row indices, shard ranges, projection and signed requests validated. Please either use vortex.open("", store=vortex_store) with the current file-prefixed store, or change from_url to construct a bucket-root store before passing the object key. The new fake-SDK assertion enforces the doubled-key call and cannot validate the native prefix contract.

[jackylee-ch]

Fixed in 918b16c — switched to vortex.open("", store=vortex_store), keeping the file-prefixed store, so it resolves to exactly the prefixed key instead of appending it again. The fake-SDK test now models the prefix contract: an empty path resolves to the single real object key; any non-empty path (the key or the full URL) resolves to a doubled, non-existent key and raises 404. So it rejects the doubled-key call the old assertion accepted, and reverting to the object-key open reproduces the 404 locally. The real pinned-SDK OSS read stays covered in the native CI lane.