Plugin: manage NetCDF-like multifile collections with kerchunk

[brendan-m-murphy]

Context

Kerchunk is a strong fit for the NetCDF-like multifile collection case from #109. It scans existing binary scientific files and writes reference mappings that let one or more source files appear as a logical Zarr-like dataset through fsspec/Zarr/xarray.

Relevant upstream docs:

• https://fsspec.github.io/kerchunk/
• https://fsspec.github.io/kerchunk/reference.html
• https://fsspec.github.io/kerchunk/tutorial.html

This should be plugin work, not a core dependency. Core should provide artifact descriptors, claims/facets, writer results, capability lookup, and read handles. A kerchunk plugin can then add optional NetCDF/HDF5/Zarr/xarray/fsspec behavior.

Goal

Design and implement an optional kerchunk-backed plugin for managing NetCDF-like multifile collections as virtual aggregate datasets.

Initial target: a managed directory or archive-derived collection of .nc / NetCDF4/HDF5-like member files can produce a kerchunk reference artifact, and that reference artifact can advertise an xarray/Zarr-style read interface without importing kerchunk/xarray/Zarr in core.

Proposed Capability Shape

Writer/converter capability:

• Input claims/facets:
  • representation=directory or archive/materialized collection input
  • interface=collection
  • collection facet with pattern, member_format, member_suffixes, optional reader_hint
• Output artifact:
  • a kerchunk reference JSON or parquet reference set as a sidecar/derived artifact
  • claims such as data_type=kerchunk-reference, interface=zarr-group, interface=xarray-dataset
  • facets for concat_dims, identical_dims, coo_map, reference_format, and source member manifest/checksums where available

Reader capability:

• Open the kerchunk reference artifact through fsspec ReferenceFileSystem / Zarr / xarray.
• Produce clear missing-plugin errors when kerchunk/xarray/zarr/fsspec extras are not installed.

Questions To Resolve

• Should the kerchunk reference be stored as a derived artifact on the same record, a managed metadata artifact, or a separate linked record?
• Should the plugin generate JSON first and defer parquet/LazyReferenceMapper support, or support both immediately?
• How should credentials/storage options be referenced without persisting secrets?
• How should update/append work for newly added members? Kerchunk has append-related support, but ogcat collection mutability should remain deliberately scoped.
• Which claim/facet names should belong to ogcat.core versus a plugin namespace such as org.fsspec.kerchunk?

Non-Goals

• Do not make kerchunk, xarray, zarr, h5py, netCDF4, or fsspec required core dependencies.
• Do not replace the generic managed collection API in Managed collections as operation targets #109.
• Do not require every collection to become a kerchunk reference.
• Do not solve mutable collection updates until Managed collections as operation targets #109/Structured artifact writer result model #117 have a clear story for collection write results and manifests.

Acceptance Criteria

• A kerchunk plugin or design spike demonstrates a tiny multifile NetCDF-like collection producing a reference artifact.
• Core ogcat remains importable and testable without kerchunk/xarray/zarr/NetCDF dependencies.
• The plugin attaches explicit claims/facets that a future registry/accessor can use for dispatch.
• The example documents how this differs from a .zarr directory store: .zarr is one directory-backed data artifact; a directory of .nc files becomes a collection only through explicit collection claims/facets.
• Tests are optional-import gated and skip cleanly when the scientific stack is unavailable.

Related

• Plan filesystem-like artifact interfaces and structured operator pipelines #108
• Managed collections as operation targets #109
• Data type, representation, interface, and facet claim schemas #116
• Structured artifact writer result model #117
• Read-side artifact handles and accessors #118
• Reader, writer, and converter capability registry #119

[brendan-m-murphy]

Boundary note from #119 review cleanup:

#119 keeps kerchunk-style behavior at the declaration level only: a plugin can declare a converter with input/output claims and required facets, but the core registry does not import kerchunk, fsspec, xarray, Zarr, or NetCDF dependencies and does not execute conversion pipelines.

For #125, the pressure test should stay optional-plugin gated and use facets for dispatch-significant details. Claim metadata is documented as descriptive in #119; facet metadata participates in selection.