[FEAT] ctdcast netCDF as CTD reference: provenance, finality check, report CTD block

[eleanorfrajka]

Summary

Let a cast use a ctdcast per-cast stage netCDF as its CTD reference instead of a raw .cnv, and carry that reference's provenance through to caldip's outputs. When the reference is a ctdcast file, caldip records the CTD sensor serials, calibration dates, conductivity slope, processing stage and data mode in the output netCDF, exposes them on the report, and can report whether each cast's reference has finished (caldip report --check). A raw-.cnv reference keeps working unchanged; the extra provenance is simply UNK and omitted.

What's changed

• readers.read_ctdcast_reference maps a ctdcast stage netCDF to caldip's canonical CTD variables for the requested ctd_sensor, masks QARTOD-fail samples, resamples to 1 Hz, and returns a provenance dict (sensor serials and cal dates, conductivity slope and whether it was applied, processing levels, processing stage, data mode, source tracking id, and the cruise verbatim).
• load_reference_data dispatches to that reader when the configured ctd_file resolves to a ctdcast stage netCDF (_is_ctdcast_nc), otherwise loads the .cnv as before.
• The output netCDF gains the CTD-reference provenance as global attributes; the overlay rule is that no UNK may overwrite a known value, and data_mode_meaning / qc_masked_flag_values are derived from their partner attribute at write time so they cannot disagree.
• caldip report --check <path> classifies each cast — final, waiting on reference, rerun needed, not run, no reference (cnv input) — comparing the config to the output and the output to the reference file, with a non-zero exit if any cast is not final. It reads recorded-vs-current values, never file mtimes.
• A cruise YAML (caldip.cruise.yaml) holds per-cruise facts (cruise, ship, year, cal_dip directory) that per-cast configs inherit; a per-cast value that disagrees warns and the cruise value wins.
• config_digest (a hash over the instrument list and clock offsets) and preferred_pair are recorded so a rerun can be triggered by a real config change and finality can gate on the reference's declared sensor pair.
• The netCDF inventory page (caldip inspect) shows the CTD reference "as recorded / now", flagging any attribute that changed since the run and the two finality gates.
• Each report cast page gains a CTD reference block: the CTD sensor used (primary/secondary) and its provenance, stated once for the cast rather than repeated per row. ctd_sensor_used is dropped from the per-stop detail table (kept in the CSV export); serials, cal dates and slope appear when the reference is a ctdcast file and are omitted for a .cnv.
• The report index always shows a Cruise column, read per cast from its {cast}_caldip.nc, so a cast's own cruise is visible even when a directory mixes cruises (the heading then reads multiple cruises).
• Every report page (index, cast, caldip inspect inventory) ends with a footer naming caldip, its version and the UTC generation time, matching the sibling oceanarray/ctdcast report footers.
• caldip stats records ctd_path as the reference path load_reference_data actually resolved (data_dir + ctd_file), not the bare filename, so the inventory recorded-vs-now view and caldip report --check can locate the reference from the project root rather than only from the cast directory.
• load_config tolerates an empty per-cast YAML (yaml.safe_load returning None), matching load_cruise_config, instead of raising AttributeError.

Breaking changes

None for machine consumers: the added netCDF global attributes are additive, and the detailed-CSV columns (including ctd_sensor_used) are unchanged. The report HTML changes — the cast page adds a CTD-reference block and drops the constant ctd_sensor_used column from the per-stop table, and the index always shows a Cruise column.

Tests

The full suite passes apart from the four pre-existing integration tests that need the optional seabirdscientific package and real instrument files. New coverage: the ctdcast reader and provenance overlay (test_ctdcast_input.py), the finality sweep and states (test_finality.py), the cruise YAML inheritance (test_cruise_config.py), the inventory recorded-vs-now table (test_inventory.py), the report CTD-reference block plus the always-on cruise column (test_report_build.py), the resolved reference path recorded by the reader (test_ctdcast_input.py), and empty-YAML config loading (test_integration.py).

Notes

The ctdcast and oceanarray repos diverged on the odb cruise label (odb2026 vs OdB); caldip carries whatever its reference/config records, verbatim, and does not normalise — reconciling the two is an upstream/downstream coordination item, not caldip's to paper over. The new provenance globals are additive; whether they warrant a schema_version bump is a release-time decision, not settled here.

[Copilot]

🟡 Changes recommended

There are correctness issues in config loading and in how ctd_path is recorded that can break --check/inventory reference resolution outside a specific working directory.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables using a ctdcast per-cast stage netCDF as the CTD reference (in addition to raw .cnv), carrying CTD-reference provenance through to caldip outputs, and adding “finality” reporting via caldip report --check plus richer report/inventory display of recorded-vs-current reference metadata.

Changes:

• Add a ctdcast reference reader (read_ctdcast_reference) with QC-masking, 1 Hz resampling, and provenance extraction; write provenance into output netCDF global attributes with safe overlay rules.
• Add finality sweep logic (caldip report --check) and inventory “as recorded / now” comparisons, plus report UI updates (CTD reference block on cast pages; Cruise column always on index; generated footer on all pages).
• Add/extend tests for ctdcast input, provenance overlay, cruise YAML inheritance, finality states, inventory comparison rendering, and report rendering changes.

File summaries

File	Description
tests/test_report_build.py	Tests CTD reference block appears once and footer presence; updates cruise column expectation.
tests/test_inventory.py	Adds fixtures/tests for recorded-vs-now reference comparison highlighting and finality gate display.
tests/test_finality.py	New tests for --check cast classification states and sweep behavior.
tests/test_ctdcast_input.py	New tests for ctdcast reader mapping, QC masking, provenance extraction, and writer coupling checks.
tests/test_cruise_config.py	New tests for cruise YAML inheritance/override and cast discovery.
README.md	Adds acknowledgements section.
docs/source/running.md	Documents footer, always-on cruise column, CTD reference block, and --check workflow.
docs/source/outputs.md	Documents new/expanded global attributes and finality-related metadata.
data/proc_calib/msm142_2026/cal_dip/castM4/castM4.caldip.yaml	Switches castM4 CTD reference from .cnv to ctdcast stage netCDF.
caldip/report/inventory.py	Implements recorded-vs-current CTD reference comparison table and finality gates display.
caldip/report/index.py	Makes Cruise column always present; simplifies row builder.
caldip/report/finality.py	Adds finality sweep logic backing caldip report --check.
caldip/report/cast.py	Adds CTD reference block rendering and removes constant cast-level columns from per-stop detail table.
caldip/report/_html.py	Adds shared “Generated by caldip … UTC” footer and a key/value table helper.
caldip/report/_data.py	Extracts CTD-reference rows from netCDF attrs for display on cast pages.
caldip/readers.py	Adds cruise YAML discovery/merge, ctdcast detection and reading, and units-driven conductivity normalization.
caldip/cli/stats.py	Writes provenance/input_mode and now passes CTD provenance into netCDF writer.
caldip/cli/report.py	Adds --check and --nc-dir to report CLI and wires to finality sweep.
caldip/_writers.py	Adds config_digest, provenance overlay into global attrs, and derived attribute coupling logic.

Review details

• Files reviewed: 19/19 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.