[PROBE: energy] Surface temperature and upward longwave consistency - #39
[PROBE: energy] Surface temperature and upward longwave consistency#39xinlan-technology wants to merge 15 commits into
Conversation
…s LF Upstream landed wflow_sbm (Flood-Lab#36), the antecedent storm-window fix (Flood-Lab#35), LF line endings for every text file with a workflow check that enforces them (Flood-Lab#37, Flood-Lab#38) and the docker build fix (Flood-Lab#42). The conflicts were the three places that count probes: the README's models table, the site's models rows in three languages, and the archive. wflow_sbm's rows take the new probe into their totals, 14 of 20, and its energy wording widens from the three heat-flux probes to the four probes that need an energy output, since it reports neither heat fluxes nor a surface temperature. Its INCOMPLETE row on energy/radiation-consistency is appended by ht run like the others. The archive keeps upstream's renormalised rows followed by this branch's, and the index carries no CRLF.
The three reference adapters take the 644 mode of the coupled family they copy. probe.yaml says again why the tolerance is 0.5 percent of the reported upward flux, which the pull request template asks to be justified there and a trimming pass had reduced to one line. The README's probes paragraph counts four of the five energy probes as needing an energy output, since pet-consistency needs none. The overflow guard in radiative_identity keeps both of its checks, now with a comment that says what each one catches: a finite but absurd temperature overflows the fourth power, and a finite but absurd rel_tol overflows the allowance, and either would otherwise slip through the comparison as a pass.
The probe README no longer counts the archived models, which the merge had made six; wflow_sbm's own results table takes the INCOMPLETE row this branch archives for it, with its energy wording widened to the four probes that need an energy output. radiative_identity drops a step-mean residual that nothing read and that sat oddly in a criterion whose point is that steps are not averaged. The criterion's unit tests lose a timestep argument no case set and a registration test that ht validate already performs.
The mean absolute residual returns to radiative_identity's diagnostics: it never decides a verdict, but it goes into every report and is the first number to look at when a model fails on a handful of steps. The overflow comment now names the two ways a bad value would pass rather than fail: an infinite allowance makes every slack zero, and an infinite residual over an infinite allowance is NaN. probe.yaml's tolerance note states the surface and sky the 1.5 percent example depends on, and says what the floor does, which is to stop the bound shrinking towards zero, not to keep it finite.
Upstream merged mass/human-abstraction (Flood-Lab#32) with its author, its reference model and its archive rows. The conflicts were again the places that count probes: the README's probes paragraph, models table and status line, the site's models rows in three languages, the archive, and CITATION.cff, where both branches appended an author. Every count now reads twenty-one, the evaluated models' denominators move with it, the energy wording stays at the four probes that need an energy output, the citation keeps its order of first merged probe, and the archive carries upstream's rows followed by this branch's. The suite is 437 tests, 21 probes and 35 models, all passing.
licm13
left a comment
There was a problem hiding this comment.
Hi @xinlan-technology, thanks for the support.
The radiative identity, CF variable semantics, numerical guards, and the construction of the positive and negative controls are sound. I reproduced the probe gate, adapter verification, validation, and the 36-case margin scan. The positive control passes and both negative controls fail with adequate margins.
I am requesting changes because the probe’s compatibility contract does not currently include the boundary inputs on which the verdict is conditional. The criterion evaluates native ts and rlus against case-provided rlds and eps, but the probe declares only the two required outputs. A model that emits both outputs while using an internal downward-longwave estimate or its own emissivity is therefore considered compatible and can receive VIOLATION against values it never consumed. That is an input incompatibility, not evidence of a radiative-identity violation.
Please extend the probe/model compatibility contract so this probe requires declared support for rlds and for the case-provided static emissivity, and add a regression test showing that a model which emits ts and rlus but does not accept those inputs is classified as INCOMPATIBLE.
Non-blocking documentation issue: the README reports an exact positive-control maximum of 3e-14, whereas the margin script produces approximately 1.8e-13–2.1e-13 under other dependency versions allowed by pyproject.toml. The result remains safely negligible, but the table should use a stable bound such as <1e-12 or state the environment used to generate the numbers.
…be re-archived Upstream stopped scoring a probe that cannot be put to a model: its verdict is N/A and a model's passes are counted out of the probes that scored it. That leaves every standing untouched by this probe, which every evaluated model is N/A on, so the README and site rows are upstream's with the probe rows and the flowchart entry laid back on top. The archive rows this branch had written as FAIL (INCOMPLETE) are regenerated by ht run under the new vocabulary for all eight evaluated models, summa and cwatm included, at the versions upstream archived last. wflow_sbm's card takes the N/A row for this probe. The diagnostics test expects NOT_SCORED where it expected FAIL.
Review of Flood-Lab#39 found a gap in the compatibility contract: the criterion judges a model's ts and rlus against the case's rlds and eps, but the probe only required the two outputs, so a model that emitted both while using its own downward longwave or its own emissivity counted as compatible and could be scored VIOLATION against values it never read. A probe can now name the case-supplied inputs its verdict rests on, requires.forcing and requires.static, and a model declares what it consumes in needs_forcing and a new needs_static; one that lacks a required declaration is N/A with reason INCOMPATIBLE. The radiation probe requires rlds and eps, its three reference models declare them, and a regression test runs the positive control with those declarations removed and gets INCOMPATIBLE, not a verdict. Existing probes require nothing, so nothing else changes. The probe README bounds the positive control's residual by 1e-12 rather than quoting 3e-14, which the reviewer measured at 2e-13 under other dependency versions the project allows.
…them Review of the declaration contract found two edges. A model that declares it reads a static.json key the case does not supply used to reach the adapter and die there with a KeyError, reported as FAIL (ERROR); the compatibility check now asks for static keys what it already asked for forcing columns, and such a case is N/A (INCOMPATIBLE) before anything runs. And verify-adapter had started refusing reference_coupled on the radiation probe because the scoring requires rlds and eps to be declared, though the adapter runs on that case perfectly well; the smoke test now asks only whether the model can be fed, and leaves what a verdict requires to run_probe. Both edges carry a regression test, and the probe's archive rows are regenerated by the final harness.
… card The final review found the contract's wording and its use disagreeing: the schema, AGENTS.md and the template defined needs_static as every static.json key an adapter reads, while the three reference manifests declare only the emissivity, as every other manifest declares nothing. The wording now says what the key is for: the static.json keys a probe may require a model to have read. The README's verdict vocabulary names the new cause of INCOMPATIBLE, the smoke-test paragraphs name static inputs beside forcing, the probe template's guidance names the requires keys the branch added, and the cwatm and summa cards, which arrived with the second merge, take the N/A row and the sentence wflow_sbm's card already had. The probe README gives the positive residual as an order of magnitude.
|
@licm13 Thanks for the review, both points are addressed in Compatibility contract. The probe now names the case inputs its verdict rests on, Residual. The README now bounds the positive control by The branch is merged with |
licm13
left a comment
There was a problem hiding this comment.
Hi @xinlan-technology, impressively efficient turnaround — both requested changes have been addressed!
The probe now explicitly declares that its verdict depends on the case-provided rlds and eps, and models must declare those inputs through needs_forcing and needs_static. Models that emit ts and rlus but do not consume those boundary inputs are now classified as N/A (INCOMPATIBLE) rather than receiving a potentially false VIOLATION. The reverse static-input check and the separation between adapter smoke testing and verdict eligibility are also covered by regression tests.
The README now reports the positive-control residual using the stable < 1e-12 bound and explains its dependence on the installed NumPy/Pandas versions.
I rechecked 17c2fbb: the targeted tests pass, the full test suite is clean apart from platform-specific skips, validation reports 21 probes and 37 models, the radiation gate separates all three reference models as declared, and the 36-case margin scan passes.
No further blocking findings. Approved.
Minor non-blocking note: the PR description still says “1e-14 to 1e-13,” while the updated README more accurately says “of order 1e-13 or smaller.” Consider aligning the PR description with the README.
There was a problem hiding this comment.
🟡 Changes recommended
Probe-required diagnostic names are not validated, allowing malformed specifications to pass validation.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds the energy/radiation-consistency probe, supporting diagnostic outputs and required static inputs across the harness.
Changes:
- Implements the radiation identity criterion, generator, and reference controls.
- Extends manifests and protocol with diagnostics and static-input declarations.
- Synchronizes tests, documentation, site content, credits, and archived results.
File summaries
| File | Description |
|---|---|
tests/test_radiation_references.py |
Tests reference-control separation. |
tests/test_radiation_probe.py |
Tests generation and compatibility. |
tests/test_radiation_consistency.py |
Tests criterion behavior and edge cases. |
tests/test_harness.py |
Tests diagnostics and static inputs. |
tests/test_docs_in_sync.py |
Includes diagnostic variables in docs checks. |
templates/probe.template.yaml |
Documents new requirement categories. |
src/hydroturing/spec.py |
Adds diagnostics and static-input metadata. |
src/hydroturing/protocol.py |
Adds diagnostics to adapter requests. |
src/hydroturing/harness.py |
Enforces declared case inputs. |
src/hydroturing/criteria/radiation.py |
Implements the radiation identity. |
src/hydroturing/criteria/__init__.py |
Registers the criterion. |
site/index.html |
Publishes translated probe details. |
scripts/radiation_margins.py |
Reproduces control margins. |
schemas/probe.schema.json |
Defines new probe requirements. |
schemas/model.schema.json |
Defines diagnostics and static needs. |
ROADMAP.md |
Marks the probe merged. |
README.md |
Updates probe and model documentation. |
probes/energy/radiation-consistency/README.md |
Documents scope and validation. |
probes/energy/radiation-consistency/probe.yaml |
Declares the probe specification. |
probes/energy/radiation-consistency/generate.py |
Generates hourly radiation cases. |
models/wflow_sbm/README.md |
Records the new N/A result. |
models/summa/README.md |
Records the new N/A result. |
models/result.csv |
Archives evaluated-model outcomes. |
models/reference_radiative/model.yaml |
Declares the positive control. |
models/reference_radiative/ht_adapter.py |
Implements consistent radiation output. |
models/reference_no_reflection/model.yaml |
Declares the reflection-negative control. |
models/reference_no_reflection/ht_adapter.py |
Omits reflected longwave. |
models/reference_air_emitter/model.yaml |
Declares the temperature-negative control. |
models/reference_air_emitter/ht_adapter.py |
Emits using air temperature. |
models/cwatm/README.md |
Records the new N/A result. |
models/_template/model.yaml |
Documents new model fields. |
docs/writing-a-probe.md |
Documents the criterion and controls. |
docs/adapting-a-model.md |
Documents static-input compatibility. |
CONTRIBUTORS.md |
Credits the probe author. |
CITATION.cff |
Adds the author citation. |
AGENTS.md |
Documents new variables and contracts. |
Review details
- Files reviewed: 36/36 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Upstream merged LISFLOOD (Flood-Lab#52). The conflicts were the README's models table, which takes upstream's lisflood row with its energy wording widened to the four probes that need an energy output, and the archive, which keeps upstream's rows followed by this branch's. lisflood is archived N/A (INCOMPLETE) on energy/radiation-consistency by ht run, and its card counts four N/A probes instead of three. The suite is 21 probes and 38 models.
Copilot's review of Flood-Lab#39 noticed that load_probe took any name under requires while load_model checks every emission against the known variables. A probe with diagnostics: [skin] passed ht validate and would have made every model INCOMPLETE for an output none could declare, and a flux asked for as a state passed as well. The probe now fails to load with the same message load_model gives, and a test tries both mistakes.
lisflood's card had one sentence left that excluded three N/A probes from its standing where the rest of the card, and the archive, say four, and its comparison with the .2 rows now names the radiation probe as the fourth; summa's card said three energy probes supply net radiation where there are now four. The required-name test's docstring and the comment beside the check say what they reject, an unknown output and a variable in the wrong category, rather than claiming both would have made every model INCOMPLETE: a flux asked for as a state is met by the flux.
|
@licm13 Thanks for the approval. Updated in ec1d1a8: merged main at 12cbaaa, archived LISFLOOD as N/A on this probe, and added validation for unknown or misclassified required outputs. The PR description is now aligned with the README. All 488 tests, the full gate, and the 36-case margin scan pass locally. |
licm13
left a comment
There was a problem hiding this comment.
Rechecked the follow-up through ec1d1a8. The new validation correctly rejects unknown required outputs and variables placed in the wrong output category, with regression tests covering both cases. The PR description is also now aligned with the README’s stable residual bound.
No new blocking findings; my approval stands.
|
@xinlan-technology Thanks for this, and for turning @licm13's review around so quickly. The probe is in good shape. I merged it with current 1.
|
@chrimerss Thanks for the detailed review. Addressed in bec106f: separated mandatory needs_* from optional uses_* inputs, with regression tests. All three references also pass energy/surface-energy-closure. The docs now clarify row-timestamp sampling and the emissivity, blackbody and linearization limits. I also fixed overflow in the mean-residual diagnostic; scoring formulas and tolerances are unchanged. All 500 tests, the full gate and the 36-case margin scan pass locally. |
Closes #22
What this probe asserts
For a uniform, opaque, snow-free gray surface with the emissivity supplied in
static.json, the surface temperature a model reports and the upward longwave it reports must describe one surface at every hourly instant:The bound is 0.5 percent of the model's own reported upward flux, not net radiation: two outputs of one model are held to one equation, and for a 290 K surface under a 300 W m-2 sky with
eps0.98, the reflected sky is about 1.5 percent of the upward flux, so a model that drops it passes at 5 percent and fails here. Every scored step must pass, so nothing cancels across hours; interval means are out of scope because mean(T^4) is not mean(T)^4. Net radiation stays a prescribed forcing and is not reconciled with the longwave components, as the proposal scoped. Passing establishes consistency between two outputs, not temperature accuracy.The three interface questions in the proposal are answered as follows.
tsis declared under a new optionaldiagnosticsoutput category (schemas,spec.py,protocol.py), so a temperature is required like a flux but never summed into a water budget as a state would be. Emissivity comes only fromstatic.json, drawn per seed in [0.95, 0.99] and held fixed.rldsis a generator column likern.needs_forcing/needs_static declare mandatory inputs; uses_forcing/uses_static declare inputs consumed whenever supplied. Probe requirements accept either declaration. The three references use optional declarations with documented defaults, so they also remain compatible with other suitable probes.
Discrimination
reference_radiativereference_air_emitterradiative_identityreference_no_reflectionradiative_identityreference_radiativeisreference_coupledwith a skin: the temperature is diagnosed from the sensible heat flux through a fixed bulk conductance, and the upward longwave is that skin's emission plus the reflected sky. Its water and energy columns arereference_coupled's exactly. The two negatives are copies of it with one line changed, emitting at the air temperature or dropping the reflected sky, andtests/test_radiation_references.pypins that they differ from it inrlusalone.Margins, measured through the real adapter path by
scripts/radiation_margins.pyand judged seed by seed: on the five gate seeds, twenty additional seeds, one drawn seed near the emissivity ceiling, and the gate seeds withepsforced to 0.99 and 0.95, the positive control's worst step is below 1e-12 of the tolerance (rounding, of order 1e-13 or smaller, varying with the numpy and pandas installed) and each negative control's worst step is at least 1.95 times it (minimum over seeds of the per-seed maximum; the floor is the forced 0.99 group).reference_air_emitterviolates 679 to 690 of 720 steps per seed,reference_no_reflectionall 720. The probe README carries the table.Validation
upstream/mainat12cbaaa(N/A verdicts, summa, cwatm, lisflood, mass/human-abstraction, LF line endings):ht validate21 probes, 38 models.pytest -q: 500 passed locally, including the line-ending check; the GitHub run on the new head awaits a maintainer's approval.ht gate: passes locally for all 21 probes; the new probe's gate is 15 adapter invocations in about a second.ht verify-adapterpasses for the three reference models, 768 rows each.scripts/radiation_margins.py: 36 cases, all separated as declared, exit 0.ht run --gate-seeds --csv models/result.csvarchived the nine evaluated models asN/A(INCOMPLETE) for missingrlusandts, so no standing changes. No physical-violation claim is made for any of them.rlusandtsrows), the probe-writing guide and the three-language site, flowchart included, are synchronised;tests/test_docs_in_sync.pypasses.Checklist
acceptedproposal issue and this PR closes itauthorsinprobe.yamlnames every author withname,affiliationandorcid, matching the proposal; CONTRIBUTORS.md and CITATION.cff are updatedht validatepassesht gate --probe energy/radiation-consistencypassesprobe.yamlradiative_identitycatches both