Statewide benchmarking evidence

[AritraDey-Dev]

Adds the curated literature evidence behind ccmmf/organization#164 and ccmmf/organization#226, exported from the benchmarking workbook.

62 evidence rows from 26 sources across the Table 6 practice by outcome cells. Every row carries the verbatim sentence, DOI, Zotero collection and source locator, so any value can be retraced without going back to the paper.

Files

file	source tab	contents
extracted_evidence.csv	extracted_evidence	62 rows, one per extracted finding
normalised_targets.csv	normalised_targets	62 rows, one per target once multi value cells are split
audited_direction_matrix_v2.csv	audited_matrix	all 18 cells classified verified, variable, disputed or unsupported
coverage.csv	coverage	which cells have evidence
source_audit.csv, carb_crosscheck.csv	as named	provenance and the CARB inventory cross check
data_processing.csv	data processing	van Kessel figure extractions with LRR and approximate LRR SEs
model_vs_evidence.csv	derived	18 cell comparison scaffold, evidence side filled and model columns empty
data_ingest_report.md		uncertainty audit and conversion record

Uncertainty handling

Reported values are preserved exactly and derived quantities sit in their own columns. Every uncertainty states its type, its scale and units, and the quantity it applies to, via val_unc_type, val_unc_value, val_unc_units, val_unc_applies_to and val_unc_derivation. lrr_se is kept separate so a treatment arm's dispersion cannot be mistaken for the contrast's.

Four corrections, all written up in data_ingest_report.md:

Poeplau and Don was recorded as standard error, but the paper says "Errors given in the text are 95% confidence intervals". It is now CI95_halfwidth with the SE derived as half width over 1.96.

Snyder had 0.57 sitting next to the LRR, implying it was the LRR's uncertainty. It is the standard deviation of the no till arm's flux. Both arms now carry their own dispersion, and no LRR SE is derivable because the source reports no n per arm.

Li and Liu are approximate SEs from confidence intervals, labelled derived_from_CI95. Li's bounds were converted to LRR before dividing by 3.92, since symmetry is more defensible on the log scale.

van Kessel overall previously carried an LRR of 0.0 taken from "no significant change", which establishes neither a zero effect nor an SE. Removed.

Anthony and Six were checked against the source PDFs and are genuinely standard errors.

What the evidence can actually support

	count
cells with a usable contrast	10 of 18
cells testable for direction and magnitude	3
cells with no usable contrast at all	8

Five of the 34 N2O rows are not N2O. Three are combined NO plus N2O emission factors and two are soil NO. Poeplau's steady state is a RothC projection rather than an observation, and Li 2024 contradicts itself between its abstract and conclusions. All of these are kept and marked not_viable with the reason recorded, rather than deleted.

[dlebauer]

I believe that we have discussed many of these points in our meeting today.

But a few requests

each CSV should have clear provenance files do not have matching worksheets in the associated spreadsheet

many rows in model_vs_evidence.csv combine results that may be best left unaggregated. Check that aggregation is appropriate, and when not, retain individual results. For example:

• flooding effect on N2O combines the effect of specialty-crop deficit-irrigation and rice flooding/rewetting treatments - this results in a “decrease” benchmark even though the rice contrast is an increase.
• tillage combines three estimates from van Kessel: overall estimate and two dry-climate subgroups; just use the most relevant (<10 y dry climate) result
• others, e.g. check if effect of N-fertilization on N2O is a viable representation of heterogeneous experiments

[divine7022]

@AritraDey-Dev on N fertilization cell. Could you add the crop specific slopes as rows in summarized_targets.csv rather than leaving them in the note and operator text? Right now upland grains 0.0017, rice 0.001 and N fixing 0.018 only exist as prose, so nothing can read them programmatically and every consumer has to hand copy them, which is how numbers drift.

The bigger issue is that the machine readable center still holds the 0.0027 all crop mean, which is the value you told me not to use because it is pulled up by the N fixers. So anything that filters the table the normal way, on use and center and spread, picks up exactly the number we agreed is wrong for a row crop panel.

This is the same structural limit that hit rice CH4: the table is keyed practice x outcome, so it cannot express a subclass. Rice CH4 has the same problem, our two dry arm should be scoring against Jiang's two event class at -0.6255 se 0.170, but the row can only carry the single event class. A crop or subclass key, or one row per class, fixes both cases and lets the scorer read them with the same filter it uses for every other cell.

On my side I only keep the mapping from our PFTs to your crop classes, no values, so whatever you deliver is the single source

[AritraDey-Dev]

got the supplement, Table S3 has SEM and n for every crop group so those rows are fittable now rather than sign checks. two corrections from it, rice is 0.0009 not 0.001 and N fixers 0.0181 not 0.018. the published SEM for the aggregate is 0.00085, which matches what I had derived from the CI.

table S4 changes what I told you though. N fixers differ from every other group but upland grain, rice and forage do not differ from each other, P of 0.193, 0.231 and 0.057. So the four way split is not supported. I added a non N fixing crops row at 0.0018, SEM 0.00048, n 221, and that is the one to use for the row crop panel with N fixers at 0.0181 for the alfalfa sites.

dataset S1 cannot check any of this. delta EF is filled for 233 site years which matches the table, but CropType only for 78, so the group means cannot be rebuilt from the released data. I am using the published table as printed.

[divine7022]

thanks!
two small things the README missed, it still says the crop class rows have no interval and are sign checks, which table S3 changed, and the subclass table doesn't list non N fixing crops

on dataset S1, agreed on using the published table as printed. worth a line in the note that the group means can't be rebuilt from the released data

[AritraDey-Dev]

thanks! always miss updating the doc 😅 .Pushed now.