Repository navigation
Add opt-in token CUSUM diagnostics to summary scans - #99
Merged
Merged
Conversation
Compose summary coverage and source-quality checks with bounded robust scoring. Add exact contributor interval arithmetic and pin text/JSON output with the feature enabled and disabled. Signed-off-by: Vijay <evijay@gmail.com>
kwisatzh
marked this pull request as ready for review
November 16, 2026 09:24
Pin a candidate whose chronological replay exceeds its target while seeded block replay passes. The regression fails when the chronological-rate safeguard is removed; production code is unchanged. Signed-off-by: Vijay <evijay@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
scan --token-cusumto show changes in observed input-plus-output token totals alongside the existing scan report. The flag is opt-in; the legacy scan findings and feature-off text/JSON outputs retain their existing behavior.Invariants And Tests
Missing or inconsistent usage remains unavailable. Missing producers, partial intervals, key/accounting changes and unsupported precision cannot become scored zeroes. The tests cover per-source no-attempt tokens, reported/request count mismatch, missing/invalid/conflicting usage, subset violations, missing quality counters, dedup coverage and restart epochs. Replacement tests exercise both input orders and retain regression refusals.
Text/JSON goldens cover the flag on and off. The OFF goldens were independently compared with stdout from the committed base. Numeric tests cover planted changes, causal references, overflow, exact interval arithmetic and share adjustment. The output/error sentinel checks use synthetic inputs.
The required pre-merge regression pins a synthetic case with 4 chronological alarms in 16 scored buckets (above a 1-in-5 target) and 5 block-resampled alarms in 32 buckets (within it). Calibration must reject this candidate. Removing only the chronological-rate safeguard makes the new test fail at that rejection assertion.
Verification
Final candidate:
9bb816170782a245033c45ebeebb8e9df259f8b2, signed and signed off, with Vijay as author and committer. Its follow-up to reviewedb5a7c23changes only the required regression test; production code and goldens are unchanged. The following local checks passed again on the final exported head using Go 1.26.11:Formatting, gocyclo on the new production files, govulncheck (
No vulnerabilities found), built-CLI goldens and source-quality predicate mutations also passed. Staticcheck is clean on change, compare and cli. Its repository-wide run reports the same seven pre-existing style findings as the committed base: S1038 in a scenario test and ST1005 in the existing operator-example/Langfuse errors. No unrelated files were changed to clear them.The first full test attempt used a private
umask 077, which changed an existing permission-test fixture from 0644 to 0600. The same failure was reproduced on the unchanged base; the runner now uses 022 for child checks while retaining private outer evidence directories. Credential code is unchanged. An initial offline tool-metadata lookup was retried with the pinned public module metadata available; product dependencies are unchanged.The final test-only head passed Linux/macOS CI, all four platform build-verification jobs, signatures, DCO, trusted API tests and CodeQL. No local Docker workload was run, and no release is requested.
Limits
The CLI diagnostic is uncalibrated and does not issue CUSUM alerts. It measures observed tokens, not cost; exact recorded-spend reader integration, contributor ranking and held-out detection calibration remain follow-on work. Complete export intervals describe the supplied observations, not proof that every physical request was captured.
Pre-merge review approved merge once this regression was added, without a second round. The test and final-head green CI conditions are met. The ordinary, exact-head squash requests returned HTTP 502; the PR remains open and no protection was bypassed. This increment stays unreleased.