Skip to content

ci: add cargo-tarpaulin coverage + cargo-crap complexity gates - #1

Open
bkataru wants to merge 2 commits into
mainfrom
ci/coverage-crap-gates
Open

bkataru wants to merge 2 commits into
mainfrom
ci/coverage-crap-gates

Conversation

@bkataru

@bkataru bkataru commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Adds the estate-standard two gates to this repo in one new job, fed by ONE instrumented suite run: cargo-tarpaulin (coverage floor) and cargo-crap (complexity ceiling), both versions pinned (tarpaulin 0.37.5, cargo-crap 0.6.1). Gate state: required-intended — forge is a PUBLIC repo, so ubuntu-latest works and branch protection IS available on this plan; the intent is for this gate to become a required check once green (operator wires it after merge).

What was added

  • New workflow .github/workflows/coverage-crap.yml (name coverage-crap). It is deliberately a SEPARATE file: .github/workflows/ci.yml is generated by forge_ci/gh-workflow-gen and hand edits there are lost on the next regeneration. The generated file is untouched.
  • One job coverage-crap: checkout → toolchain (actions-rust-lang/setup-rust-toolchain@v1, the repo's own convention, toolchain: stable + components: llvm-tools-preview) → arduino/setup-protoc@v3 (the repo needs protoc for tonic/prost in forge_app/forge_infra/forge_repo) → lockfile assertion → pinned tarball tool install with in-script PATH export + version assertions → coverage gate → CRAP gate → shell-only run summary.
  • The coverage run IS the test run: tarpaulin exits non-zero on any test failure, so the job also fails on any regression, not just on coverage drop.

Measured first-run numbers (base da9a026)

Local measurement, cargo-tarpaulin 0.37.5 --engine Ptrace --workspace --locked --out Lcov -j 2, toolchain pinned to the repo's rust-toolchain.toml (1.92), 21-crate workspace: CI verification (llvm engine) followed in the gate job itself — both runs below.

  • Coverage: 52.01% (9463/18194 lines). Suite green: 1855 tests passed, 0 failed.
  • CRAP (cargo-crap 0.6.1 on the same lcov, --workspace): 2249 functions scored; worst entries were UI::handle_subcommands 15006 (CC 122) and UI::on_command 4556 (CC 67) — both in crates/forge_main/src/ui.rs — then Renderer::render_event 3660 (CC 60, cov 0%, crates/forge_markdown_stream/src/renderer.rs:125).

Thresholds + rationale

  • COVERAGE_FLOOR: "47" — set from the first CI run of this very gate (llvm engine): measured 50.45% (8709/17263 lines) − 3pp estate margin. (Local ptrace measurement was 52.01%; the engine counts a slightly different line set.) Never ships a threshold the repo doesn't pass today.
  • CRAP_THRESHOLD: "4030" — measured max 3660 (Renderer::render_event) + ~10% headroom, after the one documented exclude below.
  • Documented CRAP exclude: crates/forge_main/src/ui.rs (--exclude 'src/ui.rs', crate-root-relative glob under --workspace; verified it is the only ui.rs in the tree). Rationale: a 3533-line TUI dispatch/event-loop surface, 93 scored functions all at 0% coverage by design (no TTY test harness drives the TUI). Its worst entry forces a ~16507 ceiling that measures nothing, so it is excluded with the same posture as other repos excluding their entry points; the coverage floor still accounts for the file. Glob note: an initial crates/forge_main/src/ui.rs pattern silently matched nothing — the pattern must be crate-root-relative.
  • Scoping check: all Rust lives in crates/* (21 workspace members); no orphan crates, no spike/experiment dirs, no non-workspace Rust (benchmarks/ and shell-plugin/ are TS/shell). benchmarks/, scripts/, templates/ contain no Rust.

Gate proof (both directions, identical exclude set)

  • cargo crap --lcov lcov.info --workspace --exclude 'src/ui.rs' --threshold 4030 --fail-above → exit 0 (passes).
  • Same command with --threshold 3000 → exit 1, "1/2156 function(s) exceed CRAP threshold 3000" (Renderer::render_event 3660). The gate can discriminate, not just pass.

Runner choice + why

ubuntu-latest (public repo, follows the repo's existing convention; the generated ci.yml also uses ubuntu-latest). Engine is --engine Llvm for estate-wide consistency with the fleet runs even though ptrace would work here; the ptrace engine is blocked in the fleet's sysbox-confined runner containers (measured on dirmacs/skywalkers run 38064108686).

DB needs

None — no Postgres/Redis/network-dependent tests (checked for DATABASE_URL/TEST_DATABASE_URL consumers: none). Tests need protoc (installed via arduino/setup-protoc@v3, same as the existing build job).

Caveats

  • Both numbers are now measured on the engine CI runs (--engine Llvm): coverage 50.45%, CRAP gate green vs ceiling 4030 (two green runs: 38069680519 at floor 49, 38070354043 at floor 47). The floor was re-based after the first run showed the llvm line set differs slightly from ptrace (17263 vs 18194 lines).
  • Job timeout is 90 minutes and test parallelism is capped (-j 4) — 21 crates on a 4-core runner is the dominant cost.
  • The summary step parses crap.json with shell only (fleet runner containers carry no python3).
  • cargo install-free: both tools install from pinned release tarballs, versions asserted in-script.
  • Local measurement needed 12G+ for the target dir on a disk that was 100% full; measurement CARGO_HOME/target were relocated to and deleted afterwards.

Ratchet rule

  • Coverage: floor rises +2pp per merged PR that raises measured coverage (target: approach 70%+); a PR that lowers coverage below the current measured level is expected to come with tests restoring it.
  • CRAP: ceiling lowered on schedule / immediately on any fail-regression once no scored function exceeds 30; the ui.rs exclusion is revisited when a TTY-driven test harness exists.

Separate workflow file (the generated ci.yml is produced by forge_ci and
hand edits there are lost on regeneration). One job, one instrumented
suite run, two gates: tarpaulin --engine Llvm gates line coverage at a
measured floor, and cargo-crap scores CRAP from the same run's lcov.

Measured on this branch's base (da9a026), tarpaulin 0.37.5 / ptrace,
21-crate workspace, 1855 tests passing: coverage 52.01% (9463/18194)
-> floor 49 (3pp engine-variance margin); worst CRAP after the
documented crates/forge_main/src/ui.rs exclusion is 3660
(Renderer::render_event) -> ceiling 4030 (+10% headroom).
@autofix-troubleshooter

Copy link
Copy Markdown

Hi! I'm the autofix logoautofix.ci troubleshooter bot.

It looks like you correctly set up a CI job that uses the autofix.ci GitHub Action, but the autofix.ci GitHub App has not been installed for this repository. This means that autofix.ci unfortunately does not have the permissions to fix this pull request. If you are the repository owner, please install the app and then restart the CI workflow! 😃

First CI run of this gate (llvm engine) measured 50.45% (8709/17263).
Align the floor with the intended measured -3pp margin on the engine CI
actually runs; the ptrace measurement (52.01%) had understated the margin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants