[systemfsoftware-maker]

Summary

Main's Release gate (run 37892133106 on 21fe21ef, #74's verdict job) failed with exit 1 (VerdictFail): planned mutant(s) missing from the shard reports naming all 114 planned mutants. The one shard's mutation-stream.jsonl holds all 114 (37 Killed, 6 Timeout, 71 CompileError, verdict score 100). The verdict read the wrong directory. This PR gives the shard reports one layout for any shard count.

Root cause

Two layout facts, both read from source and from the run itself:

1. download-artifact flattens a lone match. With pattern: mutation-shard-* and merge-multiple: false, download-artifact v6's run extracts into path/<name> only when more than one artifact matches; a single match goes straight into path. The verdict log shows it: Starting download of artifact to: /home/runner/work/xstate/xstate/.cache/shard-reports, with no mutation-shard-0/. The plan had one shard, so the jq's .cache/shard-reports/mutation-shard-0/... did not exist. With two or more shards the old jq would have resolved, which is why only a one-shard plan exposed it.
2. The plan's project paths climb out of the shard's index directory. stryker plan writes project paths relative to the plan file's directory, so a plan in .cache/ records ../packages/xstate. stryker-js 17's shard runner writes to join(<planDir>/reports/shards/<index>, project), which normalises to .cache/reports/shards/packages/xstate: no <index> directory, and every shard writes the same path. Merge applies the same join(dir, project), so the missing index cancelled out, but the per-shard output was never unique.

upload-artifact did not trim anything: the old upload had two globs (.cache/reports/shards/** and packages/*/reports/mutation-report.*), whose least common ancestor is the workspace root, so the artifact kept full paths. That matches the downloaded mutation-shard-0.

Canonical layout

• The plan lives at the repository root as stryker-plan.json (gitignored). Project paths are packages/<name>.
• Shard k writes reports/shards/<k>/packages/<name>/ (stryker-js's default --out).
• Each shard uploads only reports/shards/. A single upload path is the artifact's root, so artifact mutation-shard-<job> holds <k>/packages/<name>/.
• The verdict downloads every mutation-shard-* with merge-multiple: true into .cache/shard-reports, giving .cache/shard-reports/<k>/packages/<name>/ for one shard or many.
• The jq is .shards[] | ".cache/shard-reports/\(.index)", in plan order. It no longer needs the matrix lookup, because the artifact name no longer decides the directory.
• The per-shard HTML/JSON report moves to its own mutation-report-shard-<job> artifact, outside the mutation-shard-* pattern.

Proof against the real artifacts of run 37892133106

Source: the conductor's copies of stryker-plan and mutation-shard-0 (xstate.worktrees/.rulings/artifacts-37892133106/, laid out as gh run download -n <name> writes them). Every file matches my own gh run download 37892133106 by sha256. Each tree below is a scratch copy; the originals are untouched. Only stryker merge and stryker gate ran (inside the repo's Nix dev shell); no mutants ran.

1. Main's verdict command over CI's download layout reproduces the production failure. A pattern download that matches one artifact extracts straight into path (download-artifact v6, the artifacts.length === 1 case), so the tree is .cache/stryker-plan.json plus the shard's files directly under .cache/shard-reports/. The failure line is byte-identical to the CI log:

exit 1 (VerdictFail): planned mutant(s) missing from the shard reports: 03b7585c575b9a84, 0522ec21e015a26a, 06e55247674f863f, 09815aaf260c3253, 0ab04bec063b8809, … (all 114 ids) …[truncated] — see --reportFile or the verdict envelope on stdout
exit=1

2. This PR's verdict command over the same bytes in the layout this PR's download steps produce (stryker-plan to ., mutation-shard-* with merge-multiple into .cache/shard-reports) is also red. These artifacts came from main's plan and upload steps, not this PR's: the plan says ../packages/xstate (written to .cache/), and the shard upload is rooted at the repo, so it holds .cache/reports/shards/... instead of 1/packages/xstate/.... The verdict never sees a matching tree:

exit 1 (VerdictFail): planned mutant(s) missing from the shard reports: 03b7585c575b9a84, 0522ec21e015a26a, 06e55247674f863f, 09815aaf260c3253, 0ab04bec063b8809, … (all 114 ids) …[truncated] — see --reportFile or the verdict envelope on stdout
exit=1

3. The same bytes placed as this PR's plan and upload steps write them: exit 0 with the merged counts. Two changes. The plan sits at the root with its one project path as packages/xstate, which is what stryker plan --out stryker-plan.json writes; the 114 ids and the shard index are unchanged. The shard's packages/ directory (byte-identical) goes under .cache/shard-reports/1/, where merge-multiple puts shard 1's reports/shards/ upload. The command is this head's verdict step, byte-identical to the workflow:

set -euo pipefail
printf '{"schemaVersion":1,"survivors":[]}\n' > .cache/no-survivors.json
jq -r '.shards[] | ".cache/shard-reports/\(.index)"' stryker-plan.json > .cache/shard-dirs.txt
mapfile -t shard_dirs < .cache/shard-dirs.txt
./node_modules/.bin/stryker merge --plan stryker-plan.json --out reports/mutation "${shard_dirs[@]}"
./node_modules/.bin/stryker gate --baseline .cache/no-survivors.json

[08:17:27.117] INFO (#1): stryker merge: merged 114 mutant verdict(s) into /tmp/gate195/pr-layout/reports/mutation
[08:17:27.332] INFO (#1): stryker gate: 0 mutant(s) unchecked (in scope with no verdict)
exit=0

The merged reports/mutation/mutation.json holds 114 mutants: 37 Killed, 6 Timeout, 0 Survived, 0 NoCoverage (71 CompileError).

4. Sabotage: the mutant line for 7bd13569d6302fbb deleted from a copy of tree 3's stream: exit 1, naming the id.

exit 1 (VerdictFail): planned mutant(s) missing from the shard reports: 7bd13569d6302fbb — see --reportFile or the verdict envelope on stdout
exit=1

5. Gate check: in a copy of tree 3, one Killed mutant (48a4b106fb2a618c) changed to Survived: exit 1, naming the id.

[08:15:49.051] INFO (#1): stryker merge: merged 114 mutant verdict(s) into /tmp/gate37892133106/verdict-survivor/reports/mutation
[08:15:49.338] INFO (#1): accept the new survivors with `stryker gate --update-baseline`, or kill them before the next run
exit 1 (VerdictFail): stryker gate: 1 new survivor(s) absent from the committed baseline:
  packages/xstate/src/assert.ts:15 48a4b106fb2a618c
kill the new survivors, or accept them with `stryker gate --update-baseline` — see --reportFile or the verdict envelope on stdout

exit=1

Rows 2 and 3 are the limit of what the old run can prove: no artifact in the new layout exists until this merges. The first Release gate run on main after merge is the end-to-end check, and I will read it.

Sandbox proof (predicate 3)

sandbox-proofs/release-gate.test.ts now builds the verdict job's workspace from the workflow itself: the plan where the plan step writes it, each shard's stream at stryker-js's SHARD_OUT_MARKER (read from the installed dist), the upload root of the mutation-shard-* upload, and download-artifact's extraction rule for the verdict's download step. It runs the verdict step in the sandbox with a recording stryker stub, resolves each shard directory merge receives the way merge does (join(dir, project)), and fails naming every planned mutant whose stream isn't there. It also requires merge's --out to be the directory of the gate's GATE_REPORT_FILE, read from the same dist. It covers one shard and two shards listed in reverse matrix order. The old case pinned the jq's literal output, which is why a layout drift passed it; that case is gone.

On main's workflow (this test file only, in a scratch worktree removed afterwards), the one-shard case fails with the production error:

each release-gate planner step runs in the sandbox on only its declared hosts, and the gate logs every XS1 entry it accepted ... ok
the release-gate mutation step passes stryker every variable its main-CI guard reads, and never the local override ... ok
the release-gate verdict merges, for one shard, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
the release-gate verdict merges, for two shards whose matrix lists them in reverse plan order, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... ok
a failed shard-report merge fails the verdict step before the gate reads a merged report ... ok
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111 (merge merge --plan .cache/stryker-plan.json --out reports/mutation .cache/shard-reports/mutation-shard-0/.cache/reports/shards/1)
FAILED | 4 passed | 1 failed

On this head:

each release-gate planner step runs in the sandbox on only its declared hosts, and the gate logs every XS1 entry it accepted ... ok
the release-gate mutation step passes stryker every variable its main-CI guard reads, and never the local override ... ok
the release-gate verdict merges, for one shard, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... ok
the release-gate verdict merges, for two shards whose matrix lists them in reverse plan order, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... ok
a failed shard-report merge fails the verdict step before the gate reads a merged report ... ok
ok | 5 passed | 0 failed

Sabotages of this head's workflow, each in a scratch worktree removed afterwards:
merge-multiple: true removed:

the release-gate verdict merges, for two shards whose matrix lists them in reverse plan order, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111, 2222222222222222 (merge merge --plan stryker-plan.json --out reports/mutation .cache/shard-reports/1 .cache/shard-reports/2)
FAILED | 4 passed | 1 failed

shard upload back to the two globs:

the release-gate verdict merges, for one shard, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
the release-gate verdict merges, for two shards whose matrix lists them in reverse plan order, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111 (merge merge --plan stryker-plan.json --out reports/mutation .cache/shard-reports/1)
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111, 2222222222222222 (merge merge --plan stryker-plan.json --out reports/mutation .cache/shard-reports/1 .cache/shard-reports/2)
FAILED | 3 passed | 2 failed

jq dir .cache/shard-reports/mutation-shard-<index>:

the release-gate verdict merges, for one shard, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
the release-gate verdict merges, for two shards whose matrix lists them in reverse plan order, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111 (merge merge --plan stryker-plan.json --out reports/mutation .cache/shard-reports/mutation-shard-1)
error: Error: planned mutant(s) missing from the shard reports: 1111111111111111, 2222222222222222 (merge merge --plan stryker-plan.json --out reports/mutation .cache/shard-reports/mutation-shard-1 .cache/shard-reports/mutation-shard-2)
FAILED | 3 passed | 2 failed

--out reports/merged (control on the same scratch base: ok | 8 passed):

the release-gate verdict merges, for one shard, each shard's report directory where the shard job wrote and uploaded it, and gates the merged report with no accepted survivor ... FAILED
error: Error: merge wrote reports/merged, but the gate reads reports/mutation
error: Error: merge wrote reports/merged, but the gate reads reports/mutation
FAILED | 6 passed (10 steps) | 2 failed

Gates on f1edb29

pnpm format:check 0, deno check and deno lint on sandbox-proofs/release-gate.test.ts 0, pnpm test:sandbox 0 (8 passed). The solution doc docs/solutions/workflow-issues/sharded-mutation-gate-without-verdict.md is updated to the new layout, and its validator exits 0. No package changes, so no changeset.

Review

A local adversarial review of fdff4f1a5 found two P3s and no P0 or P1. (1) After the simplify commit the proof no longer checked merge's --out, the only link to the gate's hard-coded report path. That is fixed in f1edb2921, and the --out sabotage above is red. (2) A shard that uploads no artifact (if-no-files-found: ignore) is not modelled. That is advisory and left as is: production fails closed there, because merge raises ShardReportGap for a planned mutant with no stream, so it can't pass vacuously.

Declared (CONST-W3 / CONST-E9)

.github/workflows/release-gate.yml and its sandbox proof are judgment surfaces. They are edited here at the conductor's direction (ruling cycle 190), which owns them. This PR grades none of my capability work. timeoutMS is unchanged.

No local mutation run.

[kiro-systemf]

Fresh review PASS on f1edb29; 7/7; 0 threads; hunt clean; real-artifact proof in body.