Repository navigation
rules: pin the replay test package whole; flaky entries run three times; adr/replay-every-sample - #253
Merged
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
n-sviridenko
force-pushed
the
fix/replay-pin
branch
from
October 6, 2026 20:30
77fcf52 to
9d82a1d
Compare
This was referenced Oct 6, 2026
…ed replay test, and fails when none is green claude, codex and cursor generated_replay_test.go run a flaky: entry through replayUntilGreen (flakyRuns = 3) and fail it when no run is green, instead of passing it on one green run or skipping it. Each package has a flaky_replay_test.go for replayUntilGreen. Sloprail-Cites-User: A replay exception's reason may not move to a weaker category (flaky < untriaged < triaged), and a flaky entry runs 3 times and fails if never green; it is never skipped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah
…e, the replay files are read from the syntax tree, and the generated tests are pinned to canonical copies tools/replaycheck reads each mock's notReplaying list and generated replay test from the syntax tree and type information instead of their text, and compares replayUntilGreen and TestGeneratedReplay with the canonical copies in the rule's folder (one per mock: claude, codex, cursor), so a flaky: entry is still run three times and failed when never green. A mock's replay_allowlist_test.go and generated_replay_test.go are no longer deleted, renamed or moved: that would end the replay without removing an entry. A reason that starts with none of adapter:, mock gap:, untriaged: or flaky: is still refused. Cases for the package pin, the text a grep would miss and the map outside its file; the move cases go, as a move is now refused. Sloprail-Cites-User: The replay exception list may only shrink; adding an entry fails CI Sloprail-Cites-User: A replay exception's reason may not move to a weaker category (flaky < untriaged < triaged), and a flaky entry runs 3 times and fails if never green; it is never skipped. Sloprail-Cites-User: A replay exception's reason must start with one of adapter:, mock gap:, untriaged: or flaky:; a reason with none of these is refused. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah
…(linked to the replay-exceptions-only-shrink rule that pins the replay tests) The decision is the user's own words; the pin on generated_replay_test.go is what keeps a mock's replay test from being weakened to fewer samples. Sloprail-Cites-User: A mock's replay command replays every captured sample of a recorded run, and "replays green" means all of them do; a mock that replays only the newest sample is a gap to close, not a convention. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah
…cOS and run three times on Linux; any other -skip of a replay is refused The workflows and a mock's Makefile are judged too: a go test -skip there may name only TestGeneratedReplay/all-hooks-close-(first|second) (directly, or through TIMED_REPLAYS defined as exactly that), and a workflow that skips them must also run them on their own with -count=3. Any other skip, a widened TIMED_REPLAYS, or a skip with no run is refused. Cases for both directions. Sloprail-Cites-User: Fail, never skip Sloprail-Cites-User: A replay exception's reason may not move to a weaker category (flaky < untriaged < triaged), and a flaky entry runs 3 times and fails if never green; it is never skipped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah
…cision and requires Linux, three runs and every PR; adr/replay-every-sample drops its link to the pin rule adr/replay-exceptions-only-shrink states the timed-pair exception in the user's words (macOS e2e jobs skip it; the Linux timed-replays job runs it three times on every PR). The rule now also requires the workflow to be triggered by pull_request and the job that runs the timed pair to be on ubuntu, with cases for both. adr/replay-every-sample goes back to file-guard/adr-conformance: the pin rule does not check which samples a replay covers. Sloprail-Cites-User: Timing-sensitive replays whose recorded gaps a CI runner's timers cannot keep may run on Linux only, three times each, as long as they run on every PR. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah
…in the user's words adr-grounded refused the bullet for deciding more than the cited words (it named the tests, the platform and the job). It now says only what the user decided: every other replay skip is refused, and timing-sensitive replays may run on Linux only, three times each, on every PR. The names stay in the rule's script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah Sloprail-Cites-User: Timing-sensitive replays whose recorded gaps a CI runner's timers cannot keep may run on Linux only, three times each, as long as they run on every PR. Sloprail-Cites-User: Yes, ban other skips
n-sviridenko
force-pushed
the
fix/replay-pin
branch
from
October 7, 2026 06:23
a6b3f91 to
a36e1cc
Compare
This was referenced Oct 7, 2026
replay-exceptions-only-shrink: a build constraint on the replay test file excludes it unnoticed
#267
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three concerns, three commits.
flaky:entry throughreplayUntilGreen(3 runs) and fail it when never green (the user: "a flaky entry runs 3 times and fails if never green; it is never skipped").tools/replaycheckreads the lists and generated tests from the syntax tree and type information instead of text;replayUntilGreenandTestGeneratedReplayare pinned to canonical copies (one per mock, including cursor); a mock'sreplay_allowlist_test.goandgenerated_replay_test.goare no longer deleted, renamed or moved. A move of a list used to be permitted and compared with its old self: it is refused now.Not run yet: sr-checks, held until #239 merges (claude's generated test and list change there, so the canonical copy is regenerated after a rebase).
🤖 Generated with Claude Code
https://claude.ai/code/session_01RakB4JKMC2VyTt7wPgFSah