Repository navigation
Read the acceptance signal from a repository - #62
Merged
Merged
Conversation
The eval set scores the reviewer against defects chosen for it. This reads the other end: a repository where the action has run, and what reviewers did with the comments once they were posted. Reactions, replies, thread resolution and the share of comments nobody touched are reported side by side rather than averaged. A resolved thread means the conversation ended, which happens both when a finding is fixed and when it is dismissed; one number combining them would say nothing. Read-only, and it calls no model. The GraphQL query is written around GitHub's node limit. The first version asked for 50 pulls x 100 threads x 100 comments, which is 505,050 possible nodes against a ceiling of 500,000 — counted from the page sizes alone, so it was refused against this repository while it held no threads at all. The page sizes are now named constants with the arithmetic beside them and a test on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"What I would do next" listed learning from reactions and resolved threads. The script now exists, so the entry says what it is and that it has nothing to report yet — which is true and is the more useful thing to say. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last open item of Phase 4 in the v3 plan — the optional feedback
signal — and leaves the plan with only the measurements that need API keys.
What this adds
python -m evals.feedback --repo owner/name(ormake feedback REPO=…) readsa repository the action has already run on and reports what reviewers did with
its comments:
Broken down by severity, by category, and by month.
Read-only and free: it calls no model, and needs only a token with read
access.
Why the signals stay separate
There is no combined "accuracy" number on purpose. A resolved thread means the
conversation ended — a reviewer resolves a thread both when they fix the
problem and when they decide the bot was wrong. Averaged together the two
signals produce a figure with no meaning. The row that carries the most
information is the last one: a comment nobody touched cost the reviewer
attention and returned nothing, which is the failure this whole project is
about.
A bug the first run found
Against the real repository, the GraphQL pass failed:
GitHub counts the nodes a query could return from the page sizes alone, so
50 pulls × 100 threads × 100 comments is rejected before any node exists —
including against a repository with no review threads in it. The page sizes
are now named constants with the arithmetic (
possible_nodes()) beside them,and a test asserts it stays under the limit. Re-running afterwards, the query
is accepted.
Verification
make check— ruff clean, 250 tests pass (22 new)cirolini/genai-code-reviewwith a read token, before andafter the fix. It currently reports no comments, which is correct: v3 has
not yet posted an inline review here.
docs/results/README.mdrecords that this number is not measured yet, andwhy.
🤖 Generated with Claude Code