Read the acceptance signal from a repository

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## Unreleased
+
+### Added
+
+- **Acceptance signal.** `python -m evals.feedback --repo owner/name` (also
+  `make feedback`) reads a repository the action has run on and reports what
+  reviewers did with its comments: 👍/👎 reactions, replies, thread resolution,
+  and the share nobody engaged with. Read-only, no model call. The signals are
+  reported separately rather than averaged — a resolved thread means the
+  conversation ended, not that the finding was right.
+
 ## v3.0.0 — 2026-09-19
 
 ### Fixed

Makefile

@@ -8,7 +8,9 @@ PROVIDER ?= openai
 MODEL    ?=
 RESULTS  ?= docs/results
 
-.PHONY: help venv install lint test check eval eval-dry eval-compare clean
+REPO ?= cirolini/genai-code-review
+
+.PHONY: help venv install lint test check eval eval-dry eval-compare feedback clean
 
 help:
 	@echo "make install       install runtime and dev dependencies"
@@ -18,6 +20,7 @@ help:
 	@echo "make eval-dry      list the eval cases without calling any provider"
 	@echo "make eval          run the eval set (costs money; needs an API key)"
 	@echo "make eval-compare  eval with and without the comment budget"
+	@echo "make feedback      what reviewers did with the comments (REPO=owner/name)"
 
 $(VENV):
 	python3 -m venv $(VENV)
@@ -52,5 +55,10 @@ eval-compare: venv
 		--compare-budget \
 		--out $(RESULTS)/$(PROVIDER)-budget-comparison.md
 
+# Reads reactions, replies and thread resolution on comments the action has
+# already posted. Read-only, and free: no model is called.
+feedback: venv
+	$(PY) -m evals.feedback --repo $(REPO)
+
 clean:
 	rm -rf $(VENV) .pytest_cache **/__pycache__

README.md

@@ -220,6 +220,22 @@ reviewed. Upload it to keep a record:
 
 It contains no credentials, no diff and no source code.
 
+### Did anyone act on the comments?
+
+The run log records what the bot said. To see what reviewers did with it —
+reactions, replies and whether the thread was resolved — read a repository the
+action has already run on:
+
+```bash
+make feedback REPO=owner/name        # needs $GITHUB_TOKEN with read access
+```
+
+It is read-only and calls no model. The signals are reported side by side
+rather than averaged into one score: a resolved thread means the conversation
+ended, which happens both when a finding is fixed and when it is dismissed. The
+row that carries the most information is the share of comments nobody touched
+at all.
+
 ---
 
 ## How it works
@@ -246,6 +262,7 @@ make install   # dependencies
 make check     # ruff + pytest
 make eval-dry  # list the eval fixtures without calling a provider
 make eval      # run the eval set (needs an API key; costs money)
+make feedback  # acceptance signal from a repository the action has run on
 ```
 
 ---

docs/case-study.md

@@ -351,8 +351,11 @@ ranking should drop it.
 
 **Learn from what reviewers do with the comments.** Reactions and resolved
 threads on the bot's comments are a free signal about which findings were worth
-posting. That is the honest way to tune the budget — from what people accepted,
-not from what I guessed.
+posting, and the honest way to tune the budget: from what people accepted, not
+from what I guessed. `python -m evals.feedback` reads them. It has nothing to
+report yet, because v3 has not been running anywhere long enough to have
+produced comments for anyone to react to — which is the point at which this
+stops being a design argument and starts being an observation.
 
 **Let severity thresholds vary by path.** A `blocker` in an auth module and a
 `blocker` in a build script are not the same thing, and `min_severity` is

docs/results/README.md

@@ -114,3 +114,10 @@ meaningless. The first attempt at the Groq run lost 10 of 21 cases that way.
   billing layer, but the account used for testing has no credit, so no response
   has come back.
 - **Panel mode.** Needs two working providers in one run.
+- **Acceptance in the wild.** `python -m evals.feedback --repo owner/name`
+  reads reactions, replies and thread resolution on comments the action has
+  already posted. It reports nothing yet, because v3 has not been running long
+  enough anywhere to have produced comments for reviewers to react to. This is
+  the number that would actually settle the argument — the eval set measures
+  the bot against defects chosen for it, and this measures it against reviewers
+  who did not choose anything.