Repository navigation
fix: stop sending temperature to the Anthropic judge - #32
Merged
Merged
Conversation
Anthropic API version 2026-10-21 and SDK 1.x reject sampling parameters, so fresh installs of proofrag[anthropic] failed every generate/evaluate call. Claude now samples at its default; the judge fingerprint reads temperature=default, so Anthropic-judged baselines need one rerun before diff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
unshDee
added a commit
that referenced
this pull request
Oct 31, 2026
> Stacked on #32. Merge that first; this PR then retargets to `main`. Closes #30. ## Scorecard redesign - One headline score with a one-line verdict ("3 of 8 cases need attention · weakest metric: …"). - Compact metric bars that only turn red under 65. - Case × metric heatmap; each row expands (`<details>`, no JS) to the answer and retrieval detail. - Dark mode via `prefers-color-scheme`. The A/B report shares the same styles and bolds the better side. - Still a single self-contained HTML file. Checked light, dark, 375px width and the invalid-run banner in a browser. ## `proofrag wizard` - New optional extra: `pip install "proofrag[tui]"` (rich + typer). The core stays dependency-free; without the extra the command prints the install hint. - Four steps: docs → golden set → your app → evaluate, with live progress bars, an animated score, a results panel, and the equivalent plain CLI commands for CI. - Calls the same functions as the plain commands, so the artifacts are identical. Asks before overwriting a reviewed golden set. - **Redraw safety:** live regions are one line tall, and tables print only after bars finish. Logging handlers configured before the bar starts are re-pointed through rich while the bar is live. Otherwise each log line leaves a stale copy of the bar (reproduced in a 12-row terminal). Covered by tests in TTY and non-TTY mode. ## Other - `run --callable module:fn` now finds modules in the working directory when installed via `uv tool` / `pipx` (console scripts don't put cwd on `sys.path`). - `examples/docs-rag/naive_rag.py` exposes `ask()`. - New `docs/demo.gif` (real run of the wizard, reproducible via `vhs docs/demo.tape`), refreshed screenshots, and `docs/social-preview.png`. README image URLs stay pinned to `v0.9.0` on purpose. The new GIF shows a command 0.9.0 doesn't have, so bump the pins at the next release tag. 192 tests pass, `make lint` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Anthropic API version 2026-10-21 (and
anthropicSDK 1.x, which a freshpip install "proofrag[anthropic]"now resolves) rejectstemperature,top_pandtop_k. Everygenerateandevaluatecall failed with:Fix
temperature=0from the Anthropicmessages.createcall. OpenAI keeps it.anthropic:<model>:temperature=default, so a run can't be silently compared against a baseline judged under the old settings.Impact
Baselines judged by Anthropic need one rerun before
proofrag diffwill compare them (the fingerprint changed on purpose). Claude-judged scores may vary slightly more between runs. Noted in the CHANGELOG.Verified with a live Haiku call. 189 tests pass,
make lintclean.Worth shipping as a 0.9.1 patch.
🤖 Generated with Claude Code