Skip to content

fix: stop sending temperature to the Anthropic judge - #32

Merged
unshDee merged 1 commit into
mainfrom
fix/anthropic-no-temperature
Oct 31, 2026
Merged

unshDee merged 1 commit into
mainfrom
fix/anthropic-no-temperature

Conversation

@unshDee

@unshDee unshDee commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Problem

Anthropic API version 2026-10-21 (and anthropic SDK 1.x, which a fresh pip install "proofrag[anthropic]" now resolves) rejects temperature, top_p and top_k. Every generate and evaluate call failed with:

Messages.create() got an unexpected keyword argument 'temperature'

Fix

  • Drop temperature=0 from the Anthropic messages.create call. OpenAI keeps it.
  • The judge fingerprint now reads anthropic:<model>:temperature=default, so a run can't be silently compared against a baseline judged under the old settings.
  • Regression test asserts no sampling parameter is ever sent again.

Impact

Baselines judged by Anthropic need one rerun before proofrag diff will compare them (the fingerprint changed on purpose). Claude-judged scores may vary slightly more between runs. Noted in the CHANGELOG.

Verified with a live Haiku call. 189 tests pass, make lint clean.

Worth shipping as a 0.9.1 patch.

🤖 Generated with Claude Code

Anthropic API version 2026-10-21 and SDK 1.x reject sampling parameters, so
fresh installs of proofrag[anthropic] failed every generate/evaluate call.
Claude now samples at its default; the judge fingerprint reads
temperature=default, so Anthropic-judged baselines need one rerun before diff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 31, 2026 05:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@unshDee
unshDee merged commit 5602548 into main Oct 31, 2026
4 checks passed
@unshDee
unshDee deleted the fix/anthropic-no-temperature branch October 31, 2026 05:10
unshDee added a commit that referenced this pull request Oct 31, 2026
> Stacked on #32. Merge that first; this PR then retargets to `main`.

Closes #30.

## Scorecard redesign
- One headline score with a one-line verdict ("3 of 8 cases need
attention · weakest metric: …").
- Compact metric bars that only turn red under 65.
- Case × metric heatmap; each row expands (`<details>`, no JS) to the
answer and retrieval detail.
- Dark mode via `prefers-color-scheme`. The A/B report shares the same
styles and bolds the better side.
- Still a single self-contained HTML file. Checked light, dark, 375px
width and the invalid-run banner in a browser.

## `proofrag wizard`
- New optional extra: `pip install "proofrag[tui]"` (rich + typer). The
core stays dependency-free; without the extra the command prints the
install hint.
- Four steps: docs → golden set → your app → evaluate, with live
progress bars, an animated score, a results panel, and the equivalent
plain CLI commands for CI.
- Calls the same functions as the plain commands, so the artifacts are
identical. Asks before overwriting a reviewed golden set.
- **Redraw safety:** live regions are one line tall, and tables print
only after bars finish. Logging handlers configured before the bar
starts are re-pointed through rich while the bar is live. Otherwise each
log line leaves a stale copy of the bar (reproduced in a 12-row
terminal). Covered by tests in TTY and non-TTY mode.

## Other
- `run --callable module:fn` now finds modules in the working directory
when installed via `uv tool` / `pipx` (console scripts don't put cwd on
`sys.path`).
- `examples/docs-rag/naive_rag.py` exposes `ask()`.
- New `docs/demo.gif` (real run of the wizard, reproducible via `vhs
docs/demo.tape`), refreshed screenshots, and `docs/social-preview.png`.

README image URLs stay pinned to `v0.9.0` on purpose. The new GIF shows
a command 0.9.0 doesn't have, so bump the pins at the next release tag.

192 tests pass, `make lint` clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants