Skip to content

feat: redesign the scorecard and add the proofrag wizard TUI - #33

Merged
unshDee merged 3 commits into
mainfrom
feat/scorecard-redesign-wizard
Oct 31, 2026
Merged

unshDee merged 3 commits into
mainfrom
feat/scorecard-redesign-wizard

Conversation

@unshDee

@unshDee unshDee commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Stacked on #32. Merge that first; this PR then retargets to main.

Closes #30.

Scorecard redesign

  • One headline score with a one-line verdict ("3 of 8 cases need attention · weakest metric: …").
  • Compact metric bars that only turn red under 65.
  • Case × metric heatmap; each row expands (<details>, no JS) to the answer and retrieval detail.
  • Dark mode via prefers-color-scheme. The A/B report shares the same styles and bolds the better side.
  • Still a single self-contained HTML file. Checked light, dark, 375px width and the invalid-run banner in a browser.

proofrag wizard

  • New optional extra: pip install "proofrag[tui]" (rich + typer). The core stays dependency-free; without the extra the command prints the install hint.
  • Four steps: docs → golden set → your app → evaluate, with live progress bars, an animated score, a results panel, and the equivalent plain CLI commands for CI.
  • Calls the same functions as the plain commands, so the artifacts are identical. Asks before overwriting a reviewed golden set.
  • Redraw safety: live regions are one line tall, and tables print only after bars finish. Logging handlers configured before the bar starts are re-pointed through rich while the bar is live. Otherwise each log line leaves a stale copy of the bar (reproduced in a 12-row terminal). Covered by tests in TTY and non-TTY mode.

Other

  • run --callable module:fn now finds modules in the working directory when installed via uv tool / pipx (console scripts don't put cwd on sys.path).
  • examples/docs-rag/naive_rag.py exposes ask().
  • New docs/demo.gif (real run of the wizard, reproducible via vhs docs/demo.tape), refreshed screenshots, and docs/social-preview.png.

README image URLs stay pinned to v0.9.0 on purpose. The new GIF shows a command 0.9.0 doesn't have, so bump the pins at the next release tag.

192 tests pass, make lint clean.

🤖 Generated with Claude Code

unshDee and others added 2 commits October 31, 2026 06:00
Anthropic API version 2026-10-21 and SDK 1.x reject sampling parameters, so
fresh installs of proofrag[anthropic] failed every generate/evaluate call.
Claude now samples at its default; the judge fingerprint reads
temperature=default, so Anthropic-judged baselines need one rerun before diff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Scorecard and A/B report: one-line verdict, compact metric bars, a
  case-by-metric heatmap with expandable rows, and dark mode (closes #30).
- `proofrag wizard` in the optional `tui` extra (rich + typer): a guided run
  with one-line progress bars that route prints and earlier logging handlers
  above the bar, so long output can't make the display reprint itself.
- `run --callable` imports modules from the working directory when proofrag
  is installed as a console script (uv tool, pipx).
- naive_rag.ask() for the example; new demo GIF, screenshots, social preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 31, 2026 05:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Base automatically changed from fix/anthropic-no-temperature to main October 31, 2026 05:10
@unshDee
unshDee merged commit 7288ace into main Oct 31, 2026
4 checks passed
@unshDee
unshDee deleted the feat/scorecard-redesign-wizard branch October 31, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: dark mode for the HTML scorecard

2 participants