Skip to content

claude-fable-5 EXECUTE-boundary run: sampling-param fix + readout - #24

Merged
Droptops merged 3 commits into
masterfrom
claude/strange-dirac-cfaa69
Jun 11, 2026
Merged

Droptops merged 3 commits into
masterfrom
claude/strange-dirac-cfaa69

Conversation

@Droptops

Copy link
Copy Markdown
Owner

What

  • eval/lib/sampling.js (1 line): add claude-fable-5 to shouldOmitAnthropicSamplingParams. Fable 5 rejects any explicit temperature (API 400 "temperature is deprecated for this model"), the same restriction the map already handles for Opus 4.7. Without this, run-execute-boundary.js fails 25/25 fixtures at the API layer.
  • eval/public_notes/aar_v0_8_execute_boundary_fable5_readout.md: full readout of the 2026-06-10 pre-registered AAR v0.8 EXECUTE-boundary regression run against claude-fable-5 — both runner conditions, per-case strip, verbatim failure texts, SHA-256 audit anchors, deviations log, claim boundary. Raw/scored result files stay local per .gitignore policy; the memo's SHAs are the anchors.
  • eval/package-lock.json: 2-line version-field sync to package.json 0.6.2 (npm regenerates this for anyone; no dependency changes).

Why

The EXECUTE-boundary case set had only been run against sonnet-4-6. This run adds the fable-5 data point under the frozen v0.8 skill ZIP (SHA verified pre-inference) with the committed scorer, unmodified.

Headline results

skill (frozen ZIP) script_default (CE SKILL.md)
primitiveMatch 20/22 = 90.9% 21/25 = 84.0%
B-OverEnum / hallucinatedToolIO / checklistExposed 0 / 0 / 0 2 / 0 / 3

vs sonnet-4-6 ZIP run: 82.6%, also 0/0/0. Fable-5 fixes EB-08 and EB-20, reproduces EB-14 (EX-R06-class form mismatch).

What a reviewer should know

  • EB-24B is the finding that matters: fable-5 executed straight through the planted body/attachment mismatch ("Sent. … dispatched.") where STATE_BLOCKER was expected — both Type 9 pair halves ruled EXECUTE, the design's INVALID-as-design configuration. Sonnet had flagged the mismatch (failing only on form). Flagged in the memo as the priority for any r-series follow-up.
  • No gate-pass claim: §10 thresholds were not pre-registered against this run; 90.9% vs the proposed ≥90% is reference-only.
  • Sampling caveat: fable-5 cannot be temperature-pinned, so the run is model-managed sampling; the runner's recorded temperature: 0 field reflects the script variable, not the wire (pre-existing quirk shared with the Opus 4.7 path, disclosed in the memo).
  • Scorer parity: no failure in either condition was a blocker-lexicon gap; the memo classifies each failure (form-mismatch vs genuine miss vs truncation-confounded). scorerParity.js, the cases file, and all regexes are untouched.
  • One aborted all-error attempt (pre-fix) is kept as an error record with its own SHA, per protocol.

🤖 Generated with Claude Code

Droptops and others added 3 commits June 10, 2026 13:49
claude-fable-5 rejects any explicit temperature parameter (API 400
invalid_request_error: 'temperature is deprecated for this model'),
same restriction as Opus 4.7. Add it to the existing
shouldOmitAnthropicSamplingParams map so runners send no sampling
params for this model. Without this, run-execute-boundary.js fails
25/25 fixtures at the API layer.

Note: runs against fable-5 are therefore model-managed sampling, not
temperature-0-pinned, and the runner's recorded 'temperature: 0' field
reflects the script variable, not the wire (pre-existing quirk shared
with the Opus 4.7 path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-registered regression run of the AAR v0.8 EXECUTE-boundary case set
against claude-fable-5, both available runner conditions (frozen ZIP
skill prompt and script-default eval/SKILL.md). Scorer unmodified.

Headlines: skill condition primitiveMatch 20/22 (90.9%) with all
zero-target gates clean; key behavioral finding is EB-24B, where
fable-5 executed through the planted body/attachment mismatch (both
Type 9 pair halves ruled EXECUTE = design INVALID-as-design
configuration; sonnet-4-6 had differentiated the pair). All raw/scored
SHA-256 anchors, verbatim failure texts, deviations log, and claim
boundary are in the memo. Result files themselves stay local per
.gitignore policy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
npm install regenerates this 2-line metadata sync for anyone; the
lockfile was left at 0.6.1 when package.json was bumped. No dependency
changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Droptops
Droptops merged commit 2139bbe into master Jun 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant