test(evals): add Opus 5.5 to the multi-model evals - #317
Merged
Merged
Conversation
Add claude-opus-5-5 to both model lists in know-the-cli.eval.ts. Update @anthropic-ai/claude-agent-sdk to 0.3.280. The earlier version bundles Claude Code 2.1.258, and the API rejects Opus 5.5 for Claude Code versions before 2.1.280. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L3ocxCuThaf3J4uKnq4vxr
angeloashmore
marked this pull request as ready for review
September 22, 2026 17:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves:
Description
Before this PR, the multi-model evals in
know-the-cli.eval.tsdid not include Opus 5.5. The eval harness also could not run it. The API refuses Opus 5.5 for Claude Code versions before 2.1.280, and the Claude Agent SDK bundled 2.1.258:After this PR, both multi-model evals run
claude-opus-5-5. The Claude Agent SDK is at 0.3.280, which bundles Claude Code 2.1.280. Opus 5.5 passed both evals in a trial run.The other evals keep their default model.
evals/results.jsonis unchanged.Checklist
Preview
How to QA 1
🤖 Generated with Claude Code
https://claude.ai/code/session_01L3ocxCuThaf3J4uKnq4vxr
Note
Low Risk
Changes are limited to eval model lists and a dev-only SDK upgrade; no production CLI or runtime behavior is modified.
Overview
Adds
claude-opus-5-5to both multi-modeldescribe.forblocks inknow-the-cli.eval.ts, so the “CLI not Slice Machine” planning eval and the Slice Machine redirect eval run against Opus 5.5 alongside the other listed models.Bumps the dev dependency
@anthropic-ai/claude-agent-sdkfrom^0.3.258to^0.3.280(with matchingpackage-lock.jsonentries) so the eval harness uses a Claude Code build that the API accepts for Opus 5.5; older bundled versions were rejected with a 400.Reviewed by Cursor Bugbot for commit 35921f7. Bugbot is set up for automated code reviews on this repo. Configure here.
Footnotes
Please use these labels when submitting a review:
⚠️ #issue: Strongly suggest a change.
❓ #ask: Ask a question.
💡 #idea: Suggest an idea.
🎉 #nice: Share a compliment. ↩