Skip to content

Adopt Muse Spark 1.3 and fix browser benchmark boundaries - #138

Merged
rsproule merged 2 commits into
mainfrom
bench/correct-browser-evals
Sep 8, 2026
Merged

rsproule merged 2 commits into
mainfrom
bench/correct-browser-evals

Conversation

@rsproule

@rsproule rsproule commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • use meta/muse-spark-1.3 for the production browser subagent, preserving the benchmarked reasoning: low configuration
  • align browser eval stop points with the behavior under test instead of unrelated login, traveler-info, CAPTCHA, or payment prerequisites
  • allow named vertical search products such as Google Flights while retaining the general search-engine restriction
  • remove the consistently CAPTCHA-blocked Expedia case and clarify hotel, Yankees, Elsewhere, and Target success criteria
  • follow Eve 0.52 background subagent streams so evals attach to the real browser worker after the root turn completes
  • add an opt-in native PostgreSQL URL for local A/B runs when Docker port forwarding is unavailable

Full benchmark

Ran all 10 browser tasks, one repetition, no smoke suite, from the same main commit with only the worker model changed:

  • meta/muse-spark-1.3: 9/10, 242s median, 437s P95, $4.04
  • zai/glm-5.2: 8/10, 409s median, 875s P95, $12.07

Muse passed the corrected Google Flights, Yankees, Target, and hotel cases. Both Elsewhere attempts encountered the same Eventbrite checkout iframe error; Muse returned a bounded failure and the baseline timed out. The other baseline failure selected a hotel whose final displayed total exceeded the explicit $300 cap.

Validation

  • pnpm exec vitest run --exclude "**/.claude/**" — 82 files, 716 tests passed
  • focused model normalization and browser capability tests — 15 passed
  • pnpm exec oxlint . --deny-warnings --ignore-pattern ".claude/**"
  • formatting checks for all changed files
  • pnpm build

Known integration note

AI SDK currently logs repeated warnings that non-OpenAI reasoning parts are unsupported and skipped for Muse. Browser execution and tool calling completed successfully in the benchmark, but the warning path is noisy and should be followed up separately.

The checkout contains a pre-existing untracked .claude/worktrees tree. The aggregate local pnpm check discovers that second checkout, so validation excludes it; the repository tests and production build pass.

@rsproule rsproule changed the title Fix browser benchmark task boundaries and async worker tracing Adopt Muse Spark 1.3 and fix browser benchmark boundaries Sep 8, 2026
@rsproule
rsproule merged commit d7b5979 into main Sep 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant