Skip to content

bench(dax): declare scoring spec so run summaries carry provider rows - #558

Merged
kisernl merged 1 commit into
masterfrom
devin/1793056288-dax-scoring
Oct 27, 2026
Merged

kisernl merged 1 commit into
masterfrom
devin/1793056288-dax-scoring

Conversation

@kisernl

@kisernl kisernl commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Every sandbox-dax-daily run for 2+ weeks merges to an empty summary.results[] — workers complete (e2b builds 7/7 phases in ~83s, records land in task_results) but each worker's submitted summary carries zero provider rows because dax.bench.ts never declared a scoring spec. The platform leaderboard and the provider-weekly-composite bench both read run.summary.results, so dax contributes nothing.

Adds scoring.metrics on totalMs (the build's measured duration): ceiling: 300_000 ms — dax builds run ~20–150s, so typical providers score ~50–93 rather than clamping at 0 — with the same median .60 / p95 .25 / p99 .15 weights tti.bench.ts uses.

Verified with bench check: scoringOk: true. From the next daily run onward the run summary will carry per-provider compositeScore rows; earlier windows stay empty (the composite skips missing data rather than zeroing it).

Link to Devin session: https://app.devin.ai/sessions/6c1d53718cda4d8d87e45f292b431d89
Open in Devin Desktop: https://app.devin.ai/desktop/session/6c1d53718cda4d8d87e45f292b431d89?variant=devin
Requested by: @kisernl

dax.bench.ts had no scoring config, so each worker's submitted summary
contained zero result rows and every merged sandbox-dax-daily run has an
empty summary.results[] — the platform leaderboard and the composite
benchmark read nothing from it. Score on totalMs with a 5-min ceiling
(builds run ~20-150s) and the same median/p95/p99 weights as tti.bench.

Co-Authored-By: Noah Kiser <noah@computesdk.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@kisernl
kisernl marked this pull request as ready for review October 27, 2026 13:49
@open-cla

open-cla Bot commented Oct 27, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@kisernl
kisernl merged commit 58427b1 into master Oct 27, 2026
6 checks passed
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 promigence 99.6 0.04s 0.05s 0.05s 10/10
2 miosa 99.5 0.05s 0.05s 0.05s 10/10
3 isorun 99.5 0.05s 0.05s 0.05s 10/10
4 createos 98.9 0.10s 0.11s 0.11s 10/10
5 declaw 98.4 0.16s 0.16s 0.16s 10/10
6 daytona 98.0 0.18s 0.23s 0.23s 10/10
7 givemeanode 97.7 0.22s 0.25s 0.25s 10/10
8 superserve 97.7 0.21s 0.26s 0.26s 10/10
9 buddy 97.6 0.23s 0.26s 0.26s 10/10
10 blaxel 97.4 0.24s 0.30s 0.30s 10/10
11 arker 97.3 0.25s 0.31s 0.31s 10/10
12 beam 96.8 0.30s 0.33s 0.33s 10/10
13 vercel 95.9 0.35s 0.49s 0.49s 10/10
14 upstash 93.8 0.45s 0.89s 0.89s 10/10
15 modal 92.9 0.66s 0.78s 0.78s 10/10
16 mosaic 91.0 0.83s 1.02s 1.02s 10/10
17 novita 90.8 0.89s 0.96s 0.96s 10/10
18 cloudflare 88.7 0.85s 1.55s 1.55s 10/10
19 tenki 88.7 1.06s 1.24s 1.24s 10/10
20 sandbox0 87.9 1.18s 1.24s 1.24s 10/10
21 e2b 85.0 1.16s 2.00s 2.00s 10/10
22 microsandbox 84.4 1.53s 1.60s 1.60s 10/10
23 cloud-run 83.8 0.48s 3.33s 3.33s 10/10
24 sail 83.5 1.55s 1.79s 1.79s 10/10
25 freestyle 74.3 2.48s 2.72s 2.72s 10/10
26 codesandbox 65.5 3.14s 3.92s 3.92s 10/10
27 run-cloud 19.6 6.74s 12.93s 12.93s 10/10
28 tensorlake 8.8 1.22s 1.22s 1.22s 1/10
29 archil 0.0 0.00s 0.00s 0.00s 0/10
30 gravixlayer 0.0 0.00s 0.00s 0.00s 0/10
31 hopx 0.0 0.00s 0.00s 0.00s 0/10
32 northflank 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant