Repository navigation
chore(sandbox): remove runloop from active benchmark participants - #498
Merged
Merged
Conversation
Comment out the runloop provider entry and its CI matrix rows, matching the pattern used for other paused providers (quilt, lightning, opencomputer). The benchmarks-daily matrix is generated from this providers list, so runloop drops out of daily sandbox/TTI/DAX runs once this lands. Co-Authored-By: Noah Kiser <noah@computesdk.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
kisernl
marked this pull request as ready for review
October 9, 2026 02:03
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Devin resolved all 2 findings on 9d71f5a
Fixed by Devin (2)
- Paused scale run consumes empty VMs
- Runloop benchmark command always fails
View all findings in Devin Review
…vider - Remove the now-dead pnpm bench:runloop script (selectParticipants would reject runloop anyway); matches quilt/collimate/lelantos having no script. - start.ts: validate --provider via getProvider in parseArgs so a removed or misspelled provider exits before a platform run or VMs are created. Co-Authored-By: Noah Kiser <noah@computesdk.com>
Sandbox Benchmark ResultsBurst
View full run · SVGs available as build artifacts |
Sandbox Dax Benchmark Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comments out runloop as an active benchmark participant, matching the pattern used for other paused providers (quilt, lightning, opencomputer):
benchmarks/sandbox/providers.ts: runloop entry + import commented outbenchmarks/scale/providers.ts: runloop entry + import commented out (100k burst opt-in list).github/workflows/sandbox-tti-benchmarks.yml,sandbox-dax-benchmarks.yml,sandbox-capabilities.yml:- runloopmatrix rows commented outpackage.json: deadbench:runloopscript removed (selectParticipantswould reject runloop anyway)benchmarks/scale/scripts/start.ts:parseArgsnow validates--providerviagetProvider, so a removed or misspelled provider exits before a platform run or VMs are createdThe benchmarks-daily repo builds its matrix from
benchmarks/sandbox/providers.tsviascripts/build-daily-matrix.mjs, so runloop drops out of daily sandbox/TTI/DAX runs automatically once this merges — no benchmarks-daily change needed (verified locally:runloopis absent from the generatedincludematrix).Left intact for easy re-enablement: the
@computesdk/runloopdep,provider-vars.jsonvault keys,.env.example, and theDAX_RESOURCE_OPTIONSsizing entry.Link to Devin session: https://app.devin.ai/sessions/d5b87f8d22e5454f87d3a79eb2c4c37f
Open in Devin Desktop: https://app.devin.ai/desktop/session/d5b87f8d22e5454f87d3a79eb2c4c37f?variant=devin
Requested by: @kisernl