Repository navigation
Conversation
Adds benchmarks/sandbox/ci-capabilities.bench.ts: a real-fidelity gate that
replays the benchmarks-platform CI executor path inside one sandbox —
exec semantics, re-attach, concurrent + detached exec, workspace/runtime,
egress reachability, git clone, resource floor, verbatim dockerd bring-up
and act install scripts (ci-runtime.ts, ported from lib/ci/executor/
act-runtime.ts), runner-image pull, a detached act workflow run to green,
and a second sandbox verifying the instance deadline is honored.
Each check reports {passed, error} into a capability matrix in record.data
and results/sandbox-ci-capabilities/latest.json so a provider's readiness
(or exact gap) is visible at a glance. A provider passing every check can
be dropped into the Actions product.
Also extracts the shared LARGE_RESOURCE_OPTIONS sizing map from
dax.bench.ts into resource-options.ts and repoints the
sandbox-capabilities workflow at the new bench (120min job timeout).
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Contributor
|
Closing — this moved to the standalone repo computesdk/benchmarks-sandbox-ci-capabilities (initial commit on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
benchmarks/sandbox/ci-capabilities.bench.ts— a real-fidelity gate for Actions-product readiness. One iteration creates a sandbox, runs the full capability battery, and destroys it; a provider that passes every check can be dropped into the benchmarks-platform CI executor as-is. Check length is deliberately not a concern — the goal is fidelity, not speed.The heavy tail of the battery re-runs the same scripts the product's executor generates, ported verbatim from
lib/ci/executor/act-runtime.tsintobenchmarks/sandbox/ci-runtime.ts:dockerd— real daemon bring-up (static Docker 27.4.1 install,nohup setsidlaunch, socket poll, per-provider flags: blaxel's second daemon onunix:///var/run/docker-ce.sock+ci-docker0, cloudflare's--iptables=false --ip6tables=false --ip-forward=false)act_install— the pinnedactv0.2.89 fetch/extract/sha-marker scriptrunner_image—catthehacker/ubuntu:act-*pull (or the blaxel in-box runner image build +--pull=false)act_run— a detachedact pushrunning a real workflow to a green exit, with bridge-gateway artifact-server resolution,--bind,--action-offline-mode, and the platform's-Pimage argsThe earlier checks cover the lifecycle contract: exec semantics (env injection, exit-code/stdout-stderr separation, per-command kill),
getByIdre-attach, concurrent exec, detached-process survival, writable workspace, Node runtime (or fetch-tool bootstrap path), per-host egress reachability (github/codeload/npm/docker/platform/nodejs), a realgit clone --depth 1, and the 4 vCPU / 8 GiB / 16 GiB resource floor.deadlinecreates a second sandbox with a 4-minutetimeoutand verifies the box actually dies on schedule (catches auto-pause and ignored-TTL bugs like the namespace 1h issue); it runs even when the main sandbox fails to create.Each check runs inside its own
step()and records{passed, error}— one failure never aborts the battery, so a provider's exact gap shows up inrecord.data.featuresandresults/sandbox-ci-capabilities/latest.json. Scoring ischecksPassedPct.Supporting changes:
resource-options.ts— theLARGE_RESOURCE_OPTIONSsizing map extracted fromdax.bench.tsso this bench and dax share the 8 vCPU / 16 GiB per-provider sizingsandbox-capabilities.yml— repointed at the new bench (the oldcapabilities.bench.tswas deleted earlier), job timeout raised to 120min, results dir/artifact names renamed toci-capabilitiespackage.json—bench:sandbox:ci-capabilitiesscriptVerified:
pnpm typecheckclean; bench loads and config-validates under--dry-run. A live provider run needs vault keys, so the first real signal comes from dispatching the workflow.Link to Devin session: https://app.devin.ai/sessions/d9e314eb6c40479fbefd4d5bb85edc77
Open in Devin Desktop: https://app.devin.ai/desktop/session/d9e314eb6c40479fbefd4d5bb85edc77?variant=devin
Requested by: @kisernl