Skip to content

AI-1581: proxy CredentialBroker for sandbox code - #1104

Merged
kanat merged 13 commits into
acceleratefrom
connectors/sandbox-proxy-broker
Oct 21, 2026
Merged

kanat merged 13 commits into
acceleratefrom
connectors/sandbox-proxy-broker

Conversation

@kanat

@kanat kanat commented Oct 21, 2026

Copy link
Copy Markdown

Why

AI-1581. Wave 13's Daytona broker leaves the token with Daytona Secrets: an escaped echo reaches the sandbox (A5) and a withdrawn secret keeps working for 16-36 s. #1101 added the router's own egress proxy (internal/sandbox/egress), which resolves the token on every request and keeps no copy. This PR is its sandbox.CredentialBroker and the setting that picks it: code_sandbox.broker: proxy. The default stays daytona; nothing changes on a router that does not set it.

How it works

sequenceDiagram
    participant Session as session.Manager.box
    participant Broker as egress.Broker
    participant Redis
    participant Daytona
    participant Sandbox as Code in the sandbox
    participant Proxy as Egress proxy
    participant GitHub as api.github.com
    Session->>Broker: Prepare(grants) on the first run_code
    Broker->>Redis: Put sha256(credential) -> session, grants, placeholders (TTL 31 min)
    Broker-->>Session: env GITHUB_TOKEN=va_ph_..., HTTPS_PROXY=http://sbx:cred@proxy:8091, CA env, CA file
    Session->>Daytona: create (EnvVars, OutboundProxyUrl, DomainAllowList = proxy + config domains)
    Session->>Daytona: upload /tmp/va-egress-ca.pem, before any code
    Session->>Broker: BeforeRun on each run (withdraws a revoked grant, renews the TTL)
    Sandbox->>Proxy: CONNECT api.github.com:443, Authorization Bearer va_ph_...
    Proxy->>Redis: Get record of the credential
    Proxy->>GitHub: Authorization Bearer token resolved now
    Session->>Broker: Close deletes the record, next CONNECT is a 407
Loading

Network of a sandbox behind the proxy (orchestrator ruling 2):

sandbox_options                 allow list sent to Daytona          NO_PROXY
├─ {}                         -> DomainAllowList proxy            (none)
├─ allowed_domains D          -> DomainAllowList proxy + D        D less any entry matching a grant's host
├─ network_allow_list R       -> NetworkAllowList R + proxy IP/32  R
│   └─ proxy URL is a host name -> session refused (Daytona takes one list)
└─ network: true, no list     -> session refused (all traffic would hit the proxy)

Changes

  • internal/sandbox/egress/broker.go: Broker (Prepare, BeforeRun, Close) over the proxy's Store.
  • sandbox.Mount gains Env, Proxy, Files; sandbox.Grant gains Scheme. daytona.go sends them (EnvVars, OutboundProxyUrl, CA upload before the first run, retried on the next run), builds the allow list and NO_PROXY above.
  • config: code_sandbox.broker: daytona|proxy, code_sandbox.proxy_public_url (ROUTER_CODE_SANDBOX_PROXY_PUBLIC_URL). proxy needs it (http, host, no user) and proxy_ca_cert.
  • Caller: cmd/router sandboxBroker builds the store from Redis (refuses without redis.addr); session.Manager.broker picks egress.NewBroker and refuses a config the proxy cannot serve; NewManager refuses the proxy broker without its store.
  • brokertest.Subject.Live: the contract for a broker whose egress resolves per request.
  • Docs: connectors skill «Sandbox access», README rows, egress/AGENTS.md, brokertest/AGENTS.md.

Rules and the tests that prove them

Rule Test
The sandbox gets a placeholder va_ph_<26 base32>, never the token TestProxyBrokerContract/TestTheSandboxGetsAReferenceNeverTheToken, TestBrokerSuite/TestTheSandboxReachesAGrantedHostWithTheTokenInPlaceOfItsPlaceholder
Env names the proxy (4 proxy vars) and the CA (4 CA vars); the CA file holds the certificate only TestBrokerSuite/TestTheEnvNamesTheProxyAndTheCA
DomainAllowList / NetworkAllowList / NO_PROXY per the tree above TestProxySuite/TestTheSandboxReachesTheProxyAndTheConfigsOwnHosts, TestDaytonaSuite/TestNoProxyLeavesOutEveryDomainWhoseEntryMatchesAGrantsHost
network: true (no list) and ranges with a host-named proxy are refused at session start TestSandboxGrantsSuite/TestTheProxyBrokerRefusesAConfigItCannotServe
BeforeRun renews the record past sandbox.MaxTimeout TestBrokerSuite/TestARunAfterTheRecordExpiredRenewsIt
Close deletes the record (407 next) TestBrokerSuite/TestClosingEndsTheProxyCredential, TestProxySuite/TestClosingTheSandboxEndsItsProxyCredential
Revocation is immediate, no run in between TestBrokerSuite/TestADisconnectedConnectionIsRefusedAtTheProxy
Broker selection and wiring TestSandboxGrantsSuite/TestTheProxyBrokerSendsTheSandboxThroughTheProxy, TestBuildSessionsGivesTheProxyBrokerItsProxy, TestAManagerWithTheProxyBrokerNeedsItsStore
Config requirements TestConfigSuite/TestTheProxyBrokerNeedsTheProxysURLAndCA, TestBrokerSuite/TestAProxyURLItCannotUseIsRefused
Control: the default daytona broker sends what base sent (expected values probed on a103c19) TestSecretsSuite/TestAGrantedSandboxIsCreatedAsOnBase, TestTheDaytonaBrokerAsksNothingOfTheProxy

Mutation checks: 72 mutations over every changed condition, branch and constant (broker, daytona, session, config, cmd/router, brokertest), 72 killed. Three survivors of the first pass got tests (a run after Close, BeforeRun with no record, an IP proxy without ranges); a fourth was dead code (the proxy_public_url export), removed.

Also fixed: a flaky egress test rig. rueidis spreads keyed commands over four connections and sends PING on the first only, so the gate's recover could hand a Put a cut connection (13 failures in 30 runs before, 0 after PipelineMultiplex: -1).

Orchestrator rulings in this PR (Kanat may overturn)

  1. One PR, size stop 1400 lines (tests included).
  2. Behind the proxy the allow list is the proxy host plus the config's own domains (or ranges); NO_PROXY holds those, never an entry that matches a grant's host; network: true with broker: proxy is refused when the session starts. Interpreted as an open network (no allow list): network: true with allowed_domains is already narrowed, so it is accepted. Daytona refuses both lists at once (internal/api/configs.go:1073), so with network_allow_list the proxy joins as IP/32, and a proxy named by a host name is refused.
  3. The connectors skill «Sandbox access» says what broker: proxy needs, what the sandbox sees, the network rule, and that revocation is immediate.

Unverified

  • Daytona accepts userinfo in OutboundProxyUrl, and which of it and our HTTPS_PROXY env wins; whether Daytona sets NO_PROXY itself. Needs the live test on staging (AI-1582: port 8091 Service + CA secret).

🤖 Generated with Claude Code

kanat and others added 4 commits October 21, 2026 12:03
…er's proxy (WIP, oversize)

code_sandbox.broker: proxy gives each sandbox a proxy credential and a
va_ph_ placeholder per grant, stores the grants in Redis for the egress
proxy, and creates the Daytona sandbox with OutboundProxyUrl, the proxy
host as its only domain, and the proxy CA file. BeforeRun renews the
record; Close deletes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en network

The sandbox's allow list is the proxy and the config's domains (or its ranges and
the proxy's address); NO_PROXY holds the config's own hosts, never one matching a
grant's host. A config with network: true and no allow list, or ranges with a proxy
named by a host name, is refused when the session starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ction behind the test gate

rueidis spreads keyed commands over four connections and pings on the first only, so
the gate's recover could hand a Put a cut connection (13 failures in 30 runs before,
0 after). code_sandbox.proxy_public_url is not exported: nothing reads it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kanat and others added 9 commits October 21, 2026 13:15
…hrough a provider outage

R1.5 SSL_CERT_FILE, REQUESTS_CA_BUNDLE and GIT_SSL_CAINFO name a bundle of the image's
own roots and the CA, written by Mount.Setup before any code; NODE_EXTRA_CA_CERTS stays
on the CA alone. R1.6 the sandbox gets the CA setting's CERTIFICATE blocks alone.
R1.21 only ErrNoCredential leaves a grant out at Prepare. R1.1, R1.22, R1.25 tests.
N1.3 a run after a refused Close deletes the record; N1.4 no grant left deletes it.
R1.10 config domains naming a grant's host leave the allow list; N1.9 NO_PROXY entries
lose a trailing dot, a port and **.; R1.7, R1.8 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olds no key

R1.19 code_sandbox.broker proxy needs code_sandbox.proxy_addr (and so its CA key).
R1.6 a proxy_ca_cert holding a private key is refused. N1.18 proxy_public_url is exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1.11 sandboxBroker sets SandboxBroker, SandboxSecretPrefix and SandboxProxy, compared in
a test. R1.13 the proxy and its broker share sandboxStore. R1.23 the key never reaches
the broker's CA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1.12 NewManager refuses a proxy broker with a URL and CA but no store. R1.14 a
withdrawal reaches the session's log. R1.24 the record names the session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
N1.15, N1.16, N1.17, N1.20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d config domain is unreachable

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kanat
kanat marked this pull request as ready for review October 21, 2026 18:22
@kanat
kanat merged commit 93e1cfe into accelerate Oct 21, 2026
7 checks passed
@kanat
kanat deleted the connectors/sandbox-proxy-broker branch October 21, 2026 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant