Repository navigation
Conversation
Actors come from profile templates shared by many tasks, so the runner learns its work at runtime. Until now it pulled the task from ax-server's runner endpoint (:8081) with an Actor JWT injected by the egress gateway. ax-server now pushes the work instead. After ResumeActor it POSTs the Task, followed by its Workspace, to /runner/v1alpha1/task on the actor's port 80, or a Workspace to a setup actor at /runner/v1alpha1/setup, and polls /readyz?check=workspace. It dials the worker IP ResumeActor returned and falls back to atenet-router. Workspace setup still runs in its own setup actor, which is snapshotted once setup is done. Removed: pkg/apis/runner/v1, internal/runnerhttp, internal/lifecycle, the :8081 server and its Service and NetworkPolicy rule, the Actor JWT flags, the runner egress rule, and the TaskRow run-attempt fields. TaskRow is renamed TaskVersion.
make build-local-cluster builds ax and the AX images from the checkout, pushes them to the kind-registry container on localhost, and runs `ax local start` with them. It uses podman or docker, builds the images for the machine's architecture, starts or restarts the registry container as needed, and keeps the cluster and the registry. On a running cluster it only rolls out the new ax-server. make images now skips TLS verification when podman pushes to a registry on localhost or 127.0.0.1, which serves plain HTTP. docker already allows that, so `make images` and `make test-local-install` failed only with podman.
TestE2E_PureImageLauncher and TestE2E_PrewarmAndCustomImageSharedWorkspace waited for a task to reach Completed with status.outcome set and its actor suspended. Since tasks are pushed to runners, ax-server no longer reads a command's exit back from the runner, so those tasks stay Working and both tests timed out. Also remove the stand-in launcher image (test/e2e/launcher) that only these tests ran, and its build in hack/setup-e2e-kind.sh.
yongruilin
left a comment
There was a problem hiding this comment.
Thanks for putting up this. I left 2 questions.
A task reaches its runner only through the push that follows ResumeActor, so a resume whose push did not arrive, or whose command could not be started, left a task that looked resumed with nothing running. Resume now returns a transient error when the runner does not accept the task within WorkspaceReadyTimeout. ResumeTask returns Unavailable, the task stays Queued, and resuming again repeats the push. Resume also polls /readyz?check=workspace on every resume, not only until WorkspaceReady is True. The runner already reports a command it could not start on that check, but with prewarm most tasks have WorkspaceReady True from creation, so the report was never read. A failure reported for a task whose workspace is ready now fails the resume with ActorResumeFailed and the runner's reason. Before the workspace is known to be ready, a failure is still recorded as WorkspaceReady False with SetupFailed, as a failed setup is. The reconciler tests' mock worker is now a runner that accepts the task, since an unreachable one fails the resume. TestTaskReconciler_ResumeCommandCannotStart resumes against the real runner.
|
I tested a752865 end to end on a local cluster with the script below. It builds AX from the checkout with
Both passed. Not covered: a runner that never accepts the push (the timeout error), which cannot be provoked from outside the cluster; Output of the two cases
|
|
PTAL |
yongruilin
left a comment
There was a problem hiding this comment.
The change LGTM now.
nit: PR description's "Known limitations" section, last point "unreachable runner is soft" should be updated.
|
|
||
| // actorRequest sends an HTTP request to port 80 of a running actor: directly | ||
| // at workerAddr, the address ResumeActor returned, and through atenet-router | ||
| // (ATENET_ROUTER_ADDR) with the ate-target-actor header when workerAddr is |
There was a problem hiding this comment.
IIUIC workers are protected by network policies and the ingress should be handled via the atenet-router
There was a problem hiding this comment.
oh..right. Then it seems that the direct dial would always time out and fall back. Maybe we should drop the direct direct dial here?
There was a problem hiding this comment.
+1. newProbeHTTPClient sets a 2s timeout, so each blackholed direct dial burns 2s out of the 15s pollCtx before falling back to atenet-router.
| @@ -0,0 +1,102 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
just in case you have the time if we consolidate we avoid drifting in the future
Have hack/test-local-install.sh call hack/build-local-cluster.sh (or merge into one script with a flag/mode), passing AX_BIN="${TMP_DIR}/bin/ax" and the isolated KUBECONFIG/AX_HOME. That removes ~70 duplicated lines, gives make test-local-install the stopped-registry and Podman fixes automatically, and keeps both make build-local-cluster and make test-local-install in sync. (Note: hack/build-local-cluster.sh is also separable from PR #688 as its own small PR.)
…ckend options (ax-experiments-1rx.3) Phase A of ax-experiments-1rx.3. Times upstream ax local v0.4.0-rc.1 (Kind on Podman, Apple M5) step by step next to ate-local at bfd5042 and answers the open question in google/ax#689: - Fresh ax local start: 128-129s with the node image cached; image pulls are not the bottleneck, serial readiness waits are. One unreproduced 1059s run waited 945s silently in the control-plane rollout loop (google/ax#648). - start after stop: 170-362s, slower than delete + start, because pod-certificate volumes cannot mount until podcertificate-controller recovers and kubelet's mount backoff plus alreadyHealthy's serial 90s waits follow. - ate-local up: 2.7s warm, 7.8s with a first build. On a running Kind cluster task resume is 0.10-0.14s vs 0.45-0.61s on ate-local, so the ate-local advantage is bring-up, not the task inner loop. Explains why ate-local cannot be a cluster.Provider (Up returns a KubeContext and runLocal hard-wires Bootstrap + axdeploy.Install), proposes a local.Backend seam (Option A, after google/ax#688/#696) alongside Kind optimizations (Option B), and adds an isolated, timestamped ax local harness with a pod-state monitor. Found ax ssh failing on ate-local at bfd5042 (ax-experiments-1rx.7).
mayawang
left a comment
There was a problem hiding this comment.
Direction looks great, nice to see :8081 and the pull loop gone. Left a few inline notes on handleTaskPush in setup actors, the first-resume error path in reconciler.go, and marking the push routes internal ahead of agent-substrate/substrate#3876.
Also worth tracking a follow-up issue to bring test/e2e/launcher_test.go (launcher contract and shared-workspace isolation) back under push mode.
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| switch { | ||
| case s.pushedTask != nil && sameObject(s.pushedTask.GetMetadata(), task.GetMetadata()): |
There was a problem hiding this comment.
Should we return 409 when s.pushedSetup != nil? Temporary setup actors only run workspace setup before being deleted, and :80 is reachable from 127.0.0.1 inside the sandbox during setup.commands.
Also nit: checking non-empty metadata.name/uid in sameObject avoids matching two empty-metadata YAML bodies as equal.
| switch { | ||
| case err == nil: | ||
| workspaceReady = true | ||
| case errors.Is(err, errSetupFailed) && workspaceReady: |
There was a problem hiding this comment.
In a752865, this branch only returns an error when workspaceReady was already True. On a task's first cold resume, a typo in spec.commands still records WorkspaceReady=False/SetupFailed and returns nil from Resume. Intentional so callers inspect the condition, or should first-resume start failures fail too? Same question for errNoAddress right above on line 352.
|
|
||
| | Request | Body | Sent to | | ||
| |---|---|---| | ||
| | `POST /runner/v1alpha1/task` | The `Task` as YAML, excluding status, followed by the bound `Workspace` as a second YAML document when the task binds one | Task actors, on every resume | |
There was a problem hiding this comment.
Let's mark /runner/v1alpha1/task and /setup explicitly internal/unstable for Substrate 1.0, so non-Go runners don't treat this as a stable contract before agent-substrate/substrate#3876 lands. Fine as a follow-up too.
| const ( | ||
| // RunnerSetupPath takes a Workspace as YAML and runs its spec.setup. It is | ||
| // how a temporary workspace setup actor is told what to prepare. | ||
| RunnerSetupPath = "/runner/v1alpha1/setup" |
There was a problem hiding this comment.
Same note as docs/runner.md, worth calling these constants internal/unstable in the doc comment while we wait for agent-substrate/substrate#3876.
|
I'm merging this and @yongruilin will address some of the missing feedback in the upcoming PRs. |
Removes the runner pull protocol (
pkg/apis/runner/v1) and moves task actors to push mode. Workspace setup still runs in a separate setup actor.Why
Actors come from profile templates shared by many tasks, so the runner has to learn its task at runtime. Until now the runner pulled it from ax-server's runner endpoint (
:8081), authenticated by an Actor JWT that the egress gateway injected. That needed a second server, JWT validation, a per-actor egress rule back to ax-server, and run-attempt bookkeeping in the task row.What changes
v1alpha1.RunnerTaskPath,v1alpha1.RunnerSetupPath):POST /runner/v1alpha1/task: Task YAML, optionally followed by the bound Workspace as a second document.POST /runner/v1alpha1/setup: Workspace YAML, sent to setup actors.202accepted,200already holds the same object (atespace, name, uid),409holds different work,400bad body./readyz?check=workspacereturns503while pending,200when ready, and500with the reason when setup failed. Plain/readyzreturns200while waiting, for Substrate's WakeupProbe.internal/controller/actorhttp.go) pushes afterResumeActorand polls the workspace check. It dials the worker IPResumeActorreturned and falls back toatenet-routerwithate-target-actor. The runner never calls ax-server.TaskReconciler.Resumepushes the task on every resume. A rejected push (4xx) fails the resume withActorResumeFailed. A runner-reported setup failure setsWorkspaceReady=False/SetupFailed.WorkspaceManageruses aSetupRunner(HTTPSetupRunnerin production) in place ofSetupCoordinator.runner.Config.AwaitPushreplacesPullMode/ControlURL.ax-runnerandclaude-code-runnerwait for a push when no task is given by env or file. A newConfig.PrepareTaskhook letsclaude-code-runnerrewrite pushed tasks.pkg/apis/runner/v1internal/runnerhttp(the:8081server and its JWTs)internal/lifecycleinternal/substrate/runner_egress.gorunner/pull.goandrunner/output_buffer.go--runner-addr,--runner-endpointand Actor JWT flagsax-server-runnerService and its NetworkPolicy rule (reverts deploy, e2e: expose ax-server-runner :8081 only to atenet-egress in NetworkPolicy #631)bindGatewayis back to Gateway rules only. A task without a Gateway gets no egress policy; the runner rule existed only to reach:8081.TaskRowis renamedTaskVersion.schema.sqladds noDROP COLUMN.Multiple ax-server replicas
The replica that builds a snapshot owns the whole flow:
WorkspaceManager.Deleteremoves leftovers by prefix.Known limitations
status.outcomeis never set by the runner.ResumeTask(router auto-resume, container restart) waits for a push until the nextResumeTask.--runner-addr; cmd/ax-server always provides anHTTPSetupRunner.WorkspaceReadystaysFalse/Initializingand the resume succeeds.Testing
make verify, the three builds, andgo test ./...pass locally.AX_TEST_POSTGRES_URL,postgres:17-alpineunder podman): 10 tests pass, none skipped.readyz500HTTPSetupRunner(rejection, no address, router fallback)make test-local-install.Local cluster (
make build-local-cluster)Built ax-server and ax-runner (linux/arm64) from this branch into the
localhost:5001registry.ax local startrolled the new ax-server out onto an existingax-localKind cluster (Substrate v1.0.0, podman), with no errors in the ax-server log. Results:push-wswithsetup.filesand asetup.commandsstepax applytook 4.3 s. One setup actor (ax-ws-setup-…) was resumed, the Workspace was pushed to/runner/v1alpha1/setup, and the snapshot was tagged.Phase: Ready,SnapshotReady=True.push-warm(bindspush-ws, no profile)ax resumetook 2.1 s.WorkspaceReady=True("seeded from snapshot tag …"),Ready=True/TaskRunning. Its command readhello.txtand thebuilt.txtwritten by the setup actor, saw itsspec.env, and ran as UID 1000.push-cold(own profile, no workspace)ax resumetook 2.1 s.Ready=True/TaskRunning; the command ran with its env as UID 1000.ax suspend/ax resume push-warmReady=True/TaskRunning.ax sshworks (spec.debug: true).push-bad-wswhose setup command runsexit 7500reason reached the user.ax applyfailed withFailedPrecondition … running setup.commands[0] ("echo about to fail; exit 7"): about to fail: exit status 7, and the Workspace showedPhase: FailedwithSetupComplete/SnapshotReadyFalse/SetupFailedcarrying the same message.Unchanged from main: a setup failure the runner reports is retried once on a fresh setup actor (
maxSetupAttempts), sopush-bad-wsused two setup actors. That can help with a flakygit clone, but it is wasted work for a deterministic failure. Marking a runner-reported failure as non-retryable could be a follow-up.Fixes #685, #681.