Skip to content

all: push tasks to runners and remove the runner pull protocol - #688

Merged
rakyll merged 4 commits into
mainfrom
run
Nov 22, 2026
Merged

rakyll merged 4 commits into
mainfrom
run

Conversation

@rakyll

@rakyll rakyll commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Removes the runner pull protocol (pkg/apis/runner/v1) and moves task actors to push mode. Workspace setup still runs in a separate setup actor.

Why

Actors come from profile templates shared by many tasks, so the runner has to learn its task at runtime. Until now the runner pulled it from ax-server's runner endpoint (:8081), authenticated by an Actor JWT that the egress gateway injected. That needed a second server, JWT validation, a per-actor egress rule back to ax-server, and run-attempt bookkeeping in the task row.

What changes

  • Push endpoints on the runner's port 80 (v1alpha1.RunnerTaskPath, v1alpha1.RunnerSetupPath):
    • POST /runner/v1alpha1/task: Task YAML, optionally followed by the bound Workspace as a second document.
    • POST /runner/v1alpha1/setup: Workspace YAML, sent to setup actors.
    • Replies: 202 accepted, 200 already holds the same object (atespace, name, uid), 409 holds different work, 400 bad body.
    • /readyz?check=workspace returns 503 while pending, 200 when ready, and 500 with the reason when setup failed. Plain /readyz returns 200 while waiting, for Substrate's WakeupProbe.
  • ax-server (internal/controller/actorhttp.go) pushes after ResumeActor and polls the workspace check. It dials the worker IP ResumeActor returned and falls back to atenet-router with ate-target-actor. The runner never calls ax-server.
    • TaskReconciler.Resume pushes the task on every resume. A rejected push (4xx) fails the resume with ActorResumeFailed. A runner-reported setup failure sets WorkspaceReady=False/SetupFailed.
    • WorkspaceManager uses a SetupRunner (HTTPSetupRunner in production) in place of SetupCoordinator.
  • Runner: runner.Config.AwaitPush replaces PullMode/ControlURL. ax-runner and claude-code-runner wait for a push when no task is given by env or file. A new Config.PrepareTask hook lets claude-code-runner rewrite pushed tasks.
  • Removed:
  • Egress: bindGateway is back to Gateway rules only. A task without a Gateway gets no egress policy; the runner rule existed only to reach :8081.
  • Store: TaskRow is renamed TaskVersion. schema.sql adds no DROP COLUMN.

Multiple ax-server replicas

The replica that builds a snapshot owns the whole flow:

  • It creates a setup actor with a random suffix, resumes it, pushes, and polls that actor directly. State lives in the runner, so replicas share nothing.
  • The Workspace lock (Postgres advisory lock and lease) keeps it to one builder.
  • If the replica crashes, the setup actor is orphaned. The next attempt uses a new actor, and WorkspaceManager.Delete removes leftovers by prefix.

Known limitations

  • ax-server no longer records exit codes or attempts, so status.outcome is never set by the runner.
  • An actor woken without ResumeTask (router auto-resume, container restart) waits for a push until the next ResumeTask.
  • The push endpoints are first-assignment-wins; a runner never switches tasks.
  • Prewarm can no longer be turned off with --runner-addr; cmd/ax-server always provides an HTTPSetupRunner.
  • On resume, an unreachable runner is soft: WorkspaceReady stays False/Initializing and the resume succeeds.

Testing

  • make verify, the three builds, and go test ./... pass locally.
  • Postgres tier (AX_TEST_POSTGRES_URL, postgres:17-alpine under podman): 10 tests pass, none skipped.
  • New unit tests:
    • the metadata server push endpoints and readyz 500
    • runner push mode (task, setup, setup failure)
    • HTTPSetupRunner (rejection, no address, router fallback)
    • Resume push rejection and setup failure
  • Not run: E2E and make test-local-install.

Local cluster (make build-local-cluster)

Built ax-server and ax-runner (linux/arm64) from this branch into the localhost:5001 registry. ax local start rolled the new ax-server out onto an existing ax-local Kind cluster (Substrate v1.0.0, podman), with no errors in the ax-server log. Results:

Scenario Result
Workspace push-ws with setup.files and a setup.commands step ax apply took 4.3 s. One setup actor (ax-ws-setup-…) was resumed, the Workspace was pushed to /runner/v1alpha1/setup, and the snapshot was tagged. Phase: Ready, SnapshotReady=True.
Warm task push-warm (binds push-ws, no profile) ax resume took 2.1 s. WorkspaceReady=True ("seeded from snapshot tag …"), Ready=True/TaskRunning. Its command read hello.txt and the built.txt written by the setup actor, saw its spec.env, and ran as UID 1000.
Cold task push-cold (own profile, no workspace) ax resume took 2.1 s. Ready=True/TaskRunning; the command ran with its env as UID 1000.
ax suspend / ax resume push-warm Resume took 2.1 s. A file written before the suspend survived, the fresh runner accepted the re-pushed task, and the task returned to Ready=True/TaskRunning. ax ssh works (spec.debug: true).
Workspace push-bad-ws whose setup command runs exit 7 The runner's 500 reason reached the user. ax apply failed with FailedPrecondition … running setup.commands[0] ("echo about to fail; exit 7"): about to fail: exit status 7, and the Workspace showed Phase: Failed with SetupComplete/SnapshotReady False/SetupFailed carrying the same message.

Unchanged from main: a setup failure the runner reports is retried once on a fresh setup actor (maxSetupAttempts), so push-bad-ws used two setup actors. That can help with a flaky git clone, but it is wasted work for a deterministic failure. Marking a runner-reported failure as non-retryable could be a follow-up.

Fixes #685, #681.

Actors come from profile templates shared by many tasks, so the runner
learns its work at runtime. Until now it pulled the task from ax-server's
runner endpoint (:8081) with an Actor JWT injected by the egress gateway.

ax-server now pushes the work instead. After ResumeActor it POSTs the Task,
followed by its Workspace, to /runner/v1alpha1/task on the actor's port 80,
or a Workspace to a setup actor at /runner/v1alpha1/setup, and polls
/readyz?check=workspace. It dials the worker IP ResumeActor returned and
falls back to atenet-router. Workspace setup still runs in its own setup
actor, which is snapshotted once setup is done.

Removed: pkg/apis/runner/v1, internal/runnerhttp, internal/lifecycle, the
:8081 server and its Service and NetworkPolicy rule, the Actor JWT flags,
the runner egress rule, and the TaskRow run-attempt fields. TaskRow is
renamed TaskVersion.
make build-local-cluster builds ax and the AX images from the checkout,
pushes them to the kind-registry container on localhost, and runs
`ax local start` with them. It uses podman or docker, builds the images
for the machine's architecture, starts or restarts the registry
container as needed, and keeps the cluster and the registry. On a
running cluster it only rolls out the new ax-server.

make images now skips TLS verification when podman pushes to a registry
on localhost or 127.0.0.1, which serves plain HTTP. docker already
allows that, so `make images` and `make test-local-install` failed only
with podman.
TestE2E_PureImageLauncher and TestE2E_PrewarmAndCustomImageSharedWorkspace
waited for a task to reach Completed with status.outcome set and its actor
suspended. Since tasks are pushed to runners, ax-server no longer reads a
command's exit back from the runner, so those tasks stay Working and both
tests timed out.

Also remove the stand-in launcher image (test/e2e/launcher) that only
these tests ran, and its build in hack/setup-e2e-kind.sh.

@yongruilin yongruilin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting up this. I left 2 questions.

Comment thread internal/controller/reconciler.go Resolved
Comment thread internal/controller/reconciler.go Resolved
A task reaches its runner only through the push that follows
ResumeActor, so a resume whose push did not arrive, or whose command
could not be started, left a task that looked resumed with nothing
running.

Resume now returns a transient error when the runner does not accept
the task within WorkspaceReadyTimeout. ResumeTask returns Unavailable,
the task stays Queued, and resuming again repeats the push.

Resume also polls /readyz?check=workspace on every resume, not only
until WorkspaceReady is True. The runner already reports a command it
could not start on that check, but with prewarm most tasks have
WorkspaceReady True from creation, so the report was never read. A
failure reported for a task whose workspace is ready now fails the
resume with ActorResumeFailed and the runner's reason. Before the
workspace is known to be ready, a failure is still recorded as
WorkspaceReady False with SetupFailed, as a failed setup is.

The reconciler tests' mock worker is now a runner that accepts the
task, since an unreachable one fails the resume.
TestTaskReconciler_ResumeCommandCannotStart resumes against the real
runner.
@rakyll

rakyll commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

I tested a752865 end to end on a local cluster with the script below. It builds AX from the checkout with make build-local-cluster, then runs two tasks from the runner image it built:

  1. A task whose command starts (sleep infinity): resume, suspend, and resume again all succeed.
  2. A task whose command does not exist (ax-no-such-command): ax resume fails and names the command, and ax describe shows the reason.

Both passed. Not covered: a runner that never accepts the push (the timeout error), which cannot be provoked from outside the cluster; TestTaskReconciler_WorkspaceReady covers it.

Output of the two cases
==> 1. A task whose command starts resumes, and resumes again after a suspend.
task.ax.io/fix-688-good created
task.ax.io/fix-688-good resumed
task.ax.io/fix-688-good suspended
task.ax.io/fix-688-good resumed
Name:         fix-688-good
Atespace:     default
UID:          52a80eff-d6e0-4a58-88e3-1f2bff668ae0
State:        Working
Actor:        fix-688-good
Actor UID:    6a6c679f-0206-4b05-8c4e-7d28e346aa74
Template:     default/ax-ec286e488caa883e
Attempt:      1
Session ID:   52a80eff-d6e0-4a58-88e3-1f2bff668ae0
Image:        localhost:5001/ax-runner@sha256:4427064c77e7ae461c843261f83ea060a18217706ad1028cf9bc0d613ae85c70
Commands:     [sleep infinity]

Conditions:
  TYPE            STATUS  REASON         MESSAGE
  WorkspaceReady  True    SetupComplete  Workspace requires no setup
  Ready           True    TaskRunning    Task is running and its workspace is ready

==> 2. A task whose command does not exist fails to resume.
task.ax.io/fix-688-typo created
Error: resuming task: rpc error: code = Internal desc = resuming task on substrate: starting task command [ax-no-such-command agent.py]: exec: "ax-no-such-command": executable file not found in $PATH
Name:         fix-688-typo
Atespace:     default
UID:          1efa833b-5a1a-46a4-af11-2b6c42bf3a7c
State:        Failed
Detail:       TASK_STATUS_REASON_PROVISIONING_FAILED (starting task command [ax-no-such-command agent.py]: exec: "ax-no-such-command": executable file not found in $PATH)
Actor:        fix-688-typo
Actor UID:    0e77e2ef-e2e4-40ef-af3f-90b127a409c5
Template:     default/ax-ec286e488caa883e
Attempt:      1
Session ID:   1efa833b-5a1a-46a4-af11-2b6c42bf3a7c
Image:        localhost:5001/ax-runner@sha256:4427064c77e7ae461c843261f83ea060a18217706ad1028cf9bc0d613ae85c70
Commands:     [ax-no-such-command agent.py]

Conditions:
  TYPE            STATUS  REASON             MESSAGE
  WorkspaceReady  True    SetupComplete      Workspace requires no setup
  Ready           False   ActorResumeFailed  starting task command [ax-no-such-command agent.py]: exec: "ax-no-such-command": executable file not found in $PATH

==> PASS: a task that cannot start fails its resume, and a healthy task still resumes.
fix-688.sh (not checked in; run it from .tmp/ in a checkout)
#!/usr/bin/env bash
#
# End-to-end check of the fix for the review on https://github.com/google/ax/pull/688:
# a task whose command cannot be started (a typo in spec.commands) must not
# resume as if it were running.
#
# It builds AX from this checkout into the local cluster with
# `make build-local-cluster` (your kubeconfig and ~/.ax; the cluster stays up),
# then runs two tasks from the runner image it built:
#
#   1. A task whose command starts: resume, suspend, and resume again all
#      succeed. The runner is now polled on every resume, so this guards
#      against a healthy task failing a later resume.
#   2. A task whose command does not exist: `ax resume` fails and names the
#      command, and `ax describe` shows the reason.
#
# Not covered: a runner that never accepts the push (the timeout error), which
# cannot be provoked from outside the cluster. The unit tests cover it.
#
# Usage, from anywhere:
#   .tmp/fix-688.sh
#
# Configurable environment variables:
#   SKIP_BUILD        Set to true to reuse the running cluster and bin/ax as they are
#   AX_LOCAL_CLUSTER  Name of the local cluster (default: ax-local)
#   CONTAINER_CLI     Container engine (default: podman, else docker)
#   AX_TASK_IMAGE     Runner image for the tasks (default: the one in the local registry)
#   KEEP_TASKS        Set to true to leave the two tasks for inspection

set -euo pipefail
export GOWORK=off GOTOOLCHAIN=auto

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${REPO_ROOT}"

CONTAINER_CLI="${CONTAINER_CLI:-$(command -v podman || command -v docker || true)}"
AX="${REPO_ROOT}/bin/ax"
GOOD="fix-688-good"
TYPO="fix-688-typo"
BAD_COMMAND="ax-no-such-command"
unset AX_SERVER

log() {
  printf '\n==> %s\n' "$*" >&2
}

fail() {
  printf '\nFAIL: %s\n' "$*" >&2
  exit 1
}

cleanup() {
  local status=$?
  if ((status != 0)) && [[ -x "${AX}" ]]; then
    log "Failed. The tasks as AX reports them:"
    "${AX}" describe task "${GOOD}" >&2 || true
    "${AX}" describe task "${TYPO}" >&2 || true
  fi
  if [[ "${KEEP_TASKS:-false}" != "true" && -x "${AX}" ]]; then
    "${AX}" delete task "${GOOD}" > /dev/null 2>&1 || true
    "${AX}" delete task "${TYPO}" > /dev/null 2>&1 || true
  fi
  exit "${status}"
}
trap cleanup EXIT

if [[ "${SKIP_BUILD:-false}" != "true" ]]; then
  log "Building AX from this checkout into the local cluster (make build-local-cluster)..."
  make build-local-cluster CONTAINER_CLI="${CONTAINER_CLI}"
fi
[[ -x "${AX}" ]] || fail "${AX} not found; run without SKIP_BUILD=true"

# The runner image `make build-local-cluster` pushed, found the way
# hack/build-local-cluster.sh finds the registry.
if [[ -z "${AX_TASK_IMAGE:-}" ]]; then
  endpoint="$(sed -n 's/.*--localhost-registry-replacement=\([^[:space:]"]*\).*/\1/p' \
    internal/install/substrate/manifests/05-ate-system.yaml | head -n1)"
  [[ "${endpoint}" == *:* ]] || fail "cannot read the local registry from the embedded Substrate manifests"
  registry_port="$("${CONTAINER_CLI}" port "${endpoint%%:*}" "${endpoint##*:}/tcp" | head -n1 | sed 's/.*://')"
  [[ -n "${registry_port}" ]] || fail "the registry container '${endpoint%%:*}' is not running"
  AX_TASK_IMAGE="localhost:${registry_port}/ax-runner@$(cat bin/ax-runner.digest)"
fi
log "Tasks run from ${AX_TASK_IMAGE}"

apply_task() {
  local name="$1" commands="$2"
  "${AX}" apply -f - << EOF
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: ${name}
  atespace: default
spec:
  debug: true
  profile:
    image: "${AX_TASK_IMAGE}"
  commands: ${commands}
EOF
}

# Leftovers of an earlier run would make apply a no-op.
"${AX}" delete task "${GOOD}" > /dev/null 2>&1 || true
"${AX}" delete task "${TYPO}" > /dev/null 2>&1 || true

log "1. A task whose command starts resumes, and resumes again after a suspend."
apply_task "${GOOD}" '["sleep", "infinity"]'
"${AX}" resume task "${GOOD}" || fail "resuming ${GOOD} failed"
"${AX}" suspend task "${GOOD}" || fail "suspending ${GOOD} failed"
"${AX}" resume task "${GOOD}" || fail "resuming ${GOOD} after a suspend failed"
"${AX}" describe task "${GOOD}"

log "2. A task whose command does not exist fails to resume."
apply_task "${TYPO}" "[\"${BAD_COMMAND}\", \"agent.py\"]"
if out="$("${AX}" resume task "${TYPO}" 2>&1)"; then
  printf '%s\n' "${out}" >&2
  "${AX}" describe task "${TYPO}" >&2 || true
  fail "ax resume task ${TYPO} succeeded although '${BAD_COMMAND}' cannot be started"
fi
printf '%s\n' "${out}"
grep -q "${BAD_COMMAND}" <<< "${out}" ||
  fail "the error of ax resume does not name '${BAD_COMMAND}'"

described="$("${AX}" describe task "${TYPO}")"
printf '%s\n' "${described}"
# The Commands line names the command for every task; the Detail line carries
# the reason the resume failed.
grep '^Detail:' <<< "${described}" | grep -q "${BAD_COMMAND}" ||
  fail "ax describe task ${TYPO} does not show why the resume failed"

log "PASS: a task that cannot start fails its resume, and a healthy task still resumes."

@rakyll

rakyll commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

PTAL

@rakyll
rakyll requested a review from yongruilin November 21, 2026 23:32

@yongruilin yongruilin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change LGTM now.
nit: PR description's "Known limitations" section, last point "unreachable runner is soft" should be updated.


// actorRequest sends an HTTP request to port 80 of a running actor: directly
// at workerAddr, the address ResumeActor returned, and through atenet-router
// (ATENET_ROUTER_ADDR) with the ate-target-actor header when workerAddr is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUIC workers are protected by network policies and the ingress should be handled via the atenet-router

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh..right. Then it seems that the direct dial would always time out and fall back. Maybe we should drop the direct direct dial here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. newProbeHTTPClient sets a 2s timeout, so each blackholed direct dial burns 2s out of the 15s pollCtx before falling back to atenet-router.

@@ -0,0 +1,102 @@
#!/usr/bin/env bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just in case you have the time if we consolidate we avoid drifting in the future

Have hack/test-local-install.sh call hack/build-local-cluster.sh (or merge into one script with a flag/mode), passing AX_BIN="${TMP_DIR}/bin/ax" and the isolated KUBECONFIG/AX_HOME. That removes ~70 duplicated lines, gives make test-local-install the stopped-registry and Podman fixes automatically, and keeps both make build-local-cluster and make test-local-install in sync. (Note: hack/build-local-cluster.sh is also separable from PR #688 as its own small PR.)

ghchinoy added a commit to ghchinoy/ax-experiments that referenced this pull request Nov 22, 2026
…ckend options (ax-experiments-1rx.3)

Phase A of ax-experiments-1rx.3. Times upstream ax local v0.4.0-rc.1
(Kind on Podman, Apple M5) step by step next to ate-local at bfd5042
and answers the open question in google/ax#689:

- Fresh ax local start: 128-129s with the node image cached; image
  pulls are not the bottleneck, serial readiness waits are. One
  unreproduced 1059s run waited 945s silently in the control-plane
  rollout loop (google/ax#648).
- start after stop: 170-362s, slower than delete + start, because
  pod-certificate volumes cannot mount until podcertificate-controller
  recovers and kubelet's mount backoff plus alreadyHealthy's serial
  90s waits follow.
- ate-local up: 2.7s warm, 7.8s with a first build. On a running Kind
  cluster task resume is 0.10-0.14s vs 0.45-0.61s on ate-local, so the
  ate-local advantage is bring-up, not the task inner loop.

Explains why ate-local cannot be a cluster.Provider (Up returns a
KubeContext and runLocal hard-wires Bootstrap + axdeploy.Install),
proposes a local.Backend seam (Option A, after google/ax#688/#696)
alongside Kind optimizations (Option B), and adds an isolated,
timestamped ax local harness with a pod-state monitor.

Found ax ssh failing on ate-local at bfd5042 (ax-experiments-1rx.7).

@mayawang mayawang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direction looks great, nice to see :8081 and the pull loop gone. Left a few inline notes on handleTaskPush in setup actors, the first-resume error path in reconciler.go, and marking the push routes internal ahead of agent-substrate/substrate#3876.

Also worth tracking a follow-up issue to bring test/e2e/launcher_test.go (launcher contract and shared-workspace isolation) back under push mode.

s.mu.Lock()
defer s.mu.Unlock()
switch {
case s.pushedTask != nil && sameObject(s.pushedTask.GetMetadata(), task.GetMetadata()):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we return 409 when s.pushedSetup != nil? Temporary setup actors only run workspace setup before being deleted, and :80 is reachable from 127.0.0.1 inside the sandbox during setup.commands.

Also nit: checking non-empty metadata.name/uid in sameObject avoids matching two empty-metadata YAML bodies as equal.

switch {
case err == nil:
workspaceReady = true
case errors.Is(err, errSetupFailed) && workspaceReady:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a752865, this branch only returns an error when workspaceReady was already True. On a task's first cold resume, a typo in spec.commands still records WorkspaceReady=False/SetupFailed and returns nil from Resume. Intentional so callers inspect the condition, or should first-resume start failures fail too? Same question for errNoAddress right above on line 352.

Comment thread docs/runner.md

| Request | Body | Sent to |
|---|---|---|
| `POST /runner/v1alpha1/task` | The `Task` as YAML, excluding status, followed by the bound `Workspace` as a second YAML document when the task binds one | Task actors, on every resume |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's mark /runner/v1alpha1/task and /setup explicitly internal/unstable for Substrate 1.0, so non-Go runners don't treat this as a stable contract before agent-substrate/substrate#3876 lands. Fine as a follow-up too.

const (
// RunnerSetupPath takes a Workspace as YAML and runs its spec.setup. It is
// how a temporary workspace setup actor is told what to prepare.
RunnerSetupPath = "/runner/v1alpha1/setup"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same note as docs/runner.md, worth calling these constants internal/unstable in the doc comment while we wait for agent-substrate/substrate#3876.

@rakyll

rakyll commented Nov 22, 2026

Copy link
Copy Markdown
Member Author

I'm merging this and @yongruilin will address some of the missing feedback in the upcoming PRs.

@rakyll
rakyll merged commit 2ede873 into main Nov 22, 2026
13 checks passed
@rakyll
rakyll deleted the run branch November 22, 2026 17:59
ghchinoy added a commit to ghchinoy/ax that referenced this pull request Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runnerhttp: move internal/runnerhttp under runner/runnerhttp Redesign the runner proto

4 participants