Skip to content

Emit ax.task.state_changed after committed task writes - #665

Open
JeffLuoo wants to merge 1 commit into
google:mainfrom
JeffLuoo:t5-task-state-events
Open

JeffLuoo wants to merge 1 commit into
google:mainfrom
JeffLuoo:t5-task-state-events

Conversation

@JeffLuoo

@JeffLuoo JeffLuoo commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Cost attribution and task timelines need a record of when a task changes phase, outcome, or actor that joins with Agent Substrate's ate.actor.* events.

  • internal/taskevents wraps store.TaskApplier, the single path for task row writes (8 sites in internal/server, 2 in internal/runnerhttp). The row being replaced and the commit timestamp are captured inside the locked ApplyFunc, so no extra read is needed and records are stamped in commit order (telemetry.EmitAt).
  • After a successful commit that changes the phase or outcome, or binds or replaces the actor, it emits ax.task.state_changed (OTLP or stdout JSON) with ax.task.phase, ax.task.previous_phase, ax.task.outcome, ax.task.previous_outcome, ax.task.exit_code (not for cancelled), ax.task.generation, ax.task.id, and ate.actor.uid. Failed and no-op writes emit nothing.
  • Live actor states are not stored, so a task whose actor owns its state reports Provisioned; ate.actor.state_changed on the same ate.actor.uid carries Working/Suspended/Crashed transitions.

Related to #13 and #471.

Validation

make verify, builds of all three binaries, and go test -race ./.... Table tests run against both the memory TaskApplier and the SaveTask/UpdateTaskStatus adapter, covering create, each transition, actor bind and rotation, outcome-only changes, cancelled outcomes, delete, no-op writes, and aborted writes; separate tests cover failed commits, insert-race retries, commit-time timestamps, and concurrent Apply.

Written with help from an AI coding assistant.

Cost attribution and task timelines need a record of when a task changes
phase, outcome, or actor, joinable with Agent Substrate's actor events.

Every task row write goes through store.TaskApplier.Apply, so
internal/taskevents wraps it. The row it replaces and the commit
timestamp are read inside the locked ApplyFunc; after a successful commit
that changes the phase or outcome, or binds or replaces the actor, the
event is emitted through telemetry.EmitAt at the commit time, so a task's
records are in commit order, with the row generation to break ties. It
carries the previous and new values, ax.task.id, and ate.actor.uid.
Failed and no-op writes emit nothing. Live actor states are not stored,
so a task whose actor owns its state reports the phase Provisioned;
ate.actor.state_changed carries the rest.
@JeffLuoo
JeffLuoo force-pushed the t5-task-state-events branch from 226eb58 to d75c553 Compare November 20, 2026 16:00
@JeffLuoo
JeffLuoo requested review from erain and mayawang November 20, 2026 16:20
@mayawang

Copy link
Copy Markdown
Collaborator

Thanks Jeff! Before getting into the design: we're in the bug-bash freeze for v0.4, and task timelines / cost attribution aren't on the v0.4 or v1.0 list yet. Since this wraps the task write path, could we park it until after v1.0 and talk through where it fits then? Happy to look sooner if there's a user waiting on it

@JeffLuoo

Copy link
Copy Markdown
Collaborator Author

Thanks Maya, happy to respect the v0.4 freeze. One framing note: this isn't only for cost. It's the task-level counterpart of Substrate's ate.actor.state_changed / ate.actor.crashed. Today AX keeps no history of why a task went Failed, when a delete moved it to Terminating, or when its actor was recreated with a new UID; ax describe only shows the current state. The events carry ate.actor.uid, so operators can line up task and actor timelines and tell an AX problem from a Substrate one.

What do you think would be the best version to have this PR? No user is blocked on it, so I'll keep it rebased and we can talk through where it fits after v1.0.

@mayawang

Copy link
Copy Markdown
Collaborator

That's a good point, task history for debugging is a real gap, and lining up with the ate.actor.* events makes sense. Could you open an issue with that framing? Let's look at where it fits once the rc is out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants