Skip to content

cli.md

Latest commit

 

History

History
986 lines (870 loc) · 53.5 KB

cli.md

File metadata and controls

986 lines (870 loc) · 53.5 KB
Raw

CLI Commands

The CLI binary is ultrafuzz.

Every command accepts --project <path>. Commands that support automation accept --json and emit the ultrafuzz.cli.result.v2 envelope.

Commands

Command Purpose
ultrafuzz init Create root config plus .ultrafuzz/** product surfaces and workflow plumbing.
ultrafuzz validate Validate config, topology, prompts, path guards, agent references, and trust posture without launching agents.
ultrafuzz json validate Validate one JSON document against a strict local Draft 2020-12 schema without mutation.
ultrafuzz artifact validate Validate a declared artifact's registered schema and document-local semantic gates.
ultrafuzz run Validate, render prompts, build run evidence, compile a workflow, and launch a linked workflow run.
ultrafuzz config audit-profiles List shipped audit profiles, intended uses, and selected topologies.
ultrafuzz config audit-profile <name> Show one profile's effective project topology and settings.
ultrafuzz topology list List packaged topology assets, logical node counts, and digests.
ultrafuzz topology show <name> Show metadata and YAML for one packaged topology.
ultrafuzz topology copy <name> <path> Safely copy a packaged topology into the project.
ultrafuzz references status Report whether pinned references are present in the local digest-checked cache.
ultrafuzz references sync Explicitly fetch pinned references into the local cache.
ultrafuzz references update Rewrite the project reference catalog to newer pinned commits when requested.
ultrafuzz ps List Ultrafuzz runs and linked workflow status.
ultrafuzz inspect <run-id> Show product evidence and linked workflow details for a run.
ultrafuzz status <run-id> Show run health, progress, whether execution ended, and final report availability.
ultrafuzz stats <run-id> Derive per-node timing, token usage, cost, retry, outcome, and completeness statistics.
ultrafuzz pause <run-id> Gracefully pause an active run after its in-flight tasks finish.
ultrafuzz why <run-id> Diagnose why a run is blocked, paused, quota-parked, waiting, or unable to progress.
ultrafuzz timeline <run-id> Show checkpoint frames and fork lineage, including the frame numbers fork --frame accepts.
ultrafuzz events <run-id> Show linked workflow lifecycle events, with optional streaming.
ultrafuzz node <run-id> <node-id> Show one workflow node's status, attempts, retries, timing, and output metadata.
ultrafuzz snapshots <run-id> List durability and workspace checkpoints for recovery and time-travel diagnosis.
ultrafuzz cancel <run-id> Cancel an active run; cancellation is terminal, unlike pause.
ultrafuzz doctor Report validation, toolchain, and engine posture without changing project or run state.
ultrafuzz resume <run-id> Delegate resume for the linked workflow run after product checks.
ultrafuzz replay <run-id> Delegate replay for the linked workflow run after product checks.
ultrafuzz fork <run-id> Delegate fork for the linked workflow run after product checks.
ultrafuzz report <run-id> Locate the agent-written final report artifacts.
ultrafuzz materialize <run-id> Copy selected reviewed outputs into the target project after confirmation and path checks.
ultrafuzz clean <run-id> Remove selected generated .ultrafuzz/** paths after confirmation and path checks.
ultrafuzz dashboard Serve the local loopback dashboard and API for product state inspection and editing.
ultrafuzz eval plan Dry-run an eval suite matrix without launching workflows.
ultrafuzz eval run Launch Ultrafuzz runs for an eval suite matrix and watch local progress.
ultrafuzz eval status <id> Show disclosure-safe node progress and ETA for every row in an eval matrix.
ultrafuzz eval score <id> Score finished eval run reports against external ground truth.
ultrafuzz eval report <id> Show the scored eval run variant ranking.
ultrafuzz eval compare <id> Compare scored eval variants against a baseline variant.
ultrafuzz eval analyze <type> Generate private offline tables, provenance, score, intersection, and cost reports from a finalized handoff.
ultrafuzz eval history [id] Validate/render public eval history, or append one complete scored run.

Generated workflow-engine files are implementation plumbing. The stable product surfaces are root ultrafuzz.toml, .ultrafuzz/**, reviewed project files, and the CLI commands above.

Global Flags

Flag Meaning
--project <path> Project root. Defaults to the current working directory.
--json Emit a schema-versioned JSON envelope instead of human text.

JSON output has this shape:

{
  "schema_version": "ultrafuzz.cli.result.v2",
  "command": "init",
  "ok": true,
  "diagnostics": [],
  "data": {
    "project_root": "/project",
    "created": [],
    "preserved": [],
    "overwritten": []
  }
}

Machine consumers should read ok, diagnostics, and data. The v2 envelope is intentionally breaking and command-discriminated: every known command has a closed data shape, public diagnostics omit private implementation details, and unknown invocation failures can emit only ok: false with data: null. Only explicit operator workflow input and redacted third-party tool input/output retain deliberately open nested JSON. Version 1 is not accepted or converted.

Init

ultrafuzz init [--project <path>] [--force] [--json]

init creates or preserves:

ultrafuzz.toml
.ultrafuzz/topology.yml
.ultrafuzz/prompts/**
.ultrafuzz/references.yml
.ultrafuzz/runs/
.ultrafuzz/workspaces/
.ultrafuzz/cache/

Without --force, existing config, topology, prompts, and reference catalogs are preserved. The .smithers/agents adapter closure is not project-owned: planning accepts only the byte-exact packaged copy, so every init rewrites each adapter that differs from the current templates and leaves matching ones untouched. Refreshing the adapters after an upgrade therefore needs no --force. A symbolic link, hard link, or special file at an adapter path makes init fail with INIT_PATH_UNSAFE instead of being followed or written through. init also migrates a recognized generated smithers-orchestrator@0.32.0 manifest or a smthrs@0.34.0 manifest to the current smthrs@0.35.0 pin. The 0.34.0 migration requires the rest of the manifest to match the current generated document. Use --force to replace the other existing generated files with the current templates.

Validate

ultrafuzz validate [--project <path>] [--audit-profile <name>] \
  [--topology-path <path>] [--json]

Validation covers typed TOML config, .ultrafuzz/topology.yml, project prompt copies, safe paths, reference nodes, agent references, and trusted local execution posture. It does not launch agents. A project prompt that differs from the built-in prompt at the same path sets the prompts posture to warn with one PROMPT_DIFFERS_FROM_BUILT_IN warning per file; warnings do not fail validation.

JSON Validate

ultrafuzz json validate \
  --schema <schema.json> \
  --file <artifact.json> \
  [--ref <local-schema.json>] \
  [--max-errors <1-1000>] \
  [--json]

This is a single-document validator, separate from project-wide ultrafuzz validate. It uses strict Draft 2020-12 Ajv validation, standard formats, strict UTF-8 and JSON parsing, duplicate-key rejection, offline local references, and bounded worker execution. --ref may be repeated. No stdin, YAML, JSON5, remote schema fetching, coercion, defaults, property removal, or repair is supported.

The exit contract is:

Exit Meaning
0 The artifact conforms to the schema.
1 The artifact is unreadable, malformed, has duplicate keys, or violates the schema.
2 Invocation, schema/reference loading, schema compilation, resource limits, or validator setup failed.

Human-readable failures go to stderr. --json emits the standard ultrafuzz.cli.result.v2 envelope. Neither success nor failure changes the schema or artifact bytes.

Checked-in artifact, eval, Modal, and topology schemas are loaded from composed package-local registries. A registered schema whose filename or bytes differ from its pinned entry is a setup failure. Successful JSON output reports whether the schema was registered plus its fragment-free ID, schema SHA-256, owning package's bundle SHA-256, validator build identity, and the artifact SHA-256. The schema identities bind the producer command to the later host check, and the validator build is reported as provenance only; none of them is a mutable validation receipt.

For schema-backed producer tasks, Ultrafuzz places a run-owned trusted launcher before target-controlled PATH entries and validates a real known-valid fixture before model work. Modal uses a root-owned, read-only /usr/local/bin/ultrafuzz. Missing, stale, shadowed, or tampered validator identity is an exit-2 setup failure, not a reason to use another binary or edit the supplied schema.

Producer prompts display exact schema and contract validation commands for each JSON output. The producer must run every command after its final write and before returning; an exit 1 draft is corrected and rerun in the same session. Once the session returns, the host checks the same bytes and then applies named semantic/context gates. It never repairs, converts, normalizes, synthesizes, or falls back to another artifact, and post-session shape failure is terminal rather than a model retry.

Artifact Validate

ultrafuzz artifact validate <contract-id> <artifact-path> [--json]

Use the contract ID and path declared in the rendered Ultrafuzz Output Contract. The command applies the contract's registered JSON Schema and every document-local semantic gate, reporting failures without modifying the file. It does not authenticate workspace inputs or evaluate cross-artifact or run context. The runtime performs those publication checks after the producer finishes. Run ultrafuzz artifact validate --help for the current positional arguments.

Run

ultrafuzz run \
  [--project <path>] \
  [--run-id <id>] \
  [--input-json <strict-json> | --input-file <json-path>] \
  [--prompt <text>] \
  [--agent <agent-ref>] \
  [--model <model>] \
  [--audit-profile <name>] \
  [--topology-path <path>] \
  [--max-concurrency <n>] \
  [--json]

--input-json and --input-file are mutually exclusive. Inline input is always strict RFC 8259 JSON and is never retried as a path after a parse error. Relative file paths resolve from the project root. File input is read from one bounded, non-symlink regular-file snapshot and is rejected if it changes during the read. Both forms reject invalid UTF-8, duplicate keys, malformed JSON, and non-finite numbers. Their application data is explicitly operator-defined; Ultrafuzz validates that it is JSON but does not infer, repair, normalize, or convert its domain shape. Model-only overrides keep the configured agent and reasoning. When --agent selects another agent, backend-specific reasoning is cleared, including when --model also pins a replacement model. --max-concurrency caps workflow task submission concurrency. --audit-profile selects a profile for one command, while --topology-path atomically replaces the project or profile topology for that command.

A run ID must be new to the project. run fails with RUN_ALREADY_EXISTS before it plans anything when the run directory exists, or when the workflow engine already records a run with that ID. The second case happens after ultrafuzz clean <run>, which removes the run directory but not the engine's record. Choose another --run-id, or continue a run whose directory still exists with ultrafuzz resume <run-id>. run asks the engine only once the project has engine records, so a project's first launch skips the query.

Audit Profiles and Packaged Topologies

ultrafuzz config audit-profiles [--json]
ultrafuzz config audit-profile <name> [--project <path>] [--json]
ultrafuzz topology list [--json]
ultrafuzz topology show <name> [--json]
ultrafuzz topology copy <name> <project-relative-path> [--force] [--json]

Profile detail resolves against the selected project so explicit project settings and topology overrides are visible. topology copy stays inside the project, rejects symlink paths, and does not replace an existing file unless --force is present.

Runs require pinned reference material to already be present in the local cache when the topology uses reference nodes. Use ultrafuzz references sync as the explicit network step before run.

References

ultrafuzz references status [--project <path>] [--json]
ultrafuzz references sync [--project <path>] [--json]
ultrafuzz references update --latest [--project <path>] [--json]

status validates the reference catalog and digest-checked cache presence. sync fetches pinned catalog entries into the local cache. update requires --latest and rewrites .ultrafuzz/references.yml to current default-branch SHAs.

Run Lifecycle

ultrafuzz ps [--project <path>] [--json]
ultrafuzz inspect <run-id> [--project <path>] [--json]
ultrafuzz status <run-id> \
  [--project <path>] \
  [--window <minutes>] \
  [--watch] \
  [--interval <seconds>] \
  [--json]
ultrafuzz stats <run-id> [--project <path>] [--json]
ultrafuzz stats --bundle <report-bundle.zip> [--json]
ultrafuzz pause <run-id> [--project <path>] [--json]
ultrafuzz cancel <run-id> [--project <path>] [--json]
ultrafuzz why <run-id> [--project <path>] [--json]
ultrafuzz timeline <run-id> [--tree] [--project <path>] [--json]
ultrafuzz snapshots <run-id> [--project <path>] [--json]
ultrafuzz events <run-id> \
  [--node <node-id>] \
  [--type <category>] \
  [--since <duration>] \
  [--limit <n>] \
  [--watch] \
  [--interval <seconds>] \
  [--history] \
  [--project <path>] \
  [--json]
ultrafuzz node <run-id> <node-id> \
  [--iteration <n>] \
  [--attempts] \
  [--tools] \
  [--watch] \
  [--interval <seconds>] \
  [--project <path>] \
  [--json]
ultrafuzz resume <run-id> [--project <path>] [--max-concurrency <n>] \
  [--reset-node <workflow-node-id>] [--retry-failed] [--refresh-controller] [--json]
ultrafuzz replay <run-id> [--project <path>] [--json]
ultrafuzz fork <run-id> \
  [--project <path>] \
  [--frame <n>] \
  [--reset-node <workflow-node-id>] \
  [--label <label>] \
  [--max-concurrency <n>] \
  [--json]

status, pause, resume, replay, and fork operate on the workflow run linked from Ultrafuzz run metadata. status reports a concise health verdict and maps workflow details into the stable Ultrafuzz JSON envelope.

Detached run and resume record local running state when Smithers admits the workflow. If that workflow later terminates without another successful Ultrafuzz synchronization, raw state.json can retain that admission-time state: there is not yet a detached completion reconciler. ps therefore shows the linked workflow status when it is available; its JSON output retains both workflow_status and ultrafuzz_status so lifecycle divergence remains explicit.

A launch that fails after planning has created its run directory, but before its workflow controls are sealed (while planning, compiling the workflow, or checking its task manifest), records the run as failed with its original error. status, resume, and the other commands that read the linked workflow then report that error as RUN_LAUNCH_FAILED. Such a run never reached Smithers and cannot be resumed; fix the cause and start a new run. A launch that fails after sealing is also recorded as failed, with its error in events.jsonl, but those commands do not report it as RUN_LAUNCH_FAILED. A goal-planning topology without its vulnerability-database reference, and a missing or invalid reference cache (run ultrafuzz references sync), are rejected before the run directory is created.

resume delegates continuation to Smithers with the same Ultrafuzz and Smithers run IDs and automatically accepts changed workflow source. Control seals, link journals, controller generations, graph fingerprints, current schema bindings, and metadata projections remain provenance for inspection; they are not resume authorization. The resumed tasks keep the schemas the run was planned with: after an upgrade that changed a schema, task preparation copies, and the verifier and dependency admission validate against, the bundle sealed in the run's execution snapshot. Smithers decides which finished rows can be reused and which newly rendered or unfinished tasks run. Ultrafuzz does not rewrite historical artifacts or automatically reset, replay, timetravel, or fork completed work. The run's smithers/resolved-config.json records the execution mode its tasks were planned with, so before starting Smithers resume fails with WORKFLOW_LIFECYCLE_FAILED when that file is missing or does not parse as the current resolved-config schema, and with WORKFLOW_CLOUD_EXECUTION_REMOVED when it records mode = "cloud". A plain resume of a run whose launch workflow predates planned schema bundles fails with WORKFLOW_CONTROLLER_REFRESH_REQUIRED, also before starting Smithers, because that workflow cannot prepare a task with this release; continue such a run with resume --refresh-controller, every time. Agent adapters in the continued workflow read the run's smithers/execution-config.toml (launch gave them a copy of the same file). If resume cannot prune stale task-worktree registrations, it reports a WORKFLOW_WORKTREE_REPAIR_FAILED warning and continues.

resume --retry-failed reruns every failed task except a failed task in a failure_policy: continue group that a started task already ran without. That task was admitted while the failed one had no output, and a rerun cannot reach the outputs it already produced, so the run would read as complete over work that never used the rerun's output. resume leaves such a task failed, reports a WORKFLOW_RETRY_SKIPPED warning that names it and the tasks that ran without it, and the final report stays partial. A task whose preparation failed has not started: its preparation, which admits its inputs, reruns after the failed task, so both are retried. A task whose agent or verifier failed reruns at once, without waiting for the failed task, so it counts as started. The Modal benchmark worker does not resume a finished run whose only failures are such tasks and the tasks skipped because one of them failed.

resume runs the workflow engine from Ultrafuzz's own install: pnpm applies the committed compatibility patches (patches/) to it at install time, so resume makes no package-registry request and leaves nothing in the OS temporary directory. Resume refuses an install that lacks any of those patches, such as a plain npm install of the packed packages, and an engine inside the target project. Launch refuses the same install before it creates the run directory, because the run could not be resumed; replay and fork run the run's sealed engine instead. To fix the install, run pnpm install --frozen-lockfile && pnpm -w build in the Ultrafuzz repository checkout. The build matters because the patch registry that checks the installed files is compiled into Ultrafuzz.

Run long campaigns from a dedicated checkout or worktree, and leave its install and build alone while they run. The resumed engine and its supervisor run from that install, and the resumed workflow imports that checkout's built @ultrafuzz/runtime. A pnpm install there that changes the engine's resolved dependency tree moves the engine to a new package directory. A Smithers patch changes that tree, and so does a version change anywhere in the engine's dependency graph, such as TypeScript or React; a pull or a branch switch often brings both. pnpm then deletes the old directory, in that install or a later one: by default it clears orphaned package directories during an install once seven days have passed since it last did. From then on the running engine can fail on its next lazy import, and the supervisor's next relaunch of it fails. Run ultrafuzz resume again to continue the run from the new install. Restart long-lived Ultrafuzz processes, such as the dashboard or ultrafuzz eval run, after such an install: each resolves the engine once, so once that directory is gone their engine commands fail with restart this Ultrafuzz process.

Every task of a resumed run, with or without --refresh-controller, --retry-failed or --reset-node, and of a replay or fork, receives the run's own artifacts/<attempt-id>/prompt.rendered.md. Unless run.refresh_prompts_on_resume = false, resume first renders that file again from the project's current prompts for every task that has not finished; replay and fork never do. Before any of these commands starts an engine, it restores a missing static prompt from its launch copy in prompt-snapshots/, never over an existing file. A prompt that is still missing or cannot be read, or a runtime prompt that no longer renders, fails only its task. See Change a prompt of a running campaign.

A run that ends failed with no failed durable node was stopped by something no durable node owns: a run-level workflow runner error, such as an exception thrown while rendering the workflow, or a failed workflow task outside the durable graph. status reports it as WORKFLOW_TERMINAL_WITHOUT_FAILED_NODE, names any failed workflow tasks, and appends the runner's run-level error to the message. For a run-level error, fix that cause, then resume the same run; it continues from the tasks that already finished. While the cause persists, the resume either fails with WORKFLOW_LIFECYCLE_FAILED or the run fails again.

resume --refresh-controller first renders the currently installed Ultrafuzz controller and stock adapters beside the historical source, then delegates to that same Smithers run. It does not publish or authenticate a historical controller generation. The refreshed workflow keeps each declared output's recorded schema binding, contract digest, and validator build, so its verification markers still match the run's sealed plan after a rebuild or an upgrade that changed a schema. After such an upgrade the refresh also keeps the run's own validator launcher, which validates with the run's planned schemas, instead of switching to the installed CLI, which validates with other ones. It reports WORKFLOW_TRUSTED_CLI_UNVERIFIED when it cannot verify the launcher it keeps or the installed CLI it switches to. Refresh rejects an actively owned workflow. Because Smithers admits changed workflow source, replay determinism is the operator's responsibility; inspect the retained source and Smithers workflow hash when auditing a continuation.

status human output is watch-friendly:

Run: <run-id>
Status: running-healthy (running)
Run ended: no
Report: pending
Report reason: The run has not ended.
Report completion: UNKNOWN
Report verification: unknown
Reason: 1 running, 2 finished in last 10m
Progress: 99% (262 finished / 1 running / 1 pending / 0 failed / 264 total)
ETA: 20 minutes
Time on current step: 10 minutes on stateful-invariant-campaign
Pace: 4 finished in the last 10m

A poll that succeeds with diagnostics, such as a WORKFLOW_DEADLINE_CANCEL_FAILED or WORKFLOW_STATE_SYNC_SKIPPED warning, prints each one on stdout after these lines as <severity>: <code>: <message>, where a message can span several lines. The warnings do not change the exit status, so scripts should read them from the diagnostics array of --json output.

--watch re-polls every --interval seconds (default 30) until the run reaches a terminal state (succeeded, failed, timed-out, or canceled), needs attention, is paused, or the poll fails. With --json --watch, every poll writes one newline-delimited ultrafuzz.cli.result.v2 envelope so the stream pipes into jq and other line-oriented tools; without --watch, --json keeps the existing pretty-printed single envelope.

stats derives its snapshot on demand; it does not read or write a precomputed statistics artifact. Local-run mode synchronizes linked workflow evidence when available, then reads attempts.jsonl, usage.jsonl, state.json, graph.json, graph.fingerprint, and run.json twice as one evidence set. The command accepts the bytes only when both complete reads match, retries a recognized mutation race at most three times, and timestamps the snapshot immediately after the accepted second read. The table shows each node's status, completed and currently elapsed execution time, token components, estimated spend, model, and attempt count. JSON output uses ultrafuzz.stats.v1 inside the normal CLI envelope and additionally exposes retries, executed/reused counts, outcomes, failure categories, completeness, unattributed usage, and cumulative run accounting. Run state records a task that Smithers cancelled, for example by ultrafuzz cancel, as failed. When Smithers cancelled every failed task of a node, stats reports the node with status canceled and counts it under canceled rather than failed; status likewise counts cancelled tasks apart from failures. The closed stats v1 field pricing_complete continues to mean complete cost coverage: it is the inverse of run accounting partial_pricing. Accounting v4's own pricing_complete field independently reports whether each event had sufficient pricing evidence; an adapter-recorded cost satisfies that evidence without requiring catalog component rates.

stats --bundle reads an ultrafuzz report bundle ZIP directly without extracting it and without the original checkout, workflow backend, provider, or network. It accepts only the registered ultrafuzz.report-bundle-manifest.v3 contract and validates the current run, state, graph, graph-fingerprint, and ledger contracts against the manifest run ID. The manifest creation time anchors live elapsed calculations and must not precede any historical timestamp in the included state, attempts, usage, or accounting evidence, or be later than the host statistics clock.

A genuinely absent attempt ledger produces a warning and unavailable attempt counts and durations. A genuinely absent usage ledger produces warnings and null usage; metadata cumulative accounting is also hidden because the missing ledger cannot authenticate it. A local run that still claims accounting after losing usage.jsonl, a present-but-empty usage ledger paired with accounting, invalid UTF-8, duplicate JSON keys, malformed rows, duplicate or conflicting ledger identities, cross-run rows, and aliased or duplicate ZIP members fail the command. An artifacts/ subtree by itself is not sufficient.

Bundle creation requires current sealed workflow authority, but manifest v3 does not archive workflow-run-link-journal.json or the sealed workflow control files. Offline statistics therefore prove agreement among the bundled workflow IDs, control generation, state provenance, graph tasks, and ledgers; they cannot independently prove the historical backend link that originally authorized those IDs. The v3 manifest is a snapshot contract, not a standalone attestation of workflow-link history.

The JSON envelope carries stable machine-readable fields alongside the existing counts:

  • ended: true when local and workflow records agree that execution ended, false while the run is active, pending, or paused, and null when the records do not establish a consistent outcome. A cancellation request is not completion.
  • report: status (available, unavailable, pending, or unknown), reason, completion (complete, partial, or unknown), verification (verified, not-checked, or unknown), json_path, and markdown_path. A failed run may have an available PARTIAL report. If the report agent did not produce usable output, the report is unavailable; status does not create a replacement. Missing or unreadable publication records leave availability unknown. Unavailable and unknown reports expose no stale report paths. Verification records the check made when that report was published. Use report --require-verified when current verification is required.
  • progress: percent, finished, in_progress, pending, failed, skipped, remaining, and total. remaining is every node that is not finished, failed, or skipped, and percent is the share of nodes that are settled, so it agrees with remaining reaching zero even when nodes failed or were skipped.
  • eta: available, seconds, basis, and unavailable_reason. basis is recent-throughput when the recent activity window observed completions, run-throughput when only whole-run throughput is available, and no-remaining-nodes when nothing is left to run. unavailable_reason is no-node-counts (a live run whose snapshot reports no nodes), no-finished-nodes, no-observed-elapsed-time, run-paused (a paused run is deliberately not progressing), or run-terminal.
  • current_step: node_id, iteration, started_at, elapsed_seconds, and running_count. Elapsed time comes from Ultrafuzz's synchronized durable state.json node timestamps, and the reported step is the longest-running one. elapsed_seconds is null when no running node has a recorded start. Nodes parked on an approval, event, timer, or controller handover are excluded — they are not executing, and ultrafuzz why explains those waits.

progress and current_step are denominated differently and can legitimately disagree. progress counts the linked workflow's own tasks, which include preparation and verification work that has no durable node, while current_step counts durable Ultrafuzz nodes. Treat progress as campaign-wide completion and current_step as what is executing right now.

--watch re-synchronizes linked workflow evidence on every poll, exactly as a single status call does. Final report publication is recorded once for the current stopped run. Later polls read a small saved summary and inspect its report file identities; they do not rebuild reports, run agents, or retry tasks. Resuming work invalidates the prior final summary. If a published report file changes or disappears, status shows unknown availability.

The watch prints the final publication result, including unavailable, and stops. It may also stop for attention or a pause, so scripts must read ended rather than treating watch exit as proof of completion. A successful status query exits with code 0 even when the observed run failed or its report is unavailable; query errors remain nonzero. Use the execution status and report fields to decide whether your job meets its requirements. Use ultrafuzz report <run-id> to read an available agent-written report.

pause requests a graceful stop: no new tasks are scheduled, in-flight tasks finish, and the run settles in the resumable paused state. resume reports submitted: false (text output Run already active) instead of launching a duplicate continuation when the linked workflow is still active (its Smithers run state is running, recovering, or one of the waiting-* states), and leaves the run's recorded state and workflow deadline unchanged. A run still finishing its in-flight tasks after pause is still active; resume it again once status reports paused. Smithers also reports a run as running for up to 30 seconds after its controller process exits (its heartbeat window), so resume such a run again after that. resume --reset-node retries one failed workflow node and its dependents in the same linked run; the applied reset is recorded so retrying the command after a failed continuation resumes the already-reset run instead of repeating the reset. fork may start from a checkpoint frame and may reset one workflow node before starting the fork.

Every command in this section takes an Ultrafuzz run ID and resolves the linked workflow run from existing product evidence; none of them require the workflow-engine run ID.

cancel is terminal, unlike pause. The engine accepts a durable cancellation request before the run actually stops, so cancel distinguishes the two: a submitted request reports cancel-requested and leaves the product run nonterminal, and a confirmed cancellation reports Ultrafuzz's canonical terminal spelling canceled with its terminal timestamp. Rerunning cancel after the run has already stopped converges on the confirmed result instead of failing. Both outcomes append distinct product events. Failures use the stable WORKFLOW_CANCEL_FAILED diagnostic.

pause and cancel read run evidence the way status does, without the workflow control lock. A run whose sealed control documents diverged, for example a hand-patched published workflow or a planned graph that no longer matches the current build's artifact contracts after a rebuild, can therefore still be paused or cancelled; status reports the divergence. They start the workflow runner from the run's published execution snapshot, so, like status, they still refuse a run whose sealed execution files changed: the files the control seal lists in that snapshot, such as the run plan, agent adapters, and the runtime packages and their dependencies. A run's prompt files are not among them.

Because they take no lock, pause and cancel issued while a launch is still preparing fail without changing the run; retry once ultrafuzz run has returned. They also do not reconcile a replay or fork that was interrupted while linking its new workflow run, so they act on the workflow run it replaced, or refuse. Run ultrafuzz why <run-id> before pausing or cancelling such a run: it reconciles that link, even when it then reports a diverged control document.

why returns a deterministic diagnosis: a summary, the current node, and typed blockers with kind, node_id, iteration, reason, unblocker, waiting_since, attempt, and max_attempts. Blocker kinds are waiting-approval, waiting-event, waiting-timer, retry-backoff, retries-exhausted, dependency-failed, stale-heartbeat, engine-busy, binding, side-effect-boundary, and other. It synchronizes linked workflow evidence first, matching single-run status, and status recommends it as the next step.

The workflow runner's resume, task-retry, checkpoint-replay, log, and inspect suggestions in unblocker, the notes, and the summary are rebuilt as ultrafuzz commands for the run, without the runner's workflow path or run ID. Replaying from a checkpoint becomes ultrafuzz fork <run-id> --frame <n>. On a failed run, resuming or retrying a task becomes ultrafuzz resume <run-id> --retry-failed, because a plain resume leaves a failed node failed; on a live run, retrying one task becomes ultrafuzz resume <run-id> --reset-node <node-id>. A suggestion with no ultrafuzz equivalent, such as approving or signalling a task, stays in the runner's words with the runner called workflow runner; generated workflows never wait on either.

timeline is read-only and returns frames with frame, created_at, content_hash, and fork points. Pass a listed frame number to ultrafuzz fork <run-id> --frame <n>. --tree adds forked runs recursively under lineage, each with its own depth and frames.

snapshots is read-only and lists durability and workspace checkpoints with sequence, node_id, iteration, attempt, tier (an integer durability tier), source, label, and created_at. There is no restore or rewind command: those need separate product-state and side-effect-boundary design.

events shows the linked workflow lifecycle log, which is distinct from Ultrafuzz's own product evidence in .ultrafuzz/runs/<run-id>/events.jsonl. It defaults to lifecycle events and never requests raw agent chunks. --type is restricted to lifecycle categories (approval, frame, memory, node, revert, run, sandbox, scorer, snapshot, supervisor, timer, workflow); a raw agent or tool category is rejected with WORKFLOW_EVENTS_TYPE_UNSUPPORTED rather than silently widening the view. Without --watch it returns a bounded typed array with limit and truncated; --limit defaults to 200 and is capped at 2000. --watch streams new events incrementally rather than buffering the run, printing one redacted event per line in human mode and one newline-delimited ultrafuzz.cli.result.v2 envelope per event with --json. --history replays existing history before streaming.

node takes a workflow node ID as already reported by ultrafuzz inspect. It returns node state, status, duration, attempt counts, agent/model identity, and output metadata. Attempt and retry history requires --attempts. Tool input/output payloads appear only with explicit --tools and pass through the shared secret redaction helpers; node output itself is reported as metadata only, never inlined. --watch follows the same bounded streaming and cleanup contract as events --watch.

Doctor

ultrafuzz doctor [--project <path>] [--json]

doctor is the operational superset of validate; validate keeps its non-launching configuration contract unchanged. Doctor reports:

  • config, topology, prompt, and reference validation posture;

  • required topology backends, toolchain, and configured agent executable availability on the local PATH. Only the executables of agents the selected topology can dispatch to, including its [retry] agents fallbacks, are required; other configured profiles' executables are listed as not required;

  • the bundled workflow engine version, the version the generated project requires, and the version and bin target of the engine Ultrafuzz's own install provides;

  • npm's latest published stable engine version when the registry check is available;

  • whether that installed engine passes Ultrafuzz's version and path checks, and the posture of each compatibility patch in it. Launch and resume refuse an engine that fails either check, so each failure is reported as an error. Doctor then binds an engine that passes both for the project the way those commands do, and reports it as refused when it lies inside the project (for example, when the Ultrafuzz checkout is under --project) or when bun is not on PATH;

  • the OS temporary directory, where launch installs the controller it seals into the run: its free space and how many ultrafuzz-controller-* directories it holds, with their total size. Sizing them stops after about one second, and the size is then reported as at least the bytes counted so far. Doctor warns when the directory is a RAM-backed tmpfs or has less than 2 GiB free, and never removes those directories, because a native resume from an earlier release kept its controller there for the detached engine.

  • the provider home of each agent the selected topology can dispatch to. Agents refuse a provider home that is not a private directory the operator owns (mode 0700) below directories only the operator can write, and so does the Ultrafuzz provider-home root above one. Claude Code and Codex create ~/.claude and ~/.codex from the umask, usually 0755 or 0775, so a home either CLI created is refused. validate, and so run, report such a home as PROVIDER_HOME_UNSAFE before planning, with the directory and the command that fixes it, such as chmod 700 ~/.codex.

Doctor does not create project run state or install, upgrade, or repair local dependencies.

Diagnostics are stable: DOCTOR_AGENT_CREDENTIAL_MISSING, DOCTOR_TOOLCHAIN_MISSING, DOCTOR_TOOLCHAIN_PROBE_FAILED, DOCTOR_WORKFLOW_ENGINE_OUTDATED, DOCTOR_REGISTRY_UNAVAILABLE, and DOCTOR_TEMPORARY_DIRECTORY_CONSTRAINED. A registry or network failure produces a warning and an unknown latest version instead of failing an otherwise valid offline project. Doctor never installs, mutates, or upgrades dependencies.

Report

ultrafuzz report <run-id> [--project <path>] [--json]

report reads agent-written final report artifacts from:

.ultrafuzz/runs/<run-id>/artifacts/final-report/report.md
.ultrafuzz/runs/<run-id>/artifacts/final-report/report.json

If run metadata contains populated cumulative accounting, report --json emits REPORT_ACCOUNTING_MISMATCH diagnostics when report.md or report.json.run_metadata leaves Tokens used unavailable, non-positive, or greater than accounting.cumulative.tokens_used, or leaves Estimated spend unavailable, non-numeric, or greater than accounting.cumulative.estimated_spend_usd in the report's format. A $0.00 spend is valid. The spend must also end in + when run.json says the amount is probably low: accounting.cumulative has partial_pricing or a positive unpriced_event_count, or attempts_without_usage is present. Without accounting, no expectation applies.

Materialize

ultrafuzz materialize <run-id> \
  --copy <run-relative-source:project-relative-destination> \
  [--project <path>] \
  [--yes | --confirm] \
  [--dry-run] \
  [--force] \
  [--json]

Materialization is copy-only. Each --copy source is relative to the run root, and each destination is relative to the project root. Non-dry-run materialization requires --yes or --confirm. --force allows overwriting an existing file destination after path checks.

Patch artifacts may exist as evidence, but patch application is rejected until a safe patch applier exists.

Clean

ultrafuzz clean <run-id> \
  [--project <path>] \
  [--select <path-under-.ultrafuzz>] \
  [--yes | --confirm] \
  [--dry-run] \
  [--json]

Without --select, clean selects runs/<run-id>. Selections are relative to .ultrafuzz/ and must name generated run, artifact, or workspace directories. For a selected run that an earlier release planned for per-node Modal execution, clean, including --dry-run, reports a CLEAN_CLOUD_STORAGE_RETAINED warning, which text output prints before anything is removed. The warning names the run's Modal app, the volume and sandbox run tag the removed provider derived from its run ID, and the modal volume delete command. clean no longer removes that storage, and still deletes the run.

Dashboard

ultrafuzz dashboard \
  [--project <path>] \
  [--host 127.0.0.1] \
  [--port 3875] \
  [--run-id <run-id>] \
  [--no-live]

dashboard starts a local loopback server and prints the /dashboard URL. The API reads and edits only beta product surfaces, validates mutating saves before writing, and guards mutating requests with a per-session token.

Eval

ultrafuzz eval plan \
  [--project <path>] \
  [--suite <suite-yaml-path>] \
  [--provider none] \
  [--target-root <path>] \
  [--ground-truth-root <external-path>] \
  [--skip-target-validation] \
  [--json]
ultrafuzz eval run \
  [--project <path>] \
  [--suite <suite-yaml-path>] \
  [--provider none] \
  [--eval-run-id <id>] \
  [--row <row-id>]... \
  [--target-root <path>] \
  [--ground-truth-root <external-path>] \
  [--watch-timeout-seconds <seconds>] \
  [--no-watch] \
  [--json]
ultrafuzz eval status <eval-run-id> \
  [--project <path>] \
  [--watch] \
  [--interval <seconds>] \
  [--json]
ultrafuzz eval score <eval-run-id> [--project <path>] [--llm-judge] [--json]
ultrafuzz eval report <eval-run-id> [--project <path>] [--json]
ultrafuzz eval compare <eval-run-id> --baseline <variant-id> [--project <path>] [--json]
ultrafuzz eval bundle <eval-run-id> --output <directory> [--project <path>] [--json]
ultrafuzz eval compare <candidate-eval-run-id> \
  --against <baseline-eval-run-id> \
  [--allow-incompatible] \
  [--project <path>] \
  [--json]
ultrafuzz eval analyze all \
  --input </external/private-handoff.zip> \
  --output </external/private-analysis-directory> \
  [--project <path>] \
  [--json]
ultrafuzz eval history [eval-run-id] \
  [--project <path>] \
  [--history <history-json-path>] \
  [--charts <svg-directory>] \
  [--benchmark evmbench|ultrafuzz-bench] \
  [--lane smoke|full] \
  [--repository <public-repository-url>] \
  [--artifact <immutable-artifact-reference>] \
  [--publication-url <validated-result-url>] \
  [--check] \
  [--json]

eval analyze reads a finalized handoff ZIP directly. It accepts only the current, closed benchmark-analysis contracts. The archive contains an exact handoff/current-state.json member, optionally below one canonical top-level directory. That document uses ultrafuzz.eval.adjudication-handoff.v1 and declares the exact adjudication output path. The declared directory must contain these versioned documents:

  • finding-manifest.json — ultrafuzz.eval.finding-manifest.v1, with typed row descriptors, explicit candidate labels/titles, explicit finding IDs and severities, and declared row archive/run metadata paths;
  • instance-to-cluster.json — ultrafuzz.eval.instance-clusters.v1, with an instances array and explicit nullable match/duplicate fields; and
  • ground-truth-tp-credits.json — ultrafuzz.eval.ground-truth-credits.v1, with a typed clusters array.

Each declared row archive and nested run metadata member is read by exact path. Nested run.json must be ultrafuzz.run-metadata.v2, must identify the row's declared runId, and must contain validated accounting.cumulative evidence. The CLI rejects unmatched findings, rows, candidates, clusters or credits; count drift; duplicate projected identities; inconsistent duplicate targets; and duplicate-reference cycles before analysis.

Historical field names and shapes are not compatibility inputs. In particular, schemaVersion, unversioned array/map payloads, candidate heading, lowercase severity aliases, titles containing an inferred finding label, current-only accounting, and malformed source references are rejected. The CLI does not rewrite slashes, search ZIP/tar suffixes, supply metadata defaults, parse IDs from titles, normalize severity, filter invalid rows/references, or repair a document. A prior handoff must be regenerated against the current schemas.

Available report types are all, upset (upsert alias), scores (precision-recall-f1 alias), provenance, table, cost, and pairwise. The pairwise chart connects matched Ultrafuzz/no-fuzz rows across ground-truth TP credits, F1, and total tokens. Charts are written as PNG and editable SVG alongside CSV, JSON, and Markdown reports. provenance.json, source_manifest.json, and analysis_manifest.json are validated against their current JSON Schemas and semantic joins before they are written. Condition score summaries report the mean, median, and sample standard deviation across the declared rows.

The input and output paths are required to be outside --project. Handoff archives and generated reports may contain private target details, ground-truth findings, and adjudication evidence; the CLI refuses to place either inside the repository tree. Generated reports are private local artifacts and must not be committed or published without a separate redaction and disclosure review.

Eval suites benchmark the fuzzing pipeline against targets with known ground-truth bugs. The experiment definition lives in a committable eval YAML (default suite path from [eval].eval_config, overridable per command with --suite). Reporting accepts only none; historical provider profiles are inert and do not load credentials or enable uploads. An explicit --provider none can override a retired provider selection. Precedence for suite and reporting selection is CLI flag > env (ULTRAFUZZ_EVAL_PROVIDER, ULTRAFUZZ_EVAL_CONFIG) > ultrafuzz.toml.

plan validates config plus suite and prints the trial matrix without launching workflows. By default it also validates local target checkouts (one directory per target id under --target-root) against the pinned git refs; --skip-target-validation skips that check.

run launches Ultrafuzz runs for matrix rows (all rows, or a --row selection) and polls them to a terminal state. Reports remain in local run artifacts. --no-watch launches detached without polling.

status reads the eval matrix, its latest runs.jsonl records, and each linked durable state.json without synchronizing or changing workflow state. Every matrix row remains present in matrix order. Labels are deterministic opaque ordinals (row-01, row-02, …), including for private targets; neither table nor JSON output represents target identities, repositories, paths, refs, ground truth, findings, diagnostics, raw node output, provenance, or configuration.

Progress is the count of durable nodes in succeeded, failed, skipped, timed-out, reused-from-prior-run, or invalidated divided by all planned nodes in durable state. The output always pairs available percentages with their completed/total counts. ETA uses observed terminal-node throughput from the durable workflow start through its latest checkpoint and is explicitly an estimate because node runtimes differ. ETA is typed as unavailable when no node has completed, timestamps are missing or invalid, or an incomplete row's checkpoint is more than five minutes old. --watch refreshes the whole matrix at the requested interval while at least one row is still pending, running, paused, or not yet launched; typed invalid or inaccessible rows remain visible without making the command poll forever. Combined with --json, watch mode emits one schema-versioned JSON object per line.

score grades finished run reports against external ground truth resolved under [eval].ground_truth_root, deterministically by default and with the suite's judge model profile when --llm-judge is passed. The optional LLM judge requires an explicit ULTRAFUZZ_EVAL_JUDGE_URL and a dedicated ULTRAFUZZ_EVAL_JUDGE_API_KEY; there is no default endpoint or fallback to reporting or model-provider credentials. The URL must be HTTPS without embedded credentials, and redirects are rejected. Private targets additionally require ULTRAFUZZ_EVAL_JUDGE_ALLOW_PRIVATE_DATA=true. report shows the scored variant ranking. compare either diffs variants against a --baseline variant or compares a candidate run to an --against baseline run after verifying cohort identity, scoring identity, and variant scope; --allow-incompatible is an explicit, reported waiver.

bundle derives a versioned, self-contained analysis directory from local eval evidence. Its fixed allowlist contains aggregate terminal status, evaluation metrics, accounting, and sanitized attempt history. Raw reports, findings, diagnostics, configuration, and execution-local identifiers are not representable in the bundle. Missing optional evidence is recorded in a typed omission manifest.

history validates and regenerates deterministic public SVGs when no run ID is given. With a run ID, it accepts only a complete, successfully scored generation with immutable lineage, appends observations idempotently, and replaces history and charts together. Append mode also requires the benchmark, lane, public candidate repository, immutable source artifact, and publication URL flags. --check compares the checked-in SVGs with a fresh in-memory render and performs no writes.

Eval artifacts are written under .ultrafuzz/evals/runs/<eval-run-id>/. See Eval Suites for configuration and architecture details.