screencommander

screencommander is a macOS 14+ CLI and MCP server that gives agents an observe → decide → act loop over the desktop, accessibility-first: read the screen as structured data when possible, capture pixels when needed, and act without taking the user's mouse.

1. Observe — Retina-aware screenshots (full display or a single window), the accessibility tree as structured elements or plain text (elements, ~30–80 ms with no capture), a real-time stream of UI-change events (observe), and a pre/post frame diff on every action that answers "did that do anything?" without re-reading images.
2. Decide — deterministic coordinate mapping from capture metadata (pixels, points, or normalized → global points, window-relative included), element ids and bounds that are directly clickable, and --until predicates for blocking until the UI reaches a state.
3. Act — click, scroll, drag, hover, type, and key chords, delivered through a tier ladder: coordinate-free accessibility actions → process-targeted events posted straight to an app (cursor untouched, works on background windows) → global event synthesis. --no-cursor guarantees the pointer never moves.

Vision models work from screenshots and pixel coordinates. Non-vision models (for example codex-5.3-codex-spark) work from the element tree and text alone — same commands, same JSON envelopes. serve --mcp exposes the entire surface as MCP tools with screenshots returned in-band.

Requirements

• macOS 14.0+
• Xcode Command Line Tools (for swift)

Build

swift build

Fast Capability Guide

For a concise, operations-first guide to screencommander capabilities and reliable command patterns, see:

• SKILL.md

This is the same skill/runbook AGENTS use to quickly understand and operate the CLI.

Install

Use the reusable installer script:

scripts/install.sh --prefix /usr/local

If you prefer a user-local install without sudo:

scripts/install.sh --prefix "$HOME/.local"

The binary is installed to <prefix>/bin/screencommander. For user-local installs, ensure ~/.local/bin is on your PATH.

Permissions

screencommander requires macOS privacy permissions:

1. Screen Recording permission for screenshot

• System Settings path: Privacy & Security > Screen Recording
• Deeplink: x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture

2. Accessibility permission for click, scroll, drag, move, type, key, elements, and observe

• System Settings path: Privacy & Security > Accessibility
• Deeplink: x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility

On denial, commands fail fast with explicit remediation text and stable exit codes.

Commands

Doctor

screencommander doctor
screencommander doctor --json

Behavior:

• Reports permission status with traffic lights (🟢 granted, 🔴 denied).
• Reports active displays with IDs and bounds in points.
• Exits 0 even if permissions are missing; use output for remediation.

Screenshot

screencommander screenshot \
  --display main \
  --out ~/Library/Caches/screencommander/captures/desk.png \
  --format png \
  --meta ~/Library/Caches/screencommander/captures/desk.json \
  --cursor

JSON output variant:

screencommander screenshot --json

Behavior:

• Captures with ScreenCaptureKit SCScreenshotManager.
• Writes image and JSON metadata.
• Default image path: ~/Library/Caches/screencommander/captures/<timestamp>.png
• Default metadata path: <image>.json (for example ~/Library/Caches/screencommander/captures/<timestamp>.json)
• Also updates managed ~/Library/Caches/screencommander/last-screenshot.json by default.
• Does not prune older captures; use cleanup explicitly when you want retention.

Click

screencommander click 640 320 --space pixels

Using explicit metadata and double-right-click:

screencommander click 0.25 0.25 \
  --space normalized \
  --meta ./captures/desk.json \
  --button right \
  --double

Modifier and middle/triple-click examples:

screencommander click 640 320 --button middle
screencommander click 640 320 --modifiers cmd,shift
screencommander click 640 320 --triple

Pointer-free element clicks (no coordinates, cursor untouched):

screencommander click --element "General" --app "System Settings" --no-cursor
screencommander click --element-id 0.3.2 --app Safari
screencommander click --element "Save" --role button --via ax
screencommander click 640 320 --verify-target

Behavior:

• Defaults to metadata path ~/Library/Caches/screencommander/last-screenshot.json.
• Maps screenshot coordinates into global Quartz coordinates deterministically.
• Supports --button left|right|middle, --double, --triple, and --modifiers cmd,shift,option,ctrl.
• --element "<title/label substring>" or --element-id <id> clicks an accessibility element instead of coordinates. Resolution happens fresh at click time (ids from elements are positional). No match exits 70 (element_not_found); ambiguous substring matches exit 82 (element_ambiguous). Disambiguate with --element-id, --role, and --app.
• Element clicks use a tiered actuator, tried in order ax (AXPress/AXShowMenu — coordinate-free, background-safe) → pid (CGEvents posted to one app; cursor stays put) → global (classic path; moves the cursor). Downgrades are recorded in the result (deliveryMethod), not errors.
• Disabled elements fail with exit 72 before any fallback tier runs.
• --via ax|pid|global forces one tier with no fallback (--strict implied). --no-cursor removes the global tier so the pointer never moves. --strict turns any downgrade into exit 72 (element_not_actionable).
• Coordinate clicks keep the historical global delivery; --via pid with --app <name|pid> posts a coordinate click to one app instead.
• Plain coordinate clicks send one physical click. For human-like global clicks, --app or window metadata lets the engine activate the target before clicking, instead of firing an extra focus click.
• --verify-target (coordinate clicks) hit-tests the mapped point via accessibility first and includes the element found there in the result.
• Coordinate actions report advisory metadataFreshness in JSON when metadata is used. Add --strict-metadata to fail with stale_metadata before input injection when the metadata is known stale.
• Captures pre-action and post-action screenshots by default and prints both paths.
• Compares pre-action and post-action screenshots and reports a changed region when pixels differ. Tune with --diff-grid <1...512> and --diff-threshold <0...1>; defaults stay 64 and 0.04.
• Disable before/after capture with --no-postshot.
• Disable only frame comparison with --no-diff.

Scroll

screencommander scroll 640 800 --dy -5
screencommander scroll 640 800 --dy 300 --unit pixels
screencommander scroll 640 800 --dx 2 --dy 0

Element-targeted and pid-delivered scrolling:

screencommander scroll --element "Content" --app Safari --dy -5
screencommander scroll 640 800 --dy -5 --via pid --app Safari

Behavior:

• Maps the target point through screenshot metadata, moves the cursor there, then posts a scroll event.
• Uses line units by default; pass --unit pixels for pixel scrolling.
• Requires at least one nonzero delta across --dx and --dy.
• --element/--element-id scrolls at an element's center. There is no AX scroll action, so element scrolls try pid then global; --no-cursor restricts to pid; --via pid|global forces a tier. The result records deliveryMethod.
• Coordinate scrolls report advisory metadataFreshness; use --strict-metadata to fail before input injection when metadata is known stale.
• Captures pre-action and post-action screenshots by default (--no-postshot to disable).
• Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Drag

screencommander drag 300 400 900 400
screencommander drag 300 400 900 400 --button right --steps 20 --duration-ms 600

Behavior:

• Maps both endpoints through the same metadata and coordinate space.
• Posts mouse-down, interpolated drag events, and mouse-up.
• Defaults to --steps 12 and --duration-ms 300.
• Reports advisory metadataFreshness; use --strict-metadata to fail before input injection when metadata is known stale.
• Captures pre-action and post-action screenshots by default (--no-postshot to disable).
• Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Move

screencommander move 640 320
screencommander move 640 320 --dwell-ms 250

Behavior:

• Maps screenshot coordinates to global Quartz coordinates and posts one mouse-move event.
• Sleeps after the move when --dwell-ms is provided.
• Reports advisory metadataFreshness; use --strict-metadata to fail before input injection when metadata is known stale.
• Captures pre-action and post-action screenshots by default (--no-postshot to disable).
• Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Type

screencommander type "hello world"

With per-character delay and JSON output:

screencommander type "delayed text" --mode unicode --delay-ms 50 --json

Typing into an element without keyboard focus games:

screencommander type "user@example.com" --element "Email" --app Safari
screencommander type "hello" --element-id 0.4.1 --via ax

Behavior:

• Defaults to paste mode (cmd+v) for reliable full-text input.
• --element/--element-id targets an accessibility element: tier ax sets AXValue directly (works on background apps), falling back to focusing the element and using the keyboard path (global). type has no pid tier. --via ax|global forces a tier; --strict turns downgrades into exit 72.
• Captures pre-action and post-action screenshots by default (--no-postshot to disable).
• Compares pre-action and post-action screenshots by default (--no-diff to disable).
• Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Key

screencommander key "enter"
screencommander key "cmd+shift+4"
screencommander key "option+tab" --json
screencommander key "spotlight"
screencommander key "missioncontrol"
screencommander key "launchpad"

Supported modifier aliases include cmd|command, opt|option|alt, and ctrl|control. System/media keys are also supported (for example volumeup, volumedown, brightnessup, mute, launchpad, play, next, prev). spotlight and raycast map to cmd+space; missioncontrol maps to f3.

Behavior:

• Captures pre-action and post-action screenshots by default (--no-postshot to disable).
• Compares pre-action and post-action screenshots by default (--no-diff to disable).
• Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Keys

screencommander keys "press:cmd+tab" "press:cmd+tab"
screencommander keys "press:next" "sleep:100" "press:prev"

keys executes down/up/press/sleep steps in strict order. For repeated modifier-based shortcuts, include modifiers explicitly in each press step (for example press:cmd+tab). Standalone keys such as press:next and press:prev do not require modifiers. It captures and compares pre-action and post-action screenshots by default; use --no-postshot or --no-diff to disable those separately. Tune frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Elements

Read an app's accessibility (AX) element tree — ground-truth UI structure and text without pixels:

screencommander elements                       # frontmost app, focused window
screencommander elements --app Safari --text   # text-only view of the UI
screencommander elements --app 8412 --json     # by pid, machine-readable
screencommander elements --app "System Settings" --roles AXButton,AXTextField --visible-only

Behavior:

• Defaults to the frontmost application; target explicitly with --app <name|pid>. Ambiguous app names report candidate PIDs so callers can retry with a PID.
• Traverses the focused window by default; use --all-windows or --window-id <id> to widen or narrow.
• Each element carries a positional id (dot-joined child-index path such as 0.3.2), role, title/value/description, enabled/focused state, supported AX actions, and bounds in global points.
• When ~/Library/Caches/screencommander/last-screenshot.json exists, elements inside that screenshot also get boundsPixels in its pixel space, so elements output can drive click <x> <y> directly.
• --text prints an indented role "title": value view (also in result.text with --json) — a token-cheap way to read a screen without vision.
• --max-depth (default 40, max 200), --max-elements (default 2000, max 10000, result marked truncated when hit), --max-value-length (200), --roles, and --visible-only bound the traversal.
• Electron/Chromium apps are primed automatically (AXManualAccessibility, falling back to AXEnhancedUserInterface, restored afterwards); the result reports axPrimed.
• Apps that expose no usable AX tree fail with exit code 71 (ax_tree_unavailable).
• --profile text skips geometry and action discovery while preserving scalar text/state fields. Its empty actions means unqueried, not unsupported.
• --max-visited <1...100000> bounds visited nodes including filtered containers. --timeout-ms <1...60000> sets a cooperative AX read deadline with per-handle messaging timeouts; cancellation and deadlines are checked between synchronous calls. Partial results include visitedCount, truncated, and partialReason.

Observe

Stream real-time UI-change events for an app as NDJSON (one JSON object per line) — a push-based feed so agents can block on outcomes instead of screenshot polling:

screencommander observe --app TextEdit --events value          # value changes as you type
screencommander observe --app Finder --until 'role=AXWindow title~=Downloads' --timeout-ms 10000
screencommander observe --app Safari --events focus,window     # focus and window changes

Behavior:

• Streams NDJSON, one event per line: { ts, event, app: { pid, name }, element? }. This command always emits one-line JSON (it ignores pretty/compact).
• --events selects categories (default all): value (value changed), focus (focused element changed), window (created/moved/resized/title changed), destroy (element destroyed), app (NSWorkspace launch/activate/terminate for the target app).
• Runs until Ctrl-C (SIGINT, exit 0), --timeout-ms elapses, or --until matches.
• --until '<predicate>' stops when an element matches: whitespace-joined key<op>value conditions where key is role/title/value/id and <op> is = (exact) or ~= (case-insensitive contains), e.g. role=AXButton title~=Save. An initial tree scan makes already-true conditions return immediately. On match, a final { "matched": true, "element": ... } line is printed (exit 0).
• With --until and --timeout-ms, an unmet predicate within the timeout exits 73 (observe_timeout); a plain --timeout-ms without --until exits 0.
• Requires Accessibility permission.

Cleanup

screencommander cleanup --older-than-hours 24

Explicitly prunes managed capture artifacts (png, jpg, jpeg, json) in ~/Library/Caches/screencommander/captures older than the configured age. Screenshot and action commands never run cleanup implicitly.

Sequence

Run an ordered bundle of actions from JSON:

screencommander sequence --file ./sequence.json

Example sequence.json:

{
  "steps": [
    { "click": { "x": 935, "y": 1074, "meta": "./last-screenshot.json" } },
    { "scroll": { "x": 935, "y": 800, "dy": -4 } },
    { "move": { "x": 935, "y": 700, "dwellMS": 100 } },
    { "type": { "text": "hello from sequence", "mode": "paste" } },
    { "sleep": { "ms": 100 } },
    { "key": { "chord": "enter" } }
  ]
}

Behavior:

• Executes steps in order.
• Step keys are exactly one of click, scroll, drag, move, type, key, or sleep.
• click, scroll, and type steps accept the element-targeting fields element, elementId, role, app, via, noCursor (click/scroll), and strict, mirroring the CLI options (for example { "click": { "element": "Save", "app": "TextEdit", "via": "ax" } }).
• Coordinate click, scroll, drag, and move steps accept strictMetadata.
• Captures pre-action and post-action screenshots around each step by default.
• Disable per-step before/after capture with --no-postshot.
• Compares each step's pre-action and post-action screenshots by default. Use command-level --no-diff or a step-level noDiff: true field to disable comparison. Tune sequence-wide frame diff with --diff-grid <1...512> and --diff-threshold <0...1>.

Windows

List visible windows, optionally filtered by app:

screencommander windows
screencommander windows --app Safari
screencommander windows --json

Behavior:

• Requires Screen Recording permission.
• Without --app, lists all visible windows across all apps.
• --app accepts an app name (exact or prefix) or a PID.
• Each row shows: [windowID] AppName: "title" (WxH at X,Y, layer=N).
• Exits non-zero with code 81 if the app is not found.

Screenshot (window capture)

Capture a single window instead of the full display:

screencommander screenshot --window 12345
screencommander screenshot --window Safari

Behavior:

• --window accepts a numeric window ID or an app-name prefix (captures the frontmost window of that app).
• The sidecar metadata gains optional windowID and windowBoundsPoints fields.
• click coordinates from a window screenshot map correctly using the window bounds as the origin.
• Without --window, screenshot behavior is unchanged (full display).

Focus

Bring an app to the foreground:

screencommander focus --app Safari
screencommander focus --app 1234
screencommander focus --app Safari --json

Behavior:

• --app accepts an app name (exact or prefix) or a PID.
• Prints Focused <AppName> (was: <PriorAppName>).
• Exits non-zero with code 81 if the app is not found.

Serve (MCP server)

Run screencommander as a persistent MCP server over stdio:

screencommander serve --mcp

Register it with Claude Code:

claude mcp add screencommander -- screencommander serve --mcp

Behavior:

• Speaks the Model Context Protocol: newline-delimited JSON-RPC 2.0 on stdin/stdout (initialize, tools/list, tools/call); diagnostics go to stderr.
• Exposes every command as a tool: screenshot, click, type, key, keys, scroll, drag, move, elements, windows, focus, observe_wait, doctor, cleanup.
• One warm engine instance serves all calls — no per-action process startup — and tool results are the same JSON envelopes the CLI prints (as structuredContent plus a text block), so docs/json-output-schema.md covers both surfaces.
• Requests run in parallel by default. A client that needs a serialized follow-on request can put dependsOn in the JSON-RPC request params with the upstream request id; the follow-on starts after that upstream request completes.
• notifications/cancelled with params.requestId cancels the matching in-flight request and any queued dependents.
• Window/display enumeration is cached for 2 seconds per enumeration flavor in serve mode; simultaneous misses share one fetch. Window capture reuses the resolver's display list. Numeric window targets validate current geometry before cache reuse; app-name targets resolve fresh window ordering. Moved/resized windows and changed display bounds force a refresh. The CLI enumerates fresh. Screenshot pixels are always captured anew.
• screenshot additionally returns the capture as an in-band MCP image content block (base64 PNG), so clients get pixels without a follow-up file read.
• observe_wait wraps observe --until + timeout as a single call; an unmet predicate is an observe_timeout error (exitCode 73 in the envelope), with collected events retained in result.
• MCP elements accepts profile, maxVisited, and timeoutMs. snapshot: true retains a bounded session snapshot; since: "<snapshotId>" returns changed positional records and removedIds. Each delta is based on a fresh AX read. Missing/evicted/incompatible snapshots return the full observation with resetReason; incomplete reads cannot become baselines. IDs remain positional and must be resolved freshly before acting.
• Action tools accept optional postObserve: {"app":"<name|pid>"} to return one bounded post-delivery AX observation in the same call. Optional profile, since, maxElements, maxVisited, and timeoutMs control it. A post-read failure reports observationError while retaining the successful action result; it is not a reason to repeat the action. Immediate observation does not imply an asynchronous UI transition has settled.

Reproduce the performance comparison using bench/live_ab.py against a baseline and candidate release binary. It creates a dedicated fixture window, checks semantic equivalence, and saves paired timings and response sizes; see bench/README.md.

Scripting (JSON output)

For automation and scripts, the CLI can emit exactly one JSON object to stdout (success or error). Use this to parse results without scraping human output.

• Per-command: Add --json to any command (e.g. screencommander doctor --json).
• Global: Use --output json before the subcommand, or set SCREENCOMMANDER_OUTPUT=json so every command defaults to JSON.
• One-line output: Use --compact (or SCREENCOMMANDER_JSON_COMPACT=1) when output is JSON for smaller, faster-to-parse output.
• Precedence: Per-command --json overrides root --output over env over default (human).

Example:

# Single command
screencommander doctor --json

# Global JSON for the run
screencommander --output json screenshot --out /tmp/cap.png

# Env var for whole script
export SCREENCOMMANDER_OUTPUT=json
screencommander doctor
screencommander cleanup --json --compact

With JSON mode, stdout is exactly one JSON object: either a success envelope ("status": "ok", result, optional exitCode) or an error envelope ("status": "error", error.code, error.message, exitCode). Scripts can read stdout once and branch on status. For the full contract (envelope fields and per-command result shapes), see docs/json-output-schema.md.

For maximum speed in scripts, combine --json --compact --no-postshot (and optionally --output json or the env var) so action commands skip before/after screenshots and emit one-line JSON. Use --no-diff when you want captures but do not need frame comparison.

Metadata Schema

{
  "capturedAtISO8601": "2026-02-20T12:34:56.789Z",
  "displayID": 69733248,
  "displayBoundsPoints": { "x": 0, "y": 0, "w": 1512, "h": 982 },
  "imageSizePixels": { "w": 3024, "h": 1964 },
  "pointPixelScale": 2,
  "imagePath": "/absolute/path/to/Screenshot-20260220-123456.png"
}

Exit Codes

• 10: screen recording permission denied
• 11: accessibility permission denied
• 20: capture failed
• 21: image write failed
• 30: metadata read/write failed
• 40: invalid coordinate
• 41: mapping failed
• 50: input synthesis failed
• 60: invalid arguments or chord parse
• 70: element not found (--element/--element-id matched nothing)
• 71: target app exposes no usable accessibility (AX) tree
• 72: element not actionable (found but disabled, or action unsupported under --strict/--via)
• 73: observe --until predicate unmet within --timeout-ms
• 80: window not found (--window id/name matched nothing)
• 81: app not found (--app name/pid matched no running app)
• 82: element ambiguous (--element matched multiple best candidates; use --element-id or narrow with --role/--app)
• 83: stale metadata (--strict-metadata found display/window metadata that no longer matches the live desktop)