---
name: CUA Driver
slug: cua-driver
category: Automation
description: CUA Driver drives native macOS applications by snapshotting their accessibility tree and performing clicks, typing, and scrolling by element index. Use it for GUI tasks in macOS apps, like opening a file in TextEdit or clicking Save in Numbers.
github: "https://github.com/ThinkInAIXYZ/deepchat/tree/dev/plugins/cua/vendor/cua-driver/source/Skills/cua-driver"
language: TypeScript
stars: 6222
forks: 716
install: "npx degit https://github.com/ThinkInAIXYZ/deepchat/tree/dev/plugins/cua/vendor/cua-driver/source/Skills/cua-driver ~/.claude/skills/cua-driver"
installs_to: ~/.claude/skills/cua-driver
source_path: plugins/cua/vendor/cua-driver/source/Skills/cua-driver/SKILL.md
collection_size: 24
category_size: 1523
collection_url: "https://dirskills.com/collections/ThinkInAIXYZ/deepchat"
added: 2026-08-15T06:52:14.505Z
last_synced: 2026-08-15T06:52:14.505Z
canonical_url: "https://dirskills.com/skills/cua-driver"
---

# CUA Driver

CUA Driver drives native macOS applications by snapshotting their accessibility tree and performing clicks, typing, and scrolling by element index. Use it for GUI tasks in macOS apps, like opening a file in TextEdit or clicking Save in Numbers.

**Install:**

```bash
npx degit https://github.com/ThinkInAIXYZ/deepchat/tree/dev/plugins/cua/vendor/cua-driver/source/Skills/cua-driver ~/.claude/skills/cua-driver
```

## README

# cua-driver

Orchestrates macOS app automation via `cua-driver`. Whenever a user
asks to drive a native macOS app, follow the loop in this skill rather
than calling tools ad-hoc — the snapshot-before-action invariant is not
optional and silently breaks if you skip it.

## DeepChat MCP mode

When DeepChat auto-pins this skill because the Computer Use MCP server is
enabled, use the available Computer Use tools in the current tool list
directly. Treat examples such as `click({...})` or
`get_window_state({...})` as calls to the matching MCP tool with the same
JSON payload. Shell CLI examples are diagnostic references for
shell-based agents.

App-name resolution belongs here. When matching a user request to an
app, compare the request with each `list_apps` name and `bundle_id`.
Consider the user's language, system language, English product names,
English brand names, romanized or pinyin variants, and common
abbreviations. Treat `bundle_id` as the strongest identity signal. When
the requested app name is ambiguous, localized, translated, abbreviated,
or written in another script, call `list_apps`, resolve the most credible
bundle id, then call `launch_app` with that bundle id.

## Sparse visible UI fallback

Many media and Electron apps expose a shallow AX tree while still
showing enough pixels to act. Spotify-style snapshots with only
`AXWindow`, title-bar buttons, and a menu bar are still actionable when
the screenshot shows a visible, unambiguous target.

For a user-requested non-destructive action, continue within the same
turn through the safest visible options:

1. Re-snapshot once if the first tree looks sparse.
2. Use `zoom` on the relevant screenshot region when the screenshot is
   wide, dense, or the target is small.
3. Prefer in-window controls visible in the screenshot: cards, primary
   buttons, bottom player controls, search fields, sidebars.
4. Use pixel `click({pid, window_id, x, y})` or `click({pid, window_id,
   x, y, from_zoom: true})` for visible targets missing from AX.
5. Re-snapshot after each action and look for evidence: selected state,
   text changes, playback progress, button icon changes, new panels,
   highlighted rows, or changed window content.
6. If the first safe click has no observable effect, try the next
   visible, semantically related target before asking the user.

Ask for user confirmation only when the visible candidates are ambiguous,
the action is destructive, the target is off-screen, or the next step
requires foregrounding the app. When the AX tree is weak and the
screenshot presents clear controls, continue with the visual fallback
steps above.

Menu-bar fallback is valid when the target app is already frontmost and
the menu item names match the user's requested action. Inspect menus
such as File, Edit, View, Playback, Window, and Help for semantic
commands, then dispatch menu actions and re-snapshot. For backgrounded
apps, prefer in-window AX or pixel actions.

## The no-foreground contract — read this first

**The user's frontmost app MUST NOT change.** This is the whole
reason cua-driver exists. Users pay for the right to keep typing in
their editor while an agent drives another app in the background.
Violate this rule and every other nice property the driver gives
you (no cursor warp, no Space switch, no window raise) stops
mattering — you just shipped the Accessibility Inspector with extra
steps.

Before running any shell command, ask: **"does this raise,
activate, foreground, or make-key any app?"** If yes, don't run it.
Every one of the commands below activates the target on macOS and
is therefore forbidden unless the user **explicitly** asked for
frontmost state:

- **Every form of the `open` CLI — `open -a <App>`, `open -b
  <bundle-id>`, `open <file>`, `open <path-to-App.app>`, `open
  <url>` — always activates.** macOS routes all forms through
  LaunchServices, which unhides and foregrounds the target
  regardless of whether you passed an app name, a bundle id, a
  document, a URL, or the bundle path itself. The activation
  happens even when the only intent was "start the process."
  **Never use `open` for any app launch.** This includes launching
  a just-built .app from a local build dir (e.g. `open
  build/Build/Products/Debug/MyApp.app`) — resolve the
  `CFBundleIdentifier` from `Info.plist` and use `launch_app`
  with that id. See "The narrow carve-out" below for why
  `launch_app` is safe even when the app internally calls
  `NSApp.activate`.
- `osascript -e 'tell application "X" to activate'` —
  activates by design. Same for `... to open <file>`,
  `... to launch`, and anything with `activate` in the tell block.
- `osascript -e 'tell application "System Events" to ... frontmost'`
  in a mutating form (setting `frontmost` rather than reading it).
- AppleScript files that invoke `activate`, `launch`, or `open`
  against the target app.
- `cliclick` (moves the user's real cursor to the target coords
  before clicking — a focus-steal-equivalent even if the app's
  window state is unchanged).
- `CGEventPost` with `cghidEventTap` targeting a coordinate over
  a different app's window (warps the cursor, possibly activates
  on hit).
- `AppleScriptTask`, `NSAppleScript`, `Process` wrapping `osascript`
  that contains any of the above.
- `NSRunningApplication.activate(options:)` called from your own
  helper binary — same class.
- Dock clicks and any `open` invocation (see the first bullet —
  every form of `open` goes through LaunchServices which
  activates, full stop).
- **Keyboard shortcuts that semantically mean "focus here" —
  most notably Chrome / Safari / Arc's `⌘L` (focus omnibox) and
  Finder's `⌘⇧G` (Go to Folder).** These aren't pure key events —
  the receiving app interprets "user wants to type here" as
  activation intent and raises its window to be key. Even when
  delivered to a backgrounded pid via `hotkey`, the downstream app
  pulls focus. **For omnibox navigation specifically**, the correct
  path is `launch_app({bundle_id: "com.google.Chrome", urls:
  ["https://…"]})` — no omnibox dance, no `⌘L`, no focus-steal. Do
  NOT try `set_value` on the omnibox: Chrome's commit logic requires
  a "user-typed" signal that neither an AX value write nor
  `CGEvent.postToPid` keystrokes supply from a backgrounded pid —
  the URL lands in the field but Return fires as a no-op. See
  `WEB_APPS.md` → "Navigate to a URL" for the full pattern. The
  general principle: a shortcut that says "put my cursor inside this
  app" is a focus-steal; a shortcut that says "do this thing" (copy,
  save, quit) is fine.
- **Tab-switching shortcuts in browsers (`⌘1..⌘9`, `⌘]`, `⌘[`,
  `⌘⇧[`, `⌘⇧]`) are visibly disruptive even when delivered to a
  backgrounded pid.** The app's key handler processes the shortcut,
  the window re-renders the new tab's content, the user sees their
  tabs flipping. There is no AX-only workaround: page content (HTML,
  form state, `AXWebArea`) populates only for the focused tab;
  inspecting a background tab requires activating it, which is the
  visible flip. Observed with Dia; the same mechanic applies to every
  Chromium-family browser (Chrome, Arc, Brave, Edge).

  **Prefer the windows-over-tabs pattern**: for each URL you need to
  drive backgrounded, use `launch_app({bundle_id, urls: [url]})` —
  browsers open each URL in a new **window**. Each window has its own
  `window_id`, its own AX tree, and can be inspected / interacted with
  via `element_index` without activating or switching anything. Tabs
  are a UX grouping for humans; cua-driver workflows should default to
  windows. See `WEB_APPS.md` → "Tabs vs windows" for the full pattern.

  Tab-title enumeration (read-only) IS safe — walk a window's toolbar
  AX tree for `AXTab` / `AXRadioButton` children and read their
  `AXTitle`s. Tab switching (activating one) is not.

Reading frontmost state is fine (`osascript -e 'tell application
"System Events" to get name of first application process whose
frontmost is true'`). Mutating it is not.

**Corollary — the AXMenuBar rule.** `AXMenuBarItem` + AXPick
dispatches at the AX layer regardless of which app is frontmost,
but macOS's on-screen menu bar always belongs to the frontmost
app. If you drive a *backgrounded* app's menu bar, the AX call
succeeds but the viewer sees the dispatch rendered over the
*frontmost* app's menu bar — confusing in any observed session and
routinely a silent no-op too, because action menu items go
`DISABLED` when their owning app isn't the key window. **So: only
use menu-bar navigation when the target is already frontmost.** For
backgrounded targets, read state via in-window AX (window title,
toolbar `AXStaticText`) and dispatch via in-window `element_index`
or pixel clicks — both paths are frontmost-insensitive. Full
rationale in "Navigating native menu bars" below.

**"Open \<app\>" in user speech means launch, not activate.**
`cua-driver launch_app` is the one correct path for process
startup — it's idempotent (no-op on a running app), returns the
pid, and has an internal `FocusRestoreGuard` that catches
`NSApp.activate(ignoringOtherApps:)` calls the target makes during
`application(_:open:)` and clobbers the frontmost back to what it
was before the launch. That guard is why `launch_app` with `urls`
(e.g. `{"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}`)
is safe even for apps that normally foreground on media-load
(Chrome, Electron, media players).

## Defaults — always prefer cua-driver over shell shims

**Default transport is the `cua-driver` CLI** — `Bash` shelling out
to `cua-driver <tool-name> '<JSON-args>'`. MCP tools (prefix
`mcp__cua-driver__*`) only when the user explicitly asks for them.
CLI wins because it picks up rebuilds instantly, failures are
easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to `click(...)`, `get_window_state(...)` etc. in this
skill means `cua-driver click '{...}'` — translate to MCP form only
when MCP is requested.

Intent → tool mapping. If you find yourself reaching for the right
column, something has gone wrong — re-read "The no-foreground
contract" above:

| Intent | Use | Don't use |
|---|---|---|
| Open / launch an app | `launch_app({bundle_id})` or `launch_app({bundle_id, urls:[...]})` | `open -a`, `osascript 'tell app … to launch/activate/open'` |
| Find a pid | `list_apps` or `launch_app`'s return | `pgrep`, `ps`, `osascript frontmost` |
| Enumerate an app's windows | `list_windows({pid})` — or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` |
| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open <url>` |
| Drag / drag-and-drop / marquee select | `drag({pid, from_x, from_y, to_x, to_y})` (pixel-only — macOS AX has no semantic drag) | `cliclick dd:`, `osascript drag` |
| Screenshot | `screenshot` or the PNG in `get_window_state` | `screencapture` |
| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` |
| Hand a file/URL to an app | `launch_app({bundle_id, urls:[<path>]})` | `open -a <App> <path>`, `open <url>` |

### The narrow carve-out

The **only** legitimate use of `osascript -e 'tell app X to
activate'` is when the user **explicitly** asked for frontmost
state ("bring Chrome to the front", "make it frontmost", "I want
to see X"). Reaching for it because a tool call returned something
confusing is wrong — that's the skill's classic foot-in-the-door
failure mode and it steals focus every time.

When a cua-driver call surprises you, diagnose cua-driver first:

- **Tiny screenshot / empty `tree_markdown`?** Check
  `cua-driver get_config` → `capture_mode`. Default `"vision"` omits
  the AX tree (PNG only), `"ax"` omits the PNG, `"som"` returns
  both. If a snapshot lacks a tree, `capture_mode` is almost
  certainly `"vision"` — either reason purely from the PNG or flip
  to `"som"` / `"ax"` via `set_config`.
- **`has_screenshot: false`?** The window capture failed (transient
  race against a close, or the window has no backing store yet).
  Re-snapshot; if persistent, pick a different `window_id` via
  `list_windows`.
- **`Invalid element_index` / `No cached AX state`?** You either
  skipped `get_window_state` this turn or passed a different
  `window_id` than the one the snapshot cached against. The cache
  is keyed on `(pid, window_id)` — indices don't carry across
  windows of the same app. Re-snapshot with the same window_id
  you're about to click in.
- **Sparse Chromium AX tree?** Retry `get_window_state` once — the
  tree populates on second call.

Only after those are ruled out, and only if the user's action
genuinely needs frontmost state, fall through to the activate
fallback. Always name the focus steal in your response ("I'll
briefly bring Chrome to the front because …").

### Self-check pattern

Before every `Bash` call whose command line touches any macOS app
(launching, opening, clicking, typing, scripting, screenshotting),
run the self-check:

1. **Does this command foreground the target?** If yes — stop and
   translate to the cua-driver equivalent from the mapping table.
2. **Does this command move the user's real cursor?** (`cliclick`,
   any `CGEventPost` at `cghidEventTap` over another app's window).
   If yes — stop; use `click({pid, x, y})` which routes per-pid
   via SkyLight and never warps the cursor.
3. **Does this command bypass cua-driver entirely?** (`osascript`
   mutating GUI state, AppleScript files, external helpers.) If
   yes — stop; find the cua-driver tool that does the intent.

If all three are "no," the command is safe. If you can't answer,
default to stop and ask rather than proceed. A single `open -a`
run by accident kills the demo, the trust, and the user's in-flight
editor state.

## Prerequisites — check before starting

1. `cua-driver` is on `$PATH` (`which cua-driver`). If not, point the
   user at `scripts/install-local.sh` and stop.
2. Run `cua-driver check_permissions` (with the daemon up — see step 3).
   The default behavior also raises the system permission dialogs for
   any missing grants, so the user can grant on the spot. If either
   grant still reads `false` after that (user dismissed the dialog),
   tell them to open System Settings → Privacy & Security and grant
   Accessibility and Screen Recording to `DeepChat Computer Use.app`, then stop.
   Pass `'{"prompt":false}'` for a purely read-only status check that
   won't steal focus.
3. Start the daemon with `open -n -g -a "DeepChat Computer Use" --args serve` (the
   recommended form — goes through LaunchServices so TCC attributes
   the process to DeepChat Computer Use.app). `cua-driver serve &` also works;
   the CLI auto-relaunches through `open -n -g -a "DeepChat Computer Use"` when it
   detects a wrong-TCC context (any IDE-spawned shell: Claude Code,
   Cursor, VS Code, Conductor). Verify with `cua-driver status`.

## Using cua-driver from the shell

Tool names are `snake_case`, management subcommands are
`kebab-case` — no ambiguity. Tools invoked as `cua-driver
<tool-name> '<JSON-args>'`. Management subcommands:

- `open -n -g -a "DeepChat Computer Use" --args serve` — start persistent daemon
  (**required** for `element_index` workflows; without it each CLI
  invocation spawns a fresh process and the per-pid element cache
  dies between calls). `cua-driver serve &` also works — the CLI
  auto-relaunches via `open` when the shell's TCC context is wrong.
  Pass `--no-relaunch` / `CUA_DRIVER_NO_RELAUNCH=1` to opt out.
- `cua-driver stop` / `status`
- `cua-driver list-tools`, `describe <tool>`
- `cua-driver recording start|stop|status` — see `RECORDING.md`

Canonical multi-step workflow:

```
open -n -g -a "DeepChat Computer Use" --args serve
cua-driver launch_app '{"bundle_id":"com.apple.calculator"}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
cua-driver get_window_state '{"pid":844,"window_id":10725}'
cua-driver click '{"pid":844,"window_id":10725,"element_index":14}'
cua-driver stop
```

## Agent cursor overlay

Visual cursor overlay for demos and screen recordings. Default:
enabled. Toggle with `cua-driver set_agent_cursor_enabled
'{"enabled":true|false}'`. A triangle pointer Bezier-glides to each
click target, ring-ripples on landing, idle-hides after ~1.5s.
Motion knobs: `set_agent_cursor_motion` takes any subset of
`start_handle`, `end_handle`, `arc_size`, `arc_flow`, `spring` —
tuneable at runtime, persisted to config.

Requires an AppKit runloop, which `cua-driver serve` / `mcp`
bootstraps. One-shot CLI invocations skip the overlay entirely.

## The core invariant — snapshot before AND after every action

**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:

- **Before** — the pre-action snapshot resolves the `element_index`
  you're about to use. Indices from previous turns are stale; the
  server replaces the element index map on every snapshot, keyed
  on `(pid, window_id)`. Indices from turn N don't resolve in turn
  N+1, and indices from window A don't resolve against window B of
  the same app. Skip this and element-indexed actions fail with
  `No cached AX state`.
- **After** — the post-action snapshot verifies the action actually
  landed. Without it you can't tell a silent no-op from a real
  effect. The AX tree change (new value, new window, disappeared
  menu, disabled button, etc.) is your evidence that the action
  fired. If nothing changed, the action probably failed silently —
  say so, don't assume success.

This applies to pixel clicks too — re-snapshot after to confirm the
click landed on the intended target.

### Why window selection is the caller's job now

`get_app_state` used to pick a window for you via a max-area heuristic
that returned the wrong surface on apps with large off-screen utility
panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432
off-screen) out-area'd the visible 320×240 player window, so
`get_app_state(pid)` screenshot'd the invisible panel and clicks landed
there silently. The new `get_window_state(pid, window_id)` makes the
caller name the window explicitly — the driver validates that the
window belongs to the pid and is on the current Space, then snapshots
exactly what was asked for. Enumerate candidates via `list_windows` or
read the `windows` array `launch_app` already returns.

## Behavior matrix

Two orthogonal axes shape what the agent can do.

**capture_mode → addressing mode**

| `capture_mode` | `get_window_state` returns | Use for actions |
|---|---|---|
| **`som`** (default) | tree + screenshot | `element_index` preferred; pixel fallback |
| **`ax`** | tree only (no PNG) | `element_index` only |
| **`vision`** | PNG only (no tree) | pixel only — see [SCREENSHOT.md](./SCREENSHOT.md) |

`vision` was renamed from `screenshot` — the old name still decodes
as a deprecated alias, so an on-disk `"capture_mode": "screenshot"`
keeps working. Default is `som` so element_index clicks work the
first time a user calls `get_window_state`; the other modes are
opt-in when the caller specifically doesn't want one half of the
work. Note the tool named `screenshot` is separate (raw PNG, no AX
walk) and unrelated to the capture mode.

When a snapshot looks wrong (tiny screenshot / empty tree), check
`cua-driver get_config` for `capture_mode` before anything else.

Pure-vision mode has its own caveats — Claude Code's vision
pipeline downsamples dense text aggressively, so pixel grounding
takes multiple correction cycles on text-heavy UIs. Read
[SCREENSHOT.md](./SCREENSHOT.md) before driving anything in that
mode; it documents the iterate/annotate/verify recipe plus the
JPEG-over-PNG finding.

**Window state → what works**

| state | `get_window_state` | `click`/`set_value` (AX) | `press_key` commit (Return/Space/Tab) | pixel click |
|---|---|---|---|---|
| frontmost | ✅ | ✅ | ✅ | ✅ |
| backgrounded / visible | ✅ | ✅ | ✅ | ✅ |
| **minimized** (Dock genie