# Droid Control

Terminal, browser, and desktop automation. Record demos, verify behavior claims, and run QA flows.

Droid Control lets Droids *operate* software: launch apps, type commands, click buttons, record what happens, and produce polished video evidence of it. Built by Droids, for Droids.

## What you get

<CardGroup cols={3}>
  <Card title="Verify claims" icon="magnifying-glass-chart">
    Test whether a behavior claim is true and produce evidence either way. No staging, no advocacy, just investigation.
  </Card>
  <Card title="Run QA flows" icon="vial-circle-check">
    Drive terminal CLIs, web apps, or Electron apps through end-to-end flows. Report pass/fail with annotated screenshots.
  </Card>
  <Card title="Record demos" icon="clapperboard">
    Generate polished before/after comparison videos of PRs, complete with title cards, keystroke overlays, and window chrome.
  </Card>
</CardGroup>

## Get started

<Tabs>
  <Tab title="UI">
    Run `/plugins` in a Droid session, go to the **Browse** tab, find **droid-control**, and install it.
  </Tab>
  <Tab title="CLI">
    ```bash
    droid plugin marketplace add https://github.com/Factory-AI/factory-plugins
    droid plugin install droid-control@factory-plugins --scope user
    ```
  </Tab>
</Tabs>

For video rendering, Remotion dependencies need a one-time install after adding the plugin:

```bash
droid plugin list --scope user
cd <plugin-path>/remotion && npm install
```

<Note>
You also need the runtime tools for your use case (tuistory, agent-browser, ffmpeg, etc.). See [Prerequisites](#prerequisites) for per-use-case install commands.
</Note>

## Commands

Droid Control adds three slash commands. Each handles the full workflow end-to-end: planning, execution, recording, and reporting.

<Tabs>
  <Tab title="/demo">
    Record a demo video of a feature or PR.

    ```
    /demo pr-1847
    ```

    Accepts a PR number, GitHub URL, or free-text description. Comparison PRs get side-by-side layout by default; new features get single-branch.

    Add flags for extra polish:

    ```
    /demo pr-1847 -- showcase, keys
    ```

    | Flag | Effect |
    |------|--------|
    | `showcase` | Cinematic preset with warm backgrounds and film grain |
    | `keys` | Keystroke overlay pills showing user actions |

    ### How it works

    <Steps>
      <Step title="Understands the change">
        Fetches the PR description, diff, and linked ticket. For each change, identifies what needs to be proven and what a viewer could confuse it with.
      </Step>
      <Step title="Plans the interaction">
        Scripts a sequence of actions that produces visible evidence the feature works. Both branches run identical interactions so only the behavior differs. Presents the plan and waits for your approval before recording.
      </Step>
      <Step title="Captures both branches">
        Launches recorded sessions on the baseline and candidate branches in parallel using worker subagents.
      </Step>
      <Step title="Composes the video">
        Renders a polished video via Remotion with title cards, window chrome, and effects. Six visual presets range from cinematic (`factory`) to utilitarian (`minimal`).
      </Step>
      <Step title="Verifies the output">
        Checks the final video against the original commitments before delivering.
      </Step>
    </Steps>
  </Tab>
  <Tab title="/verify">
    Test a specific behavior claim and report findings with evidence.

    ```
    /verify "ESC cancels streaming in bash mode"
    ```

    Also accepts a PR reference with an optional claim:

    ```
    /verify 11386 -- the fork flag creates a new session
    ```

    If given a PR number alone, Droid fetches the PR and identifies the most important testable claim.

    <Tip>
    The droid is framed as an **investigator**, not an advocate. If the claim is false, that's a valid finding. Anti-fabrication rules prevent staging evidence to match expected outcomes.
    </Tip>

    ### How it works

    <Steps>
      <Step title="Determines what to test">
        Identifies the specific behavior to observe and what evidence type is needed: text snapshots for functional claims, screenshots for visual claims, or raw byte captures for encoding claims.
      </Step>
      <Step title="Captures the evidence">
        Launches the app, runs the minimal interaction sequence that demonstrates the behavior, and captures the result. If the behavior contradicts the claim, that is evidence, not an error.
      </Step>
      <Step title="Reports the finding">
        Delivers a structured report with a **CONFIRMED**, **REFUTED**, or **INCONCLUSIVE** conclusion, along with all captured evidence inline.
      </Step>
    </Steps>
  </Tab>
  <Tab title="/qa-test">
    Run automated QA against terminal CLIs, web apps, or Electron apps.

    ```
    /qa-test https://app.example.com -- login, create a project, invite a member
    ```

    Also accepts a CLI command, Electron app name, PR reference, or free-text description. Test steps after `--` are optional. Droid designs a reasonable flow if none are provided.

    ### How it works

    <Steps>
      <Step title="Defines the test plan">
        Determines the target (web, terminal, or Electron), designs test steps from your instructions or the app's UI, and identifies what evidence to capture at each step.
      </Step>
      <Step title="Drives the flow">
        Launches the app and executes each step, capturing screenshots (browser) or text snapshots (terminal) along the way. If a step fails, it records the failure and continues for maximum coverage.
      </Step>
      <Step title="Reports results">
        Delivers a step-level pass/fail table with inline evidence and a summary of any issues found.
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Example output

Every video below was planned, recorded, and rendered entirely by a Droid.

<Tabs>
  <Tab title="CLI: single-branch">
    <Frame caption="Demo of the /cwd command. Factory preset, single-branch layout.">
      <video autoPlay muted loop playsInline>
        <source src="/docs-assets/images/features/droid-control-demo-single.mp4" type="video/mp4" />
      </video>
    </Frame>
  </Tab>
  <Tab title="CLI: before/after">
    <Frame caption="Before/after comparison of a bash mode output redesign. Factory preset, side-by-side layout.">
      <video autoPlay muted loop playsInline>
        <source src="/docs-assets/images/features/droid-control-demo-comparison.mp4" type="video/mp4" />
      </video>
    </Frame>
  </Tab>
  <Tab title="Web: single-branch">
    <Frame caption="Browser automation demo of a web app. Recorded and rendered by a Droid.">
      <video autoPlay muted loop playsInline>
        <source src="/docs-assets/images/features/droid-control-web-single.mp4" type="video/mp4" />
      </video>
    </Frame>
  </Tab>
  <Tab title="Web: before/after">
    <Frame caption="Before/after comparison of a web app change. Side-by-side layout.">
      <video autoPlay muted loop playsInline>
        <source src="/docs-assets/images/features/droid-control-web-comparison.mp4" type="video/mp4" />
      </video>
    </Frame>
  </Tab>
</Tabs>

<LabeledDivider label='Under the hood' />

## Automation drivers

Droid Control supports four automation backends. The right one is selected automatically based on what you're targeting.

<CardGroup cols={2}>
  <Card title="tuistory" icon="terminal">
    **Virtual PTY automation.** Default for terminal work. Playwright-style CLI with asciinema recording and forced truecolor output.
  </Card>
  <Card title="true-input" icon="keyboard">
    **Real terminal emulator.** Headless Wayland compositor (Linux), KVM/QEMU (Windows), or QEMU monitor (macOS). For when you need real rendering evidence.
  </Card>
  <Card title="agent-browser" icon="globe">
    **Web and Electron apps.** Playwright-backed CLI with Chrome DevTools Protocol support. Navigates pages, fills forms, clicks buttons, captures screenshots.
  </Card>
  <Card title="desktop-control" icon="desktop">
    **Native desktop apps.** Accessibility-tree snapshots, element or pixel actions, and per-window screenshots via [trycua/cua](https://github.com/trycua/cua) `cua-driver`, all without stealing focus. macOS and Windows are production tier; Linux is pre-release.
  </Card>
</CardGroup>

## Video rendering

Demo and showcase videos are rendered with [Remotion](https://www.remotion.dev/), a React-based video engine. The plugin includes 23 visual components and 6 presets.

### Visual presets

| Preset | Look | Best for |
|--------|------|----------|
| `factory` | Warm black, traffic lights, amber glow | Official Factory content |
| `factory-hero` | Same + gradient background | Landing pages, social |
| `hero` | Cool gradient, generous margins | Non-Factory marketing |
| `macos` | Dark, clean frame | General-purpose demos |
| `presentation` | Black, generous margins | Slide decks, talks |
| `minimal` | No window bar, tight margins | Docs embeds, inline clips |

### Automatic layers

- Warm radial backgrounds, floating particles, film grain overlay, color grading
- Configurable title-to-content transition (`motion-blur`, `flash`, `whip-pan`, `light-leak`, `glitch-lite`)
- Animated window chrome with traffic lights and glassmorphic borders
- Auto-scaled title/subtitle text

### Effect layers

- Spotlight overlays to highlight specific regions
- Directed zoom for small text or details
- Keystroke pills showing user actions
- Section headers and transition sweeps
- Syntax-highlighted code annotations for source-change overlays

## Architecture

The plugin uses a composition architecture with three layers:

{/* sweep-allow: term-bullets */}

- **Orchestrator**: Routes each request through three independent lookups (target, stage, artifact) to determine which skills to load.
- **11 atom skills**: Self-contained background knowledge loaded on demand, split into drivers, targets, stages, and polish.
- **3 commands**: Parse arguments into commitments, then delegate to atoms via hybrid handoffs.

Every workflow flows through **capture → compose → verify**. Commands declare *what* to produce; atoms own *how*. Skills chain through explicit handoffs rather than hardcoded pipelines, so the droid follows the flow naturally.

<Card title="Architecture deep dive" href="https://github.com/Factory-AI/factory-plugins/blob/master/plugins/droid-control/ARCHITECTURE.md" icon="sitemap">
  Design rationale: UX for droids, waterfall routing, task delegation, and hybrid handoffs.
</Card>

<LabeledDivider label='Setup reference' />

## Prerequisites

Only install what you need for your use case.

### Terminal demos (tuistory)

```bash
npm install -g tuistory                                # virtual PTY driver
pip install asciinema                                   # terminal recording
cargo install --git https://github.com/asciinema/agg   # .cast → .gif converter
sudo apt-get install -y ffmpeg                          # video processing
```

### Web/Electron automation (agent-browser)

```bash
agent-browser install   # downloads Chromium
```

### Native desktop apps (desktop-control)

```bash
curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash
cua-driver skills install   # upstream skill pack (deep tool reference)
```

Windows installs via PowerShell: `irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex`. macOS additionally needs `cua-driver permissions grant` (Accessibility + Screen Recording).

### Real terminal emulator (true-input)

| Platform | Required tools |
|----------|----------------|
| Linux/Wayland | `cage`, `wtype`, a Wayland-compatible terminal |
| Windows (KVM) | `libvirt`, `qemu`, KVM VM with SSH |
| macOS (QEMU) | `qemu`, `socat`, macOS VM with SSH |

### Video composition (showcase)

Requires Node.js >= 18, Chrome/Chromium, `ffmpeg`, `ffprobe`, and `agg`.

<RelatedLinks>
  <RelatedLink href="https://github.com/Factory-AI/factory-plugins/tree/master/plugins/droid-control" title='Source code'>
    Full plugin source: skills, commands, scripts, and Remotion components.
  </RelatedLink>
  <RelatedLink href='/harness/plugins' title='Plugins'>
    Learn how plugins work, how to install them, and how to build your own.
  </RelatedLink>
  <RelatedLink href="https://github.com/Factory-AI/factory-plugins/blob/master/plugins/droid-control/README.md" title='README'>
    Quick start, command reference, and prerequisites.
  </RelatedLink>
  <RelatedLink href='/droid-computers/overview' title='Droid Computers'>
    Persistent environments for running terminal and browser automation.
  </RelatedLink>
</RelatedLinks>