Codex Computer Use with Paperclip

Short answer: Paperclip doesn't ship a dedicated "computer use" adapter today — the built-in adapter list is claudelocal, codexlocal, geminilocal, cursor, opencodelocal, pilocal, hermeslocal, openclawgateway, process, and http. None of them are explicitly a "screen control" adapter (see the Adapters Overview (/docs/adapters/overview)).

That said, there are two practical paths to roughly what you're describing — Paperclip orchestrates the heartbeat, and the underlying CLI (or process you wrap) does the desktop work.

Option A — let the underlying CLI do the controlling, via its own tools/MCP

The codexlocal adapter (/docs/adapters/codex-local) runs codex exec --json on the same Mac mini. Whatever Codex CLI itself can do — tool calls, MCP servers you've connected to it — is what runs. If/when Codex CLI exposes computer-use tools to your install, the agent will inherit them.

The same pattern applies to Claude Code via claudelocal (/docs/adapters/claude-local) — it's the Claude Code CLI running locally, and any MCP servers you've registered (including a browser-controlling MCP) are available to the agent during a heartbeat. The "Claude for Chrome" extension is a separate Claude.ai feature; it doesn't plug into Paperclip directly, but a Chrome-controlling MCP attached to Claude Code gets you the equivalent capability inside Paperclip runs.

Option B — wrap your own controller behind process or http

If you have a script that does the desktop automation (AppleScript, a Playwright runner, etc.), the process adapter (/docs/adapters/process) runs it as a shell command per heartbeat, and the http adapter (/docs/adapters/http) calls a webhook you control. Both are noted as "Coming soon" in the agent-config UI dropdown but fully functional via the API or an imported company export.

On the per-task control question ("ticket → agent uses computer to do it"): Paperclip's task model doesn't itself drive a mouse — it hands the task description to the adapter, the adapter runs the CLI, the CLI does whatever its tools allow. So the right question is "which CLI/tool stack do I want on the Mac mini?", and Paperclip is the orchestrator on top.

Curious to hear if anyone in the community has wired up a computer-use MCP to Claude Code or Codex inside a Paperclip company — would be a great use-case writeup.

> But you said the CLI can use it's "Computer Use" to talk to it. I don't think the CLI has computer use, only the Desktop App does.

You're absolutely right—the standalone [Codex CLI](https://developers.openai.com/codex/cli/features) doesn't have native "Computer Use" capabilities. Only the [Codex desktop app](https://developers.openai.com/codex/app/computer-use) can see your screen, move the cursor, and click on apps. \[1, 2, 3\]

However, the "CLI" can still trigger these actions if you use the version of the tool bundled within the app package rather than a standalone Homebrew or manual install. \[3\]

Here is how that "theoretical" conversation between the CLI and the Desktop app actually works:

The App-Bundled CLI: If you run the CLI from /Applications/Codex.app/Contents/Resources/codex, it inherits the "Computer Use" permissions of the parent app. In this setup, a script running in your terminal could technically command the desktop agent to open the GUI and start clicking. \[3\] The Model Context Protocol (MCP): The Desktop app acts as an MCP server. This allows the CLI to "call" the app's internal tools (like the screen-operator) as if they were functions. \[1\] Shared Session State: Both interfaces share a [synced history](https://openai.com/index/codex-for-almost-everything/). If the CLI agent "opens" the app via a shell command, it can see the desktop app's state because they are both logged into the same [OpenAI session](https://openai.com/index/introducing-the-codex-app/). \[4, 5\]

The Bottom Line: A standalone CLI cannot use your computer. But a CLI agent running inside the [Codex environment](https://developers.openai.com/codex/app/features) uses the Desktop app as its "hands and eyes" via the Computer Use plugin. \[1, 3, 6\]

Would you like the exact shell path or command to check if your CLI is the "app-bundled" version that supports these features?

\[1\] <https://developers.openai.com>

\[2\] <https://ai.sulat.com>

\[3\] <https://github.com>

> Is this possible? - maybe...\ > \ > I mean, If a paperclip agent (Codex CLI) want's to use the computer, could it in theory open the Codex Desktop app and talk to it via an API ?

In theory, yes. A paperclip agent (like a Codex CLI-driven bot) could interact with the Codex Desktop app, but not through a standard "REST API." Instead, it would use two modern integration methods: Computer Use and the Model Context Protocol (MCP). \[1, 2\]

1. Computer Use: The "Visual" API

The Codex Desktop app has a native Computer Use plugin. This allows one agent to literally "see" and "click" on the screen. \[1, 3, 4, 5\]

Opening the App: The CLI agent could execute a shell command like open -a Codex (on macOS) or start codex (on Windows). Talking to It: Once open, the CLI agent can use its "Computer Use" capability to type prompts into the Desktop app’s chat window and "read" the visual output from the screen. This is essentially a GUI-based API. \[4, 6, 7, 8, 9\]

2. MCP: The "Structural" API

The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is the formal way these tools "talk" to each other now. \[10\]

Codex as a Server: You can run the [Codex CLI as an MCP server](https://developers.openai.com/cookbook/examples/codex/codexmcpagentssdk/buildingconsistentworkflowscodexcliagentssdk). This exposes specific tools—like codex() and codex-reply()—to other agents or applications. Cross-Surface Sync: The Codex Desktop app automatically picks up session history and configuration from the CLI. This means a CLI agent can start a task, and the Desktop app will "know" about it instantly, allowing them to collaborate on the same project. \[6, 11, 12, 13\]

3. Practical Hurdles

Sandboxing: Both the CLI and Desktop app use strict [sandboxes](https://developers.openai.com/codex/agent-approvals-security). You would need to explicitly enable networkaccess or use specific approval rules in your config.toml to let one agent "control" another system-level app. Permissions: For the "Computer Use" method, you must manually grant Accessibility and Screen Recording permissions in your OS settings. \[3, 10, 14, 15, 16\]