Computer Use Linux is a Rust MCP server and CLI that lets an agent control a real Linux desktop. Any MCP host (Codex Desktop's Linux build, Claude Desktop, Hermes Agent, or a custom client) spawns it over stdio and can read accessibility trees, list and focus windows, take screenshots, click, drag, scroll, type, and invoke semantic accessibility actions. It targets GNOME, KDE/KWin, Hyprland, niri, i3, and COSMIC, Wayland first with partial X11 support, and was extracted from the Linux distribution of Codex Desktop, which still bundles it as a plugin.
Most computer-use MCP servers are macOS-only or drive xdotool against an X11 root window. This one uses the freedesktop RemoteDesktop portal for pointer actions on Wayland with ydotool (uinput) as the deterministic fallback, and routes literal text through KDE clipboard integration, wtype, or portal keysyms depending on the compositor. Window targeting tries the GNOME Shell extension, GNOME Shell Introspect, the COSMIC helper, KWin DBus scripting, hyprctl, the niri IPC socket, i3 IPC, and generic X11/EWMH in order and reports which backend won. Tools accept role, name, text, and state selectors backed by AT-SPI, with pixel coordinates as a fallback for canvas and game surfaces.
| Desktop or session | Window backend |
|---|---|
| GNOME Wayland | GNOME Shell extension, Introspect fallback |
| GNOME X11 | Introspect, then generic X11/EWMH |
| KDE Plasma / KWin | Temporary KWin DBus scripting |
| Hyprland | hyprctl |
| niri | niri msg, direct JSON IPC fallback |
| i3 | i3 IPC |
| COSMIC Wayland | Bundled COSMIC helper |
| Sway / generic wlroots | No dedicated window backend yet |
| Generic X11 / XFCE / other EWMH | wmctrl plus xprop |
A doctor tool returns one JSON readiness report covering platform, portals, AT-SPI, windowing, input, and blockers with a recommended next step. Screenshots are size-bounded before they return to the host. Every tool carries MCP ToolAnnotations that separate read-only observation, local setup mutators, UI state mutators, and desktop action mutators, and CI fails if the published annotations drift. A shell execution tool is absent unless the operator opts in. The binary opens no network listener, makes no outbound connections, and ships no telemetry. A click-through on-screen indicator shows the agent's cursor movements, keycaps, and typed text, masking password fields. It is distributed as a Claude Code and Codex plugin marketplace, a Pi Coding Agent package, a Hermes skill, a Rust crate, an npm wrapper, and prebuilt binaries.
Features
- Diagnostics: doctor returns a JSON readiness report, and setup tools enable AT-SPI and install the GNOME Shell extension for window targeting
- Discovery: lists running apps and compositor windows with title, app id, focus state, client type, and bounds, and returns a combined screenshot plus accessibility tree per app
- Screenshots: bounded PNG or JPEG captures of the screen or a single window, with size, scale, and quality controls
- Input: click, drag, scroll, key chords, and literal text, by element index, semantic selector, or coordinates, optionally targeted at a window or terminal
- Semantic actions: invokes AT-SPI actions (press, activate, toggle) and writes values into text fields, sliders, and spinners
- Window navigation: focus, move, and resize windows by id, pid, app id, class, title, or terminal selector
- Terminal targeting: cross-references terminal windows with their TTY and foreground process so input can target the terminal running a given command
- Safety annotations: MCP ToolAnnotations classify every tool as read-only, setup, UI state, or desktop action mutator, with shell execution off unless opted in
- On-screen indicator: a click-through overlay shows cursor glides, click ripples, keycaps, and typed text, masked for password fields
- No network: no listeners, no outbound connections, and no telemetry, only local DBus and uinput transports
- Host integrations: Claude Code and Codex plugin, bundled in Codex Desktop for Linux, a Pi Coding Agent package, a Hermes skill, and any generic stdio MCP client
