Computer Use Linux is a Rust server for the Model Context Protocol (MCP), plus a matching command-line tool, that gives an agent control of a live Linux desktop. Any MCP host can spawn it and then read accessibility trees, list and focus windows, capture screenshots, click, drag, scroll, type, and invoke semantic accessibility actions. The crate ships the main binary and a small computer-use-linux-cosmic helper used only for COSMIC Wayland window management.
Input is Wayland-first. Pointer actions can go through the org.freedesktop.portal.RemoteDesktop interface, with ydotool / ydotoold (uinput) as the fallback; literal text prefers wtype on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, org.freedesktop.portal.Screenshot otherwise, and fall back to spawning gnome-screenshot in background or systemd contexts where both DBus paths are denied.
Window targeting is compositor-aware: the window registry tries a GNOME Shell extension, GNOME Shell Introspect, the COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, i3 IPC, and generic X11/EWMH in order, then reports which backend answered or why each one failed. Element targeting is semantic rather than positional — click, performaction, and setvalue accept role / name / text / states selectors backed by AT-SPI, with pixel coordinates kept as a fallback for canvas, games, and X clients without ATK.
A doctor command returns one structured JSON readiness document covering platform, portals, AT-SPI, windowing, input, and a readiness summary with explicit blockers and a recommended next step, so a host can surface the state of a machine without parsing prose. Shell execution is not part of the default surface: runshell is registered only when the server is started with COMPUTERUSELINUXENABLE_SHELL=1, and it runs with the server user's own file and network authority under a fixed non-login shell, a 30-second default timeout, and 8 MiB stream limits.
| Desktop or session | Window backend |
|---|---|
| GNOME Wayland | GNOME Shell extension, org.gnome.Shell.Introspect fallback |
| GNOME X11 | org.gnome.Shell.Introspect, then generic X11/EWMH |
| KDE Plasma / KWin | temporary KWin DBus scripting |
| Hyprland | hyprctl clients -j and hyprctl dispatch focuswindow |
| i3 | i3-msg, optional xprop for PID hydration |
| COSMIC Wayland | computer-use-linux-cosmic helper |
| Sway / generic wlroots | no dedicated backend yet |
| Generic X11 / XFCE / other EWMH | wmctrl plus xprop |
Features
- Discovery tools: listapps, listwindows, focusedwindow, and getapp_state, which returns a screenshot plus an accessibility tree with element indices the input tools accept
- Bounded screenshots: capped at 1920 px and 2 MiB by default, with maxwidth, maxbytes, scale, format, and quality overrides and metadata for converting preview pixels back to desktop coordinates
- Input tools: click, drag, scroll, presskey, and typetext, targetable at a window or terminal, with focused-element feedback and warnings when no editable element holds focus
- Semantic actions: performaction invokes any AT-SPI action exposed by an element, and setvalue writes to text fields, sliders, and spinners
- Window navigation: activatewindow by window id, pid, app id, wmclass, or title, plus movewindow and resizewindow for recovering off-screen windows
- Setup helpers: setupaccessibility enables the GNOME toolkit-accessibility setting and setupwindow_targeting installs the bundled GNOME Shell extension
- Safety annotations: tools/list marks each tool read-only, setup-mutating, UI-mutating, or destructive so a host can prompt before committing state
- CLI parity: the same capabilities run as subcommands for scripting and debugging
