OpenGUI

OpenGUI is an agent framework for operating Android apps through their graphical interface on real phones and emulators. It reads the current screen, plans the next step, performs taps and other mobile actions, and returns a structured result. Example tasks include collecting posts on X, summarizing Reddit or Hacker News threads, UI regression testing, and repetitive in-app workflows, with human confirmation recommended before publishing, messaging, or account changes.

The full stack has a backend and an Android client. In the backend, a main graph handles planning, a Plan Supervisor keeps task state and continuation, an executor subgraph runs screenshot, vision, action, and call-user loops against the live device, and a Summarizer closes each run with a report of what happened and what still needs attention. The Android client uses the standard AccessibilityService APIs for screenshots and actions, so no root access or unlocked bootloader is needed. Phones can stay on standby and receive work through Feishu, Telegram, Discord, or a REST API.

Planning and visual execution can use different model providers. The recommended order for visual GUI execution is Doubao, then Qwen, then OpenAI vision models, with Grok experimental, and any chosen model must accept image input and tool calls. A cost-saving profile pairs Qwen for planning with Doubao for vision, and a higher-cost profile uses Claude Opus throughout.

OpenGUI also ships as a plugin for DeepSeek Harness and a connector for WorkBuddy, both of which add phone operation without running the full backend. Coding agents such as Claude Code, Codex, and OpenCode can bootstrap the full stack from a bundled skill and then dispatch Android tasks through a local CLI.