Midscene.js is a vision-based GUI agent and testing framework for end-to-end (E2E) tests of web, mobile, and desktop apps. You describe UI steps and expected results in natural language, and the agent carries them out by looking at screenshots, acting on what it sees, and checking the visible result.
Because elements are located by appearance and position, tests can target icon-only buttons, custom controls, canvas content, and elements inside cross-origin iframes without selectors or semantic annotations, and assertions can check colors, highlights, layout, and visual feedback. Sending screenshots instead of large DOM trees keeps model costs low: in the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo passed 58 of 60 tasks for $0.59 in total model cost.
The same agent APIs work across platforms: web through Playwright, Puppeteer, or desktop Chrome, Android through ADB, iOS through WebDriverAgent, HarmonyOS NEXT through HDC, desktop apps on macOS, Windows, and Linux (including over RDP), and custom interfaces that supply screenshots and actions. Midscene Test (beta) organizes cases in YAML and wraps shared business steps (data preparation, API calls, cleanup) in TypeScript nodes, and Midscene Skills let AI coding agents operate and verify applications.
Supported models include Qwen, Doubao Seed, DeepSeek, GLM vision models, Gemini Flash, and UI-TARS, including open-source models you can self-host.
| Benchmark | Pass@1 | Model used in the reported run |
|---|---|---|
| AndroidWorld | 93.1% | Gemini-3.5-Flash |
| MobileWorld | 78.6% | Gemini-3.6-Flash |
| AppControlBench | 96.7% | Doubao Seed 2.1 Turbo |
Features
- Natural-language steps: aiAct for autonomous flows, aiTap and aiInput for single actions, aiAssert for assertions, aiWaitFor for conditions, and aiQuery for structured extraction
- Vision-based locating: finds elements by appearance and position, including canvas content and cross-origin iframes
- Visual assertions: checks colors, selection highlights, layout, and visual feedback described in natural language
- Cross-platform: web, Android, iOS, HarmonyOS NEXT, desktop apps, and custom interfaces through one API
- Test framework integration: works inside existing Playwright, Puppeteer, or Vitest tests
- Midscene Test: YAML test cases with reusable TypeScript nodes and a generated node spec for developers and agents
- HTML reports: screenshots, element locations, the AI decision process, and action and assertion results for each run
- Playground and Chrome extension: try and refine instructions against a live interface before writing tests
- Midscene Skills: lets AI coding agents operate applications, verify interfaces, and inspect results
- Model choice: Qwen, Doubao Seed, DeepSeek, GLM, Gemini, and UI-TARS, including self-hosted open-source models
