Midscene.js is a vision-driven GUI agent for end-to-end testing and UI automation. Instead of reading the DOM or accessibility tree, it works from a screenshot and natural-language steps, so it can target elements that structure-based tools miss: icon-only buttons, custom controls, , native apps, and cross-origin iframes. Because it operates on what is rendered, it can also assert on colors, highlights, layout, and visual state rather than only the presence of a DOM node.
The same API and test suite run across surfaces. On the web you add Midscene to a Playwright or Puppeteer suite or drive your own Chrome through Bridge Mode; on desktop it automates macOS, Windows, and Linux apps; on mobile it automates Android, iOS, and HarmonyOS on real devices and simulators. Any interface you can screenshot can be automated through one API.
Automation is written with the JavaScript SDK or in YAML, or handed to AI agents through Midscene Skills. Methods such as aiAct, aiQuery, and aiAssert cover acting, extracting data, and asserting. Element localization runs purely on multimodal vision models with strong UI grounding, including Qwen3.x, Doubao-Seed, GLM-4.6V, Gemini, and UI-TARS, with self-hostable open-source options; DOM context can be opted in for data extraction when needed.
Features
- Vision-based targeting: locates elements from the screenshot alone, with no selectors or annotations to maintain
- Visual assertions: verifies colors, highlights, layout, and rendered state, not just DOM presence
- Cross-platform: one API for web, desktop (macOS, Windows, Linux), and mobile (Android, iOS, HarmonyOS), plus any screenshotable interface
- Web integrations: works inside Playwright or Puppeteer suites, or drives an existing Chrome via Bridge Mode
- Authoring options: JavaScript SDK, YAML scripts, or delegation to AI agents through Midscene Skills
- Core methods: aiAct, aiQuery, aiAssert, and related calls for action, extraction, and verification
- Model choice: runs on multimodal models with UI grounding, including self-hosted open-source options, and can include DOM for page understanding
