MicrosoftOmniParser turns a user-interface screenshot into structured elements: detected interactive regions plus a functional description of each icon. A vision model consuming that structure can name the region it wants to act on rather than infer coordinates from raw pixels.
Two model families make up the release: an interactive-region detector and an icon-caption model, published as V1, V1.5 and V2 checkpoints. V1.5 added finer-grained detection of small icons and a prediction of whether each element is interactable. V2 reports 39.5% on the ScreenSpot Pro grounding benchmark, and a later addition is a YOLOv9-E interactive region detector distributed as an inference-only weight. A Gradio demo and an example notebook run the parser directly.
OmniTool pairs OmniParser with a Windows 11 virtual machine so a vision model of your choice can operate it. Models supported out of the box are OpenAI (4o, o1, o3-mini), DeepSeek (R1), Qwen (2.5VL) and Anthropic computer use. Trajectories can be logged locally, which allows the parser and the tool to feed a training-data pipeline for a domain-specific agent.
Features
- Interactive region detection: locates the clickable regions of a screenshot, with a separate YOLOv9-E detector available as an inference-only weight
- Icon captioning: a caption model describes what each detected icon does, giving the downstream model semantic handles on the screen
- Interactability prediction: V1.5 onward predicts whether a detected element can be interacted with
- Grounding benchmark result: V2 reports 39.5% on the ScreenSpot Pro grounding benchmark
- OmniTool VM control: drives a Windows 11 virtual machine with OpenAI, DeepSeek, Qwen or Anthropic computer-use models
- Trajectory logging: local logging of runs, usable as the input to a training-data pipeline
- Demos: a Gradio app and an example notebook exercise the parser end to end
