Agent-E automates actions on a user's computer, currently focused on the web browser. You describe a task in natural language and the agent carries it out: filling in web forms from information about you or from another site, searching and sorting products on e-commerce sites, locating details on pages such as sports scores or university contact information, playing and controlling YouTube videos, running web searches, filtering issues on project-management platforms such as JIRA, and suggesting products for a stated need.
The system is built on the AG2 agent framework (formerly AutoGen) and splits work between agents and skills. A user-proxy agent executes skills, a browser-navigation agent holds the browser skills, and a planner agent decides the steps ahead. Each skill is an atomic action that returns a natural-language description of its outcome rather than a bare status value, so the model can follow what happened and correct course. Sensing skills such as geturl and getdomwithcontenttype report the state of the page. Action skills such as click, entertext, and entertextandclick change it. Skills are configured rather than written as code by the model, and extra skill directories or files can be loaded through configuration.
DOM distillation trims the page down to the elements relevant to the task and hands the model a compact JSON snapshot instead of full HTML. The distillation works from the DOM accessibility tree, and an injected mmid attribute on every element gives the model a stable handle to reference in generated queries. Three content types are available: text only, input fields, and all fields.
Agent-E runs against a local Chrome profile or a Playwright browser, and the model is selected through environment variables (model name, API key, and an optional base URL, API type, and API version for Azure- or other-hosted endpoints). A FastAPI wrapper accepts POST requests and streams results back, and per-request llm_config overrides can set separate configurations for the planner and browser-navigation agents. The test suite builds on Web Arena and runs JSON-defined task files that double as examples.
Features
- Natural-language browser tasks: a task description drives form filling, search, navigation, media playback, and issue triage without a scripted click path
- Skills library: sensing skills (geturl, getdomwithcontenttype) and action skills (click, entertext, entertextandclick) as atomic, conversational actions
- DOM distillation: an accessibility-tree-based JSON snapshot of only the relevant elements, addressed through an injected mmid attribute
- Agent split: user proxy, browser-navigation agent, and an optional planner agent, configured on top of AG2
- Custom skills: additional skill directories or .py files load at startup through an environment variable
- HTTP API: a FastAPI wrapper takes POST task requests, streams results, and accepts per-request model configuration for each agent role
- Browser choice: a local Chrome profile or Playwright-managed browsers
- Test harness: Web Arena-derived JSON task files, run in batches by index with optional screenshots after each operation
