Activity Frames records screen activity locally and compiles it into episodic memory an agent can read. A capture engine stores thousands of per-second snapshot rows a day (app, window title, page, accessibility tree, input events), and the compiler turns those instants into bounded records called activity frames: a session with an app, a site, start and end times, the page entities visited, input counts, and pointers back to the raw capture rows that produced it.
The same data compiles into a compact context block for a system prompt. In the published example, a day of activity compiles to 1,371 tokens against 247,563 tokens of raw snapshot rows, in under a second and with no language model in the compile path.
Recurring activity also compiles into replayable workflows. aframes steps expands one frame into its ordered click-by-click script (focus changes, clicks grounded by element name and role, typed runs), so a computer-use agent can repeat a demonstrated task instead of re-deriving it from screenshots. On the expected path the replay makes zero model calls, and anything unexpected halts and asks. The project measures this re-derivation cost as a Routine Overhead Ratio and publishes the instrument, measurements, and a deterministic browser executor alongside the package.
A two-tier schema contract separates what is measured from what is inferred. Tier 1, the package itself, contains only what deterministic code can derive from capture data (sessions, durations, typed page entities, input volume, coverage gaps), with no intent labels. Tier 2 extensions that add interpretation must namespace it, tag confidence as high, medium, or speculative, and link evidence. Capture, storage, and compilation stay on the machine. Typed-text content is opt-in at the output, audio is off by default, and the compiler opens the capture database read-only.
Features
- Activity frames: snapshot rows compile into bounded sessions with app, site, duration, page entities, input counts, and evidence pointers
- Context blocks: a paste-ready block summarizing a day or the last N hours for a system prompt
- Replayable steps: one frame expands into an ordered click-and-type script an agent can execute with slot substitution
- Pattern detection: repeated clicks, action sequences, URL patterns, app-switching loops, and daily habits
- MCP server: six tools: get_context, get_activity, get_steps, get_day_summary, get_patterns, and get_communications
- Python API: an ActivityLog class returning structured day, recent-hours, and context output
- CLI: record, per-day and per-app views, communications surfaces, patterns, steps, and an MCP stdio server
- Entity parsers: typed page entities for LinkedIn, GitHub, GitLab, Google properties, YouTube, X, Reddit, Notion, Figma, Linear, Slack, Discord, Stripe, Atlassian, and others, with a generic fallback
- Local-only operation: capture, storage, and compilation stay on the machine, with on-device OCR and no model in the compile path
- Bring your own recorder: any capture database with compatible frames, ui_events, and elements tables can be used in place of the built-in recorder
