Activity Frames records screen activity locally and compiles it into episodic memory an agent can read. A capture engine stores thousands of per-second snapshot rows a day — app, window title, page, accessibility tree, input events — and the compiler turns those instants into bounded records called activity frames: a session with an app, a site, start and end times, the page entities visited, input counts, and pointers back to the raw capture rows that produced it.
The same data compiles into a compact context block for a system prompt. In the published example, a day of activity compiles to 1,371 tokens against 247,563 tokens of raw snapshot rows, in under a second and with no language model in the compile path.
Recurring activity also compiles into replayable workflows. aframes steps expands one frame into its ordered click-by-click script — focus changes, clicks grounded by element name and role, typed runs — so a computer-use agent can repeat a demonstrated task instead of re-deriving it from screenshots. On the expected path the replay makes zero model calls; anything unexpected halts and asks. The project measures this re-derivation cost as a Routine Overhead Ratio and publishes the instrument, measurements, and a deterministic browser executor alongside the package.
A two-tier schema contract separates what is measured from what is inferred. Tier 1, the package itself, contains only what deterministic code can derive from capture data — sessions, durations, typed page entities, input volume, coverage gaps — with no intent labels. Tier 2 extensions that add interpretation must namespace it, tag confidence as high, medium, or speculative, and link evidence. Capture, storage, and compilation stay on the machine; typed-text content is opt-in at the output, audio is off by default, and the compiler opens the capture database read-only.
Features
- Activity frames: snapshot rows compile into bounded sessions with app, site, duration, page entities, input counts, and evidence pointers
- Context blocks: a paste-ready block summarising a day or the last N hours for a system prompt
- Replayable steps: one frame expands into an ordered click-and-type script an agent can execute with slot substitution
- Pattern detection: repeated clicks, action sequences, URL patterns, app-switching loops, and daily habits
- MCP server: six tools — getcontext, getactivity, getsteps, getdaysummary, getpatterns, and get_communications
- Python API: an ActivityLog class returning structured day, recent-hours, and context output
- CLI: record, per-day and per-app views, communications surfaces, patterns, steps, and an MCP stdio server
- Entity parsers: typed page entities for LinkedIn, GitHub, GitLab, Google properties, YouTube, X, Reddit, Notion, Figma, Linear, Slack, Discord, Stripe, Atlassian, and others, with a generic fallback
- Local-only operation: capture, storage, and compilation stay on the machine, with on-device OCR and no model in the compile path
- Bring your own recorder: point $AFRAMESDB at any capture database with compatible frames, uievents, and elements tables
