Source-linked AI summary
Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Nossa Iyamu
TL;DR
Computer-use agents lack memory of users’ passive screen activity, leaving routines to be re-derived at inference cost. The paper deterministically compiles capture streams into typed, evidence-linked activity frames and evaluates their compact context and demand-side cost measurements. On a single-user corpus, the compiled block supports high-accuracy answering while remaining reproducible and model-free in construction.
Problem
Agent memory records what users tell models rather than what users do, despite existing capture systems producing abundant raw activity data.
Method
A zero-model deterministic compiler converts local snapshot streams into bounded activity frames with typed pages, timing, input volume, provenance, and evidence pointers.
Results
Compiled context outperforms raw rows and LLM summaries for downstream question answering at both model tiers, while reducing prompt-ready context by 86× with 68 ms full-day compilation.
Takeaways & Limitations
The compiler supplies demand-side measurements of routine overhead and recurrence and enables deterministic replay with the model out of the loop.
Takeaways & Limitations
The benchmark uses two model tiers on one single-user corpus with one answering pass, and screen tenure is an imperfect proxy for attention.
Abstract
from arXiv · showhide
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
1 Introduction
The paper addresses the gap between abundant screen capture and agent-usable episodic memory by deterministically compiling snapshots into auditable activity frames. It evaluates the resulting compact context and extends the compiler into a demand-side instrument for routine cost and recurrence.
- The missing middle: Deterministic compilation converts snapshot streams into bounded activity frames with application, site, timing, active time, input volume, and raw-row evidence pointers.The same database and window produce the same document without a model, making memory reproducible, cacheable, and auditable.
- The missing middle: The two-tier schema separates measured activity from optional inferred labels, which must be namespaced, confidence-tagged, and evidence-linked.The measured tier remains valid when inferred content is removed.
- Evaluation: 86× reduction and 68 ms full-day compilation characterize the compiler’s prompt-ready context efficiency and runtime on the reported corpus.The implementation also reports byte-identical reproducibility and other corpus-level measurements.
- Evaluation: Compiled context improves question answering over raw rows and LLM summaries at both model tiers, with a mid-tier model matching a frontier model.The evaluation uses an independent oracle and releases a rerunnable harness.
- Demand-side measurement: The compiler measures Routine Overhead Ratio R and desktop routine recurrence h from passively captured human activity rather than agent rollouts.This supplies demand-side cost parameters for routines before delegation.
2 Related Work
Prior work builds memory, routines, and task structure from agent traces, screenshots, or interaction logs, but the paper positions activity frames as deterministic typed memory from passive human activity. Its contribution also targets trust and cost measurement through measured evidence and a demand-side view.
- Memory and activity capture: Existing memory systems externalize conversations, tool calls, task traces, or agent trajectories, whereas activity frames compile what the human did.The related systems include virtualized context, temporal knowledge graphs, execution-trace memory, and hierarchical procedural memory.
- Memory and activity capture: Recent capture-memory systems place model calls inside construction, while this paper keeps memory construction deterministic.The cited systems include multi-agent screenshot memory, local vision-language summaries, model-based annotation, and LLM trace summarization.
- Task structure: Interaction-log research has long extracted tasks and routines, but those pipelines generally end in task models or human-oriented representations rather than typed agent memory.The paper connects its method to TaskTracer, SWISH, lifelogging, sessionization, process mining, robotic process mining, and data-to-text work.
- Trust and cost: Evidence pointers and a hard measured/inferred boundary address risks that poisoned or hallucinated labels could be mistaken for facts.These properties are presented as structural safeguards for agent memory.
- Skills and cost: Trajectory-based skill induction and acquisition channels observe routines only after agents incur execution cost and only for tasks they can complete.The paper contrasts this supply-side view with passive pre-delegation human activity.
3 The Activity Frames Schema
The activity-frames schema represents a query window as a provenance-bearing document of measured, reproducible activity, while permitting evidence-linked inference as a removable second tier. It preserves page typing and input volume without silently emitting typed content.
- Schema principles: Every standard field is deterministically derived from capture data, documents are reproducible, frames point to raw rows, and blind spots are reported explicitly.The schema avoids intent labels because code cannot observe intent.
- Document structure: A document contains coverage, chronological frames, blind spots, and provenance metadata; each frame is keyed by application and site.Coverage includes first and last activity, active minutes, span, and capture gaps over five minutes.
- Typed page references: Browser frames use deterministic URL parsing to produce typed references such as profiles, repositories, pull requests, documents, and email.Unmatched URLs fall back to a generic domain reference, so typing is total and lossless.
- Measured and inferred tiers: Inference is permitted only in a namespaced block with a confidence tag and evidence links, and consumers can strip it to retain measured data.The boundary is schema-enforced rather than dependent on a particular inference method.
- Privacy rule: Input volume records keystroke, click, and copy counts, while input content is excluded by default and requires explicit operator opt-in.A conforming producer cannot silently emit typed content.
4 Deterministic Compilation
Deterministic compilation transforms event-driven, multi-monitor capture streams into bounded activity frames using fixed, auditable rules for dwell, session gaps, interruptions, input attribution, and URL typing.
- Frame construction: Activity frames are segmented independently per monitor from snapshot and input streams, using the application-site pair as the frame key.The compiler emits bounded stretches with timing and provenance rather than relying on learned interpretation.
- Segmentation rules: 90 s is the maximum dwell credit per frame, while gaps above 300 s close the frame and become candidate coverage gaps.Dwell uses min(∆t, 90 s), and no dwell is credited across a session gap.
- Segmentation rules: 20 s is the flicker-merge threshold: an A → B → A sequence folds B into A as a measured interruption when no session break intervenes.The detour remains recorded rather than being discarded.
- Input attribution: Input events are re-attributed to the temporally nearest snapshot, while click resolution records exact, tolerance, or zone confidence and leaves unresolved clicks unguessed.Keyboard-layout mismatches are repaired only through an explicit operator-supplied translation map.
- Entity typing: URL typing applies pure, layered functions from bespoke parsers through generic detectors and heuristics to a total fallback.Because the layers depend only on the URL, coverage additions remain reviewable line by line.
5 Reference Implementation
The reference implementation packages local capture and deterministic compilation into an open Python system that can emit prompt-ready activity documents through standard interfaces.
- System components: The implementation provides an on-demand local capture engine, a zero-dependency read-only compiler, and an MCP server.The capture engine records activity entirely on-device, with audio capture off by default.
- Outputs: The compiler emits JSON, YAML, Markdown, and compact plaintext context blocks designed for system prompts.Operators can also point it at an existing compatible recorder database instead of using the provisioned capture engine.
6 Empirical Characterization
The compiler is characterized on a frozen single-user corpus using token cost, runtime, reproducibility, entity typing, and downstream question answering. Compiled activity frames sharply reduce context size while preserving high answer accuracy across model tiers, though the corpus and dwell measure impose clear scope boundaries.
- Corpus and scope: 61 calendar days produced 109,735 snapshot rows, 214,360 input events, and 8.4M element-tree rows across 54 applications; overhead measurements use a later 51-active-day freeze.The corpus is explicitly reported as a mechanism characterization, not a user study.
- Token cost: 86× fewer tokens reduced one full day from 126,812 raw-row tokens to a 1,469-token compact context block without model calls.The compiled schema-v1 document required 34,815 tokens, or 3.6× fewer than raw rows.
- Downstream question answering: 98.4% accuracy was achieved from the compiled block at both model tiers, while summary accuracy ranged from 66.1% to 80.4% under the reported benchmark settings.The compiled block also removed the capability gap between the mid-tier and frontier models; raw-row and summary baselines improved with model strength but never caught it.
- Compilation: 68 ms was the median end-to-end compilation time, and independent compilations were byte-identical after excluding the generation timestamp.This supports rebuilding episodic memory on every query while preserving deterministic output.
- Entity typing: 81.3% of 5,120 distinct URLs received non-generic typed references spanning 46 kinds, with the remaining 18.7% falling back to generic domain references.Coverage expands incrementally through one pure function per site rather than a learned component.
- Duration distribution: 0.5 minutes was the median active frame duration across 17,514 frames, rising to 0.9 minutes after excluding single-snapshot transits.Single-snapshot transits comprised 52% of sub-minute frames, while dual-monitor stretches accounted for a further 20%.
7 The Routine Overhead Ratio
The paper uses deterministic compilation to measure routine-replay costs from passive human activity, reporting modeled overhead ratios of 60–343× and recurrence rates that constrain realistic fleet savings.
- Measurement instrument: The compiler doubles as a demand-side instrument for measuring Routine Overhead Ratio R and routine recurrence h from passive human activity rather than agent rollouts.These measurements are reported on one user’s real-work corpus and are intended as observed parameters for agent-cost models.
- Measurement instrument: A routine is a recurring action n-gram of 3–60 UI actions, appearing at least three times and naming at least two distinct targets.Sessions are split at inter-action gaps over 90 seconds to avoid degenerate repetition patterns.
- Cost model: At a typical 1512 × 982 capture, the modeled screenshot-driven agent cost is about 2,500 tokens per step, comprising 1,979 image, 350 context-read, and 180 reasoning tokens.The numerator is an upper bound because it assumes fresh screenshots and no cross-step prompt caching.
- Results: 60× is the median Rinject, while the minimal-script ceiling reaches Rinfo = 343× across the 20 most frequent action routines.The 20 routines are concentrated in compose-message and window-close families, so Rinject reflects frequent micro-routines rather than diverse tasks.
- Results: 99.7% is the median ceiling’s per-covered-step recovery, but measured median guard coverage is only 0.415 and replay remains exposed to interface drift.The recovery is therefore a covered-step ceiling, not a fleet-wide saving.
- Fleet implication: The modeled three-arm comparison reports 83.3% token savings for plan injection and 40.8% for local replay relative to the screenshot baseline.The live accessibility-tree comparison saved only approximately 14%, so the larger savings remain modeled ceilings pending live billing.
- Fleet implication: 9.0% in-sample and 7.7% out-of-sample are the delegable recurrence rates, yielding an honest all-fleet token ceiling near 9.0% and 7.7%, respectively.The 7.7% estimate uses a same-user temporal holdout; page revisitation rates are not treated as delegable task recurrence.
8 Privacy, Trust, and Limitations
The paper keeps capture and compilation local while separating measured facts from inference and documenting limits around reproducibility, coverage, attention, matching, and empirical scope.
- Privacy and trust: The entire pipeline runs locally, with read-only compilation, operator-controlled artifact sharing, typed text excluded by default, and audio capture disabled by default.Compilation itself uses no model.
- Privacy and trust: Evidence pointers make episodes auditable against raw rows, while the measured/inferred boundary lets consumers strip away interpretation and retain the measured core.These properties are presented as defenses against unverifiable or interpretation-disguised memories.
- Scope: The empirical characterization uses one professional’s machine and one capture-engine database layout, so cadence, coverage, and portability may differ across users and platforms.The paper identifies multi-user replication and source adapters as future work.
- Measurement limits: 45% of credited active time ends in heartbeat rows, and uninterrupted heartbeat runs reach 42 minutes, so dwell measures screen presence rather than attention.The measure includes reading and watching, but also time spent away from an awake display.
- Evaluation limits: The downstream benchmark evaluates two agent tiers on one single-user corpus with one answering pass each, and stronger models narrow the summary gap.Frames measure attention episodes rather than complete tasks; task-level structure is assigned to tier-2 inference.
- Cost limits: R’s numerator and the three-arm dollar comparison are modeled at list prices rather than billed, while denominator artifacts and median guard coverage of 0.415 are measured.A single owner-authorized live execution confirms the replay side, but full live three-arm billing remains reserved.
- Operational limits: Accessibility trees are present on 81.5% of frames, and approximately 82% entity-typing accuracy propagates into match precision q, so replay coverage and savings are bounded.Frames without structured targets fall back to coordinates, while typing errors can cause false matches or missed matches.
- Reproducibility: The paper scopes determinism to compilation over stored capture, not back to pixels, because OCR can vary across models or rescans.On-device OCR is the only learned component in the pipeline.
9 Conclusion
Activity frames turn passive screen capture into deterministic, auditable episodic memory and also expose routine cost parameters, while the paper reserves broader validation and live billing.
- Conclusion: At 68 ms and zero tokens per day, deterministic activity frames make episodic memory infrastructure rather than inference.The measured/inferred boundary preserves a place for interpretation without contaminating measured facts.
- Conclusion: The compiler also supplies first single-user readings of R and h from recurring passive activity, with modeled numerator costs awaiting multi-user replication and live billing.The schema, compilation rules, implementation, and evaluation harness are open.