PaperlayerView exact revision on arXiv

Source-linked AI summary

EchoPath: Execution-Level Replayable Memory for GUI Agents

Yao Zhao, Aditya Shanmugham, Swastik Roy, Yanxun Xu

arXiv:2609.16635v1cs.AI

TL;DR

Repeated GUI workflows make fresh observe-plan-ground-act loops inefficient, while trajectory-context memories lack direct executability and coordinate replay can be brittle. EchoPath makes validated trajectories callable through compatibility-gated replay and visual target re-aiming. In matched second-pass evaluation, it completed 145/159 tasks (91.2%) at a median 20,370 tokens and 127.5 seconds, versus Synapse at 91.8%, 586,386 median tokens, and 315.7 seconds.

  • Problem

    Repeated enterprise GUI tasks are inefficient with fresh observe-plan-ground-act loops, while trajectory memories are not directly executable or easily auditable and raw-coordinate replay is brittle.

    Source: §1 · pp. 1–2

  • Method

    EchoPath converts validated GUI trajectories into callable memories and compatibility-gates replay, using visual evidence to bind stored actions to the current GUI.

    Source: §1 · p. 2, §2.2 · p. 6

  • Results

    20,370 median tokens and 127.5 seconds for EchoPath with 145/159 tasks completed (91.2%), versus Synapse at 586,386 median tokens, 315.7 seconds, and 91.8% success.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Validated execution experience can serve as a controllable, replayable memory for recurrent GUI work, with fallback when direct replay is unsafe.

    Source: §5 · p. 11

    EchoPath is best suited to stable work environments; diagnostics do not establish robustness under broad interface drift or enterprise-scale deployment.

    Source: §5 · p. 11

Abstract

from arXiv · show

Computer-use agents increasingly operate browsers, software, and desktop applications via CLI or API portals, but graphical user interface (GUI) still plays an important role in common industrial production scenarios. GUI agents commonly employ fresh observe-plan-ground-act loops, which is inefficient for enterprise tasks that repeatedly update records, process forms, configure tools, and export reports. We introduce EchoPath, a model-agnostic harness that converts artifact-validated GUI trajectories into standardized, parameter-controlled callable memories, analogous to Model Context Protocol (MCP)-style tool calls rather than unstructured experience records. Each memory stores task-intent keys, application and state preconditions, flexible input parameters, GUI evidence, validation provenance, and lifecycle state, so the host agent invokes a targeted procedure only when it can be deterministically replayed in the current runtime. The core mechanism enabling replay is an image-based target-reaiming algorithm that treats stored coordinates as visual evidence, matches the remembered GUI target against the current screen, and emits corrected operation coordinates before execution. During replay, EchoPath rebinds only declared modifiable inputs and rejects ambiguous or incompatible steps to bounded grounding repair or fresh planning. In experiments with real computer-use tasks, EchoPath reduced median token cost by more than 90% and median execution time by about 60%. These results support a bounded form of enterprise GUI memory: validated execution experience can become a controllable callable asset for recurrent work rather than only context for another reasoning pass.

1 Introduction

Repeated enterprise GUI work makes fresh observe-plan-ground-act loops inefficient, while existing memory and replay approaches trade executability against robustness and inspectability. EchoPath turns validated GUI trajectories into callable, parameter-controlled memories with compatibility checks, visual re-aiming, and governed lifecycle controls.

1 Introduction

  • Repeated enterprise workflows make fresh observe-plan-ground-act loops inefficient, while operational settings also prioritize cost, latency, auditability, privacy, and control.Existing trajectory memories provide context for another model-driven plan but are not directly executable or easily auditable; raw-coordinate replay is brittle under interface changes.Source: §1 · pp. 1–2
  • EchoPath converts validated GUI trajectories into standardized, parameter-controlled callable memories rather than unstructured experience records.The schema is model-agnostic and environment-agnostic, analogous to an MCP-style tool call.Source: §1 · p. 2
  • A memory runs only when task intent and application/state preconditions match, with flexible inputs declared and visual evidence used for pointer actions.Ambiguous targets, failed preconditions, or changes to fixed fields lead to bounded grounding repair or ordinary planning.Source: §1 · p. 2
  • EchoPath combines deterministic replay and controlled adaptation with image-based target re-aiming, rejecting ambiguous or incompatible steps to fallback.Only declared parameter paths may be rebound; fixed navigation and structural actions remain unchanged.Source: §1 · p. 2
  • The contributions include a callable replay schema, the IBTR algorithm, a model-agnostic harness, and a governed memory lifecycle.Lifecycle controls include artifact-gated promotion, branching, repair, merging, deprecation, quarantine, and lineage tracking.Source: §1 · p. 2

2 Methods

EchoPath organizes recurrent GUI operation as a memory-based control loop, creating reusable episodes only from validated work. Replay binds a stored action path to the current GUI when its state and targets are compatible, with bounded repair or planning fallback when they are not.

2 Methods

  • EchoPath frames GUI operation as a five-stage memory-based loop: task input, retrieval and gating, plan-or-replay selection, execution, and artifact evaluation.ActionLens provides a common boundary for fresh planned execution and replay, recording actions and visual evidence.Source: §2 · pp. 2–3

2.1 Problem formulation

  • A GUI task combines the user instruction, desktop context, and external artifact evaluator; a completed run becomes eligible for memory only if evaluation passes.The task model also represents latent desktop state, observations, and normalized action parameters.Source: §2.1 · pp. 3–4
  • Each replayable episode stores retrieval keys, application context, preconditions and expected effects, executable actions, evidence, optional flexible bindings, validation, reports, and lifecycle state.Only active episodes are exposed for default replay retrieval.Source: §2.1 · p. 4

2.2 EchoPath Action Replay

  • Action replay replaces fresh per-step planning and grounding with compatibility-gated instantiation of a stored trajectory, using visual and coordinate evidence to re-aim pointer actions.IBTR matches a remembered target against the current screenshot and binds it in the current coordinate frame rather than treating stored coordinates as portable targets.Source: §2.2 · pp. 5–6
  • Ambiguous local targets can trigger bounded grounding repair, while structural incompatibility rejects direct replay and returns control to planning.Examples include a shifted or ambiguous target for local repair, and a wrong application, absent document state, or needed change to a fixed parameter for structural failure.Source: §2.2 · p. 6

3 Experiments

The experiments use a paired two-pass OSWorld-Verified design: first construct candidate memories, then rerun the same task entries to test retrieval, gating, rebinding, and execution. Additional controlled analyses probe retrieval under clutter, visual target binding, storage, state compatibility, and flexible-input enforcement.

3.1 Main Experiment Setup

  • The paired two-pass OSWorld-Verified study constructs memories on the first pass and tests their retrieval, gating, rebinding, and execution on the same task entries in the second.The second-pass resolution changes from 1920×1080 to 1600×900 to test corrected coordinates from visual re-aiming or bounded repair rather than coordinate copying.Source: §3.1 · p. 7
  • The study reports first-pass token use, runtime, and consolidation cost, then second-pass recall, replay success, target-reaiming hit rate, token use, and execution time.Synapse is included as a planning-augmentation baseline for comparison with replayable execution.Source: §3.1 · p. 7

3.2 Core Functionality Analysis

  • Controlled analyses probe retrieval accuracy and latency as realistic distractor memories are added, and assess visual target matching on 200 coordinate-based cases under original and transformed resolutions.The transformed-resolution diagnostic resizes screenshots by a uniform scale from 0.6 to 1.6 and applies the same factor to documented coordinates.Source: §3.2 · pp. 7–8
  • Other diagnostics measure consolidated-memory storage, start-state gate accuracy across compatible, partially changed, and incompatible screenshots, and whether declared input substitutions are accepted while fixed-field mutations are rejected.Storage analysis relates retained action steps to megabytes; state-gate testing uses 50 memories, and flexible-input testing uses 50 active memories.Source: §3.2 · p. 8

4 Results

Across 159 active memories, EchoPath reports matched replay results alongside offline retrieval, visual re-aiming, state-gating, and flexible-input diagnostics.

Construction and storage

  • Across 159 active memories, median first-pass task cost was about 572k tokens and 4.5 minutes; memory footprints stayed below 1 MB even for long trajectories.Tasks recorded 2–29 GUI actions, most often 4–13, and consolidation reduced about 30% of exploration steps to final memory trajectories.Source: §4 · p. 8, Figure 3 · p. 9, Figure 4 · p. 10

Second-pass replay

  • 145/159 matched tasks (91.2%) were completed by Codex EchoPath at 20,370 median tokens and 127.5 seconds, versus Synapse's 91.8%, 586,386 median tokens, and 315.7 seconds.The replay table reports second-pass memory replay results.Source: Table 1 · p. 8, §4 · p. 9

Target re-aiming

  • For 200 coordinate-based actions, original-screen matching accepted 95.5% with 0-pixel deviation for every accepted match; under random scaling, 188/190 accepted cases were within 2 pixels.The visual re-aiming table covers the offline diagnostic; rejections clustered around repeated or visually near-duplicate targets.Source: Table 2 · p. 9, §4 · p. 9

Retrieval scalability

  • Correct retrieval remained 100% with no wrong selections or misses as the active repository grew from 159 to 659 memories; mean lookup time rose from 216.5 to 615.4 ms.Median lookup time increased from 208.9 to 573.8 ms, and maximum lookup time reached 1,232.0 ms.Source: §4 · p. 9, Figure 4 · p. 10

State gate and flexible inputs

  • The state gate accepted 49 compatible and 46 partially changed starts and rejected 39 incompatible starts out of 50 each; flexible rebinding accepted and rejected all 50 valid and invalid cases.Flexible substitutions were accepted only for declared inputs, while non-flexible mutations were rejected with 100% accuracy.Source: Table 3 · p. 10, §4 · p. 10

5 Discussion and conclusion

EchoPath turns validated GUI trajectories into governed, replayable action memories that can replace repeated observe–plan–ground–act loops when task and state conditions are compatible. The evidence is bounded by replay’s vulnerability to interface drift and by first-pass acquisition costs; proposed next steps address both.

5 Discussion and conclusion

  • EchoPath combines replayable action memory, image-based target re-aiming, retrieval, compatibility gating, and fallback to make memory a controllable systems layer.The framework is intended to reduce repeated model reasoning and grounding while preserving fallback when direct replay is unsafe.Source: §5 · p. 11
  • Under compatible state conditions, validated execution can replace repeated observe–plan–ground–act loops rather than merely provide context for another reasoning pass.The paper presents this as a system claim, with computational efficiency as the main practical implication.Source: §5 · p. 11
  • Replay is best suited to controlled application and display environments because visual re-aiming remains vulnerable to layout, localization, scaling, and duplicate-target changes.The diagnostics do not yet establish live robustness under broad interface drift or enterprise-scale deployment.Source: §5 · p. 11
  • Future work targets stronger semantic UI understanding and state contracts, user-guided or incremental memory acquisition, and repository repair, privacy, and governance.Interactive demonstrations and incremental consolidation are proposed to reduce collection costs for long-trajectory procedures.Source: §5 · p. 11

A.1 Computer-use agents and benchmarks

Prior work spans realistic computer-use benchmarks, experience-augmented planning and procedural memory, and visual record-and-replay systems. EchoPath’s related-work framing isolates recurrent, artifact-validated execution as the question of whether a successful procedure can be replayed under matched conditions.

A.1 Computer-use agents and benchmarks

  • OSWorld, Windows Agent Arena, AndroidWorld, and OSWorld 2.0 [16] motivate reproducible initial states, external outcome checks, and second-run cost measurement.OSWorld-Human [4] adds human-reference trajectories and efficiency metrics, identifying online model calls as a major source of latency.Source: §A.1 · p. 17
  • Cradle and Agent S [18] combine computer-use planning and memory, while EchoPath isolates replay of successful procedures under matched first- and second-run conditions.GUI grounding work includes Set-of-Mark [19], OmniParser [20], SeeClick, CogAgent, and OS-Atlas [23]; EchoPath stores target evidence for image-matching re-aiming with agentic fallback.Source: §A.1 · p. 17

A.2 Experience-augmented planning and procedural memory

  • Experience-augmented systems retain trajectories, reflections, workflow summaries, or skills as context [7], while callable skills such as Voyager and SkillWeaver [31] may still rely on fresh planning.EchoPath tests whether a complete, artifact-validated episode can bypass per-step planning and grounding when compatibility and re-aiming checks pass.Source: §A.2 · pp. 17–18

A.3 Programming by demonstration and replay

  • Sikuli, Ringer, and WebRobot establish the visual record-and-replay lineage; EchoPath adds agent-generated trajectories, artifact gates, retrieval, and an explicit memory lifecycle.Its lifecycle includes promotion, repair, branching, merge, and retirement.Source: §A.3 · p. 18

A.4 Memory for GUI operation

GUI-memory research includes trajectory banks, visual and symbolic retrieval, trust gates, and reusable workflows. EchoPath distinguishes its systems focus from broader memory algorithms and positions runtime-state auditing as complementary rather than as a GUI-memory performance baseline.

A.4 Memory for GUI operation

  • GUI memory spans filtered trajectory banks, latent or graph-based retrieval, trust-gated readout, parameterized templates, and utility-based retention [33] [34] [35] [36] [37] [38] [39].The surveyed systems use varied representations and selection mechanisms, including visual trajectories, linked memory, task-relevant values, and reusable units.Source: §A.4 · p. 18
  • Executable Agentic Memory [40] uses a structured knowledge graph and value-guided search; EchoPath instead emphasizes transparent action programs, replay-time re-aiming, and artifact evidence without memory-model training.The paper frames this as a systems and evaluation distinction, not a claim to a more general memory algorithm.Source: §A.4 · p. 18

A.5 Verification, visual evidence, and runtime state

  • VerificAgent [41] adds expert seeding and human fact-checking before deployment, while AGMem [42] reports that full-screen visual history can worsen some GUI-agent failure modes.AGMem instead stores local image crops associated with successful actions or recovery.Source: §A.5 · p. 18
  • OpenRath [43] makes runtime state and replay explicit but leaves memory quality for follow-on evaluation, so it is a complementary runtime-state comparison, not a GUI-memory performance baseline.Its Session tracks conversation chunks, workspace placement, lineage, token usage, tool evidence, and memory-event boundaries.Source: §A.5 · pp. 18–19

B Methodology Details

ActionLens provides a common execution boundary for freshly planned actions and replay, recording normalized GUI operations and evidence independently of host-agent traces. Its capture protocol records before-and-after state and coordinate context, while failed traces remain diagnostic rather than automatically promoted to active memory.

B.1 ActionLens: wrapped GUI operation

  • ActionLens records normalized GUI operations that actually touch the desktop, making execution observable and replayable independently of host-agent logs or private reasoning traces.The host agent may plan tasks and choose actions, while ActionLens records the executed operations.Source: §B.1 · p. 19
  • The controller-compatible wrapper preserves the underlying action space and can wrap local PyAutoGUI or remote VM controllers while recording evidence for each executed action.The common function-call interface keeps replay independent of agent frameworks’ internal trace formats.Source: §B.1 · p. 19
  • Each wrapped run stores a manifest, action and observation logs, lifecycle events, and screenshots; standalone observations are recorded separately from action steps.The manifest includes run, source-agent, task, controller, status, output-path, and record-count metadata.Source: §B.1 · p. 19
  • For each action, ActionLens captures before and after screenshots and records timing, parameters, executor output, errors, and coordinate context for later replay.Coordinate context includes logical screen size, mouse position, screenshot dimensions, optional region, and inferred pixel-to-controller scale factors.Source: §B.1 · p. 19
  • Failed actions are recorded with error status and an attempted after-state capture, but successful GUI execution alone does not promote a trace to active memory.Promotion is gated by external artifact evaluation.Source: §B.1 · p. 19

B.2 Memory retrieval, and consolidation

EchoPath retrieves replayable memories through intent matching followed by compatibility checks, and turns evaluated trajectories into governed, auditable memory paths. Failed retrieval falls back to regular planning.

Task retrieval

  • Retrieval ranks active paths against normalized task-intent phrases, then checks each candidate’s application, validation, reasoning, lifecycle, action support, and replay evidence before selection.The first-stage intent threshold is λ = 0.18; an intent hit alone does not establish replayability.Source: §B.2 · pp. 19–20
  • Replay requires all binary gates to pass and a compatibility score of at least θ = 0.75, with task-intent match, artifact validation, and lifecycle status among the gates.The score weights intent similarity at 0.40 and lifecycle and artifact validation at 0.15 each.Source: §B.2 · p. 20
  • If no memory passes the gates, EchoPath records a miss or rejection and starts regular planning with ActionLens.Source: §B.2 · p. 20

Consolidation

  • Memory lifecycle operations support promotion, quarantine or repair branching, evaluated path merging, deprecation, deterministic diff, and export; screenshot evidence is stored separately as content-addressed blobs referenced by records.The blob references preserve audit evidence while avoiding duplicate screenshot storage.Source: §B.2 · pp. 20–21

Trajectory representation and consolidation

  • A trajectory becomes an action graph whose shared GUI-operation prefixes link to task-specific branches, while consolidation selects an ordered subsequence of successful steps and drops exploratory or redundant actions.The graph retains source-step lineage, reuse counts, and lifecycle references.Source: §B.2 · pp. 20–21

Consolidation

  • Promotion requires a passed artifact evaluation named by an evaluator, viable reasoning, and no failed or unsupported action or remaining blocker; rejected candidates remain available for diagnosis or repair.Source: §B.2 · p. 21

B.3 Image-based target-reaiming Algorithm

The target-reaiming algorithm searches the current screenshot for stored GUI crops at multiple scales, scores candidate matches, and rejects matches that are weak or ambiguous before emitting revised pointer coordinates.

Matching stored targets

  • Stored target crops act as visual fingerprints that are searched across the current screenshot at multiple scales, with the click ratio locating the target point within the matched crop.The implementation tries target, context, and wide-context crop variants, beginning with the target crop.Source: §B.3 · p. 21

Candidate scoring

  • For crops with sufficient contrast, matching uses zero-mean normalized correlation; nearly constant crops instead use normalized squared difference, with both branches oriented so larger values indicate greater similarity.The flatness test avoids undefined correlation scores for uniform regions.Source: §B.3 · pp. 21–22

Ambiguity checking

  • Because repeated controls can create competing matches, the algorithm suppresses nearby candidates and compares the best match with the strongest remaining non-overlapping candidate.The second candidate’s score is zero if none exists.Source: §B.3 · p. 22
  • A match is accepted only when γ⋆ ≥ η and Δγ ≥ ηmargin; the current defaults are η = 0.78 and ηmargin = 0.02, otherwise replay is rejected before a pointer event.Rejection allows step-local repair or higher-level fallback.Source: §B.3 · p. 22

Coordinate output

  • For an accepted match, the algorithm derives a target point from the matched crop and click ratio, converts it using current image-to-controller scale factors, and rejects coordinates outside the screen.The final action retains its original primitive and non-coordinate parameters while replacing the pointer coordinates.Source: §B.3 · pp. 22–23

Algorithm

  • Algorithm 1 searches crop and scale candidates, tracks the best and strongest non-overlapping match, applies confidence and margin thresholds, then outputs revised coordinates and a match report.Source: §B.3 · p. 23

C Experiment Setup

The supplement describes an AWS virtual-machine setup and prompt modules for planning, grounding, verification, trajectory consolidation, and controlled replay adaptation.

C Experiment Setup

  • Experiments used Amazon AWS Virtual Machines, and EchoPath requires no GPU to initialize and run with customized agentic frameworks.Source: C Experiment Setup · p. 23

D.1 Desktop Next-Step Planning

  • The desktop planner receives a screenshot, recent action history, and task guidance, then returns one next GUI step; prompt runtime fields are inserted into JSON payloads.The prompt requires visible evidence for completion and specifies allowed action forms.Source: §D.1 · p. 24, D Prompts Used by Major Pipeline Modules · p. 24

D.2 Image Grounding

  • The grounding agent returns coordinates only for a clear target, and bounded replay fallback may instead choose one safe control to expose a target hidden by a transient blocker.Source: §D.2 · p. 25

D.3 Step Verification

  • The step verifier compares before-and-after screenshots and distinguishes success, visible progress, pending operations, and blocked actions.Source: §D.3 · pp. 25–26

D.4 Goal Verification

  • The goal verifier marks completion only when the requested end state is visible and ready or non-visual evidence proves the requested result.It rejects inferred completion from app status or assumed focus.Source: §D.4 · p. 26

D.5 Memory Trajectory Consolidation

  • The consolidator selects an ordered subsequence of necessary successful actions, preserves state-preparation dependencies, marks only task-specific text inputs as flexible, and recommends promotion only for a viable replay.Source: §D.5 · pp. 26–27

D.6 Replay-Time Flexible-Input Reasoning

  • Replay-time reasoning may execute a memory unchanged, modify only listed flexible inputs, or reject it when the task requires changing fixed actions, navigation, application, or unsafe behavior.Source: §D.6 · p. 27

D.7 Replay-Step Grounding Fallback

  • When re-aiming fails, the fallback is limited to the current replay action: prefer the remembered target, or choose one safe blocker-dismissal control rather than planning the task anew.Source: §D.7 · p. 28

Related papers

Loading 2609.16635v1…