PaperlayerView exact revision on arXiv

Source-linked AI summary

Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control

Longtao Zheng, Rundong Wang, Xinrun Wang, Bo An

arXiv:2306.07863v3cs.AI

TL;DR

LLM-based computer-control agents are constrained by context limits, incomplete trajectory exemplars, and task-specific retrieval, motivating better generalization and long-horizon decision-making. SYNAPSE combines state abstraction, trajectory-as-exemplar prompting, and exemplar memory, achieving strong results across MiniWoB++ and Mind2Web, including 99.2% success on 64 MiniWoB++ tasks from 48-task demonstrations and a 56% Mind2Web improvement over MindAct.

  • Problem

    Limited context, complex states, incomplete trajectory exemplars, and task-specific demonstrations constrain LLM-based computer-control agents.

  • Method

    SYNAPSE abstracts raw states, prompts with complete successful trajectories, and retrieves relevant exemplars from memory using similarity search.

  • Results

    99.2% average success rate was achieved across 64 MiniWoB++ tasks using demonstrations from only 48 tasks, while Mind2Web step success rate improved by 56% over MindAct.

  • Takeaways & Limitations

    SYNAPSE generalizes beyond task-specific demonstrations and supports long-horizon computer-control tasks across standard and real-world web benchmarks.

  • Takeaways & Limitations

    SYNAPSE has high inference latency and depends on exemplar quality.

Abstract

from arXiv · show

Building agents with large language models (LLMs) for computer control is a burgeoning research area, where the agent receives computer states and performs actions to complete complex tasks. Previous computer agents have demonstrated the benefits of in-context learning (ICL); however, their performance is hindered by several issues. First, the limited context length of LLMs and complex computer states restrict the number of exemplars, as a single webpage can consume the entire context. Second, the exemplars in current methods, such as high-level plans and multi-choice questions, cannot represent complete trajectories, leading to suboptimal performance in long-horizon tasks. Third, existing computer agents rely on task-specific exemplars and overlook the similarity among tasks, resulting in poor generalization to novel tasks. To address these challenges, we introduce Synapse, a computer agent featuring three key components: i) state abstraction, which filters out task-irrelevant information from raw states, allowing more exemplars within the limited context, ii) trajectory-as-exemplar prompting, which prompts the LLM with complete trajectories of the abstracted states and actions to improve multi-step decision-making, and iii) exemplar memory, which stores the embeddings of exemplars and retrieves them via similarity search for generalization to novel tasks. We evaluate Synapse on MiniWoB++, a standard task suite, and Mind2Web, a real-world website benchmark. In MiniWoB++, Synapse achieves a 99.2% average success rate (a 10% relative improvement) across 64 tasks using demonstrations from only 48 tasks. Notably, Synapse is the first ICL method to solve the book-flight task in MiniWoB++. Synapse also exhibits a 56% relative improvement in average step success rate over the previous state-of-the-art prompting scheme in Mind2Web.

1 INTRODUCTION

SYNAPSE addresses context, trajectory-representation, and exemplar-generalization challenges in LLM-based computer control with state abstraction, trajectory-level prompting, and memory. It achieves strong results on MiniWoB++ and Mind2Web, including long-horizon and novel-task performance.

  • Motivation: LLM-based computer agents face limited context, complex states, incomplete exemplar structures, and task-specific demonstrations.These constraints hinder few-shot learning, long-horizon decision-making, and generalization to novel tasks.
  • Approach: SYNAPSE combines state abstraction, trajectory-as-exemplar prompting, and exemplar memory for computer control.The components reduce state length, represent complete successful trajectories, and retrieve relevant exemplars through similarity search.
  • Results: 99.2% average success rate was achieved across 64 MiniWoB++ tasks using demonstrations from only 48 tasks.SYNAPSE is reported as the first ICL method to achieve human-level performance on the benchmark without self-correction.
  • Results: SYNAPSE solved the book-flight task, which requires long-horizon decision-making under intricate states.The result addresses a task previous ICL methods had failed to solve within MiniWoB++.
  • Results: 32%, 50%, and 56% improvements in Mind2Web step success rate were obtained over MindAct with GPT-3.5 as components were added.With CodeLlama-7B, SYNAPSE achieved an average 2.5× step success rate compared to MindAct.

2 RELATED WORK

Prior computer-control agents progressed from BC and RL toward stronger learned systems, but achieving human-level performance could require extensive demonstrations. LLM-based computer-control work therefore motivates more data-efficient approaches.

  • Building Agents with LLMs: LLM-based agent research also combines language-model planning with embedding, value-function, or environment-feedback mechanisms.These approaches generally focus on high-level semantic plans while relying on BC or RL for low-level motor control.
  • Agents for Computer Control: MiniWoB++ became a standard benchmark for agents performing human-like computer interactions.Early approaches primarily used behavioral cloning and reinforcement learning.
  • Agents for Computer Control: Early BC and RL methods were insufficient for human-level performance on MiniWoB++.CC-Net achieved human-level performance but required an extensive dataset of 2.4 million demonstrations.

3 SYNAPSE

SYNAPSE converts raw computer states into concise observations, prompts with complete trajectories, and retrieves similar exemplars from memory. This pipeline targets context limits, sequential decision-making, and generalization across related tasks.

  • Pipeline: SYNAPSE retrieves relevant few-shot exemplars by similarity search over embeddings of task metadata.Metadata can include task descriptions, initial states, website names, domains, and related task information.
  • State Abstraction: State abstraction extracts task-relevant information from raw states to reduce the length of each state.The method uses few-shot prompts with state-observation pairs or other abstraction forms to produce clean observations.
  • Trajectory-as-Exemplar Prompting: Existing step-wise plans and MCQ exemplars omit complete trajectories and can accumulate errors through one-action-per-query generation.These limitations are especially problematic for long tasks with many steps or repeated actions.
  • Trajectory-as-Exemplar Prompting: SYNAPSE can generate consecutive actions before requesting a new state when the trajectory exemplar supports temporal action abstraction.In the terminal example, it generates type(ls) and press(enter) before pausing for the next state.
  • Trajectory-as-Exemplar Prompting: Trajectory-as-exemplar prompting formats successful demonstrations and current interaction histories as sequences of tasks, observations, and actions.The LLM generates the next action from these trajectories while the current trajectory is updated with subsequent observations and actions.
  • Exemplar Memory: Task-specific exemplar selection prevents prior agents from exploiting similarities among related tasks.SYNAPSE stores metadata embeddings with corresponding abstraction prompts and trajectories, then retrieves relevant exemplars through vector similarity.

4 EVALUATION

SYNAPSE is evaluated on MiniWoB++ and Mind2Web through benchmark comparisons and component ablations. It achieves human-level MiniWoB++ performance, improves Mind2Web step success, and shows distinct benefits from state abstraction, trajectory prompting, and exemplar memory.

  • MiniWoB++ evaluation: 99.2% mean success rate gives SYNAPSE human-level performance on MiniWoB++, outperforming all baselines across tasks.The evaluation compares against BC+RL, fine-tuning, and ICL methods.
  • MiniWoB++ evaluation: SYNAPSE solves 64 MiniWoB++ tasks using demonstrations from only 48 tasks, including tasks that previous ICL methods could not handle.These include long-horizon or context-intensive tasks such as book-flight and click-pie.
  • Mind2Web evaluation: 56% improvement in average Step SR over MindAct is achieved on Mind2Web with GPT-3.5 after incrementally adding SYNAPSE components.The evaluation also reports 32% and 50% improvements as components are added incrementally, and 2.5× average Step SR with CodeLlama-7B.
  • Ablation studies: State abstraction improves performance under complex or ambiguous observations, raising email-inbox-nl-turk success from 52% to 100%.In Mind2Web, using fewer top-ranked elements with direct generation improves success rate by 32% over MCQ with the top-50 elements, while recall drops from 86% to 53%.
  • Ablation studies: 19% higher average step success rate follows from adding TaE prompting to state abstraction, while TaE outperforms MindAct by 50% across three generalization levels.On MiniWoB++, TaE is especially beneficial for long-horizon and repeated-action tasks such as guess-number, use-spinner, and use-autocomplete.
  • Ablation studies: Exemplar memory supports generalization to unseen or related tasks, yielding an almost 100% average success rate on 16 unseen MiniWoB++ tasks.In Mind2Web, memory adds a 6% Step SR improvement for cross-task and cross-website generalization, but provides no benefit for entirely unseen domains.

5 DISCUSSION, LIMITATIONS & FUTURE WORK

SYNAPSE combines state abstraction, trajectory-level prompting, and exemplar memory to address context limits, incomplete exemplars, and task-specific generalization. It achieves strong benchmark performance but remains limited by inference latency, exemplar quality, and its text-only scope.

  • Discussion: SYNAPSE addresses limited context, incomplete exemplar structures, and task-specific exemplars through state abstraction, trajectory-as-exemplar prompting, and exemplar memory.State abstraction reduces state tokens, trajectory-level exemplars support multi-step decisions, and memory retrieval supports generalization.
  • Discussion: 99.2% average success rate across 64 MiniWoB++ tasks and a 56% relative improvement on Mind2Web establish the method’s reported benchmark performance.The MiniWoB++ result uses no self-correction, while the Mind2Web result spans cross-task, cross-website, and cross-domain generalization.
  • Limitations & Future Work: High inference latency, dependence on exemplar quality, and text-only operation constrain the current framework.The authors suggest distillation, instruction tuning, improved retrieval, compositional generalization, and multimodal extensions as future directions.

A ADDITIONAL RESULTS

Additional results compare SYNAPSE with humans and several baselines at the task level. The table reports per-task mean success rates, while the figure filters and orders task-wise success-rate differences.

  • Additional Results: Table 3 compares per-task mean success rates for SYNAPSE, humans, and baseline methods.The table also reports each method’s average success rate and number of solved tasks.
  • Additional Results: Figure 6 compares task-wise success-rate differences among SYNAPSE, humans, AdaPlanner, Pix2Act, and WebN-T5.Tasks absent from other methods or showing differences below 0.05 are excluded, and included bars are sorted by ascending difference.

B.1 MINIWOB++

The MiniWoB++ setup uses raw HTML states and a fixed keyboard-and-mouse action space, with exemplars varying by task complexity. State abstraction uses explicit or implicit prompts to produce cleaner observations.

  • Environment Setup: MiniWoB++ experiments test 64 tasks, with demonstrations for 48 tasks and 16 unseen tasks.The setup follows the legacy branch and reports the unseen task names explicitly.
  • Exemplars: Complex tasks such as book-flight use more exemplars, including five in one configuration, which lowers average data efficiency.SYNAPSE does not rely on the self-correction mechanisms used by RCI.
  • Action Space: The agent represents states with raw HTML and uses click-xpath, move-mouse, type, press, and click-options actions.Code generation grounds LLM-produced actions, and exemplars specify selecting the first matching element.
  • State Abstraction: State abstraction uses state-observation pairs when context permits multiple states and can also infer cleaned observations from trajectories with LLMs.The states come from the environment, while human-provided cleaned observations accompany collected trajectories.

B.2 MIND2WEB

Mind2Web evaluates open-domain web navigation across diverse websites and domains, with test sets designed to measure increasingly broad generalization. Its tasks provide high-level goals rather than step-by-step instructions.

  • Dataset: Mind2Web contains over 2,000 tasks from 137 websites spanning 31 domains.The dataset supports interactions beyond basic searching or reading, including clicking, selecting, and typing.
  • Generalization: Mind2Web tests cross-task, cross-website, and cross-domain generalization using tasks from familiar sites, unseen sites in familiar domains, and unseen domains.These splits measure navigation to related tasks, new websites, and entirely new domains.

C PROMPTS

The prompts section illustrates SYNAPSE through terminal examples, showing how state abstraction and trajectory-as-exemplar prompting transform raw computer states into task-relevant observations and actions. It also records a text-transformation failure and the limited Mind2Web demonstration setting.

  • C PROMPTS: The section states that all trajectories are open sourced, while the displayed prompts provide concrete illustrations of SYNAPSE’s prompting process.The examples are presented as demonstrations of prompting and LLM responses.
  • C.1 TERMINAL: SYNAPSE completes the terminal task in four steps by combining two state-abstraction steps with two TaE-prompting steps.The task requires listing files before deleting the file with the specified extension.
  • C.1.1 STEP 1: STATE ABSTRACTION (EXPLICIT): State abstraction converts raw terminal states into task-relevant observations, retaining the instruction and relevant interface contents while reducing distracting information.The examples include task descriptions, terminal structure, available commands, login messages, and the active prompt.
  • C.1.3 STEP 3: STATE ABSTRACTION (EXPLICIT): For a .gpg task, the prompted trajectory shows the model receiving the task and prior observation-action sequence before selecting the matching file after listing the directory.The example identifies window.gpg among index.rb and media.html, then supplies the complete filename for deletion.
  • C.1.2 STEP 2: TAE PROMPTING: The examples demonstrate that TaE prompting links an initial task description to successive observations and actions rather than presenting only an isolated plan.The terminal trajectories explicitly show the assistant’s commands and the user’s resulting terminal state.
  • C.3 TEXT-TRANSFORM (FAILURE CASE): A demonstrated MiniWoB++ failure occurs when the LLM recognizes jrpf as jrfp while typing transformed text into a text field.The resulting action types jrfp and submits it, illustrating an error in text transformation.
  • C.4 MIND2WEB: The Mind2Web examples cover only one step because real-world tasks and websites are complicated, using an element-ranking model for observations and different k values for retrieved context.The setup uses k = 3 for memory trajectories and prior observations and k = 5 for the current observation.
Loading 2306.07863v3…