PaperlayerView exact revision on arXiv

Source-linked AI summary

A Zero-Shot Language Agent for Computer Control with Structured Reflection

Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, Yang Li

arXiv:2310.08740v3cs.CLcs.SYeess.SY

TL;DR

The paper asks how a computer-control agent can learn new tasks without expert traces, particularly when screen representations may expose information not yet visible to a user. It proposes zero-shot executable-action planning with structured reflection, reporting better results than prior systems on easy tasks and comparable performance on harder tasks. The authors also identify a rare memory-management corner case and continued difficulty with end-to-end tasks.

  • Problem

    Many computer-control methods rely on expert traces, and some screen representations include information not yet visible to users; whether an agent can learn and improve without such traces remains a challenge.

    Source: §1 · p. 1

  • Method

    The agent plans executable actions in a partially observed environment and uses structured self-reflection and thought management to learn from mistakes across attempts.

    Source: §1 · p. 2, §5 · p. 5, §5.1 · p. 5

  • Results

    On easy MiniWoB++ tasks, the agent often outperforms prior systems, while on more complex tasks it performs on par with prior best models.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The results show that a zero-shot agent can handle easy MiniWoB++ tasks with often one planning pass and match the best LLM-based model on complex tasks through planning and reflection.

    Source: §8 · p. 9

    End-to-end tasks such as book-flight remain a significant challenge for the zero-shot agent.

    Source: §9.5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MiniWoB++, we show that our zero-shot agent often outperforms recent SoTAs, with more efficient reasoning. For tasks with more complexity, our reflective agent performs on par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen information.

1 Introduction

The paper addresses whether an agent can learn computer control without expert traces, while avoiding screen representations that expose information unavailable to a user. It proposes a zero-shot agent and reports strong results on easy MiniWoB++ tasks and comparable performance on harder ones.

1 Introduction

  • Prior MiniWoB++ approaches use extensive task traces for supervision or prompting, while autonomous learning without those traces remains an open question.These approaches have reported over 90% task completion on dozens of tasks.Source: §1 · p. 1
  • Some prior screen representations include HTML information not yet visible on screen, which may be unavailable in general use.Manual examination found this multi-screen information in the HTML of 5 of 13 relatively challenging MiniWoB++ tasks.Source: §1 · pp. 1–2
  • The zero-shot agent solves almost all easy MiniWoB++ tasks and performs comparably to prior few/many-shot state-of-the-art on more challenging tasks.The authors describe the approach as a first zero-shot design for computer control tasks.Source: §1 · p. 2
  • The proposed agent uses a unified instruction prompt across tasks, compact screen representations, executable-action planning, and structured reflection to improve from exploration failures.For harder tasks, it uses a few rounds of attempts to learn from failures.Source: §1 · pp. 1–2

2 Background

The background reviews LLM planning and reflection, then situates the work among computer-control approaches evaluated on MiniWoB++. It also introduces tables comparing prior reflective planners and MiniWoB systems.

Planning & Reflection

  • Prior reflective-planning work includes intermediate-thought action planning and iterative self-critique, while this agent uses an efficient staged planner.The passage names REACT (Yao et al., 2023), REFLEXION (Shinn et al., 2023), SELF-REFINE (Madaan et al., 2023), and SWIFTSAGE (Lin et al., 2023).Source: §2 · p. 2
  • Table 1 compares prior reflective planners by task action count and whether they require a planner call for each action.Its caption defines N as the number of actions to complete a task and marks per-action planner calls as inefficient.Source: Table 1 · p. 2

Language/Vision Agent for Computer Control

  • Table 2 compares prior work on MiniWoB++, with screen consistency discussed separately in Section 3.1.The passage notes that no public CC-Net screen representation was available when the paper was written.Source: Table 2 · p. 2, §2 · p. 2
  • MiniWoB++ provides live fundamental computer tasks, with prior approaches using human annotation for behavior cloning or reinforcement learning and prompted LLM inference for action generation.The passage distinguishes multimodal or vision-input models from off-the-shelf LLM agents.Source: §2 · p. 2

3 Environment Interface

The interface represents the environment as a screen observation, supports action execution, and uses feedback to guide subsequent behavior. The design limits observations to available information and compresses HTML for the agent.

3 Environment Interface

  • The interface lets the agent comprehend screen representations, execute actions, and react to environment feedback.These roles are associated with screen comprehension in Sections 3.1 and 3.2, action execution in Section 3.3, and feedback in Section 3.4.Source: §3 · p. 3

3.1 Treatment of Screens

  • HTML elements are expanded only when the agent takes the UI action that reveals them, so the agent must act with limited information.The design relaxes assumptions about the environment by avoiding expanded HTML that requires a UI action before it is available.Source: §3.1 · p. 3

3.2 Compact Screen Representation

  • The compact representation heuristically simplifies HTML while retaining each leaf element’s id, class, text, placeholder, value, and position on a 3x3 screen grid.The paper motivates compression by the verbosity of raw HTML and the context-length limits of LLMs.Source: §3.2 · p. 3

3.3 Action Space

  • The agent outputs commands for three action types, grounds elements through XPath, uses compact ids for clicks, and implements typing as a click followed by key presses.Table 3 presents action types and example commands.Source: §3.3 · p. 3, Table 3 · p. 3

3.4 Environment Feedback

  • Environment feedback classifies trial endings as correct, cycle, no change, incomplete, exception, or failed, with conditions associated with reflection prompts.Cycle and no-change conditions compare current HTML with prior screens; exceptions include unsuccessful action grounding.Source: §3.4 · p. 3

4 Planning Strategies

The section reviews iterative and plan-then-adapt strategies before presenting staged planning, which generates executable actions visible on the current screen and repeats across screens.

4 Planning Strategies

  • The section frames planning as issuing actions for a goal based on prior interaction, represented as a sequence of state-action pairs.Figure 3 depicts a successful reflection trial on use-autocomplete; its step actions are paraphrased.Source: §4 · p. 4, Figure 3 · p. 4

4.1 Iterative Planning

  • Iterative planning generates and executes one atomic action at a time, then observes the resulting state.It suits exploratory environments; in responsive environments, a long interaction history can connect readily to Chain-of-Thought reasoning.Source: §4.1 · p. 4

4.2 Plan-Then-Adapt

  • Plan-then-adapt turns a rough initial plan into executable actions, but may require trace examples and risk hallucinating actions for unobserved states.The initial plan and adaptation use the same LLM.Source: §4.2 · p. 4

4.3 Staged Plan-And-Follow

  • Staged planning maximizes executable actions visible on the current screen, follows that plan, then replans on later screens.Each stage generates k executable actions for the current state; the method omits former states to make inference more efficient.Source: §4.3 · pp. 4–5

5 Structured Self-Reflection

The agent reflects on failed trials and stores action corrections for later attempts. Structured thought management constrains retries, tracks failed actions, and clears later reflections when an earlier correction changes.

5 Structured Self-Reflection

  • After a negative signal, the agent reviews its trajectory, proposes an improved action, and retries the task.Reflections record wrong-action-to-replacement pairs and are added to the planner prompt in the follow-up trial.Source: §5 · p. 5

5.1 Structured Thought Management

  • Reflection memory can become combinatorial across trials, and corrections after an earlier critical mistake may be outdated.Without memory management, the agent may repeat mistakes, fail to reach the reflected step, or cycle among previously wrong actions.Source: §5.1 · p. 5
  • Structured thought management enforces reflected actions, records failed actions, and clears later entries when an earlier reflection changes.It uses a disabled-action set and has memory size N, avoiding a bound based on the LLM input limit.Source: §5.1 · pp. 5–6

5.2 Constraining Action Space

  • For click actions, previously failed options are disabled while their element information remains visible; non-click actions instead force the reflected action at its indexed step.The authors report no positive preliminary impact from prompting the LLM to invalidate non-click actions.Source: §5.2 · p. 6

6 Experiments

On complex multi-screen tasks, the zero-shot agent improves through reflection and performs comparably to prior best models despite lacking their expert traces or additional screen information. On easier tasks, staged planning achieves high completion while reducing LLM queries.

6 Experiments

  • The experiments categorize task complexity, test staged planning, and then examine whether the zero-shot agent learns from mistakes.Prompts and complete results are provided in the appendices.Source: §6 · p. 6

6.1 Setup

  • The evaluation divides 43 MiniWoB++ tasks into 1-screen-1-step, 1-screen-n-step, and n-screen-n-step categories based on state updates and required actions.Each task uses 25 random seeds for completion-rate evaluation; FLAN-PaLM2 L runs at temperature 0.Source: §6.1 · p. 6, Table 4 · p. 6

6.2 Model Comparison

  • Comparisons include supervised and prompted models, with shared-task results reported to address differences in the task sets used by prior work.For few-shot comparisons, the authors focus on agents with reflection capacity.Source: §6.2 · pp. 6–7

6.3 Single Step Tasks

  • 99.6% completion over 3 trials on 1-screen-1-step tasks approaches RCI’s 99.8% with one round of plan-level reflection and few-shot traces.Without reflection, the agent correctly completed nine tasks at 100%; the ambiguous click-widget task improved to 96% after three reflection rounds.Source: §6.3 · p. 7, Figure 4 · p. 7

6.4 Iterative Planning v.s. Staged Planning

  • Staged planning can execute all planned actions for a screen, substantially reducing LLM queries while retaining high completion on 1-screen-n-step tasks.The agent achieved 94.0% completion in one trial and 96.2% in three trials.Source: §6.4 · p. 7, Table 5 · p. 7

6.5 Reflective Planning on Challenging Tasks

  • After weak first-trial performance on challenging multi-screen tasks, reflection improves completion by avoiding recorded negative signals, and performance remains comparable to prior best models despite their information advantages.The agent continues improving through T = 5; the authors note that prior few-shot models use inconsistent screens and that the comparison is therefore disadvantageous to their agent.Source: §6.5 · p. 7, Table 6 · p. 7, Figure 6 · p. 8

7 Analysis & Discussions

Across ablations, structured reflection and action constraints are beneficial, and trial-to-trial differences on multi-screen tasks are statistically significant. The analysis also identifies context-length and screen-complexity limits on staged planning.

7.1 Ablation on Reflection Strategies

  • Structured reflection is a beneficial add-on in a comparison of both mechanisms against a setting without structured thought management.The comparison uses five challenging tasks and 25 seeds per setting.Source: §7.1 · p. 8, Figure 7 · p. 8

7.2 Ablation on Action Constraint

  • Deleting an element’s id in the HTML pseudo code heuristically discourages the agent from repeating the corresponding action, and the ablation finds this constraint beneficial.The change is described as minimal.Source: §7.2 · p. 8

7.3 Statistical Significance over Trials

  • p < 0.05: t-tests find significant differences across trial counts on n-screen-n-step tasks using 325 samples per comparison.The 325 samples comprise 13 tasks with 25 predictions each.Source: §7.3 · p. 8, Table 7 · p. 8

7.4 Planning Call Reduction

  • Planning-call reduction is evaluated on long 1-screen-n-step traces, comparing iterative and staged planning calls for successful first trials and all trials.The study defines long traces as requiring at least seven actions and evaluates with 25 examples per task at T = 3.Source: Table 8 · p. 8, §7.4 · p. 8

7.5 Compact Screen & Input Length Limit

  • Long HTML representations can exceed the reflection agent’s context limit, while exposing unseen elements can simplify multi-screen navigation and obscure its difficulty.For social-media-all, the authors keep the first screen when constructing the reflection prompt because nuanced actions do not substantially change it.Source: §7.5 · pp. 8–9

7.6 Capacity of Staged Planning

  • Staged planning handles many actions well on flat, less ambiguous screens, but complex candidate layouts sharply reduce one-pass planning capacity, making reflection helpful.The probes compare click-checkboxes and social-media; relaxing RCI’s candidate constraint increases planning difficulty.Source: §7.6 · p. 9, Table 9 · p. 9

8 Conclusions

The paper presents a zero-shot agent for computer control that combines efficient planning with structured thought management. On MiniWoB++, it reports strong results on simple tasks and performance on par with the best LLM-based model on complex tasks.

8 Conclusions

  • The agent is presented as the first zero-shot agent for computer control tasks.Its workflow is designed to generalize across easy and complex tasks through efficient planning and structured thought management.Source: §8 · p. 9
  • On simple MiniWoB++ tasks, often one planning query outperforms both the best iterative planning agent and supervised state-of-the-art models.The conclusion attributes this to an efficient planning approach.Source: §8 · p. 9
  • On complex MiniWoB++ tasks, the agent performs on par with the best LLM-based model through more efficient planning and reflection, without manually crafted trace prompts or ad-hoc environment feedback.Source: §8 · p. 9

9 Limitations

The limitations describe evaluation scope, possible extensions, and remaining challenges, including a rare reflection-memory loop and difficulty with end-to-end tasks. The appendix passages also identify prompts used for action planning and reflection.

9.1 Other LLM Choices

  • Evaluations focused on PaLM-2, although the authors believe their findings may reasonably adapt to other LLMs.Source: §9.1 · p. 9

9.2 Other Modalities of Input

  • The proposal could also benefit multimodal use cases, where large multimodal models can accept screen images and explicit reasoning remains in language.Source: §9.2 · p. 9

9.3 Integration Zero-shot Chain-of-Thought

  • Integrating zero-shot chain-of-thought methods is a possible direction, but extracted prior knowledge needs evaluation because computer-control tasks differ from general-domain tasks.The authors leave this direction for future work.Source: §9.3 · p. 10

9.4 Constraining Space for Non-click Actions

  • Disabling failed click actions by removing their element id can help avoid repeated mistakes, but this constraint applies only to click actions.Source: §9.4 · p. 10

9.5 More End-to-end Tasks

  • End-to-end tasks such as book-flight remain a significant challenge for the zero-shot agent.The passage contrasts this with few-shot approaches that extract informative reference traces.Source: §9.5 · p. 10

9.6 Higher-order Action Cycle

  • In rare cases, clearing earlier reflection memory can make the agent loop between two different failed traces.The authors suggest additional trace memory as a possible way to address this corner case.Source: §9.6 · p. 10

A Prompt for Action Space; B Prompt for Staged Planning; C Prompt for Action Paraphrase

D Status Prompt Prefix; E Prompt for Reflection

F Completion Rate Tables

The appendix reports completion rates across 43 MiniWoB++ tasks, organized by task category and screen/step structure.

F Completion Rate Tables

G Task-wise Statistical Significance

The appendix describes a task-wise significance test comparing T = 1 with T = 5 on n-screen-n-step tasks. It specifies a one-tailed McNemar test but provides no test outcomes in these passages.

G Task-wise Statistical Significance

Related papers

Loading 2310.08740v3…