PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models can Solve Computer Tasks

Geunwoo Kim, Pierre Baldi, Stephen McAleer

arXiv:2303.17491v3cs.CLcs.AIcs.HCcs.LG

TL;DR

Computer-task agents need to solve new natural-language tasks without large expert datasets or task-specific rewards. The paper introduces Recursive Criticism and Improvement, a prompting method that critiques and revises LLM outputs. RCI outperforms prior computer-task approaches and improves language-model reasoning, especially when combined with chain-of-thought prompting.

  • Problem

    Natural-language computer agents should generalize to new tasks, but prior approaches require extensive expert demonstrations or task-specific reward functions.

  • Method

    RCI prompts a pre-trained LLM to generate an output, identify problems in it, and produce an improved output, including across computer-task grounding stages.

  • Results

    RCI surpasses existing SL, RL, and LLM approaches on MiniWoB++ and improves reasoning over zero-shot and CoT with external feedback, with RCI+CoT performing best.

  • Takeaways & Limitations

    RCI provides a practical approach for natural-language computer-task execution using few demonstrations and no task-specific reward function.

  • Takeaways & Limitations

    The study focuses primarily on InstructGPT-3+RLHF models, leaving RCI’s generalization to other models unexplored.

Abstract

from arXiv · show

Agents capable of carrying out general tasks on a computer can improve efficiency and productivity by automating repetitive tasks and assisting in complex problem-solving. Ideally, such agents should be able to solve new computer tasks presented to them through natural language commands. However, previous approaches to this problem require large amounts of expert demonstrations and task-specific reward functions, both of which are impractical for new tasks. In this work, we show that a pre-trained large language model (LLM) agent can execute computer tasks guided by natural language using a simple prompting scheme where the agent Recursively Criticizes and Improves its output (RCI). The RCI approach significantly outperforms existing LLM methods for automating computer tasks and surpasses supervised learning (SL) and reinforcement learning (RL) approaches on the MiniWoB++ benchmark. We compare multiple LLMs and find that RCI with the InstructGPT-3+RLHF LLM is state-of-the-art on MiniWoB++, using only a handful of demonstrations per task rather than tens of thousands, and without a task-specific reward function. Furthermore, we demonstrate RCI prompting's effectiveness in enhancing LLMs' reasoning abilities on a suite of natural language reasoning tasks, outperforming chain of thought (CoT) prompting with external feedback. We find that RCI combined with CoT performs better than either separately. Our code can be found here: https://github.com/posgnu/rci-agent.

1 Introduction

The paper targets natural-language computer-task agents that can generalize beyond task-specific training. It introduces RCI, which improves LLM outputs through criticism and revision, outperforming prior approaches on computer tasks and reasoning benchmarks.

  • Natural-language computer agents could automate repetitive work and assist with complex problem-solving, potentially increasing productivity.
  • LLM agents acting through keyboard and mouse actions struggle with task, state, and agent grounding, while prior SL+RL methods require task-specific data and rewards.
  • RCI first generates an output, prompts the LLM to identify problems, and then generates an updated output conditioned on the critique.
  • RCI surpasses existing supervised learning, reinforcement learning, and LLM approaches on MiniWoB++, using few demonstrations per task without task-specific reward functions.
  • RCI improves LLM reasoning over zero-shot prompting and slightly over chain-of-thought prompting with external feedback, while RCI combined with CoT performs best.

2 Methods

RCI decomposes computer-task action selection into planning, environmental grounding, and executable formatting. It applies critique and improvement across these stages to produce more task-appropriate, state-feasible, and admissible actions.

  • 2.1 RCI Prompting: RCI critiques a language model’s previous output and generates an improved output based on the identified problems.
  • 2.2 RCI for Computer Tasks: Computer-task action selection is decomposed into task grounding, state grounding, and agent grounding.
  • Action selection: At each step, the LLM receives the task and current environment state, samples a natural-language action, and passes it to the instruction-following agent.
  • Task grounding: Task grounding generates an actionable plan and conditions subsequent actions on it; one explicit RCI pass suffices for most MiniWoB++ tasks.
  • State grounding: State grounding maps abstract task-grounded actions to concrete HTML elements in the current environment state so they become feasible to execute.
  • Agent grounding: Agent grounding uses implicit RCI to ensure state-grounded actions are admissible and correctly formatted for the computer agent.

3 Evaluation

The evaluation compares RCI with prompting and learning baselines on reasoning benchmarks and MiniWoB++, examining performance, data requirements, grounding components, and language-model choice. RCI improves reasoning accuracy, reaches strong MiniWoB++ performance with limited demonstrations, and benefits from grounding and stronger underlying models.

  • 3.1 Reasoning tasks: RCI substantially outperforms standard prompting across all eight arithmetic and commonsense reasoning benchmarks.The evaluated datasets include GSM8K, MultiArith, AddSub, AQUA, SVAMP, SingleEq, CommonSenseQA, and StrategyQA.
  • 3.1 Reasoning tasks: RCI outperforms Zero-Shot CoT and Few-Shot CoT on four of five arithmetic tasks, while combining RCI with CoT achieves the highest scores on four tasks.The exception is MultiArith, where standard prompting already reaches 96.06% accuracy and RCI adds no significant gain.
  • 3.2.2 Outperforming baselines on MiniWoB++ task suite: RCI achieves a 41% higher MiniWoB++ score than WebN-T5-3B and outperforms CC-Net without dictionary-based typing by 6%.WebN-T5-3B uses 12K expert demonstrations, whereas the comparison with CC-Net uses the fully generative typing setting.
  • 3.2.2 Outperforming baselines on MiniWoB++ task suite: RCI surpasses category-level state-of-the-art by 37 percentage points in supervised learning and 27 percentage points in reinforcement learning, while exceeding combined SL+RL by less than 1 percentage point.The combined SL+RL comparison uses substantially more expert data and online interactions.
  • 3.2.2 Outperforming baselines on MiniWoB++ task suite: RCI reaches comparable MiniWoB++ performance using the least data, despite competing methods using thousands or millions of expert demonstrations.CC-Net uses 2.4 million demonstrations, while WebN-T5-3B uses 12,000 expert demonstrations.
  • 3.2.4 Ablating the groundings: Grounding effects vary by task: state grounding helps when initial states lack sufficient information, task grounding helps long-horizon tasks, and agent grounding improves action feasibility.Removing agent grounding decreases success by 60% on simpler tasks.
  • 3.2.5 Ablating the language model: RCI performance depends on the underlying language model, with experiments comparing GPT-3, InstructGPT-3, and InstructGPT-3 + RLHF across task difficulties.The primary MiniWoB++ evaluation uses gpt-3.5-turbo and gpt-4, while earlier models support ablation studies.

4 Limitations

The authors identify limitations involving model coverage, context length, action diversity, planning horizon, and understanding complex interfaces.

  • RCI is evaluated primarily with InstructGPT-3 + RLHF models, leaving its generalization to other models unexplored.
  • Limited context length constrains the agent’s ability to handle extensive HTML states effectively.The authors suggest architectural adjustments or novel parsing methods may be needed.
  • An action space mainly restricted to clicks and typing limits the agent’s web-navigation capabilities.
  • The agent’s short-term decision focus overlooks long-term strategy in tasks requiring coordinated action sequences.
  • Contemporary web interfaces can contain nuances that are difficult for LLMs to understand from HTML alone.The authors suggest adding more modalities to the state definition.

5 Discussion

The discussion contrasts RCI’s few-shot, reward-free setting with task-dependent approaches while noting current capability and cost limits and broader deployment risks.

  • LLMs can solve novel tasks in zero-shot or few-shot settings without task-dependent expert data or reward functions.
  • Current LLMs remain less capable than task-dependent SL+RL approaches on some computer tasks.
  • RCI costs more to run than approaches that sample from the LLM only once.
  • The paper’s reasoning results suggest that improving LLMs’ ability to think ahead remains important for difficult computer tasks.
  • Allowing LLMs to act on computers creates risks including cyberattacks, disinformation, job displacement, prompt injection, and runaway events.

B.1 Automated computer tasks

Automated computer-task research spans reinforcement and imitation learning, LLM-based action systems, and reasoning architectures, with RCI evaluated across multiple models.

  • B.1 Automated computer tasks: MiniWoB++ research has used reinforcement learning and imitation learning, but human-level performance required 6,300 hours of expert demonstrations.
  • B.1 Automated computer tasks: The proposed agent requires less than two demonstrations per task on average, unlike approaches that fine-tune LLMs with substantial expert data.
  • B.2 LLMs with actions: SayCan grounds candidate actions by multiplying each action’s probability under FLAN by a value function indicating action suitability.
  • B.2 LLMs with actions: Inner Monologue adds a feedback loop for state grounding but still relies on a pre-trained language-conditioned robot policy.
  • B.1 Automated computer tasks: RCI is evaluated on MiniWoB++ with gpt-3.5-turbo and gpt-4, alongside other models in ablation studies.

C.2 Reasoning tasks

The reasoning evaluation covers arithmetic and commonsense datasets and excludes some computer tasks whose HTML exceeds language-model context limits.

  • The reasoning evaluation uses eight datasets spanning arithmetic and commonsense reasoning.
  • Arithmetic evaluation includes SingleEq, AddSub, MultiArith, AQuA, GSM8K, and SVAMP.
  • Commonsense evaluation includes CommonsenseQA and StrategyQA.
  • Several MiniWoB++ tasks were excluded because their HTML exceeded language models’ maximum context length.

C.4 MiniWoB++ task selection for ablation studies

The ablation study selects MiniWoB++ tasks and organizes them by difficulty. This classification structures comparisons across task levels.

  • The ablation study uses a subset of MiniWoB++ tasks selected from three difficulty levels.The levels are based on success rates achieved by the agent with gpt-3.5-turbo.
  • The task-selection procedure links ablation coverage to the agent’s observed success rates.
  • Table 4 classifies the tasks used in the ablation study according to difficulty.

C.5 Modifications on MiniWoB++ tasks

The evaluation modifies MiniWoB++ tasks to accommodate language models with limited context length. These changes are documented in Table 5.

  • The MiniWoB++ benchmark was modified for evaluation with language models having limited context length.
  • The modifications are intended to make the benchmark suitable for the evaluated language-model setting.
  • Table 5 summarizes the modifications applied to MiniWoB++ tasks.

D Prompts for MiniWoB++ tasks

The MiniWoB++ prompts decompose computer-task action selection into task, state, and agent grounding. Examples show how grounding refines plans and actions, while task-specific instructions address execution details.

  • D Prompts for MiniWoB++ tasks: The agent specification defines the setup used for prompting MiniWoB++ tasks.
  • D Prompts for MiniWoB++ tasks: Task grounding prompts the model to generate a high-level plan for solving the task.
  • D Prompts for MiniWoB++ tasks: The prompts include task-specific instructions for interpreting webpage outputs and selecting the required action.
  • D Prompts for MiniWoB++ tasks: State grounding connects task-level actions to executable elements in the current environment state.
  • D Prompts for MiniWoB++ tasks: Agent grounding formats the selected action into the output representation expected by the environment.
  • E Examples of cross grounding: State grounding can remove superfluous instructions from an initial action plan.
  • E Examples of cross grounding: State grounding can restore omitted punctuation so a generated action matches the action specified in the plan.

F Failure modes on MiniWoB++ tasks

The MiniWoB++ failure analysis identifies limitations in state grounding, visual perception, long-horizon planning, unexpected-event handling, strategic reasoning, and precise interaction control.

  • click-dialog-2: State grounding can produce task-irrelevant actions in click-dialog-2, so the state-grounding step is excluded for that task.
  • click-menu: Explanatory few-shot trajectories improve sample efficiency for click-menu, where hovering is required before selecting the target component.
  • count-shape: The agent struggles with tasks requiring visual perception of rendered shapes on a canvas because it relies on HTML observations.
  • guess-number: Long-horizon guess-number tasks can exceed 50 steps, causing actions to deviate from the original plan and produce irreparable failure.
  • unexpected pop-up tasks: Unexpected pop-up handling is not reliably executed even when state grounding should generate an instruction to close the window.
  • tic-tac-toe: Tic-tac-toe failures reflect missing defensive play and failure to account for attacks being blocked by the opponent.
  • use-autocomplete: Autocomplete failures arise when selecting words ending with specified characters requires pressing the down arrow an exact number of times.

G Additional results

Additional results report reasoning-benchmark performance without external feedback and define task-level comparisons against state-of-the-art baselines in MiniWoB++.

  • Without external feedback, RCI prompting on reasoning benchmarks performs equivalently to or below zero-shot approaches.
  • Figure 10 plots task-level residual performance by subtracting each state-of-the-art baseline’s performance from the agent’s performance.
  • Table 18 provides comprehensive task-level success-rate evaluations for baseline models across MiniWoB++ tasks.
  • The table includes outcomes using GPT-4 for some tasks, highlights those results in blue, and sources baseline performance from prior studies.
  • Average success rates for violet-highlighted tasks are reported in Figure 3, alongside the supervised-learning state-of-the-art baseline.
Loading 2303.17491v3…