PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Evaluation and Refinement of Digital Agents

Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, Alane Suhr

arXiv:2404.06474v3cs.AI

TL;DR

Digital agents still make mistakes, while evaluating their trajectories and improving them without extra supervision remains a challenge. This paper develops domain-general automatic evaluators and uses them for inference-time guidance and filtered behavior cloning. The evaluators improve agent performance across web and device-control settings.

  • Problem

    Digital agents still make mistakes on simple tasks, and evaluating their performance and failure modes matters for understanding them and supporting safe deployment.

    Source: §1 · p. 1

  • Method

    The paper develops domain-general neural models to evaluate instructions and agent trajectories, then uses their judgments for inference-time guidance or filtered behavior cloning.

    Source: §1 · p. 1

  • Results

    The evaluators improve the best-performing GPT-4 WebArena agent’s success rate by up to 29% and boost a strong device-control policy by 75% via filtered behavior cloning.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Automated evaluation models can characterize and improve digital-agent performance without expert demonstrations or hand-designed evaluation functions.

    Source: §1 · p. 1

    Current evaluators are still imperfect, and the experiments focus on Reflexion and filtered behavior cloning while discarding generated language explanations.

    Source: Limitations and Future Work · p. 10

Abstract

from arXiv · show

We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics. Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance. Without any additional supervision, we improve state-of-the-art performance by 29% on the popular benchmark WebArena, and achieve around 75% relative improvement in device control settings.

1 Introduction

Digital agents can still fail on simple tasks, making evaluation important for understanding their failure modes and safe deployment. This work uses automated evaluators to assess and improve agent performance without extra supervision.

1 Introduction

  • Automated evaluation models can both characterize and improve agent performance without expert demonstrations or hand-designed evaluation functions.The introduction frames evaluation as important for understanding failures and supporting safe real-world deployment.Source: §1 · p. 1
  • The evaluators assess user instructions and arbitrary agent trajectories with domain-general neural models.The paper compares an end-to-end vision-language approach with a modular caption-then-reason approach, trading off performance, cost, and transparency.Source: §1 · p. 1
  • Up to 82.1% accuracy on WebArena and 92.9% on Android-in-the-Wild measure agreement with oracle evaluation metrics.The evaluators also improve existing agents through inference-time guidance and filtered behavior cloning without extra supervision.Source: §1 · p. 1
  • Figure 1 connects trajectory evaluation to two refinement routes: Reflexion reward and filtered behavior cloning.The evaluator scores a digital agent’s trajectory, and those scores can be used to enhance model performance.Source: Figure 1 · p. 2

2 Related Work

Prior digital-agent research spans agent architectures, evaluation benchmarks, and methods for refinement. This work positions model-based evaluation of arbitrary instructions and trajectories as a third approach, alongside simulation- and demonstration-based benchmarks.

Digital Agents

  • Earlier digital-agent work includes semantic parsing, reinforcement learning, and imitation learning, followed by renewed interest enabled by pretrained language and multimodal models.Recent benchmarks and approaches address instruction-conditioned web navigation and device control.Source: §2 · p. 2
  • Digital-agent methods have shifted from semantic parsing, reinforcement learning, and imitation learning toward pretrained language and vision-language models and inference-time refinement.Related approaches include exploration, plan revision, error correction, and self-critique; OS-Copilot’s critic does not evaluate executed behavior or support model training.Source: §2 · p. 2

Autonomous Refinement and Evaluation

  • Reflexion uses an external evaluation function for inference-time improvement, while this work studies refinement without that requirement.The authors also distinguish their work from WebVoyager, which does not deeply analyze judgment quality or explore evaluator use for agent improvement.Source: §2 · p. 3

Digital Agent Benchmarks

  • Simulation-based benchmarks enable execution of arbitrary policies, but simulator design, task curation, and hand-crafted evaluators limit how well they mirror real-world diversity.WebArena and VisualWebArena improve on earlier environments in coverage, realism, or generalizability, though they remain imperfect.Source: §2 · p. 3
  • Demonstration-based benchmarks use reference action matching, which requires human demonstrations and does not account for sequential outcomes or valid deviations from the reference.Mind2Web contains over 2K trajectories and AitW 715K, and both are also used to evaluate agents.Source: §2 · p. 3
  • The paper proposes a third approach: directly evaluating arbitrary instructions and agent trajectories with a model.This frames model-based trajectory evaluation alongside simulation-based and demonstration-based benchmarks.Source: §2 · p. 3

3 Domain-General Evaluators

The evaluators map an instruction, action sequence, and screenshot states to trajectory-level or per-step judgments. The paper implements this with either an end-to-end vision-language model or a modular caption-then-reason pipeline, with the modular approach facing screenshot-description limitations.

3 Domain-General Evaluators

  • The evaluator takes a user instruction, text-form actions, and screenshot states, and returns scalar evaluations for the trajectory’s steps.The paper assumes the instruction and actions are text and each state is represented as a screenshot image.Source: §3 · p. 3
  • Trajectory-level evaluation assigns a binary outcome to the final step, while per-step evaluation labels success, progress, or actions that do not contribute to the objective.Trajectory-level evaluation queries the model once; per-step evaluation queries it n times.Source: §3 · pp. 3–4
  • The end-to-end and modular evaluators trade off performance, cost, and transparency, with the modular design supporting explainability and low-cost local deployment.The end-to-end option can use advanced VLMs, while the modular approach can compose open-weight models with slightly weaker performance.Source: §3 · p. 4

3.1 End-to-End Approach

  • The end-to-end approach prompts a pretrained vision-language model to reason from instructions and screenshots before producing an evaluation.The experiments use GPT-4V.Source: §3.1 · p. 4

3.2 Modular Caption-then-Reason Approach

  • The modular approach has a vision-language model describe observations, then a language model evaluates those descriptions alongside actions and the instruction.The reasoner produces a text-based thought and final evaluation; the captioner is not given the original instruction because that can exacerbate hallucinations.Source: §3.2 · pp. 4–5
  • The modular approach risks losing task-relevant image details, and current open-weight vision-language models struggle to produce detailed screenshot descriptions out of the box.The authors fine-tune an open-weight captioner on 1,263 screenshot-description examples, with OCR results as an additional input.Source: §3.2 · p. 4

4 Experiments and Results

The experiments test evaluator agreement with benchmark or human judgments, then use evaluators for inference-time refinement and filtered behavior cloning across web and device-control settings. They report strong agreement and performance gains, while error analysis identifies reasoning and task-specification errors.

4 Experiments and Results

  • The experiments cover WebArena oracle evaluation, human-judged AitW trajectories, and filtered behavior cloning on iOS and Android.AitW experiments sampled 120 test tasks; the iOS task set contained 132 tasks.Source: §4 · p. 5, §4.2 · p. 8

4.1 Automatic Evaluation

  • WebArena evaluator accuracy was 82.1% for Captioner + GPT-4, 80.6% for GPT-4V, and 74.4% for Captioner + Mixtral; on AitW, Captioner + Mixtral reached 92.9%.On AitW, replacing Mixtral with GPT-4 reduced accuracy to 89.8%.Source: §4.1 · pp. 6–7
  • All three evaluators had 100% Kendall correlation with human judgments, compared with 66.7% for action matching.The authors note that single-step action matching misses error propagation and distribution shift captured by whole-trajectory success metrics.Source: §4.1 · p. 7

4.2 Autonomous Refinement

  • GPT-4V-guided Reflexion improved WebArena relative success rate by 29%, while Captioner + Mixtral improved it by 16%.All evaluator variants improved performance without hand-designed evaluation functions.Source: §4.2 · p. 8
  • Filtered BC improved iOS successes from 8 to 14 tasks (75% relative) and Android successes from 15 to 26.0 ± 0.8 (73% relative).The Android result was averaged over three training runs; its evaluation set contained 96 tasks.Source: §4.2 · p. 9

4.3 Error Analysis

  • Other errors included lost caption information (10%) or end-to-end screenshot understanding (5%); about 10% of correct evaluations had incorrect reasoning.The authors manually categorized sampled successful and erroneous WebArena evaluations.Source: §4.3 · p. 9
  • Reasoning errors dominated sampled WebArena evaluator failures: 50% for GPT-4V/GPT-4 methods and 70% for Mixtral-Captioner; task ambiguity rates were 30% and 10%, respectively.The error categories were assessed sequentially, so Mixtral-Captioner’s lower task-ambiguity rate largely reflected more earlier-category errors.Source: §4.3 · p. 9

5 Conclusion

The paper develops automatic methods to evaluate and refine digital agents, with end-to-end and modular designs that trade off performance, cost, and modularity. The evaluators support refinement without extra supervision and improve results in web and device-control settings.

5 Conclusion

  • The proposed methods provide trajectory-level or per-step evaluation through end-to-end or modular designs, trading off performance, cost, and modularity.The modular approach combines a vision-language model with a language model; the end-to-end approach uses a pretrained vision-language model.Source: §5 · p. 10
  • 29%: Reflexion’s best-performing agent gains up to 29% success rate with the evaluators as its reward function.The evaluators also improve a strong device-control policy by 75% in a domain-transfer task via filtered behavior cloning, without extra supervision.Source: §5 · p. 10
  • The evaluators refine existing agents through inference-time guidance and filtered behavior cloning without extra supervision.The paper reports refinement both through Reflexion and training with filtered behavior cloning.Source: §5 · p. 10

Limitations and Future Work

The authors identify imperfect evaluator performance and noisy-supervision robustness as areas for further work. They also limit their study to two refinement approaches and leave evaluators’ generated explanations unused.

Limitations and Future Work

  • Current evaluators remain imperfect, and better action-space representations or stronger base models may improve outcomes.The authors say such evaluator improvements will likely translate directly to improved outcomes.Source: Limitations and Future Work · p. 10
  • The study focuses on Reflexion and filtered behavior cloning, leaving broader refinement algorithms for future exploration.The authors identify scaling experiments and developing robust, efficient training and inference-time algorithms under noisy supervision as future directions.Source: Limitations and Future Work · p. 10
  • The evaluators use binary or ternary judgments and discard their language-based explanations.Future work could leverage explanations for language supervision or scalable oversight of agent behavior.Source: Limitations and Future Work · p. 10

Ethics Statement

The authors frame domain-general evaluators as a first step toward understanding digital agents’ risks beyond simulated environments. They say human evaluation and oversight will remain important for mitigating potential harms.

Ethics Statement

  • The work is presented as a first step toward understanding agent risks outside simulated environments such as WebArena.The authors note that real-world agents with unconstrained browser access may reach users’ passwords, financial information, or social media messages.Source: Ethics Statement · p. 10
  • Human evaluation and oversight will also be important for mitigating potential harms from future systems.The authors say autonomous evaluation should supplement, rather than supplant, human efforts.Source: Ethics Statement · p. 10
  • The authors identify better understanding of real-world failure modes as critical to safe deployment of digital agents.They describe current agents as mostly research artifacts and discuss potential security risks as such agents improve and are increasingly deployed.Source: Ethics Statement · p. 10

A Experiment Details

This section introduces the paper’s experiment details.

A Experiment Details

  • The section provides details about the experiments.The passage directs readers to the official code reference.Source: A Experiment Details · p. 16

A.1 End-to-End Approach

The end-to-end evaluator uses gpt-4-1106-vision-preview through the OpenAI API, with screenshots supplied at high resolution.

A.1 End-to-End Approach

  • Screenshots are fed without resizing in “high-resolution” to gpt-4-1106-vision-preview through the OpenAI API.The model is run at temperature 0, with other parameters left at their defaults.Source: §A.1 · p. 16
  • Evaluation uses temperature 0, while other model parameters remain at their defaults.Source: §A.1 · p. 16
  • Environment-specific prompt templates are provided in Figures 12 and 13.Source: §A.1 · p. 16

A.2 Modular Caption-then-Reason Approach

The modular approach combines screenshot descriptions with textual actions and language-model reasoning; its action-string representation loses pixel-click coordinates when images are textualized.

A.2 Modular Caption-then-Reason Approach

  • The screenshot dataset is primarily built by random subsampling source datasets, with 50 additional in-house screenshots captured for iOS because online resources are limited.Source: §A.2 · p. 16
  • String-form actions lose click coordinates [x, y] for pixel-based policies when the image is represented textually.Transforming pixel-localized actions into text more adequately is left for future work.Source: §A.2 · p. 16
  • GPT-4V produces dense screenshot-caption demonstrations, which are manually corrected or filtered when apparent errors occur.The captioning queries use temperature 0 and default settings for other parameters.Source: §A.2 · p. 16
  • The Qwen-VL-chat captioner is fine-tuned with EasyOCR text-recognition results as additional input during training and inference.Fine-tuning runs for 3 epochs with batch size 72; the optimizer is AdamW, with a cosine learning-rate schedule starting at 1e-5.Source: §A.2 · p. 16
  • After screenshot descriptions and actions are obtained, a language model—Mixtral-8x7B-Instruct-v0.1 or gpt-4-turbo-preview—is queried to evaluate the trajectory.The prompts use temperature 0 and default settings for other parameters.Source: §A.2 · p. 16

A.3 Evaluation on WebArena

The WebArena evaluation uses released GPT4-0613 + CoT - v2 trajectories and compares evaluator predictions with the oracle evaluator.

A.4 Evaluation on WebArena

  • The evaluators’ predictions are compared with the oracle evaluator using confusion matrices shown in Figure 4 (left).Source: §A.3 · p. 16
  • Evaluation uses the GPT4-0613 + CoT - v2 trajectories released by WebArena.Source: §A.3 · p. 16
  • Figure 4 (left) presents the confusion matrices for this oracle comparison.Source: §A.3 · p. 16

A.4 Evaluation on Android

The Android evaluation uses an Android Studio emulator and 120 sampled tasks, then compares evaluator predictions with human judgments.

A.4 Evaluation on Android

  • Android is simulated with Android Studio’s built-in Pixel 4 emulator running API version 33, with agent execution through a Python API based on Appium.AndroidEnv is not used because it lacks support for typing actions.Source: §A.4 · p. 16
  • The 120 evaluation tasks are sampled evenly and randomly from the General, WebShopping, and GoogleApps Android-in-the-Wild test subsets, 40 per subset.Install tasks are excluded because they require credit card information, and single-step tasks are outside the trajectory-level focus.Source: §A.4 · p. 17
  • The evaluators are compared with human judgments using confusion matrices presented in Figure 4 (right).All policies use temperature 0 during evaluation.Source: §A.4 · p. 17

A.5 Refinement on WebArena

This section describes a Reflexion setup for WebArena and refinement setups for iOS and Android agents. It illustrates post-task reflection in which an agent diagnoses failed attempts and proposes a changed strategy for retrying.

A.5 Refinement on WebArena

  • Reflexion combines an Actor, Evaluator, and Self-Reflection module; failed attempts trigger reflections stored in memory to guide later trials.The Actor generates actions from observations, and the Evaluator scores how well the trajectory meets the goal. The evaluator variants receive webpage snapshots rather than the DOM tree.Source: §A.5 · p. 17

A.6 Refinement on iOS

  • The iOS experiments use 132 task instructions, with 80 for data collection and training and 52 held out for testing.They use an iPhone 13 emulator running iOS 16; swiping up is translated to a left or right swipe half the time during data collection and always to the right during evaluation. CogAgent is fine-tuned with LoRA for 3,000 steps.Source: §A.6 · p. 17

A.7 Refinement on Android

  • For Android refinement, the authors use AutoUI-base with tasks sampled from the corresponding part of the AitW General split.Training uses temperature 1, learning rate 3e −3, and batch size 128.Source: §A.7 · p. 17

A.5 Refinement on WebArena

  • The reflection prompt asks the agent to explain why it failed and devise a concise new plan that accounts for the mistake.It supplies the task objective, interaction history, status, and memory of previous attempts.Source: §A.7 · pp. 31–32
  • After an unsuccessful Magento task, the agent says it should search the REPORT tab for best-selling products next time.Source: §A.7 · p. 33
  • After its answer about a fingerprint-resistant review failed, the agent concludes it should read all reviews next time.Source: §A.7 · p. 35
Loading 2404.06474v3…