PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Distillation of Web Agent Capabilities Enables Generalization

Xing Han Lù, Siva Reddy

arXiv:2604.07776v1cs.LG

TL;DR

Frontier web agents are costly to deploy locally, while small open-weight models lag in capability. The paper structures synthetic trajectory generation around modular annotation roles, using a single frontier teacher to train a 9B student. The resulting model reaches 41.5% on WebArena and improves across unseen environments, including an 18.2pp gain on WorkArena L1.

  • Problem

    Small open-weight web agents trail frontier models by over 22 percentage points, while frontier systems require expensive API access and transmit user data to third-party servers.

  • Method

    AGENT-AS-ANNOTATORS replaces human task design, annotation, and supervision with modular LLM components that generate, execute, and filter web-agent trajectories.

  • Results

    41.5% on WebArena enables a 9B open-weight student to surpass GPT-4o and Claude 3.5 Sonnet, with improvements transferring across five benchmarks.

  • Takeaways & Limitations

    Structured trajectory synthesis from one frontier teacher produces generalizable web interaction capabilities in a locally deployable 9B model.

  • Takeaways & Limitations

    The persona module lacks a controlled no-persona comparison, and teacher comparisons use only the Gemini family.

Abstract

from arXiv · show

Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

1 Introduction

AGENT-AS-ANNOTATORS addresses the capability gap between frontier web agents and small open-weight models by structuring trajectory synthesis around modular annotation roles. Using Gemini 3 Pro, it produces filtered training data that yields a competitive 9B student with cross-environment transfer.

  • Motivation: Small open-weight web agents trail frontier models by over 22 percentage points while frontier systems require costly third-party API access.These constraints motivate local capability distillation from a frontier teacher.
  • Motivation: Agentic capability distillation uses a frontier model as teacher to generate training trajectories for a smaller student.Existing web-agent pipelines differ in component design, making systematic comparison difficult.
  • Framework: AGENT-AS-ANNOTATORS replaces human Task Designer, Annotator, and Supervisor roles with Persona and Task Generators, an Agent, and a Judge.The framework organizes synthetic trajectory generation by analogy to human annotation.
  • Results: 3,000 tasks across six WebArena environments produced a 9B student that reaches 41.5% on WebArena and transfers to unseen enterprise platforms.The implementation uses Gemini 3 Pro as teacher, and only successful trajectories train the student.

2 Related Work

Prior work advances web-agent evaluation and synthetic trajectory generation through increasingly realistic environments and scalable collection methods. AGENT-AS-ANNOTATORS sits within this literature while framing differing synthesis designs and evaluation hints as comparable pipeline choices.

  • Web-agent benchmarks: Web-agent benchmarks progressed from synthetic micro-environments to realistic self-hosted applications and cross-website generalization.WebArena evaluates six web applications, while VisualWebArena adds vision-grounded tasks and WorkArena variants target enterprise workflows.
  • Synthetic trajectory generation: InSTA scales grounded task proposal and execution to 150K websites while filtering trajectories with an LLM judge.Its design emphasizes website scale rather than the persona-driven diversity used here.
  • Synthetic trajectory generation: Other synthesis pipelines retroactively label exploration, replay tutorials, refine exploration, or search URL graphs, making their designs difficult to compare directly.AGENT-AS-ANNOTATORS maps these approaches onto common modules.
  • Knowledge distillation: Knowledge distillation and curated synthetic data show that stronger teachers and data quality can train weaker models effectively.Related work spans Self-Instruct, persona-driven diversity, reasoning distillation, and curated data.
  • Evaluation and self-improvement: AGENT-AS-ANNOTATORS augments LLM-based evaluation with task-specific evaluation hints, while RL-based web-agent methods provide complementary alternatives.The framework treats evaluation reliability and reinforcement learning as related but distinct directions.

3 The AGENT-AS-ANNOTATORS Framework

AGENT-AS-ANNOTATORS formalizes web-agent data creation as two phases that mirror three human annotation roles. Its modules separate task grounding and diversity from fresh-environment execution and hint-based success verification.

  • Pipeline comparison: AGENT-AS-ANNOTATORS is the only mapped pipeline that instantiates all six framework modules.The framework exposes design choices for systematic comparison and links them to downstream consequences.
  • Role mapping: The framework maps human Task Designer, Annotator, and Supervisor roles to modular LLM components.Prior trajectory synthesis pipelines can be expressed as framework instantiations with different module choices.
  • Role mapping: AGENT-AS-ANNOTATORS is organized into task synthesis and trajectory collection-and-filtering phases.The two phases mirror the division of labor in human annotation.
  • Task synthesis: The Persona Generator creates varied user profiles, while the Task Generator explores environments and produces grounded task intents paired with evaluation hints.Persona diversity induces varied task distributions within the same environment.
  • Trajectory collection and filtering: The Agent solves each task in a freshly reset environment using only the task intent, while the Judge evaluates the trajectory against evaluation hints.This separation prevents the Agent from accessing exploration data or hints during task solving.
  • Trajectory collection and filtering: Only successful trajectories are retained for student training, and grounded task definitions uniquely support generating evaluation hints alongside intents.The framework makes this task-grounding distinction explicit across prior pipelines.

4 Experimental Setup

The experiments synthesize and filter web-agent trajectories across six WebArena environments, convert successful interactions into supervised examples, and evaluate a 9B student across five BrowserGym benchmarks. The setup uses Gemini 3 Pro for task generation, acting, and judging, with structured reasoning traces retained for SFT.

  • Dataset construction: A3-SYNTH contains 3,000 tasks generated from 250 personas across six self-hosted WebArena environments.Each persona is assigned to all six environments, with two task intents synthesized per exploration.
  • Teacher model: Gemini 3 Pro serves as teacher for both task synthesis and trajectory collection, using a reduced thinking budget and concise reasoning traces.The same frontier model fills both roles in the pipeline.
  • Judge: The Judge receives the full trajectory, task intent, and structured evaluation hints describing expected outcomes.Hints include concrete end states such as confirmation messages or newly created files.
  • Filtering and data statistics: 2,322 successful trajectories yield 16,353 observation-action training examples after Judge filtering.Success rates range from 69–85% across environments, with an average of 7.0 steps per trajectory.
  • Training format: Successful trajectories are formatted as multi-turn SFT examples pairing observations and task goals with structured reasoning followed by actions.Cross-entropy loss is applied only to assistant tokens.
  • Student training: Qwen3.5-9B is fine-tuned for two epochs using multimodal text-and-image inputs and distributed training across 4–8 GPUs.The configuration uses a 1 × 10^-5 learning rate, cosine annealing, batch size 32, and maximum sequence length 8,192.
  • Evaluation: Evaluation uses BrowserGym across WebArena, VisualWebArena, WorkArena, WorkArena++, and MiniWoB.Official splits are used where available, while WorkArena L1 and MiniWoB use their full task sets.

5 Results

The 9B student achieves strong WebArena performance, transfers to unseen enterprise interfaces and other benchmarks, and benefits from higher-quality structured trajectories and pipeline components.

  • Cross-benchmark performance: 41.5% on WebArena: A3-Qwen3.5-9B surpasses GPT-4o and Claude 3.5 Sonnet under the same evaluation protocol.It also exceeds Go-Browse’s 21.7% open-weight result, though that comparison is confounded by different agent harnesses and observation formats.
  • Cross-benchmark performance: +18.2pp on WorkArena L1: performance rises from 33.3% to 51.5% on an unseen enterprise platform.The result is consistent with transfer of general web interaction primitives across different layouts and navigation patterns.
  • Cross-benchmark performance: 41.5% vs. 41.5% on WebArena: the fine-tuned 9B model exactly matches the 3× larger Qwen3.5-27B.Performance remains competitive on VisualWebArena but trails on WorkArena++ L2, where longer compositional tasks benefit from larger capacity.
  • Cross-benchmark performance: +18.8pp on GitLab and +18.0pp on Shopping Admin are the largest per-site gains from fine-tuning.Smaller students also improve on WebArena, with gains of +11.1pp at 4B and +6.1pp at 2B, showing diminishing returns at 2B.
  • Qualitative behavior: 2 actions: A3-Qwen3.5-9B reaches the correct pending order directly, whereas the base model takes 10 actions and returns the wrong order.The example illustrates more efficient navigation on a Shopping Admin task.
  • Teacher quality: 69–85% across environments: Gemini 3 Pro with reduced thinking substantially outperforms Gemini 3 Flash at 17–53% in trajectory success.Flash nevertheless produces more observation-action pairs because failed trajectories tend to be longer, supporting trajectory quality over quantity.
  • Teacher quality: Lower reasoning budgets improve A3-SYNTH success rates across all six environments, with teacher success also higher under reduced thinking.The authors suggest concise traces or reduced overthinking may explain the reversal relative to typical test-time-compute scaling.
  • Ablations: 4.5pp and 7.9pp: removing Judge filtering and reasoning traces, respectively, reduces WebArena performance despite increased data in the filtering ablation.Matched-scale variants also show a 2.4pp advantage for persona-driven data over no-hints data.

6 Discussion

The results suggest that six environments support transfer across several generalization axes, while the study identifies limits in persona ablation, data scaling, validation, teacher coverage, and training method.

  • Generalization: +18.2pp, +7.5pp, +5.4pp, and +5.8pp: gains extend to unseen enterprise, longer-horizon enterprise, visual, and simplified HTML tasks.These results span WorkArena L1, WorkArena++ L2, VisualWebArena, and MiniWoB.
  • Limitations and scope: The persona module lacks a controlled no-persona comparison because regenerating exploration and task synthesis without personas was left for future work.This limits direct attribution of gains to the persona component.
  • Limitations and scope: Diminishing returns at 2,322 trajectories suggest that further scaling within the same six environments may require new task-generation strategies.The authors also note that Judge false-positive rates lack validation against human labels.
  • Limitations and scope: Teacher comparisons use only the Gemini family, and the model is trained with SFT only rather than combined reinforcement learning.Other teacher families and RL combinations are identified as future extensions.
  • Future directions: Broader environment coverage and iterative self-improvement are proposed as extensions to the depth-focused synthesis approach.These directions would test complementary coverage strategies and further student-driven improvement.

7 Conclusion

Structured, quality-filtered trajectory synthesis from one frontier teacher produces a competitive 9B open-weight web agent that transfers beyond its training environments.

  • Conclusion: 2,322 carefully filtered trajectories suffice to produce a 9B model that surpasses GPT-4o and Claude 3.5 Sonnet on WebArena.The model also transfers to enterprise, visual, and micro-task environments never seen during training.
  • Conclusion: Each pipeline module—Judge filtering, evaluation hints, and intact reasoning traces—contributes measurably to the outcome.The conclusion frames data quality as more important than quantity.
  • Conclusion: Reducing the teacher’s reasoning budget improves trajectory quality and generation cost, suggesting that capable distillation need not be expensive.The authors will release the dataset, pipeline code, checkpoint, and evaluation resources.

Reproducibility Statement

The study specifies its evaluation stack and planned releases to support reproduction of the data, model, and measurements.

  • Reproducibility: All experiments use BrowserGym and AgentLab for evaluation, with hyperparameters and training details specified in Section 4.2 and Appendix E.6.The release includes the full A3-SYNTH dataset, generation code, fine-tuned checkpoint, and evaluation scripts and configurations.

Ethics Statement

The paper identifies dual-use risks from deploying autonomous web agents and limits experiments to self-hosted or dedicated evaluation environments. It also discloses substantial use of Claude for writing, figures, code, and analysis while attributing research ideas and scientific claims to the authors.

  • Dual-use risk: Autonomous web-agent capabilities could support legitimate tasks but also enable spam, scraping, or interactions violating website terms of service.The paper frames this as a dual-use risk of lowering the barrier to deployment.
  • Risk mitigation: Experiments used only self-hosted WebArena and dedicated WorkArena and MiniWoB instances, so no production website was affected.The authors describe this as a partial mitigation and encourage safeguards before production deployment.
  • AI-assistance disclosure: Claude was used extensively for paper editing, figure and table generation, data-processing and evaluation code, and experimental analysis.The authors state that they originated and verified the research ideas, experimental design, and scientific claims.

B.2 Per-Benchmark Detailed Results

A3 fine-tuning improves performance across five benchmarks, including unseen enterprise, visual, and atomic web-interaction settings. Gains are strongest on WorkArena L1, while longer compositional WorkArena++ L2 tasks remain difficult and comparisons carry evaluation caveats.

  • Cross-benchmark overview: All five benchmarks show improvement after fine-tuning, with four evaluated out-of-distribution relative to WebArena-based training.Figure 4 visualizes improvements across all five benchmarks; MiniWoB is explicitly fully out-of-distribution.
  • VisualWebArena: +7.5pp on VisualWebArena raises full-benchmark success from 26.2% to 33.7%.The full evaluation contains 910 tasks spanning classifieds, shopping, and Reddit sites.
  • WorkArena L1: +18.2pp on WorkArena L1 raises success from 33.3% to 51.5% on unseen ServiceNow enterprise tasks.The benchmark covers 330 tasks involving record creation, filtering, sorting, ordering, charts, and navigation.
  • WorkArena++ L2: A3-Qwen3.5-9B improves WorkArena++ L2 from 2.2% to 9.7%, a 4.4× relative improvement, but compositional tasks remain challenging.L2 tasks require longer interaction horizons of up to 50 steps; dashboard tasks are 0% for all models.
  • MiniWoB: +5.8pp on MiniWoB reflects improved atomic web skills, especially Type and Workflow tasks.The A3 model also demonstrates more efficient navigation and correct HTML form-input conventions in qualitative examples.
  • Model scale: Distillation benefits persist across student scales: WebArena gains are +10.5pp for 9B, +11.1pp for 4B, and +6.1pp for 2B.The 9B fine-tuned model matches the 3× larger Qwen3.5-27B at 41.5% on WebArena.
  • Ecosystem comparisons: A3-Qwen3.5-9B reaches 41.5% on WebArena, exceeding GPT-4o by 10.0pp and Claude 3.5 Sonnet by 5.5pp under the same protocol.The result surpasses models 7–45× larger on WebArena, although published comparisons retain setup differences.
  • Published leaderboard comparisons: The model achieves 42.1% on the full 812-task WebArena benchmark, nearly doubling Go-Browse’s 21.7% open-weight SFT result.Go-Browse uses different student, teacher, and evaluation setups, so the comparison reflects multiple differences.

C.1 Judge Filtering Ablation

Judge filtering improves student performance despite reducing training-set size, while intact reasoning traces and sufficient data scale further strengthen WebArena results.

  • Judge filtering: The Judge acts as a quality gate that improves training-data signal-to-noise ratio rather than merely removing clearly failed trajectories.
  • Data scaling: WebArena performance rises from 32.0% at 285 trajectories to 41.5% at 2,322, with gains diminishing at larger data scales.The increments are +5.0pp, +3.2pp, and +1.3pp across successive data increases.
  • Reasoning traces: Removing reasoning traces lowers WebArena success from 41.5% to 33.6%, while truncating them to 500 or 250 characters performs even worse.Truncated-500 reaches 31.0% and Truncated-250 reaches 26.8%, suggesting incoherent cutoffs degrade the training signal.
  • Teacher thinking budget: A higher teacher thinking budget produces a 24.4% student result versus 41.5% for reduced thinking, despite traces not being longer.The authors hypothesize that more internal deliberation yields less reasoning in the visible trace available for student learning.
  • Evaluation hints: Removing evaluation hints reduces matched-size performance from 37.8% to 35.4%, a 2.4pp drop associated with improved Judge identification of false positives.Without hints, the Judge’s verdict flipped for 21.3% of re-judged trajectories, including 144 additional false positives caught with hints.

D Exploration Results

Exploration supplies the Task Generator with environment familiarity for persona-driven task synthesis, but exploration success varies substantially by model type.

  • Exploration process: Exploration gives the Task Generator site familiarity, with trajectories stored as context for generating tasks grounded in web content and functionality.An LLM agent navigates for up to 20 steps while assigned a persona.
  • Evaluation: Exploration success requires at least 10 navigation steps followed by the exact termination string, rather than measuring exploration quality directly.Downstream task quality implicitly reflects the quality of exploration.
  • Results: Proprietary models achieve 87–93% exploration success, compared with 70–75% for open-weight models across 1,500 tasks.Premature termination and failure to produce the exact termination string are the primary open-weight failure modes.
  • Task synthesis: Persona diversity broadens task coverage by combining varied user characteristics with the affordances of each web environment.The pipeline generates 250 personas and assigns each across six environments, producing persona-environment exploration pairs.
  • Task synthesis: Task Generator instructions require abstract, multi-action, and creative intents grounded in an assigned persona and exploration trajectory.The instructions are adapted from WebArena annotation guidelines and supplied in the generator’s system prompt.
  • Judging: The Judge asks the success question last after considering side effects, looping, and optimality, reducing confirmation bias in success judgments.

E.5 Training Data Statistics

A3-SYNTH provides diverse, multimodal supervised training data from six WebArena environments, evaluated alongside benchmarks spanning realistic, visual, enterprise, and atomic web tasks.

  • Data format: Each training example pairs a system role and action format with an observation containing the page state, task goal, and optional screenshot, followed by reasoning and action.Observations include the accessibility tree, URL, and optionally a screenshot; assistant outputs use structured reasoning tags.
  • Data statistics: Click and fill actions comprise 65.3% and 23.4% of the Pro training distribution, respectively.
  • Data statistics: All responses contain explicit <thought> reasoning, while <think> step-by-step reasoning appears in 81.4% of responses.The median lengths are 996 characters for <thought> and 459 characters for <think> when present.
  • Task coverage: Teacher success rates vary by site, reaching 85.4% on Wikipedia and 69.0% on Reddit.Reddit’s multi-step interactions are described as more failure-prone than Wikipedia’s editing and search tasks.
  • Evaluation relationship: A3-SYNTH uses the same six WebArena environments as testing but distinct intents, interaction sequences, and success criteria from the 381 human-authored test tasks.The environments, data, and user accounts are shared, while the specific goals remain novel.
  • Benchmarks: The evaluation suite includes realistic WebArena tasks, visually grounded VisualWebArena tasks, enterprise WorkArena tasks, and simplified MiniWoB interactions.
  • Evaluation setup: BrowserGym and AgentLab provide the unified environment interface and evaluation framework used across experiments.BrowserGym wraps each benchmark as an OpenAI Gymnasium environment.
Loading 2604.07776v1…