Source-linked AI summary
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso
TL;DR
Multimodal web agents still struggle with unseen websites and domains, including enterprise-specific and proprietary platforms. AdaptAgent uses few-shot human demonstrations with in-context learning or meta-learning, and reports improved task success over non-adapted state-of-the-art agents on two benchmarks.
Problem
Current multimodal web agents struggle to generalize to unseen websites and domains, including proprietary workflows, while fine-tuning is costly.
Source: §1 · p. 2Method
Results
3.36% to 7.21% absolute task-success gains and 21.03% to 65.75% relative gains are reported on unseen websites and domains over current performance.
Source: §1 · p. 2Takeaways & Limitations
The results indicate few-shot demonstrations complement large-scale pre-training and fine-tuning, while gains from more examples tend to saturate.
Source: §6 · pp. 8–9The best-performing agent achieves less than 25% overall task success on both benchmarks, with room for improvement on long action sequences and complex visual layouts.
Source: §6 · p. 9
Abstract
from arXiv · showhide
State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.
1 Introduction
Multimodal web agents have advanced through prompting and large-scale training, but still face limited generalization to unseen tasks, websites, and domains. AdaptAgent uses few human demonstrations to adapt proprietary and open-weights agents, improving task success on unseen settings.
1 Introduction
- Web agents have progressed through prompting and extensive pre-training and fine-tuning, while multimodal agents also use GUI visuals alongside HTML/DOM information.The section frames generalization to new tasks, websites, and domains as important to broad applicability.Source: §1 · p. 1
- The paper targets unseen tasks and proprietary workflows because current agents have limited generalizability and fine-tuning is costly.It asks whether agents can adapt to unseen websites and domains using only one or two human demonstrations.Source: §1 · p. 2
- 3.36% to 7.21% absolute task-success gains, or 21.03% to 65.75% relative gains, are reported for AdaptAgent using one or two demonstrations on unseen websites and domains.The framework uses in-context learning for proprietary MLLMs and meta-learning for open-weights MLLMs.Source: §1 · p. 2
- The analyses find multimodal demonstrations outperform text-only ones, meta-learning data selection affects generalization, and additional examples bring higher costs and saturating gains.These analyses are presented as recommendations for researchers and practitioners.Source: §1 · p. 2
- The authors position few-shot demonstrations as a complementary direction to large-scale pre-training and fine-tuning for improving web-agent generalizability.This is the paper’s stated broader implication of the empirical findings.Source: §1 · p. 2
2 Related Work
Related work spans UI and web agents, few-shot learning with LLMs, and learning from demonstrations. The paper builds on these lines while focusing on adapting agents to unseen web settings with few examples.
UI/Web Agents
- Prior UI/web-agent work progressed from reinforcement-learning systems using language and GUI models to multimodal LLM agents that reason over user instructions and interfaces.The paper organizes related work into UI/web agents, few-shot learning with LLMs, and learning from demonstrations.Source: §2 · p. 2
- Many state-of-the-art web agents use pretrained LLMs with GUI images, prior actions, annotations, and HTML/DOM, while some train on HTML attention or next-action prediction.These training approaches can require substantial data and resources, may be limited to simpler environments, and may need human supervision during exploration.Source: §2 · p. 2
Few-Shot Learning with LLMs
- Few-shot LLM work includes in-context learning and preference-tuning methods, but preference-tuning approaches may not directly fit precise action prediction.The paper adopts model-agnostic meta-learning for quickly adapting web agents using demonstrations.Source: §2 · p. 3
Learning from Demonstrations
- Learning from demonstrations includes imitation learning and inverse reinforcement learning, with less exploration of its application to web agents than to robotics and autonomous systems.The paper explores demonstrations as a way to improve web-agent performance in unseen environments.Source: §2 · p. 3
3 Few-Shot Adaptation with Human Demonstrations
AdaptAgent addresses whether a few human demonstrations can help web agents generalize beyond the demonstrated task and website. It uses in-context examples for proprietary models and meta-learning for open-weights models.
3 Few-Shot Adaptation with Human Demonstrations
- The central challenge is whether demonstrations on one task and website can support related tasks on similar sites or even unrelated domains.The section contrasts this goal with the possibility that demonstrations only teach the demonstrated or closely related task, and notes that collecting many demonstrations is costly.Source: §3 · pp. 3–4
Methods for learning with human demonstrations
- AdaptAgent uses SeeAct with GPT-4o as its proprietary-model baseline and adapts it with multimodal in-context examples, while CogAgent is adapted through meta-learning.CogAgent is an 18B multimodal LLM with a visual backbone for GUI images.Source: §3 · p. 4
In-context learning with SeeAct + GPT-4o
- For SeeAct, a human demonstration is represented as visual snapshots, filtered HTML elements, and human selections, then added to the original prompt.The prompt guides GPT-4o to iteratively choose the next action from the current GUI state.Source: §3 · p. 4
Meta-learning with CogAgent
- For CogAgent, the method applies model-agnostic meta-learning, using task-specific fine-tuning followed by updates informed by held-out task losses.The paper uses FOMAML because computing full meta-gradients is challenging for LLMs with billions of parameters; its first-order approximation omits second-order gradients.Source: §3 · p. 4
4 Experimental Protocol and Details
The experiments assess adaptation across Mind2Web and VisualWebArena using settings for cross-task, cross-website, and cross-domain evaluation. Protocols differ for proprietary and open-weights models, and performance is compared with established baselines.
Datasets
- Mind2Web has 1,009 training tasks from 73 websites and test subsets for cross-task, cross-website, and cross-domain generalization; VisualWebArena contains 910 tasks across three websites.The authors amend Mind2Web’s overlapping cross-task evaluation set to evaluate adaptability properly.Source: §4 · p. 5
Experimental Protocol
- For GPT-4o, evaluation adds one relevant in-context demonstration from the target website or domain; CogAgent meta-training samples four tasks per Mind2Web website.The meta-trained CogAgent adapts to new websites or domains using two tasks, with VisualWebArena adaptation using two tasks from each website; results average five runs with different task selections.Source: §4 · p. 5
Baselines
- The baselines include zero-shot SeeAct + GPT-4o, SeeAct* with Set-of-Mark prompting, pretrained CogAgent, and conventional CogAgent fine-tuning variants.CogAgent-FT uses about 3.4 times as many examples as the data-equivalent CogAgent-FT (DE) and CogAgent-FOMAML settings.Source: §4 · p. 5
Evaluation metrics
- Mind2Web evaluation reports element accuracy, operation F1, step success rate, and overall task success rate, while VisualWebArena centers on overall success rate.VisualWebArena has human trajectories for 233 tasks, enabling granular metrics on that subset.Source: §4 · p. 6
5 Results
AdaptAgent improves task success on unseen websites and domains across Mind2Web and VisualWebArena using few-shot adaptation. Additional analyses find multimodal demonstrations outperform text-only ones, while meta-learning strategy and demonstration count affect performance.
Few-shot human demonstrations unlock complementary gains in agent’s performance
- Across unseen-site and unseen-domain settings, adaptation raised Mind2Web overall success rate by 4.18%–7.21% and VisualWebArena overall success rate by 3.36%–5.11%.These gains correspond to relative increases of 21.03%–45.40% on Mind2Web and 28.32%–65.75% on VisualWebArena over state-of-the-art approaches.Source: Table 1 · p. 6, §5 · pp. 6–7
More multimodal demonstrations help boost agent’s performance
- For SeeAct*, adding 1, 3, 5, or 10 in-context demonstrations slightly improved performance across settings, but gains were minimal as more examples were added.The analysis sampled 30 Mind2Web tasks across cross-task, cross-website, and cross-domain sets; longer prompts incur higher computational costs.Source: Figure 2 · p. 7, §5 · p. 8
Multimodal demonstrations are more effective than text-only demonstrations
- Multimodal in-context demonstrations improved performance over text-only versions by 0.95%–3.78% across settings in both benchmarks.The gains corresponded to relative increases of 4.29%–23.76%; multimodal examples include visual webpage snapshots.Source: Table 2 · p. 7, Figure 2 · p. 7, §5 · p. 7
Few-shot human demonstrations unlock complementary gains in agent’s performance
- CogAgent-FOMAML outperformed CogAgent-FT, trained on about 3.4× more examples, across tests except Mind2Web cross-task, where it beat the data-equivalent baseline.The exception was CogAgent-FT (DE), trained with data equivalent to CogAgent-FOMAML.Source: §5 · p. 7
Data selection strategies for meta-learning influence adaptability in different settings
- Intra-website meta-learning selections favored cross-website generalization, inter-website selections favored cross-domain generalization, and the hybrid strategy balanced both.The authors used the hybrid strategy for their main results.Source: Figure 2 · p. 7, Table 3 · p. 8, §5 · p. 8
6 Discussion and Limitations
AdaptAgent uses few-shot human demonstrations to adapt proprietary and open-weights web agents to unseen websites and domains. The authors report benefits over existing strategies alongside substantial room for improvement on both benchmarks.
6 Discussion and Limitations
- AdaptAgent adapts proprietary and open-weights multimodal web agents to unseen websites and domains using few-shot human demonstrations.The authors describe the framework as complementary to gains from larger pretrained models or fine-tuning on larger datasets.Source: §6 · pp. 8–9
- Meta-learning data-selection trade-offs can influence generalizability, while additional in-context multimodal examples boost proprietary agents with gains that tend to saturate at higher example counts.Source: §6 · p. 9
- The best-performing agent achieved less than 25% overall task success on both Mind2Web and VisualWebArena, with room to improve on long action sequences and complex visual layouts.Source: §6 · p. 9
Disclaimer
The disclaimer limits the document's purpose and disclaims responsibility for the information. It also excludes investment-related uses and solicitation.
Disclaimer
- The document is for informational purposes and disclaims representations, warranties, and liability concerning the information's completeness, accuracy, or reliability.Source: Disclaimer · p. 9
- The document is not investment research or advice, nor a recommendation, offer, or solicitation to buy or sell a financial product or security.Source: Disclaimer · p. 9
- It is not intended for evaluating transactions and does not constitute a solicitation where solicitation would be unlawful.Source: Disclaimer · p. 9
A.1.1 UI/Web Agents
Prior web-agent approaches use pretrained language models and GUI or HTML/DOM context, while some train models to predict actions. The section presents few-shot human demonstrations as a way to adapt agents to unseen settings without costly retraining.
A.1.1 UI/Web Agents
- Most state-of-the-art web agents use pretrained LLMs with GUI images, prior actions, annotations, and HTML/DOM context when available.Pix2Act and WebAgent train LLMs to attend to HTML or generate next actions, including through self-supervision and reinforcement learning.Source: §A.1.1 · p. 13
- Prior approaches often require substantial training resources and curated data, and their performance in unseen settings has been lacking.The section also notes that exploration-based training requires ongoing human supervision to avoid risky outcomes.Source: §A.1.1 · p. 13
- Few-shot human demonstrations are proposed to help GUI and web agents adapt efficiently to unseen settings without costly retraining.The approach is also intended to support complex workflows, proprietary settings, and custom information from human experts.Source: §A.1.1 · p. 13
A.1.2 Few-shot learning with LLMs
Few-shot learning seeks data-efficient adaptation to new tasks, but existing methods largely address preference tuning. The section motivates meta-learning for web agents that must predict concrete actions.
A.1.2 Few-shot learning with LLMs
- Few-shot alignment aims to adapt models with a limited number of examples, unlike data-hungry RLHF [38] and DPO [40].Source: §A.1.2 · pp. 13–14
- In-context learning [8] supports few-shot adaptation but can be tedious and sensitive to variations [27] [49].Source: §A.1.2 · p. 14
- GPO [61] and DITTO [50] show promise for few-shot preference tuning, but their fit for precise action prediction is unclear.The authors adopt model-agnostic meta-learning [16] to train web agents for quick adaptation.Source: §A.1.2 · p. 14
A.1.3 Learning from demonstrations
The appendix reviews learning from demonstrations and details the MAML adaptation procedure, benchmark settings, implementation, and difficulty-stratified results. It also shows how SeeAct’s prompt incorporates a multimodal demonstration.
A.1.3 Learning from demonstrations
- Learning from demonstrations can train agents by imitating actions or inferring objectives; the paper connects this approach to web agents’ perception, reasoning, and execution.Imitation learning maps demonstrations to actions, whereas inverse reinforcement learning learns a reward function; web agents share core capabilities with real-world task robots.Source: §A.1.3 · p. 14
A.2 First-order approximation of MAML for Multimodal Web Agents
- MAML adapts parameters on website tasks, then updates the initialization using held-out tasks from the same website; the first-order approximation drops second-order derivatives.The inner loop fine-tunes on web tasks, while gradients for the update are evaluated at the adapted parameters.Source: §A.2 · p. 15
A.3.1 Mind2Web
- Mind2Web separates familiar-site cross-task evaluation from unseen-site and unseen-domain tests, and amends its cross-task split to reduce train-test task overlap.The benchmark’s training set contains 1,009 tasks across 73 websites; the amendment moves low-similarity tasks into the cross-task set while preserving split sizes.Source: §A.3.1 · pp. 15–16
A.3.2 VisualWebArena
- VisualWebArena contains 910 tasks across three websites; 233 have human trajectories for step-level evaluation, while the rest use overall live-environment success.Source: §A.3.2 · p. 16
A.4 Implementation Details
- The CogAgent experiments use LoRA with rank 20 and learning rate 1e-5; meta-learning and adaptation use one gradient step per step of two sampled tasks.The meta-batch size is one website, and conventional fine-tuning runs for two epochs.Source: §A.4 · pp. 16–17
A.5 Results stratified by sequence and visual difficulty levels
- Adaptation improved success across sequence and visual difficulty levels; 1-shot multimodal SeeAct* performed best, with especially strong hard-sequence gains in VisualWebArena.For hard-sequence tasks, SeeAct* rose from 1.7% to 5.2% in human-trajectory evaluation and from 0.9% to 5.9% in live-environment evaluation; CogAgent’s hard-task gains were minimal, especially in VisualWebArena.Source: §A.5 · p. 17, Table 4 · p. 17
A.6 1-ICMD Prompt for SeeAct and SeeAct*
- SeeAct’s prompt adds a multimodal in-context example that pairs a task description with GUI snapshots and action records.The example includes HTML elements and human selections alongside the visual snapshots.Source: §A.6 · pp. 17–18