Source-linked AI summary
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan
TL;DR
Existing benchmarks do not jointly provide rich real-world language, interactive grounding, scalable data collection, and automatically computable feedback. WebShop addresses this gap with a simulated e-commerce environment and evaluates agents that navigate, search, inspect, customize, and purchase products. The best model achieves 28.7% task success, exceeding heuristics but remaining below human experts.
Problem
Existing interactive grounding benchmarks lack rich real-world language or are difficult to scale because data and feedback require substantial human involvement.
Method
WebShop simulates online shopping with real-world products, crowdsourced instructions, multiple webpage types, and diverse search and selection actions.
Results
The best agent achieves a 28.7% success rate, compared with 9.6% for heuristics and 59.6% for human experts.
Takeaways & Limitations
WebShop provides a benchmark for studying language grounding and sequential decision making in realistic web interactions, including sim-to-real transfer.
Takeaways & Limitations
Human performance estimates are affected by some crowdsource workers’ lack of patience and consistency, although the model–human gap remains significant.
Abstract
from arXiv · showhide
Existing benchmarks for grounding language in interactive environments either lack real-world linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. To bridge this gap, we develop WebShop -- a simulated e-commerce website environment with $1.18$ million real-world products and $12,087$ crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate multiple types of webpages and issue diverse actions to find, customize, and purchase an item. WebShop provides several challenges for language grounding including understanding compositional instructions, query (re-)formulation, comprehending and acting on noisy text in webpages, and performing strategic exploration. We collect over $1,600$ human demonstrations for the task, and train and evaluate a diverse range of agents using reinforcement learning, imitation learning, and pre-trained image and language models. Our best model achieves a task success rate of $29\%$, which outperforms rule-based heuristics ($9.6\%$) but is far lower than human expert performance ($59\%$). We also analyze agent and human trajectories and ablate various model components to provide insights for developing future agents with stronger language understanding and decision making abilities. Finally, we show that agents trained on WebShop exhibit non-trivial sim-to-real transfer when evaluated on amazon.com and ebay.com, indicating the potential value of WebShop in developing practical web-based agents that can operate in the wild.
1 Introduction
WebShop addresses the need for scalable interactive environments that combine realistic language with automatically computable feedback. It models online shopping as a multi-step language-grounding and decision-making task over a large product and instruction collection.
- Existing benchmarks often lack rich real-world language, interactive grounding, scalability, or automatically computable feedback.WebShop is motivated by combining these properties in one environment.
- The web offers a scalable, semantic, interactive, dynamic, and realistic environment with natural text, images, and interactive elements.
- WebShop requires agents to interpret instructions, search and compare products, inspect descriptions, select options, and purchase matching items.Agents may need multiple searches and backtracking between pages.
- WebShop contains over one million scraped products, over 12 thousand crowdsourced instructions, and a diverse action space involving text queries and buttons.
- The best agent reaches an average score of 62.4 and a 28.7% success rate, compared with 45.6 and 9.6% for heuristics and 82.1 and 59.6% for human experts.
2 Related Work
Prior web benchmarks cover navigation, webpage interaction, classification, retrieval, and question answering, but WebShop targets realistic, long-horizon shopping decisions with richer language and actions.
- WikiNav restricts reinforcement-learning agents to following hyperlinks or stopping, making it purely navigational.
- World of Bits and MiniWoB support webpage interaction using pixel or DOM observations and methods such as exploration and curriculum learning.
- Existing webpage datasets commonly address single-decision supervised tasks such as predicting elements, generating API calls, or semantic parsing.
- WebShop combines realistic webpage text and images with a rich interaction space for long-range sequential decision making.
- Web-based NLP systems mainly use search engines to retrieve evidence for information extraction, retrieval, question answering, or dialogue.
3 The WebShop Environment
WebShop is a large-scale interactive e-commerce environment in which agents interpret natural-language product instructions, search and navigate webpages, customize items, and purchase products. Its design combines realistic products and instructions with scalable annotations, deterministic interactions, and multiple language-grounding challenges.
- Environment: WebShop contains over 1.1 million real-world products and requires agents to find and purchase products matching natural-language specifications.The environment is built from 1,181,436 Amazon products and supports product-finding tasks based on natural-language instructions.
- Design goals: The environment separates website transitions from task-specific instructions and rewards, enabling extension to new tasks and domains.This modularity is intended to support interdisciplinary methods addressing multiple WebShop challenges simultaneously.
- State and actions: The environment models webpages as four page types: search, results, item, and item-detail pages.Products expose text fields, prices, options, images, and hidden attributes used for reward calculation.
- State and actions: Agents perform either text searches or button-selection actions, with search restricted to search pages and clicks used elsewhere.Click actions deterministically transition between webpage types rather than representing low-level mouse movements.
- Instructions and reward: Each instruction specifies attributes, buying options, and a maximum price, while the terminal reward scores attribute, option, price, and product-type alignment.Success is defined as receiving reward 1, although multiple products may satisfy a general instruction.
- Data and challenges: WebShop uses deterministic BM25 retrieval, scalable human-written instructions, and human trajectories to support reproducible training and evaluation.The benchmark also targets query generation and reformulation, strategic exploration, robust language understanding, and long-term memory.
4 Methods
The methods combine imitation learning and reinforcement learning with pretrained language and vision models for search generation and action selection. The resulting modular agents generate queries, score available actions, and can be refined through online reinforcement learning.
- Approach: WebShop agents combine pretrained language and image models with imitation learning and reinforcement learning.The method family includes both RL and IL agents and uses pretrained models for representing and generating text.
- Search imitation: The search model learns to generate search actions from instructions using 1,421 instruction-search pairs collected from human trajectories.Search is treated as a sequence-to-sequence generation problem using a fine-tuned BART model.
- Choice imitation: The choice model predicts a distribution over available click actions from observations and human-selected actions.It trains on 9,558 trajectory samples and uses BERT representations with cross-attention to score candidate actions.
- Multimodal architecture: The choice architecture combines visual and textual representations with action representations before producing scalar action logits.ResNet-50 processes images into features that are projected and concatenated with BERT observation representations.
- Interaction policy: During interaction, BART generates top-5 search queries while the BERT policy samples actions on other pages to encourage diverse behavior.The pipeline uses beam search for query generation and samples from the action distribution rather than always selecting the highest-probability action.
- RL fine-tuning: Online RL fine-tunes the choice model while freezing BART-generated top-10 search candidates and optimizing policy-gradient, value, and entropy losses.The entropy term is added to prevent premature convergence, and the total RL objective sums the three losses.
5 Experiments
Experiments show that imitation learning and reinforcement-learning finetuning substantially outperform the rule baseline, but remain well below human performance. Analyses identify language pretraining, option selection, exploration, memory, and search reformulation as important challenges, while zero-shot transfer demonstrates practical value on Amazon and eBay.
- Task performance: 10% success rate and 45.6 score are achieved by the rule baseline, while imitation learning reaches a score of 59.9.The rule baseline cannot resolve language-specified options or explore more products.
- Task performance: 62.4 score is reached after reinforcement-learning finetuning, while success rate slightly decreases from 29.1% to 28.7%.The best model remains below expert humans’ 59.6% success rate and the average human’s 50%.
- IL ablations: Nearly two-thirds of success rate is lost without language-pretrained choice initialization, whereas replacing learned search generation with a rule lowers reward and success rate by around 3 points.Choosing the right options is more critical than expanding the search space.
- RL ablations: Direct reinforcement-learning training from pre-trained BERT performs worse than the rule baseline, indicating that imitation-learning warm-starting is critical.An RNN-based RL model also has success rate more than 10% below the IL+RL model and shows greater variance.
- Human expert vs. agents: Humans outperform agents most strongly on option score, with a 28% gap, and use longer, more exploratory trajectories with more items and searches.Human examples also show query reformulation for abbreviated product terms and long-term memory when revisiting earlier products.
- Effect of RL fine-tuning after IL: RL finetuning reduces average trajectory length from 9.4 to 4.8 and lowers option score from 45.2 to 38.9 while improving attribute, type, and price scores.The analysis points to balancing exploration and exploitation during RL.
- Zero-shot sim-to-real transfer: On Amazon, IL+RL achieves a Score of 65.9 and SR of 25%, outperforming the Rule baseline’s Score of 45.8 and SR of 19% without fine-tuning.On eBay, IL+RL reaches Score 62.3 and SR 21%, versus the rule baseline’s Score 31.7 and SR 7%.
6 Discussion
WebShop is a scalable benchmark for sequential decision making and language grounding in e-commerce, with modularity for new tasks and domains. Its analyses identify challenges and directions for stronger web-agent language understanding and decision making.
- Benchmark: WebShop provides a large-scale e-commerce environment for evaluating sequential decision making and language grounding.The benchmark models interaction with an online shopping website.
- Results: Agents trained on WebShop show promising sim-to-real transfer to real-world shopping websites.
- Research challenges: Trajectory analyses identify challenges involving webpage semantics, query reformulation, long-horizon exploration, and memory.
- Extensions: WebShop’s modular design supports incorporating new web tasks and domains.
Checklist
The paper documents dataset construction, crowdsourced instruction collection, environment design, and benchmark analysis. It also reports the search-engine behavior that makes query reformulation important for task completion.
- Data construction: The dataset contains 1,181,436 products scraped from Amazon across five categories and 313 associated sub-categories.Products with duplicate titles or IDs were filtered, without extra filtering to avoid selection bias.
- Search engine: Directly searching the full instruction finds the corresponding item in ranks 1–10 nearly one-third of the time but misses it beyond rank 50 more than half the time.The result motivates query reformulation based on the instruction.
- Instruction collection: Workers wrote instructions from product titles, categories, attributes, and options after qualification tasks.The study qualified 213 workers and paid $0.15 per example.
- Human behavior: Human trajectory analysis contrasts impatient workers who commit after one search with experts who refine searches and select correct options.
A.7 Reward Verification
The reward verification study compares automatic and manually rescored human trajectories, finding close agreement while identifying systematic under-scoring from exact matching. The model architecture uses cross-attention to score actions from observations and action text.
- Reward verification: Average trajectories received automatic and manual scores of 74.9 and 76.3, respectively, with Pearson correlation 0.856.
- Reward verification: Expert trajectories received automatic and manual scores of 81.5 and 89.9, respectively, with Pearson correlation 0.773.
- Reward verification: 87.0% of average-worker automatic scores were within 10% of manual scores.Synonyms and lexically similar words were the main source of mismatch.
- Reward limitation: The reward function consistently under-scores selected products because it relies on exact matching criteria.The paper proposes recognizing synonyms within the context of a product and goal instruction.
- Model architecture: Cross-attention computes observation–action interactions, transforms action-token representations, and mean-pools them into an action score.The layer uses contextualized observation and action token embeddings.
C.1 IL Training Details
The supplementary training details describe computational settings for imitation and reinforcement learning, evaluation-time search sampling, and oracle-based analysis of search versus action choice.
- IL Training Details: Imitation-learning training uses a batch size of 1, 32 gradient-accumulation steps, learning rate 2 × 10^-5, and 10 epochs.Training takes around two hours on one RTX 2080 GPU.
- RL Training Details: Reinforcement-learning models train for 100,000 steps in four parallel environments with backpropagation through time every eight steps.
- RL Training Details: RL training takes around 27 hours for Transformer models on one RTX 3090 GPU and around 20 hours for GRU models on one RTX 2080 GPU.
- Analysis: The Choice oracle uses hidden rewards, attributes, and options to disentangle learning to search from choosing the right actions.
- Evaluation: Testing compares beam search with top-1 selection for both search and choice models, with the search model generating top-5 queries.Top-5 queries are sampled uniformly to increase search diversity.
C.4 Image Ablation
The image ablation evaluates an imitation-learning agent across multiple random-seed trials and examines transfer procedures from WebShop to real shopping websites. Removing images only slightly reduces performance because the current instructions and rewards are textual.
- Three random-seed trials compare the imitation-learning model with an image-ablated variant over 500 test cases.
- Removing images only slightly hurts overall performance but significantly reduces variance.
- The limited image effect is consistent with instructions and rewards that currently use textual information.
- The sim-to-real procedure translates real Amazon URLs into the model’s text observations and valid actions, then maps outputs back to Amazon URLs until a buy-now action.
- The implementation converts Amazon HTML into page information, WebShop HTML, text observations, and valid actions for the model.
D.2 Sim-to-real Transfer Results
WebShop-trained agents show transfer to real Amazon and eBay webpages, while transfer is stronger to Amazon because eBay differs more in product coverage and search behavior. The transferred exploratory behavior is retained despite shifts in products and search dynamics.
- Transfer results closely match WebShop’s reported numbers, supporting the environment’s promise for grounded agents on real web environments.
- Amazon transfer is better than eBay transfer because eBay has a larger product gap and a weaker search engine.
- An example IL-agent trajectory on Amazon records actions while marking options in red and attributes in blue.
- Exploratory behavior learned in WebShop is retained during transfer despite domain shifts in products and search-engine dynamics.
- The results indicate an opportunity for sim-to-real-trained agents to transfer to other real-world web tasks.
E Potential Societal Impacts and Limitations
The paper identifies representation, labeling, safety, and task-semantics limitations in WebShop. Its data and processing remain potentially biased, while real-world deployment requires controlled interfaces and richer instruction and reward semantics.
- Potential Societal Impacts and Limitations: WebShop may raise concerns about diversity, fairness, and representation despite being designed to reduce human data-collection effort.
- Diversity and representation in data collection: The product data is biased toward the USA and English and may represent only a subset of products users want to buy.
- Bias in data processing: Manual attribute labeling may reflect annotators’ experience, including uneven knowledge across product attributes.
- Safety for developing web agents: WebShop provides a controlled simulation for safer web-agent development, while preliminary real-site transfer restricts actions to fixed URL updates and holds form submissions inside the interface.
- Limitations in the current task: Current instructions are limited by simple or generic attributes and overly specific options, which may let agents use special options as product cues while ignoring other instruction content.
- Task procedure: The search interface returns up to 10 results per page and supports navigation to subsequent pages for further exploration.
- Task procedure: A product match requires the detail-page text to contain the instruction information and corresponding options to be selected when available.
- Task procedure: The task instructs users to refine queries and inspect multiple product pages rather than copying the full instruction or selecting the first result automatically.