Source-linked AI summary
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing Gong
TL;DR
Existing web-agent benchmarks evaluate digital interaction without egocentric physical-world grounding. Ego2Web couples first-person videos with online tasks and introduces Ego2WebJudge; agents show substantial room for improvement, while the judge reaches 84.0% agreement with GPT-4o human evaluation.
Problem
Existing web-agent benchmarks focus on web-page understanding and interaction, providing no setting where egocentric visual perception informs online actions.
Method
Ego2Web pairs real-world first-person videos with downstream web tasks, using automatic generation, human verification, and refinement, while Ego2WebJudge provides multimodal LLM-based evaluation.
Results
Agents show clear room for improvement in visual grounding, reasoning, and perception–action integration; Ego2WebJudge achieves 84.0% agreement with human judgments using GPT-4o.
Takeaways & Limitations
The benchmark provides a realistic, scalable setting for testing multimodal agents that connect egocentric perception with online web actions.
Abstract
from arXiv · showhide
Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
1. Introduction
Ego2Web addresses the missing connection between egocentric visual perception and online web action by pairing first-person videos with visually grounded web tasks. It introduces a human-verified benchmark and evaluation framework, while experiments reveal substantial room for agent improvement.
- Motivation: Existing web-agent benchmarks evaluate digital interaction without measuring whether users’ real-world visual surroundings inform online actions.This gap matters for workflows such as identifying an object physically and purchasing it online.
- Benchmark: Ego2Web couples egocentric videos with downstream web tasks requiring agents to ground visual cues and connect them to symbolic web concepts.The benchmark spans e-commerce, media retrieval, knowledge lookup, and local/maps services.
- Data construction: The benchmark is constructed through automatic task synthesis followed by human verification and refinement for visual grounding, web feasibility, and instruction quality.Video profiles are generated from structured clip-level captions before an LLM produces web instructions.
- Evaluation: Ego2WebJudge evaluates live web trajectories by comparing instructions, actions, screenshots, final responses, and annotated video evidence.This makes evaluation sensitive to both web execution and the visual grounding required by the task.
- Results: About 40% gap from oracle performance remains across evaluated agents according to human evaluation, indicating substantial room for improvement.The reported gap is observed across all agents evaluated on Ego2Web.
- Contributions: Ego2Web introduces 500 video–instruction pairs and reveals limitations in agents’ visual grounding, reasoning, and perception–action alignment.The benchmark is intended to support development of multimodal agents operating across physical and digital contexts.
2. Related Works
Prior work advances egocentric video reasoning and interactive web agents separately, but existing benchmarks do not connect first-person visual understanding to online web decision-making.
- Egocentric video understanding: Egocentric video benchmarks study human-centric activities, intentions, object interactions, temporal order, causal relations, and higher-level video-language reasoning.Examples include Ego4D, EPIC-KITCHENS, EgoExo4D, EgoSchema, EgoThink, and EgoPlan.
- Egocentric video understanding: Existing egocentric video benchmarks evaluate perception and reasoning in isolation without connecting video understanding to online web decisions.This distinguishes Ego2Web from prior first-person video datasets.
- Ego2Web’s position: Ego2Web combines real-world egocentric video with executable web tasks, creating a testbed for multimodal agents grounded in physical-world perception.Its distinguishing feature is the connection between first-person visual understanding and online action.
- Web agents: Web-agent benchmarks have progressed from sandboxed browser interaction toward richer visual inputs and interactive online environments.WebArena, VisualWebArena, and VideoWebArena represent stages in this progression.
3. Ego2Web: From Video Perception to Web Reasoning and Actions
Ego2Web defines tasks that transform first-person video evidence into web actions and constructs them through a semi-automatic MLLM–LLM pipeline with human quality control.
- Task definition: Given an egocentric video and instruction, an agent must execute browser actions that reach a specified goal state.The task tests both visual perceptual understanding and web execution reasoning.
- Video processing: The dataset pipeline converts curated first-person videos into structured clip-level visual metadata and video profiles.The videos cover contexts including household, shopping, travel, and office scenes.
- Task synthesis: An LLM conditions on visual metadata and selected active websites to synthesize instructions linking recognized physical content to digital actions.Examples include recognizing a brand or matching clothing color and logo on a shopping website.
- Dataset coverage: The benchmark covers diverse high-level task types and website domains, including e-commerce, media retrieval, knowledge bases, and local/map services.The distribution is summarized across major platforms and fine-grained website domains.
- Quality control: Human annotators retain samples only after checking visual grounding, web feasibility, and instruction quality.They may edit both generated instructions and visual-cue annotations.
4. Ego2WebJudge: Automatic Online Evaluation Grounded in Visual Cues
Ego2WebJudge extends online web-agent evaluation with annotated egocentric visual evidence, enabling success judgments that account for both web outcomes and physical-world content. Its pipeline extracts task key points, selects relevant screenshots, and compares the resulting trajectory with the video evidence.
- Motivation: Prior automatic evaluators rely on screenshots and action trajectories, leaving evaluation blind to task-relevant information in egocentric videos.Ego2WebJudge is introduced to address this missing visual grounding.
- Framework: Ego2WebJudge takes the instruction, action sequence, screenshots, and annotated visual evidence as inputs and classifies the outcome as success or failure.The judge checks whether the final web result satisfies task key points and matches objects, scenes, or brands in the video.
- Task examples: The benchmark’s examples pair sampled egocentric frames with web instructions and mark essential visual cues such as a mint toaster or black backpack.These annotations identify the physical evidence that downstream web actions must use.
- Evaluation stages: The evaluation pipeline identifies critical instruction key points before selecting screenshots whose visual content is relevant to the task.Key-point extraction supports multi-step instructions, while relevance filtering reduces irrelevant trajectory context.
- Evaluation stages: Relevant screenshots, action history, generated keypoints, and annotated video keyframes are integrated for final outcome judgment.The comparison determines whether the web result is visually consistent with the real-world content observed in the video.
5. Experiments
Experiments show substantial limitations in current web agents, with performance depending strongly on visual-input fidelity and evaluation quality. Ego2WebJudge closely tracks human judgments while raw video substantially outperforms caption-only perception.
- Experimental Setup: The experiments evaluate six mainstream web agents using human evaluation and Ego2WebJudge with multiple multimodal LLM judges.The setup includes live online evaluation and ablations of visual perception.
- Main Results: 58.6% human-evaluated SR makes BU-Gemini-3-Flash the strongest agent, with 57.2% under Qwen3-VL-Flash and 48.2% under Gemini-2.5 Pro.It outperforms the other evaluated agents under every automatic judge.
- Domain Analysis: The strongest agent reaches 38.2% in E-Commerce, 50.7% in Media Retrieval, 75.0% in Knowledge Lookup, and 48.3% in Local/Maps.Knowledge Lookup is easiest on average, while Local/Maps and E-Commerce are more challenging.
- Evaluation: 84.0% agreement rate with GPT-4o and 80.8% with Gemini-2.5 Pro make Ego2WebJudge more aligned with humans than WebVoyager and WebJudge.The reported comparison is 84.0% versus 74.7% for WebVoyager and 78.4% for WebJudge with GPT-4o.
- Ablation Studies: Raw video raises overall SR to 48.2%, compared with 23.6% for detailed captions and 4.4% without visual input.The hierarchy holds across domains, with especially large gains in Knowledge Lookup and Local/Maps.
- Error Analysis: Manual inspection identifies object misidentification, temporal misunderstanding, cross-modal retrieval failure, and other constraints as recurring errors.The sampled failures include 36% object misidentification and 18% temporal and action misunderstanding.
6. Conclusion
Ego2Web connects egocentric visual perception with online web-agent execution through human-verified video–instruction pairs and live evaluation. Its experiments expose room for improvement in visual grounding, reasoning, and perception–action integration.
- Conclusion: Ego2Web combines 500 automatically generated and human-verified video–instruction pairs with live online web evaluation.The benchmark is designed to connect real-world understanding with online web tasks.
- Conclusion: Ego2WebJudge uses grounded visual cues to assess task success and achieves high agreement with human evaluation.The framework is intended to support consistent and efficient assessment.
- Conclusion: Experiments reveal clear room for improvement in agents’ visual grounding, reasoning, and perception–action integration.The conclusion frames these as limitations observed across leading multimodal agents.
A. More Statistics of Ego2Web
Ego2Web builds its video data from standardized EgoSchema clips derived from Ego4D and distributes tasks across five web categories and 18 widely used websites.
- Video Statistics: Ego4D provides over 3,600 hours of first-person video with dense timestamped narrations covering thousands of actions and objects.These narrations provide fine-grained grounding for video reasoning.
- Video Statistics: EgoSchema filters Ego4D into non-overlapping three-minute clips containing at least 30 human-annotated narrations.The filtering targets consistent duration and rich temporal supervision.
- Web Distribution: The benchmark contains 230 E-Commerce, 132 Media Retrieval, 92 Knowledge Lookup, 31 Local/Maps, and 15 Other tasks across 18 websites.E-commerce and media platforms dominate the distribution.
B. Implementation Details
The implementation generates structured video descriptions and video-dependent web instructions, then evaluates agents against strict visual-grounding criteria. Prompts require explicit visual anchors, verifiable goals, and evidence-based success judgments.
- Video Captioning: The video-captioning prompt requires global scene context and local object descriptions with distinguishing attributes, actions, and temporal progression.Structured output includes a video description and object entries.
- Task Generation: Candidate videos must contain clear visual cues that naturally support a web task and cannot be solved using captions alone.Suitability requires objects, activities, or scenes linked to an online information or action need.
- Task Generation: Generated instructions must depend on at least two visual cues, specify a Must Match list, and define a clear, verifiable goal.Instructions also include timestamps, allowed domains, and why the task is video-dependent.
- Automatic Evaluation: Ego2WebJudge receives video evidence, task instructions, completion key points, action history, and relevant webpage snapshots before making a binary success judgment.The evaluator checks whether the web result is correctly grounded in the egocentric evidence.
- Evaluation Criteria: The evaluation protocol marks uncertain, ambiguous, mismatched, or visually unsupported results as failures rather than assuming correctness.It rejects reliance on textual claims, webpage titles, search queries, or approximate matches.
- Evaluation Criteria: Required filters, sorting operations, and numerical constraints must be visibly and exactly satisfied for success.Examples include exact price ranges, years, ratings, and bed counts.
- Evaluation Criteria: Visual grounding checks object identity, category, brand, color, quantity, state, material, shape, text, spatial relations, and actions.Similar-looking objects, related events, unsupported claims, and partial matches are treated as failures.
- Implementation Details: Agents without raw-video access receive detailed timestamped captions instead, including examples generated for Claude and GPT-5.4.The caption examples describe scene context, temporal progression, and local interactions.