MMInA: Benchmarking Multihop Multimodal Internet Agents

1 Introduction

• Real-world web tasks require long-range planning and multimodal reasoning, while existing benchmarks and many agents focus on single-site or textual tasks.The benchmark tasks average 2.85 website hops, with the longest requiring 10 hops.Source: §1 · pp. 1–2, Figure 1 · p. 2
• MMInA offers 1,050 human-written tasks spanning multimodal inputs and multiple website hops on real-world, evolving websites.Tasks include activities such as e-commerce navigation, information extraction, and comparison across web sources.Source: §1 · p. 2
• GPT-4V achieves 21.8% success, below human performance of 96.3%, and agents often fail early on multihop tasks.The paper also reports that replaying past action trajectories significantly improves both single-hop and multihop performance.Source: §1 · p. 2
• The holistic evaluation protocol measures success at both the hop and overall task levels, providing more granular assessment than task success alone.Hop success rates support analysis of how agent behavior relates to hop length.Source: §1 · p. 3
• The proposed memory-augmented method uses past action histories to enhance agent performance.The method replays past action trajectories.Source: §1 · p. 3

2 Related Works

• Existing agent benchmarks commonly use curated textual interfaces, and the cited works do not evaluate LLMs or LMMs in web-browsing environments with a web-agent-specific metric.The section discusses tool-use benchmarks including BOLAA, as well as multimodal generalist benchmarks OpenAGI and GAIA.Source: §2 · p. 3
• Webshop, Mind2Web, CogAgent, SeeAct, and WebVoyager are among the cited web-agent efforts, while WebArena provides a standalone multicategory website environment.VisualWebArena builds on WebArena by introducing visual cues.Source: §2 · p. 3
• The section characterizes existing benchmark tasks as oversimplified because completion often requires only one website, unlike natural long-horizon browsing.Source: §2 · p. 3

3.1 Environment

• Browsing is modeled as a partially observable process in which agents receive page observations and histories, then take web actions or provide textual answers.Source: §3.1 · p. 3

3.2 Dataset Construction

• MMInA combines tasks spanning 1–10 hops, 2,989 hops, and 14 dynamic real-world websites, with manually crafted questions diversifying task style, scope, and content.Source: §3.2 · p. 4
• For dynamic websites, multihop success requires visiting all required websites in order; annotators record shortest successful paths as subsets of successful agent trajectories.Source: §3.2 · p. 5

3.3 Multimodal Web Content

• Tasks require agents to use both images and text across multiple turns, such as locating specified products and comparing their visual and textual information.Source: §3.3 · p. 5

3.4 Multihop Cross-website Browsing

• Multihop tasks move agents to the next website after each completed hop, modeling a sequence of actions used to address a high-level task.Source: §3.4 · p. 5

3.5 Evaluation

• Single-hop responses are scored by essential-keyword inclusion or GPT-3.5-Turbo fuzzy matching, while multihop tasks require every hop to be completed in sequence.Source: §3.5 · pp. 5–6

4.1 Baselines

• The evaluation compares LLM and LMM reasoning backbones, heuristic-based WebShop and CogAgent agents, and human baselines; human test takers consistently outperform web agents.Text-based models were evaluated with text-only or caption-augmented inputs, while multimodal models received image and text inputs.Source: §4.1 · p. 6

4.2 Main Results

• Current agents struggle on multihop tasks, and GPT-4V and Gemini-Pro-Vision fail on early hops more often as task hop counts increase.The authors report that performance drops sharply as the number of hops increases.Source: §4.2 · p. 6, Table 3 · p. 8
• Multimodal models generally perform better on hop and task metrics, while long-context models excel on structured webpage tasks and DeepSeek-R1 performs well on single-hop tasks.The authors also report degraded performance for DeepSeek-R1 on multihop tasks requiring longer context.Source: §4.2 · p. 7

4.3 Why are Multihop Web Tasks Challenging?

• In multihop tasks, agents may switch to alternate websites after failing instead of retrying the expected site, causing excessive exploration and lower task completion.Limited memory and missed hop termination conditions can also leave agents repeating actions or lingering in completed hops.Source: §4.3 · p. 7
• After aligning first hops semantically, the authors find first-hop success declines as total task hop count rises, attributing this to enlarged search spaces and weak zero-shot long-context reasoning.The authors describe multihop complexity as involving task-flow management beyond the sum of single-hop performances.Source: §4.3 · p. 8

4.4 Memory-augmented Agents

• The proposed agents use semantic, episodic, and procedural memory; replaying past action trajectories for similar tasks improves performance.Semantic memory stores general world knowledge, episodic memory holds ongoing-task trajectories, and procedural memory encodes completed task sequences and outcomes.Source: §4.4 · p. 8

5 Conclusion, Challenges, and Outlook

• MMInA benchmarks agents on 1,050 multimodal multihop tasks across 14 diverse websites and evaluates LLMs, LMMs, and human baselines.The benchmark uses real-world websites.Source: §5 · p. 9
• Its holistic evaluation assesses both task-level and hop-level success rates for multihop tasks.Source: §5 · p. 9
• The paper proposes a flexible memory-augmented approach intended to improve agents’ procedural memory and performance.Source: §5 · p. 9
• Future work will consider an evaluation method focused on actions to guide agent operations directly.Source: §5 · p. 9

• Web-page protections make direct image fetching from HTML exceptionally challenging, so the benchmark uses one offline standalone website and one open-source website.Source: §6 · p. 9

7 Ethical Considerations

• Bias in base multimodal models can produce inaccurate or unfair outcomes, making training-data representativeness relevant to avoiding biased behavior.Source: §7 · p. 9

A.1 Environment & Parameter Settings

• The evaluation used a 1280 × 2048 viewport and supplied webpage accessibility trees as text input, with default pretrained-model or API-model parameters.Source: §A.1 · p. 12

A.2 Computing Resources

• Inference for each MMInA epoch took 4–8 hours on one Nvidia RTX6000 Ada GPU with 48 GB memory.Source: §A.2 · p. 12

Hop Analysis

• In GPT-4V hop analysis, first-hop success declined as total hop count increased, while rates for tasks with more than seven hops fluctuated because there were fewer such tasks.Source: §A.3 · p. 12

Ablation of agents’ memory

• For simpler shopping or Wikipedia tasks, K=1 or 2 tended to yield better results, while larger histories offered diminishing returns and introduced decision-making biases and disturbances.Source: §A.3 · p. 12
• Procedural memory improved agents’ action prediction and execution, while replaying the last K task trajectories expands prompt length by K.Source: §A.3 · p. 12

B.1.1 Annotators

• Three trained annotators, varying in age and gender and proficient in web browsing, divided the dataset and cross-validated task diversity and answer accuracy.Source: §B.1.1 · p. 12

B.1.2 Anontation Protocals

• Final trajectories include all required website nodes, but agents may freely visit unnecessary websites before completing tasks.Source: §B.1.2 · p. 12

B.1.3 Data Statistics

• MMInA contains 1,050 tasks spanning travel and cooking, while agent actions generally increase with hop count, with 4-hop comparisons as an exception.The dataset also includes 108 QA pairs filtered from WebQA; Table A1 reports GPT-4V success rates by hop count.Source: §B.1.3 · pp. 12–13, Table A1 · p. 13

C More Related Works

• The reviewed multimodal datasets provide large collections of image-text pairs, which the paper identifies as fundamental to advances in multimodal AI.Examples include web-scraped pairs, billions of CLIP-filtered pairs, and multimodal instruction-response data.Source: C More Related Works · p. 13

B.2.1 Website Links

• All MMInA websites are evolving real-world sites except Shopping, and the evaluation protocol can support additional websites.Kiwix library updates may change Wikipedia URLs, but the authors say this does not affect their experiments.Source: Table A2 · p. 14, C More Related Works · p. 14

B.2.2 Comparision with other benchmarks

• MMInA requires multimodal inputs at multiple task steps, making it a more challenging multimodal benchmark than VWA.Source: Table A3 · p. 14

C More Related Works

• The review describes instruction tuning for LLMs and LMMs, with multimodal examples pairing instructions and multimodal inputs with ground-truth responses.It represents a sample as instruction, multimodal input, and response.Source: C More Related Works · p. 14
• Related work expands VQA instruction data by enriching short answers and generating additional examples from initial examples.The review describes both answer transformation and LLaVA’s use of GPT-4 to create data from image descriptions and bounding-box outlines.Source: C More Related Works · p. 15
• Web GUI-agent research spans HTML and screenshot-based interaction, including systems combining screenshots, action history, and textual website representations.The review also describes work that uses visual neighbors of webpage elements to inform representations.Source: C More Related Works · p. 15
• The mobile-agent review covers screenshot-based perception, OCR, self-planning, and agents that operate apps through actions such as tapping and swiping.Source: C More Related Works · pp. 15–16

C.1 Other Benchmarks

• Webshop uses simulated e-commerce with real-world products and crowdsourced textual instructions, but its single-website design limits task variety and does not evaluate open-ended real-world actions.Source: §C.1 · p. 16
• Mind2Web targets generalist web agents through an instruction-following dataset, while SeeAct adds visual understanding of rendered pages to generate action plans.Source: §C.1 · p. 16
• WebVoyager uses webpage screenshots and JavaScript tools to identify interactive elements, while WebArena and VisualWebArena provide interactive multi-category sites and visual cues.Source: §C.1 · p. 16
• The review characterizes existing benchmark tasks as single-website and oversimplified relative to natural long-horizon browsing across multiple websites.Source: §C.1 · p. 16