PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, Jie Tang

arXiv:2408.06327v1cs.AIcs.CLcs.CV

TL;DR

Existing benchmarks do not comprehensively evaluate LMMs as visual foundation agents across realistic, multitask environments. VAB addresses this with a diverse interactive benchmark and hybrid trajectory training set, showing substantial but incomplete progress: behavior cloning improves open models, while proprietary models still retain an advantage.

  • Problem

    Existing LMM benchmarks insufficiently test high-level reasoning and planning across diverse realistic visual-agent environments.

    Source: §1 · p. 2

  • Method

    VAB benchmarks and trains visual agents across embodied, GUI, and visual-design scenarios using interactive evaluation and 4,482 hybrid-curated trajectories.

    Source: §1 · pp. 2–3

  • Results

    Behavior cloning substantially improves open LMMs, while top proprietary models solve 36.2% of challenging problems with prompting and remain ahead of fine-tuned open models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VAB provides a foundation for comparing and improving visual agents across multiple domains, with behavior cloning as a useful first step toward stronger agents.

    Source: §1 · p. 3, §8 · p. 12

    Fine-tuned open LMMs still perform below proprietary models, and some VAB design principles may become obsolete as the field advances.

    Source: §2 · p. 4, §8 · p. 12

Abstract

from arXiv · show

Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents. These agents are postulated to excel across a myriad of tasks, potentially approaching general artificial intelligence. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across nine proprietary LMM APIs and eight open models, we demonstrate the considerable yet still developing agent capabilities of these models. Additionally, VAB constructs a trajectory training set constructed through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, promoting substantial performance improvements in LMMs through behavior cloning. Our work not only aims to benchmark existing models but also provides a solid foundation for future development into visual foundation agents. Code, train \& test data, and part of fine-tuned open LMMs are available at \url{https://github.com/THUDM/VisualAgentBench}.

1 Introduction

Existing LMM evaluations rarely test high-level reasoning and planning across realistic visual-agent environments. VAB addresses this gap with a multitask benchmark, interactive evaluation, and trajectory training data, while showing strong but still limited model capabilities.

Research gap

  • Existing evaluations focus mainly on VQA, OCR, exams, or isolated environments, leaving multitask visual-agent reasoning and planning insufficiently measured.Prompting-only evaluations also underserve open LMMs with limited instruction-following capabilities.Source: §1 · p. 2

Contributions

  • VAB introduces a systematic benchmark spanning embodied, GUI, and visual-design tasks for training and evaluating visual foundation agents.It includes five datasets across three scenarios and standardized prompting and data formats.Source: §1 · pp. 2–3

Evaluation

  • Interactive evaluation measures agents while they engage directly with environments, making VAB more realistic than benchmarks based only on offline trajectories.Task performance is measured with environment-specific judge functions.Source: §1 · pp. 2–3

Results

  • 36.2% of challenging problems are solved by gpt-4o with prompting, although proprietary models remain far from practical deployability.This result reflects evaluation across nine proprietary APIs and eight open LMMs.Source: §1 · p. 3

Training

  • Behavior cloning on 4,482 hybrid-curated trajectories substantially improves open LMMs, with most surpassing gemini-1.0-pro and qwen-vl-max while approaching gemini-1.5-pro.The trajectories combine Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations.Source: §1 · p. 3

2 Problem Formulation and VAB Design Features

VAB formulates visual foundation agents as multimodal interactive decision-makers and designs evaluation around vision-centric perception, high-level decisions, interaction, and trainable trajectories. Its principles are intended to accommodate evolving LMM capabilities.

Problem formulation

  • In the POMDP formulation, visual observations extend LLM-agent settings while requiring LMMs to reconcile multimodal understanding with high-level reasoning.The formulation includes state, action, transition, reward, instruction, and observation spaces.Source: §2 · p. 3

Design Features

  • VAB emphasizes vision-centric tasks and simplifies low-level control so LMMs can be evaluated primarily on high-level planning and decision sequences.This design reflects LMM strengths in commonsense, knowledge, and instruction following.Source: §2 · p. 3
  • Interactive evaluation accommodates tasks with multiple valid ways to achieve their goals.VAB follows the execution-based evaluation practice used for real-world agent tasks.Source: §2 · p. 3
  • Hybrid trajectory construction provides instructions, trajectories, and reward functions for behavior cloning, which improves open LMM performance on VAB.Creating such datasets is costly because it requires valid environment-specific demonstrations and reward functions.Source: §2 · p. 4

Limitations

  • The design principles may become obsolete as LMM capabilities advance, motivating continued VAB updates.This is an explicit scope boundary of the benchmark design.Source: §2 · p. 4

3 VISUALAGENTBENCH: Tasks and Environments

VAB covers embodied, GUI, and visual-design agents through diverse interactive environments and tasks. Its environments target household and open-world action, interface interaction, and fine-grained visual reasoning.

3 VISUALAGENTBENCH: Tasks and Environments

  • VAB spans household and game environments, mobile and web GUIs, and frontend CSS debugging across five tasks.The benchmark combines adapted existing resources with newly created tasks and interactive environments.Source: §3 · p. 4

3.1 Embodied Agent

  • VAB-OmniGibson evaluates household agents in a high-fidelity simulator using high-level actions and task success rate.Tasks include placing objects into specified locations amid diverse scenes and realistic physical effects.Source: §3.1 · p. 4
  • VAB-Minecraft tests generalized embodied abilities through tasks such as item collection and hostile-mob combat, scored by task success rate.Agents use provided scripts or the Steve-1 low-level controller.Source: §3.1 · p. 4

3.2 GUI Agent

  • The GUI suite evaluates practical interaction in Android and web environments, where agents must interpret information-dense interfaces and layouts.VAB-Mobile uses reproducible Android Virtual Device tasks, while WebArena-Lite adapts 165 tasks across five websites for multimodal evaluation.Source: §3.2 · p. 5

3.3 Visual Design Agent

  • VAB-CSS evaluates iterative frontend style editing, requiring fine-grained visual grounding and reasoning across rendering outcomes.Success rate measures whether the final rendering matches the target design.Source: §3.3 · p. 5

4 Methodology for VAB Data Collection

VAB uses a two-stage process to create executable, diverse task instances, then curates training trajectories with environment-specific hybrid strategies. This methodology spans five datasets and balances cost, adaptability, versatility, and flexibility across trajectory sources.

4.1 Task Instance Collection: Prototyping and Instantiation

  • VAB first prototypes high-level goals, then instantiates them with concrete environmental items and conditions while assigning judging functions and rephrasing instructions for expression diversity.This process is intended to ensure task quality, feasibility, and executability.Source: §4 · p. 5, §4.1 · pp. 5–6
  • The five datasets use environment-specific instance construction, including randomized scenes, game configurations, application groundings, web items, and CSS corruptions.The resulting collections include 992 OmniGibson instances, 628 Minecraft instances, 1,186 WebArena-Lite training instances, and 1,210 CSS instances; Mobile contributes 119 test instructions and about 70 training prototypes.Source: §4.1 · p. 6

4.2 Training Trajectory Collection: 3-Leveled Strategies

  • VAB curates training trajectories with three strategies: program-based solvers, LMM agent bootstrapping, and human demonstrations.Programs automate prototype-specific tasks, bootstrapping uses prompted LMM agents with optional memory, and humans annotate scenarios requiring inaccessible or authenticated environments.Source: §4.2 · p. 7
  • Strategy selection trades off average cost, adaptability, versatility, and flexibility, with programs cheapest, bootstrapping most adaptable and flexible, and humans most versatile.These dimensions determine whether a method can scale, fit an environment, cover diverse tasks, or support trial-and-error trajectories.Source: §4.2 · p. 7
  • VAB combines strategies by environment: programs for OmniGibson, bootstrapping for Minecraft, and primarily human demonstrations with some bootstrapping for Mobile.The choices reflect hardware constraints, random exploration, authentication requirements, interface accessibility, and desired trajectory flexibility.Source: §4.2 · pp. 7–8
  • The methodology supplies dataset statistics and benchmark-comparison information alongside interactive evaluation and training-trajectory resources.The cited tables summarize VAB’s datasets and compare test instances, training trajectories, reinforcement-learning setups, and average trajectory turns across benchmarks.Source: Table 2 · p. 8, Table 3 · p. 8
  • WebArena-Lite uses program-based solvers because human annotation is inefficient on its interfaces and prompted LMMs achieve less than 20% success.CSS uses bootstrapping for flexible trial-and-error debugging, while gpt-4o initially resolves fewer than 40% of tasks and receives target-rule hints after five failed steps.Source: §4.2 · p. 8

5 Baseline Experiment

VAB evaluates visual foundation agents through standardized interactive prompting and multitask fine-tuning across diverse environments. Existing models remain substantially challenged, while trajectory supervision markedly improves open LMM performance but does not close the proprietary-model gap.

5.1 Setup

  • VAB evaluates proprietary APIs with prompting and open LMMs with multitask fine-tuning across its benchmark environments.Table 4 reports success rate as the proportion of successful tasks, while open models were fine-tuned because they could not reliably follow VAB system prompts.Source: Table 4 · p. 9, §5.1 · p. 9, §5.2 · p. 10
  • The evaluation uses multi-round agent interactions, retaining only the latest image for Embodied and GUI tasks but preserving image history for CSS.This design reflects current APIs’ limited multi-image support and the importance of comparing visual inputs in CSS.Source: §5.1 · p. 9

5.2 Main Results

  • 36.2% overall success rate is achieved by gpt-4o, yet proprietary LMM performance remains far from practical deployment on VAB.Most proprietary models exceed 20% with prompting, while claude-3.5-sonnet trails gpt-4o and gpt-4o-mini is inferior to gpt-4o.Source: §5.2 · p. 10
  • Trajectory SFT substantially improves open LMM agents, with InternVL-2 outperforming gemini-1.0-pro across evaluated environments and claude-3-opus on CSS.Open LMMs rarely follow VAB system prompts without fine-tuning and can reach 0% success in preliminary trials.Source: §5.2 · p. 10
  • The proprietary–open gap remains large after trajectory SFT, but larger backbones and more high-quality trajectories may narrow or reverse it.The paper identifies VAB as a testbed for practical visual-agent performance while noting compute limits restricted testing of larger open backbones.Source: §5.2 · p. 10

6 Analysis

VAB analyzes visual grounding and planning as central challenges for multimodal agents. Results show dependence on explicit grounding aids, comparable direct-action and ReAct performance, and benefits from training on error-recovery trajectories.

6.1 Visual Grounding Analysis

  • Object labels significantly improve embodied-agent performance, revealing grounding failures despite strong image captioning and object-recognition abilities.Figure 4 compares VAB-OmniGibson results with and without annotated object labels.Source: §6.1 · p. 10, Figure 4 · p. 11
  • CogVLM2 struggles to output GUI bounding boxes directly, making Set-of-Marks instrumental for visual grounding.Figure 5 compares SoM and REC for GUI agents trained on CogVLM2.Source: §6.1 · p. 10, Figure 5 · p. 11
  • VAB-CSS uses a relatively lenient visual-difference setting by supplying natural-language adjustment descriptions instead of requiring direct screenshot comparison.Table 5 reports that removing the description sharply reduces performance.Source: §6.1 · p. 11, Table 5 · p. 12

6.2 Performance on Planning

  • Directly generating an action can match or exceed ReAct when gpt-4o and claude-3.5-sonnet serve as agent backbones.Table 6 compares direct action generation with the ReAct framework.Source: §6.2 · p. 11, Table 6 · p. 12
  • Training with error-recovery scenarios improves fine-tuned open LMMs, while gpt-4o recovers robustly except on often-irreversible GUI tasks.Figure 6 compares overall success with success when incorrect actions occur across tasks.Source: §6.2 · p. 11, Figure 6 · p. 11

7 Related Work

Related work establishes LMM-based visual agents across embodied, GUI, and visual-design settings while highlighting challenges in grounding, instruction following, and long-term planning. Existing benchmarks remain narrow or prompting-only, motivating VAB’s cross-domain training and evaluation design.

LMM-as-Visual-Agent

  • LMM-based visual agents target embodied, GUI, and visual-design tasks but still face grounding, instruction-following, and long-term-planning challenges.Prior systems commonly use task-specific training or reinforcement learning because general-purpose LMMs lack strong zero-shot capabilities.Source: §7 · p. 12

Benchmarking LMM-based visual agents

  • Existing benchmarks cover individual web, mobile, desktop, game, or household domains but rarely provide comprehensive cross-application evaluation.Many also focus only on prompting, whereas VAB adds training data for open-source foundation LMMs.Source: §7 · p. 12

8 Conclusion

VAB-OmniGibson combines realistic household environments, high-level embodied planning, and curated trajectories for training and evaluation. Its action space and task construction support solvable, multi-step activities while behavior cloning remains below proprietary-model performance.

8 Conclusion

  • Open LMMs fine-tuned on the provided trajectories still perform below proprietary LMMs such as gpt-4o.The authors identify behavior cloning as a first step and future reinforcement learning in interactive environments as a direction.Source: §8 · p. 12

A VAB-OmniGibson

  • VAB-OmniGibson uses OmniGibson to evaluate LMMs as high-level planners for realistic household activities.The agent receives egocentric images, annotated objects, instructions, and environment feedback, then selects low-level actions.Source: §A.1 · p. 19

A.1 Detailed Description

  • 181 test instances cover 45 manually adapted household activities selected for embodied planning evaluation.Activities are instantiated across several scenes and judged by whether all goal conditions are met within 100 turns.Source: §A.1 · p. 19
  • 872 training trajectories combine rule-based solving, error-recovery augmentation, and LMM bootstrapping across 901 task instances.The rule-based solver decomposes long-horizon activities into subtasks, while added failures train recovery from placing objects into closed containers.Source: §A.1 · p. 19

A.2 Actions

  • Twenty low-level actions let agents grasp, navigate, manipulate, and operate household objects, including opening, closing, and appliance controls.The action set is designed so all testing instances are solvable.Source: §A.2 · pp. 19–20

A.3 Rule-based Solver for Training Trajectory Collection

This section describes rule-based and prompt-guided procedures for collecting training trajectories and evaluating visual agents across VAB-OmniGibson, VAB-Minecraft, and VAB-Mobile. It covers goal decomposition, interaction protocols, benchmark pipelines, action spaces, and evaluation settings.

A.3 Rule-based Solver for Training Trajectory Collection

  • The VAB-OmniGibson rule-based solver decomposes each BDDL goal into state-identification or object-position subgoals and fulfills them sequentially through navigation, manipulation, and state-changing actions.State goals require locating and modifying an object, while positional goals require finding, grasping, moving, and placing it relative to another object.Source: §A.3 · p. 20

A.4 Prompt Example

  • The VAB-OmniGibson prompt requires exactly one predefined action per turn, with observations, thoughts, and an action code constrained by visibility, reachability, room navigation, and failure recovery reminders.The interface supplies task goals, reachable rooms, action feedback, held-object state, current room, and vision inputs with boxed object identifiers.Source: §A.4 · pp. 21, 23, 24

B VAB-Minecraft

  • VAB-Minecraft adapts the JARVIS-1 pipeline [65] into a standardized high-level planning benchmark for LMMs, using item-obtaining tasks that require task decomposition and ingredient collection.The benchmark targets generalized embodied planning in Minecraft’s diverse open-world environment.Source: B VAB-Minecraft · p. 24, §B.1 · p. 24

B.1 Detailed Description

  • VAB-Minecraft contains 116 manually annotated solvable test tasks spanning six material levels and resources from 11 plant types, 4 animal types, and 6 hostile-mob types; success means obtaining the target item within 100 rounds.Training uses 512 newly designed instances yielding 176 successful GPT-4-turbo trajectories and 596 JARVIS-1 instances yielding 206 successful GPT-4o trajectories, for 382 total.Source: §B.1 · p. 25

B.2 Actions

  • The Minecraft action space has six actions: craft, smelt, equip, teleport_to_spawn, look_up, and execute, with four adapted from JARVIS-1 [65] and two newly implemented.teleport_to_spawn supports ingredient collection and recovery, look_up provides crafting or smelting guidance, and execute delegates basic tasks to Steve-1 [30].Source: §B.2 · p. 25

B.3 Prompt Example

  • The VAB-Minecraft prompt exposes one predefined function per turn for survival-mode item acquisition, while examples specify task goals, initial inventory and resources, action feedback, and environmental context.The cake example initializes buckets, logs, mature wheat, sugar cane, chickens, and cows so the necessary ingredients are conveniently available.Source: §B.3 · p. 25, Figure 8 · p. 26
  • B.3 Prompt Example: The rule-based solver uses look_up for item knowledge and execute for simple verb-object subtasks, while crafting and smelting use craft instead.Guidance recommends equipping suitable tools, decomposing complex tasks, checking failures, and changing decisions rather than repeating failed actions.Source: §B.3 · pp. 27–29
  • B.3 Prompt Example: The agent receives task, state, orientation, and vision information, then outputs an observation, thought, and exactly one action function per turn.The vision input is summarized in the observation because images are not retained across dialogue turns.Source: §B.3 · pp. 29–30

C.1 Detailed Description

  • C.1 Detailed Description: Android Eval contains 119 tasks across 8 apps and supports ReAct [72] and SeeAct [76] evaluation in reproducible virtual machines.Preloaded histories, offline data, randomized operations, and noninitial device states reduce network or temporal effects while enabling more complex state-based task recognition.Source: §C.1 · p. 30

C VAB-Mobile

  • C VAB-Mobile: VAB-Mobile requires agents to complete diverse user tasks through predefined actions.Available actions include tapping, typing, long pressing, swiping, finishing, pressing back, and pressing home.Source: §C.2 · p. 30

C.3 Metrics

The GUI evaluation defines task-completion metrics and standardized smartphone and web interaction prompts. WebArena-Lite adapts realistic web tasks into a multimodal benchmark with annotated elements, predefined actions, and generated training trajectories.

C.3 Metrics

  • GUI success is measured from device or screen state for operation tasks, while query tasks are scored against ground-truth answers.The metric is oriented directly toward whether the task was completed.Source: §C.3 · p. 30

C.4 Prompt Example

  • Smartphone prompts provide labeled screenshots, require one function call per step, and expose actions such as tap, text entry, long press, and swipe.The prompt instructs agents to finish when complete and vary operations when progress stalls.Source: §C.4 · p. 31

D WebArena-Lite

  • WebArena-Lite evaluates high-level user tasks across simulated Gitlab, map, forum, shopping, and content-management websites.The environment also provides tools such as a sketch pad and calculator and supports varied user roles.Source: §D.1 · p. 32

D.1 Detailed Description

  • WebArena-Lite contains 165 adapted multimodal tasks and 1,186 valid training samples generated from rule-based Playwright solvers.Thirty tasks were manually fixed, cross-website tasks were removed, and training instructions use valid and invalid items with varied phrasing.Source: §D.1 · p. 32
  • A WebArena-Lite example uses screenshot annotations to guide editing a user status through the avatar and “(21) Edit Status” control.The SoM approach highlights actionable elements and supplies identifiers for interaction.Source: Figure 9 · p. 33

D.2 Actions

  • Web interaction is restricted to predefined actions including click, hover, type, press, scroll, tab control, navigation, and stopping.These actions are provided to keep interaction practical despite complex real-world webpages.Source: §D.2 · p. 33

D.3 Metrics

WebArena-Lite scores whether agents complete tasks rather than how they execute them. Its success-rate evaluation covers answer matching, navigation endpoints, and webpage content modification.

D.3 Metrics

  • WebArena-Lite uses Success Rate because multiple execution paths may complete the same web task.The metric ignores the agent’s execution trajectory and checks task completion instead.Source: §D.3 · p. 33
  • Question-answering tasks are scored by string matching, while webpage-navigation tasks are judged by the URL at termination.These categories preserve task-specific completion criteria.Source: §D.3 · p. 33
  • Content-modification tasks are evaluated by extracting webpage content and checking whether it satisfies the expected configuration.This applies to tasks that change the environment rather than return an answer or navigate to a page.Source: §D.3 · p. 34
  • Answer matching uses exact_match, must_include, or fuzzy_match patterns depending on the evaluation requirement.Fuzzy matching uses LLM assistance such as GPT-4 to determine correctness.Source: §D.3 · p. 34
  • Selecting metrics by task type supports construction of a comprehensive and relatively accurate test set.Source: §D.3 · p. 34

D.4 Task Amendment

This material amends WebArena tasks for reliable natural-language evaluation and describes prompting and task design for VAB-CSS.

D.4 Task Amendment

  • 39 of 165 selected WebArena tasks were corrected for typos, inaccurate answers, or scoring criteria, with exact-match requirements changed to inclusion or fuzzy matching.Source: §D.4 · p. 34

D.5 Prompt Example

  • The web-agent prompt supplies objectives, accessibility trees, URLs, open tabs, previous actions, and a structured action vocabulary including navigation, page operations, tabs, and stopping.Source: §D.5 · p. 36

E VAB-CSS

  • VAB-CSS evaluates multi-round CSS debugging on complete frontend projects using a constrained toolkit and a definitive success metric rather than whole-HTML generation.Source: §E.1 · p. 37, Table 8 · p. 37

E.1 Detailed Description

  • VAB-CSS is designed around iterative CSS refinement in a realistic multi-file frontend setting, addressing limitations of single-round datasets and qualitative or continuous-only evaluation.Source: §E.1 · p. 37, Table 8 · p. 37

E.2 Data Collection

VAB-CSS constructs manageable visual-debugging tasks through single-property corruption, human descriptions, and bootstrapped training trajectories.

E.2 Data Collection

  • Each task corrupts one categorical CSS property by changing or removing its value, while noting that even a single corruption can substantially alter the rendered appearance.Source: §E.2 · pp. 37–38
  • Human annotators describe differences between target and current renderings because existing LMMs struggle to identify those visual discrepancies directly.Source: §E.2 · p. 38, Figure 10 · p. 38
  • Training trajectories primarily retain successful gpt-4o attempts, supplemented by lenient-setting successes combined with standard-setting failures to model trial and error.Source: §E.2 · p. 38

E.3 Actions

  • Agents receive screenshots and HTML plus tools to locate, inspect, edit, and revert CSS rules without knowing which project file contains the rule.Source: §E.3 · pp. 38–39

E.4 Metrics

VAB-CSS defines success through a perceptual SSIM threshold and uses prompts that enforce sequential tool-based CSS editing.

E.4 Metrics

  • A VAB-CSS task succeeds when final SSIM similarity exceeds 0.9, rather than requiring the stricter perfect similarity of 1.0.The 0.9 threshold is an empirical choice based on observations.Source: §E.4 · p. 39, §E.5 · p. 39
  • Success Rate measures the proportion of successfully fixed test tasks, while Improve Rate measures whether final rendering is more similar to the target than the initial rendering.Source: §E.4 · p. 39

E.5 Prompt Example

  • The CSS-agent prompt requires one action at a time, valid selectors, iterative screenshot-based decisions, and optional reversal of incorrect edits.Source: §E.5 · pp. 39–40
  • The prompt prioritizes major layout issues while instructing agents to ignore imperfect text font and color properties.Source: §E.5 · p. 40

Related papers

Loading 2408.06327v1…