PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

Ziyan Jiang, Jingbo Yang, Jiabao Ji, Yujian Liu, Qiucheng Wu, Tommi Jaakkola, Yang Zhang, Shiyu Chang

arXiv:2609.40325v1cs.AI

TL;DR

Interactive world auditing requires agents to couple visual reasoning with actions that gather evidence, a capability not comprehensively evaluated by existing benchmarks. WorldAuditBench evaluates VLM-only and two-stage VLA–VLM auditors on interactive 3D anomaly tasks. VLM auditors outperform the two-stage auditors, but both remain far below human performance.

  • Problem

    Existing benchmarks leave comprehensive evaluation of agents that must act to gather evidence and identify diverse interactive-world anomalies underexplored.

    Source: §1 · pp. 1–2

  • Method

    WorldAuditBench provides 213 tasks across 13 environments and evaluates single-VLM auditors alongside two-stage VLA–VLM auditors that explore first and analyze trajectories afterward.

    Source: §1 · p. 2, §4.1 · pp. 5–6

  • Results

    Across 213 tasks, VLM auditors achieve 28.2% to 42.3% success versus 6.6% to 17.4% for two-stage VLA–VLM auditors; both fall below humans at 83.4%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    WorldAuditBench provides a testbed for studying agents that form hypotheses from observations, test them through action, and connect evidence across viewpoints and time.

    Source: §7 · p. 9

    Agents remain challenged by deciding which observations to seek and whether those observations support a conclusion.

    Source: §7 · p. 9

Abstract

from arXiv · show

As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or objects inconsistent with the surrounding scene. Multimodal AI systems, including vision-language models (VLMs) and vision-language-action models (VLAs), have shown potential for automating this task. However, 3D world auditing is complex, requiring the close coupling of two distinct capabilities: action, to navigate the 3D world and search for anomalies systematically and efficiently; and visual reasoning, to understand the environment and identify anomalies from multimodal observations. It remains largely unexplored whether multimodal agents can effectively couple these two capabilities, using visual reasoning to identify potential anomalies while taking actions to validate them. In this paper, we introduce WorldAuditBench, a benchmark for 3D world auditing comprising 213 anomaly tasks across 13 environments built with Unreal Engine 5 and Three.js, spanning five anomaly families. We evaluate five frontier models under a fixed exploration budget using two auditing paradigms: VLA-based exploration followed by VLM-based anomaly identification, and an end-to-end VLM agent in which visual reasoning directly guides action selection. Across the evaluated models and two paradigms, success rates range from 6.6% to 42.3%, substantially below human performance (83.4%). Through the task of world auditing, WorldAuditBench provides a testbed for studying how multimodal agents couple action and visual reasoning in interactive 3D environments, while highlighting current limitations in their ability to gather and interpret evidence during exploration.

1 INTRODUCTION

World auditing couples visual reasoning with actions that gather evidence, but existing benchmarks leave comprehensive evaluation of this challenge underexplored. WorldAuditBench evaluates two auditing paradigms and finds that both trail human performance.

1 INTRODUCTION

  • Auditing requires agents to navigate and reason together because some defects emerge only through interaction or comparisons across viewpoints and time.For example, testing whether an apparent barrier has a collision defect requires moving toward and attempting to cross it.Source: §1 · pp. 1–2
  • WorldAuditBench contains 213 tasks across 13 environments and five anomaly families, evaluating anomaly identification through visual inspection, interaction, and repeated observation.The benchmark compares interactive anomaly identification across related benchmarks and simulation environments.Source: §1 · pp. 2–3, Table 1 · p. 3
  • The single-VLM auditor selects actions while inspecting observations, whereas the two-stage VLA–VLM auditor analyzes a fixed exploration trajectory offline.This distinction determines whether anomaly analysis can guide exploration.Source: §1 · p. 2
  • Across 213 tasks, VLM agents achieve success rates of 28.2% to 42.3%, versus 6.6% to 17.4% for two-stage VLA–VLM auditors; both fall below humans at 83.4%.The paper reports that using observations to guide further exploration helps gather evidence to verify suspected defects.Source: §1 · pp. 2–3
  • Analysis identifies VLA exploration failures in reaching anomalies in large worlds and finds that VLM memory for storing and retrieving observations substantially improves discovery.The authors also report a substantial performance gap relative to human auditors.Source: §1 · p. 3

2 RELATED WORK

Prior work evaluates visual understanding using fixed images or prerecorded video, and embodied benchmarks commonly assess interactive task execution or information gathering. World auditing instead requires agents to actively gather evidence to expose and verify diverse anomalies.

2 RELATED WORK

  • Video-MME, VideoGameBunny, GlitchBench, VideoGameQA-Bench, and VideoGlitchBench primarily assess recognition, localization, description, or reporting from visual assets supplied in advance.These settings mainly test answers based on available visual evidence.Source: §2 · p. 3
  • Embodied environments and benchmarks evaluate perception, interaction, task execution, and information gathering, while FlySearch includes anomaly search for out-of-place objects.These works mainly focus on task completion or locating visually identifiable anomalies.Source: §2 · pp. 3–4
  • World auditing targets anomalies that may require active physical tests or comparisons across viewpoints and time, making evidence gathering part of identification.This contrasts with benchmarks that provide observations in advance or focus on task completion.Source: §2 · pp. 3–4

3 WORLDAUDITBENCH DATASET

WorldAuditBench is a manually constructed, human-validated benchmark of interactive 3D tasks. Its five anomaly families differ in the observations and actions needed for detection.

3.1 DATASET OVERVIEW

  • WorldAuditBench comprises 213 single-anomaly tasks across 13 interactive environments, spanning five scene types and implementations in Unreal Engine 5 and Three.js.The environments comprise 126 Unreal Engine 5 tasks and 87 Three.js tasks.Source: §3.1 · p. 4

3.2 ANOMALY TAXONOMY

  • The taxonomy groups anomalies into static physics, interactive physics, spatial consistency, temporal consistency, and semantic consistency, with detection needs ranging from a single view to deliberate actions or revisits.The five families are further divided into fifteen categories; examples include object support, collision, viewpoint-dependent appearance, changes over time, and context-inconsistent configurations.Source: §3.2 · p. 4

3.3 TASK CONSTRUCTION

  • The authors collect 13 environments across indoor, urban, historical, industrial, and natural settings, selecting bounded regions in larger environments to yield 27 scenes.The environments are built with Unreal Engine 5 and Three.js.Source: §3.3 · p. 4
  • For each hand-crafted task, the authors define a bug-free scene, exploration region, starting pose, and interactions, then perturb the scene and annotate the anomaly and its expected normal behavior.Perturbations include changes to placement, collision, appearance, interaction responses, or behavior over time.Source: §3.3 · pp. 4–5
  • Independent judges explore each task and rate whether its anomaly is observable, verifiable, and correctly annotated; failures or uncertain ratings prompt revision until two judges pass the same version.The task-construction process collects scenes, introduces anomalies, and iteratively refines tasks through human review.Source: §3.3 · p. 5, Figure 2 · p. 5

4 AUDITING PARADIGMS

WorldAuditBench compares auditors that couple visual reasoning with action during exploration against a two-stage setup that analyzes a completed trajectory. The task formalizes exploration as a history of actions and observations ending in an evidence-supported anomaly report.

4.1 TASK DEFINITION

  • Auditors receive instructions, task context, and an initial first-person RGB view, then report the target anomaly with supporting observations.Source: §4.1 · p. 5
  • The trajectory τ records the initial input, actions, resulting observations, and final anomaly report; each action depends on the preceding history.Source: §4.1 · pp. 5–6

4.2 VLM-ONLY PARADIGM

  • The VLM-only auditor reasons over the observations collected so far to choose actions that investigate suspected anomalies.Its tools support navigation, interaction, memory retrieval, and linking observations to reports.Source: §4.2 · p. 6

4.3 VLA–VLM PARADIGM

  • In the VLA–VLM setup, a VLA explores first and a VLM analyzes the recorded trajectory afterward, so intermediate analysis cannot redirect exploration.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments compare five model-and-harness combinations under two exploration budgets and evaluate reports with a VLM judge. Performance varies across models and anomaly families, with temporal consistency especially difficult.

5.1 EXPERIMENTAL SETUP AND METRICS

  • The five evaluated model-and-harness combinations use 40 actions for VLM-only auditing and 60 seconds of sampled VLA exploration for VLA–VLM auditing.Recorded VLA trajectories are shared across VLM backbones, but the paradigms have different exploration budgets.Source: §5.1 · p. 7
  • Approximately ten computer science PhD students each evaluate tasks for up to ten minutes, and a VLM judge scores reports against task rubrics.Success rate is the percentage of tasks with the target anomaly correctly identified.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • GPT-6 Astra reaches 42.3% SR among VLM auditors, while temporal-consistency SR is at or below 12.5% for every model and paradigm.Gemini reaches 32.4% and Claude 28.2% SR among VLM auditors.Source: §5.2 · p. 7

6 ABLATION STUDIES

The ablations examine how exploration conditions and coexisting anomalies affect auditing, and separate failures to expose anomalies from failures to identify them once exposed.

6.1 FACTORS AFFECTING AUDITING PERFORMANCE

  • Moving closer raises VLM-only SR from 33.3% to 38.1% and VLA–VLM SR from 5.6% to 7.1% on the studied Unreal tasks.The study uses Gemini 3.8 Flash on 126 Unreal tasks and reduces the navigable route to about one third of its original length.Source: §6.1 · p. 8
  • At 1.5× exploration budget, SR reaches 34.9% for VLM and 7.1% for VLA–VLM.Source: §6.1 · p. 8

6.2 AUDITING MULTIPLE ANOMALIES IN A SCENE

  • With two or three anomalies, all-anomaly success is 7.1% and 4.8%, while anchor success is 42.9% and 47.6%, respectively.Target recall is 29.8% and 29.4% in those conditions.Source: Figure 5 · p. 8, §6.2 · p. 9

6.3 ANOMALY EXPOSURE AND IDENTIFICATION

  • On exposed tasks, VLM-only success is 38.5% with geometric exposure and 62.1% with human-rated exposure, versus 26.0% and 47.3% for VLA–VLM.The analysis attributes the VLM-only advantage partly to inspecting additional views, testing suspected anomalies, and retrieving earlier frames and notes.Source: Figure 6 · p. 9, §6.3 · p. 9

7 CONCLUSION

WorldAuditBench tests how multimodal agents combine action and visual reasoning in interactive 3D auditing. The experiments find low agent performance relative to humans and identify evidence gathering and interpretation as central challenges.

7 CONCLUSION

  • WorldAuditBench contains 213 tasks across 13 environments and five anomaly families for studying action and visual reasoning in interactive 3D worlds.Source: §7 · p. 9
  • VLM-only auditors generally outperform VLA–VLM auditors, but both remain far below human performance.Source: §7 · p. 9
  • Reduced task guidance and coexisting anomalies further challenge discovery, underscoring the difficulty of choosing observations and deciding whether they support a conclusion.Source: §7 · p. 9

AI USE STATEMENT

The authors used generative AI to assist with manuscript drafting, polishing, and formatting, while retaining responsibility for the final work.

AI USE STATEMENT

  • AI tools assisted with manuscript drafting, polishing, and formatting, including figures and tables; authors reviewed and verified AI-assisted text and code.The authors state that they take responsibility for the final content, claims, code, and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The benchmark uses simulated 3D environments, with voluntary and informed human participation in validation and the auditing baseline.

ETHICS STATEMENT

  • Human participation in task validation and the auditing baseline was voluntary and informed; benchmark releases will retain required source-license notices and attributions.Environment sources and licenses are documented in Appendix A.2.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

WorldAuditBench reports that its code, task configurations, and runnable environment packages will be released under applicable licenses, alongside experimental documentation to support reproduction.

REPRODUCIBILITY STATEMENT

  • The authors plan to release evaluation code, task configurations, and runnable environment packages, with prompts and settings documented to support reproduction.Releases are subject to the applicable licenses.Source: REPRODUCIBILITY STATEMENT · p. 10

A ENVIRONMENTS

  • The benchmark contains 213 tasks across 13 environments and five scene categories, with 126 tasks in Unreal Engine 5 and 87 in Three.js.Table 4 summarizes the environments, rendering engines, and task counts.Source: A ENVIRONMENTS · p. 15, Table 4 · p. 15

A.1 ENVIRONMENT EXAMPLES

  • The environments span indoor, city, historical, industrial, and natural settings, including interactive domestic, transit, and courtyard scenes alongside navigation-and-inspection settings.Available interactions vary by scene: for example, selected Subway tasks include interactive benches, while Sponza Atrium offers navigation and visual inspection only.Source: A ENVIRONMENTS · p. 15, Residential House · p. 15, Sponza Atrium · p. 16
  • The 13 environments comprise 27 scenes: 21 bounded regions across seven Unreal Engine 5 environments and one scene in each of six Three.js environments.The example views are grouped by environment and labeled by scene.Source: §A.1 · p. 15

A.2 SOURCES AND LICENSES

  • Unreal Engine environment packages will omit original project source files and standalone third-party assets; Three.js components retain their original licensing terms and notices.The release plan preserves required copyright notices, creator attributions, and modification notices.Source: §A.2 · p. 19

B DATASET CREATION

The appendix documents how WorldAuditBench tasks are constructed, annotated, validated, and evaluated, and describes the agents’ inputs, tools, and reporting workflow.

B DATASET CREATION

  • The dataset contains 213 manually constructed single-anomaly tasks across 13 environments: 126 in Unreal Engine 5 and 87 in Three.js.Each task is checked for correct loading, target accessibility, and occurrence of the intended anomaly before human validation.Source: B DATASET CREATION · p. 20, §B.1 · p. 20

B.1 SCENE AND TASK CONSTRUCTION

  • Tasks begin from a normal configuration, specify starting conditions and available interactions, and introduce one anomaly from five families covering physics, spatial, temporal, and semantic consistency.The taxonomy distinguishes static from interactive physics anomalies and includes appearance changes across viewpoints, visibility, or lighting.Source: §B.1 · p. 20

B.2 TASK ANNOTATIONS

  • Each task pairs a scene description and ground-truth anomaly category with a rubric describing the target and expected normal appearance or behavior; the rubric is withheld from auditors.The rubric provides a common reference for human validation and evaluation of agent reports.Source: §B.2 · p. 20

B.3 HUMAN VALIDATION

  • Independent reviewers inspect tasks in a browser, assess observability and rubric fit, rate quality and difficulty, and may comment on ambiguous behavior or problems.Acceptance requires at least two distinct reviewers to rate the same scene configuration Pass; revisions to the anomaly or rubric require renewed validation.Source: §B.3 · p. 21, Figure 7 · p. 21

C.1 AGENT INPUTS AND INSTRUCTIONS

  • Auditors receive instructions, a scene description, an anomaly-type hint, a matching in-context example, and an initial first-person RGB observation, but not the target identity, location, or rubric.The instructions tell agents to gather evidence, avoid treating every unusual object as a bug, and use the full 40-action budget for exploration and verification.Source: §C.1 · p. 22

C.2 ACTIONS AND TOOL USE

  • Movement, viewpoint changes, interactions, and waiting consume environment steps, while observation, memory, and reporting operations are recorded separately from that step cost.Table 6 lists the available actions and tools, their inputs, and their effects.Source: §C.2 · p. 23, Table 6 · p. 23

C.3 AGENT OUTPUT

C.4 VLM AS JUDGE

  • A VLM judge scores reports against the task rubric using the report, summary, and cited visual evidence; its judgments show high overall agreement with human judgments across both engine types.The scoring rules specify when visual evidence is required and distinguish rubric match from evidence that fails to establish a required condition.Source: §C.4 · pp. 28–29, Table 7 · p. 29

C.5 ANOMALY TAXONOMY AND FULL IN-CONTEXT EXAMPLES

The taxonomy organizes fifteen anomaly categories into five families, with a demonstration for each category. The examples illustrate how anomalies can be identified through visual comparison, interaction, or repeated observation.

Taxonomy overview

  • The taxonomy groups fifteen anomaly categories into five families and links each category to an in-context demonstration; demonstration tasks are excluded from evaluation.Table 8 summarizes the taxonomy and corresponding examples.Source: §C.5 · p. 29, Table 8 · p. 29

Static physics

  • Static physics examples include scale or proportion mismatches, missing support, and solid objects intersecting or misaligned with one another.Multiple viewpoints can help establish an intersection or overlap.Source: §C.5 · pp. 30–31

Interactive physics

  • Interactive physics covers missing or unexpected collisions and object trajectories disproportionate to the contact and physical conditions.The abnormal-trajectory example contrasts a clean control tipping about 90° with a faulty version overturning about 180° under the same contact.Source: §C.5 · pp. 31–32

Spatial consistency

  • Spatial consistency anomalies include visibility contradictions, viewpoint- or distance-dependent appearance, displaced shadows, and inappropriate material properties.Examples use occlusion, fixed scene references, object position, and nearby matching surfaces to reveal inconsistencies.Source: §C.5 · pp. 33–34

Temporal consistency

  • Temporal consistency covers unexplained existence or static-attribute changes and abrupt or inconsistent operational-state transitions.Examples compare an object’s location before and after leaving and returning, or contrast abrupt day–night alternation with a gradual normal cycle.Source: §C.5 · pp. 35–36

Semantic consistency

  • Semantic consistency includes configurations that conflict with an object’s function and objects incompatible with the scene’s historical era.Historical incompatibility is assessed in light of the established setting and any explanatory premise.Source: §C.5 · p. 36

D FULL RESULTS

The qualitative audits include correct reports of a closing door, traversable lockers, and a historically mismatched appliance, as well as failures to gather or interpret evidence. The geometric-coverage analysis measures proximity and viewing direction, not anomaly recognition.

D FULL RESULTS

  • Table 9 reports success rate and inference cost by rendering engine and approach, with task counts by anomaly family; pooled results are reported in Table 2.Bold and underline mark the best and second-best results within each engine and approach.Source: Table 9 · p. 37

E GEOMETRIC COVERAGE

  • Geometric coverage counts a task when a screenshot meets both environment-specific distance limits and the viewing-direction threshold.The distance limits range from 7 to 16 metres; the target must be within approximately 41.4° horizontally and 32° vertically of the camera’s viewing direction.Source: E GEOMETRIC COVERAGE · p. 38
  • Coverage uses recorded positions and viewing directions, so an occluded target can count as covered without showing that its anomaly occurred or was recognized.Both auditors use the same target locations and thresholds.Source: E GEOMETRIC COVERAGE · p. 38

F QUALITATIVE AUDITING TRAJECTORIES

  • The qualitative section presents three successful and three failed audits from the Unreal Engine main evaluation, each run with a 40-action budget.The cases illustrate different outcomes and are not intended to estimate how often each failure occurs.Source: F QUALITATIVE AUDITING TRAJECTORIES · p. 39

Success cases

Failure: Missing Before-and-After Evidence

  • The plant-disappearance audit failed because broad room comparisons did not establish a clear before-and-after view of the target plant.The auditor reported no anomalies after revisiting the room.Source: Failure: Missing Before-and-After… · p. 43

Failure: Mistaking Advertisement Cycling For An Anomaly

  • The poster audit mistook normal advertisement cycling for an anomaly, producing a false positive while leaving the target disappearance unidentified.The auditor compared changing advertisement content rather than establishing that the target poster disappeared.Source: Failure: Mistaking Advertisement… · p. 44

Failure: Misidentifying The Anomalous Object

  • The floating-bench audit reported the advertising column as floating and incorrectly concluded that the benches were grounded.The case illustrates a false positive on the wrong object alongside a missed target anomaly.Source: Failure: Misidentifying The Anomalous… · p. 45
Loading 2609.40325v1…