PaperlayerView exact revision on arXiv

Source-linked AI summary

OneScene-Bench: Do Vision-Language Models Integrate Evidence Across Views?

Swathi Suhas, Anurag Das, Bernt Schiele, Jonas Fischer

arXiv:2610.38926v1cs.CV

TL;DR

Multi-image evaluations can allow correct answers without requiring cross-view correspondence, leaving it unclear whether VLMs integrate observations into a coherent scene-level representation. OneScene-Bench controls when integration is necessary across counting, consistency, and spatial reasoning, and finds persistent difficulty with complementary-view counting and consistency. Its results show a gap between processing multiple images and integrating evidence across them.

  • Problem

    Existing multi-image evaluations may permit correct answers through independent per-view processing, so they do not establish whether VLMs integrate evidence across views.

    Source: §1 · p. 1

  • Method

    OneScene-Bench controls overlap, edit visibility, and spatial evidence in real-world multi-view counting, consistency, and spatial-reasoning tasks.

    Source: §1 · p. 2

  • Results

    4.5–50.0% accuracy on complementary-view counting, versus 45.5–81.5% on complete-overlap examples; cross-view consistency remains around 50%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results reveal a persistent gap between processing multiple images and integrating them into coherent scene-level reasoning.

    Source: §5 · p. 8

    The benchmark covers three scene-level capabilities and 1,300 manually verified examples from DL3DV-10K.

    Source: §5 · p. 8

Abstract

from arXiv · show

Vision-language models (VLMs) increasingly process multiple views of the same scene, but it remains unclear whether they integrate these observations into a coherent scene-level representation. Existing multi-image evaluations do not isolate this capability, as tasks may admit correct solutions through independent per-view processing without requiring cross-view correspondence. We introduce OneScene-Bench, a controlled benchmark designed to explicitly distinguish when cross-view integration is necessary. Our benchmark evaluates three complementary capabilities on real-world multi-view scenes: unique object counting, cross-view consistency, and spatial reasoning. For counting, we control object overlap across views to separate cases solvable by simple per-view aggregation from those requiring instance-level correspondence. For consistency, we introduce localized edits in shared and view-specific regions, testing whether models distinguish genuine scene contradictions from differences caused by viewpoint and visibility. For spatial reasoning, we construct object relations that require combining spatial evidence across viewpoints to compare physical proximity. We evaluate twelve state-of-the-art open-weight and proprietary VLMs and find that counting accuracy drops substantially on complementary views, ranging from 4.5-50.0%, compared with 45.5-81.5% on complete-overlap examples. Moreover, cross-view consistency remains around 50% across models, including recent proprietary VLMs, despite substantially stronger performance on spatial reasoning. Our results reveal a substantial gap between processing multiple images and integrating evidence across them, highlighting the need for evaluations that explicitly test scene-level multi-view understanding. Code and data are available at https://github.com/swathisuhas/OneScene-Bench.

1 Introduction

Multi-view inputs do not by themselves show that a VLM integrates evidence across views: some questions can be answered through independent per-view processing. OneScene-Bench controls when cross-view integration is necessary and finds substantial weaknesses in counting and consistency.

1 Introduction

  • Correct answers on complete- or subset-overlap counting can come from taking the larger per-view count, but complementary views require recognizing shared instances.Thus, multi-view benchmark success alone does not establish cross-view integration.Source: §1 · pp. 1–2
  • A reliable evaluation must distinguish questions requiring cross-view integration from those solvable through independent per-view processing.The paper notes this capability matters in embodied settings such as robotics and autonomous systems.Source: §1 · pp. 1–2
  • OneScene-Bench uses overlap-controlled counting, localized consistency edits, and spatial-reasoning questions to test whether models integrate views of the same scene.The examples come from real-world multi-view captures and are manually verified.Source: §1 · p. 2
  • Across twelve open-weight and proprietary VLMs, counting accuracy is 45.5–81.5% on complete-overlap examples but 4.5–50.0% on complementary views requiring cross-view correspondence and deduplication; cross-view consistency stays around 50%.Source: §1 · p. 2
  • The benchmark also analyzes aggregation strategies, image-order sensitivity, no-image bias, explicit reasoning, and scaling with additional views.These analyses accompany the twelve-model open-ended evaluation.Source: §1 · pp. 2–3

2 Related Work

Prior work spans broad multi-image evaluation and more specialized multi-view benchmarks, but these evaluations generally do not isolate whether a correct answer requires integrating views of the same scene.

2 Related Work

  • Multi-image benchmarks cover perception, image sequences, multimodal contexts, and diverse cross-image reasoning, while multi-view benchmarks target settings including robotics, urban scenes, real-world geometry, and synthetic 3D.Examples include BLINK [14], Mementos [36], MileBench, MV-RoboBench [12], and All-Angles Bench [38].Source: §2 · p. 3
  • Existing multi-view benchmarks generally do not separate questions solvable through independent per-view processing from those requiring cross-view integration.The paper identifies this distinction as a gap in prior evaluations.Source: §2 · p. 3
  • OneScene-Bench controls overlap, edit visibility, and relational evidence to test whether models reconcile observations into a coherent representation of one physical scene.The related work contrasts this design with benchmarks that cover important multi-view capabilities without explicitly isolating the integration requirement.Source: §2 · p. 3

3 OneScene-Bench

OneScene-Bench is a human-verified benchmark built to control whether answering questions about a scene requires cross-view integration. It covers counting, consistency, spatial reasoning, and an additional counting-scaling evaluation.

3 OneScene-Bench

  • The 1,300-question core benchmark uses two views per example: 700 counting, 300 consistency, and 300 spatial-reasoning questions from DL3DV-10K scenes.Outputs are integer counts, YES/NO judgments, or choices between reference objects; annotations are manually verified.Source: §3 · pp. 3–4

3.1 Multi-View Counting

  • Complementary-view counting requires matching repeated instances across views, whereas complete- and subset-overlap counts can be recovered from the larger per-view set.The counting set contains 200 complete-overlap, 300 subset-overlap, and 200 complementary-view examples.Source: §3.1 · pp. 4–5

3.2 Cross-View Consistency

  • Consistency examples contrast edits in view-specific regions, which leave views compatible, with shared observable-region edits, which create contradictions.Both conditions contain image-level differences, so detecting differences alone cannot distinguish them.Source: §3.2 · p. 5

3.3 Multi-View Spatial Reasoning

  • Spatial questions ask whether B is physically closer to A or C, with objects sometimes distributed across the two views.The 300 questions favor cases where combining views informs the comparison beyond either view alone.Source: §3.3 · p. 5

3.4 Multi-View Scaling

  • The scaling extension evaluates 200 complete-overlap questions with two to six views, adding 800 configurations beyond the core benchmark.All target instances remain visible in each selected view, while additional views increase the correspondences models must resolve.Source: §3.4 · p. 5

3 OneScene-Bench

  • Across the benchmark, counting degrades on complementary views, consistency remains around 50%, and spatial reasoning is substantially stronger.The table reports accuracy for twelve open-weight and proprietary VLMs.Source: Table 1 · p. 6

4 Evaluation

OneScene-Bench evaluates five multi-view settings with shared instructions and inputs, while judge-based scoring allows semantically equivalent answers. Models perform much better on some tasks than on cross-view consistency and complementary-view counting, and diagnostics reveal limits in their use of visual evidence.

4.1 Evaluation Setup

  • All models receive the same task instructions and images across five settings, with GPT-4o judging whether responses reach the correct conclusion.Judge-based accuracy accepts semantically equivalent answers and minor response-format differences.Source: §4.1 · p. 6

4.2 Benchmark Evaluation Results

  • Counting accuracy falls on complementary views, consistency stays near chance, and spatial reasoning is substantially stronger for most models.Qwen3-VL-32B-Instruct scores 70.0% on complete overlap and 70.7% on subset overlap, but 34.0% on complementary views; consistency ranges from 47.0% to 55.7%.Source: §4.2 · p. 6

4.3 Diagnostic Analyses

  • Without images, models retain non-trivial accuracy, exposing language and answer-distribution priors; LLaVA-OneVision-2 gets 50% consistency accuracy by always answering YES.Table 2 contrasts no-image accuracy with image-based performance; visual evidence substantially improves counting and spatial reasoning.Source: §4.3 · p. 7, Table 2 · p. 7
  • Reasoning reduces counting accuracy across nearly all tested variants and settings, with only a 1.5-point gain for Kimi-VL on complementary views.The figure’s comparison covers matched reasoning-enabled and instruction-tuned models, plus direct and thinking-enabled MiniCPM-V-4.5 inference.Source: §4.3 · p. 7, Figure 3 · p. 7
  • Joint counting predictions agree more often with the maximum than the sum of per-view counts, even when maximum aggregation fails on complementary views.Complete- and subset-overlap examples can be solved by maximum aggregation when individual-view counts are correct; complementary views require evidence from both images.Source: §4.3 · p. 7, Table 3 · p. 8
  • With scene-level count fixed, complete-overlap accuracy initially rises as views increase but peaks before six views for all four models.The reported pattern suggests added evidence helps only until integrating more views becomes increasingly difficult.Source: Figure 3 · p. 7, §4.3 · p. 8

5 Discussion

The results show a gap between processing multiple images and coherently reasoning across them: models struggle most when they must match and deduplicate objects across complementary views. Strong spatial reasoning does not guarantee reliable scene-level integration; the benchmark itself covers three capabilities and 1,300 manually verified examples from DL3DV-10K.

5 Discussion

  • Models can perform spatial reasoning while still struggling with object correspondence, deduplication, and cross-view consistency.The paper frames these capabilities as essential for reliable and accurate 3D reasoning.Source: §5 · p. 8
  • The benchmark covers three scene-level capabilities and 1,300 manually verified examples from DL3DV-10K.The authors prioritize careful human verification while balancing evaluation breadth against computational and API costs; broader tasks, environments, and model coverage remain future work.Source: §5 · p. 8

6 Conclusion

OneScene-Bench evaluates whether VLMs integrate multiple views to reason about a scene. Across open-weight and proprietary models, the authors report persistent difficulty with object correspondence, complementary evidence, and cross-view inconsistencies.

6 Conclusion

  • Models struggle to establish object correspondence, reconcile complementary evidence, and detect inconsistencies across viewpoints.The benchmark provides a controlled framework for measuring these capabilities and tracking progress toward more reliable scene-level multi-view understanding.Source: §6 · p. 8
  • The authors conclude that models of different architectures, both closed and open, are usually unable to perform basic visual reasoning across multiple views.They describe multiple views as the prevalent representation format for 3D scenes.Source: §6 · p. 8
  • OneScene-Bench is designed to track progress toward more complex 3D reasoning with VLMs.The authors state that closing the gap in multi-view visual reasoning becomes essential for that goal.Source: §6 · p. 8

A Detailed Dataset Construction

The appendix details benchmark construction, candidate mining, and manual verification, then reports that predictions are generally robust to input-image order.

A.1 Counting Candidate Mining

  • Counting candidates came from two frames sampled 10–25 frames apart at DL3DV-10K locations, with Qwen3-VL-32B-Instruct proposing countable categories.Locations were skipped after up to 10 unsuccessful attempts; manual overlap labels yielded 200 complete-overlap, 300 subset-overlap, and 200 complementary-view questions.Source: §A.1 · p. 12

A.2 Consistency Construction

  • Consistency examples use localized edits in shared or view-specific regions to distinguish scene contradictions from viewpoint-dependent differences.SIFT correspondences filtered with RANSAC identify candidate regions; an edit in a view-specific region is labeled consistent, while an edit in a shared region appearing in only one view is labeled inconsistent.Source: §A.2 · p. 12

A.3 Spatial Reasoning Candidate Mining

  • Spatial-reasoning candidates ask whether object B is physically closer to A or C, using three objects identified across two views.Candidates were retained when the objects and proximity relation were unambiguous, with 300 questions constructed.Source: §A.3 · p. 12

B.1 Image Order Sensitivity

  • 18 of 20 model–task combinations changed by at most four percentage points when image order was reversed.Gemma-4 on complete-overlap counting and LLaVA-OneVision-2 on spatial reasoning showed the largest changes: 6.0 and 9.0 points, respectively.Source: Table 4 · p. 13, §B.1 · p. 13

A.4 Verification Procedure

  • Reviewers checked images, referenced objects, and annotations, rejecting or correcting ambiguous identity, correspondence, visibility, or answers.Task-specific checks covered counting correspondences and overlap, consistency of edits with shared or view-specific regions, and spatial object references and proximity.Source: §A.4 · p. 13

B.2 No-Image Evaluation Setting

The no-image setting records whether models explicitly refuse questions without visual evidence, while the appendix documents evaluation and judging prompts. The reported refusals vary substantially across models and tasks.

B.2 No-Image Evaluation Setting

  • Gemma-4-31B-IT refused all consistency questions and 293 of 300 spatial-reasoning questions, but only 21 of 700 counting questions.Qwen3-VL-32B-Instruct, InternVL3.5-38B-HF, and LLaVA-OneVision-2-8B-Instruct refused none; refusals required explicitly stating that the question could not be answered without visual information.Source: §B.2 · p. 13, Table 5 · p. 14

C Evaluation Prompts

  • All evaluated models receive the same task prompts, and an LLM judge evaluates raw responses against human-verified answers.The counting prompt requests a single scene-level integer; the consistency prompt asks for YES or NO, and the spatial prompt asks whether B is closer to A or C.Source: C Evaluation Prompts · p. 14, §C.1 · p. 14

C.1 Task Prompts

  • The counting prompt uses the target object category in place of {object_name}, with the same prompt across all three counting configurations.For the multi-image ablation, the prompt refers to multiple input images rather than two, while its other instructions remain unchanged.Source: §C.1 · p. 14

C.2 LLM-as-a-Judge Prompts

  • The LLM judge accepts semantically equivalent final answers and marks conflicting answers incorrect when no unambiguous final answer is given.The judging prompts compare each raw response with a human-verified count, consistency label, or spatial answer; the consistency prompt treats YES/SAME/CONSISTENT and NO/DIFFERENT/INCONSISTENT as equivalent.Source: §C.2 · pp. 15–16

D Screenshots from the Dataset Verification Tool

Three figures show the interfaces reviewers used to verify benchmark examples for counting, consistency, and spatial reasoning.

D Screenshots from the Dataset Verification Tool

E Qualitative Samples

Figure 7 presents representative questions and model predictions across OneScene-Bench’s five tasks. It shows the evaluated overlap configurations for counting and marks answer correctness.

E Qualitative Samples

  • Figure 7 covers complete-overlap, subset-overlap, and complementary-view counting, cross-view consistency, and multi-view spatial reasoning.Questions are slightly shortened for presentation while preserving their original meaning.Source: Figure 7 · p. 20
  • The figure shows answers from four VLMs, with green checks and red crosses indicating correct and incorrect predictions.The ground-truth answer is marked with a green tick.Source: Figure 7 · p. 20, E Qualitative Samples · p. 20

F Compute Resources

Open-source models were evaluated mainly on A100 GPUs, while closed-source models used paid APIs. The paper also identifies the data and model licenses governing its existing assets.

F Compute Resources

  • Open-source model evaluation used primarily NVIDIA A100 GPUs with 80 GB of memory and took about 2 hours per model per task; closed-source models used paid APIs.Source: F Compute Resources · p. 20

G Licenses for Existing Assets

  • OneScene-Bench uses DL3DV-10K under its Terms of Use, and model assets and proprietary APIs were used under their respective licenses or terms of service.The listed open-weight model licenses include Apache License 2.0 and MIT.Source: G Licenses for Existing Assets · p. 20

Related papers

Loading 2610.38926v1…