MVVBench: Benchmarking 4D Reasoning in Vision-Language Models

1 Introduction

• Existing video benchmarks test temporal reasoning from one viewpoint, while multi-view image benchmarks test static spatial understanding; rigorous evaluation of their joint, monocular-ambiguous setting is missing.Prior multi-view QA efforts also relied on VLM-generated questions, limiting how far they could probe beyond the generator’s comprehension.Source: §1 · pp. 1–2
• MVVBench contains 1,323 human-authored question–answer pairs from 608 real-world multi-camera scenes, each unanswerable from a single designated view but uniquely solvable across views.Its six categories cover attribute identification, relative distance, relative camera pose, and compositional counting.Source: §1 · p. 2
• Diagnostic experiments reveal cross-view correspondence and temporally coherent reasoning difficulties, while multi-view-specific chain-of-thought, TADSG, and preliminary verifiable-reward reinforcement learning explore ways to elicit competence.The prompting and TADSG strategies are inference-time approaches that do not require retraining.Source: §1 · p. 2

2 Related Work

• Video and 3D VLMs largely target either temporal reasoning from a single viewpoint or static 3D awareness, leaving concurrent dynamic camera streams underexplored.The broader video and multi-view fields have developed along these separate tracks.Source: §2 · p. 2
• Existing multi-view evaluations focus on static scenes, and VLM-generated questions can cap benchmark difficulty at the generator’s own competence; MVVBench instead requires dynamic temporal reasoning and cross-view integration.The cited prior examples include All-Angles Bench and E3VQA for static multi-view evaluation.Source: §2 · p. 2
• Generic chain-of-thought, tool use, and scene graphs do not directly address cross-view identity and temporal coherence, motivating multi-view-guided reasoning and query-conditioned temporal scaffolding.The proposed approaches provide explicit cross-view identity without retraining.Source: §2 · p. 3

3.1 Data Collection

• The benchmark draws 608 scenes and 3,082 videos from Ego-Exo4D, Panoptic Studio, and MMPTRACK, selected for varied multi-camera configurations.The sources contribute 496, 44, and 68 scenes, respectively.Source: §3.1 · p. 3
• Experts author open-ended questions under a monocular-ambiguity constraint, making answers uniquely determinable only by integrating evidence across views.The annotation process includes iterative human verification; 641 questions underwent at least one revision cycle, followed by independent validation of the full set.Source: §3.1 · pp. 3–4
• Six categories test implicit and explicit attribute identification and relative distance, relative camera pose, and compositional counting; 1,091 of 1,323 questions are dynamic.Counting spans entities conditioned on attributes, actions, or temporal windows.Source: §3.1 · pp. 3–4

3.2 Benchmark Conversion

• Open-ended questions are converted to multiple choice through answer-type classification, with counting, descriptive, and binary items handled through tailored conversion branches.The conversion pipeline includes a final human verification step.Source: §3.2 · p. 4, Figure 2 · p. 4
• Counting distractors use nearest integers, binary questions use two options unless reframed, and descriptive distractors are iterated through a VLM generator–judge loop before human verification.The counting distractors require no video access; descriptive distractors are conditioned on the video.Source: §3.2 · p. 4, Figure 2 · p. 4

4.1 Experimental setup

• The evaluation spans 33 VLMs and includes a human upper bound measured by one evaluator who averaged 60.34 seconds per question.The diagnostic table varies visual access, temporal localization, and reasoning-model setting for Qwen3-VL-4B.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 Main Results

• GPT-5-mini reaches 54.2, while human performance is 80.3; many VLMs remain near the 35.8 chance-level lower bound, and no evaluated model exceeds 55.Counting is especially difficult: humans reach 90.1, while no VLM exceeds 65.4.Source: §4.2 · p. 6
• Performance varies sharply across model families and categories, and larger variants do not consistently outperform smaller ones.The irregular patterns suggest MVVBench probes capabilities not well captured by standard evaluation suites.Source: §4.2 · p. 6

4.3 Diagnostic Analysis

• Text-only scoring is 37.49, close to the 35.81 random-guess baseline and below the 42.78 visual baseline, indicating that linguistic priors alone do not solve the benchmark.Providing all videos raises the Overall score from 42.78 to 43.31, but can expose an unselected view that bypasses reasoning over the designated views.Source: §4.3 · p. 7
• Providing the annotated temporal window raises the Overall score to 43.69, showing that identifying when the relevant event occurs is a substantial bottleneck.The windows can also isolate cross-view reasoning from temporal localization, or support future evaluation of multi-view temporal grounding.Source: §4.3 · p. 7

4.4 Structured Reasoning for Multi-View Video

• Naive CoT does not help, whereas MV-Guided CoT raises overall accuracy to 44.82 by prompting sequential view inspection and cross-referencing.Models still make cross-view identity errors, and all methods degrade on dynamic questions relative to static ones.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 Enhancing the performance in MVVBench

• TADSG uses person detection and re-identification to establish global identities, then answers from a query-conditioned, target-centric dynamic scene graph.It activates only when correspondence and target identity are sufficiently reliable, otherwise falling back to the prompting baseline.Source: §4.5 · pp. 8–9
• After DAPO post-training with verifiable exact-match rewards, Qwen3-VL-4B-Instruct accuracy rises from 43.67% to 47.67% on the held-out evaluation.The preliminary study trains on 954 two-view questions and holds out 300.Source: §4.5 · p. 9

5 Conclusion

• MVVBench evaluates joint spatial and temporal reasoning with 1,323 human-authored QA pairs across 608 real-world multi-camera scenes, while most models remain near chance and the best trails humans by over 25 points.Temporal localization is a key bottleneck, and naive chain-of-thought does not help.Source: §5 · p. 9
• MV-Guided CoT and TADSG yield consistent gains by scaffolding cross-view correspondence and evidence aggregation.The conclusion presents these as structured reasoning approaches rather than retraining.Source: §5 · p. 9
• The benchmark focuses on human-centric scenes, TADSG relies on person-specific detection and re-identification, and six categories may omit fine-grained action or causal reasoning.The paper identifies broader domain coverage and more general cross-view correspondence as future-work directions.Source: §5 · p. 9
• The benchmark is intended to support more reliable reasoning over dynamic multi-camera scenes, with potential applications including embodied perception, assistive robotics, safety monitoring, and failure analysis.The authors caution that stronger multi-view video understanding could also be misused in surveillance or tracking, and emphasize privacy, consent, licensing, and identity-tracking risks.Source: §5 · p. 9

A Further Details on MVVBench

• The six categories cover implicit and explicit attribute identification, implicit and explicit relative distance, relative camera pose, and counting.They probe identity and context across views, sensor-relative spatial judgments, camera geometry, and counts conditioned on attributes, actions, or time windows.Source: A Further Details on MVVBench · p. 14
• The Figure 4 taxonomy is descriptive rather than canonical, retaining original categories as inner-ring labels and assigning each question one rule-based outer-ring subcategory.The assignment uses linguistic form and intended reasoning type.Source: A Further Details on MVVBench · p. 14
• Counting is partitioned into total, filtered, and conditional counts, while relative distance is grouped by the compared entities or camera distance.Conditional counting ties the count to a temporal or situational clause; view-indexed distance adds camera distance for comparisons involving a referenced object or region.Source: A Further Details on MVVBench · pp. 14–15
• Attribute-identification subcategories include action, object, color, existence, comparison, and residual state questions.View-indexed attribute questions additionally use a Cross-view Location bucket, which accounts for most of them.Source: A Further Details on MVVBench · p. 15
• Relative-pose questions are classified as Camera Position when they specify camera-to-camera relations, and nearly all current questions fit that bucket.A fallback Spatial Relation bucket is also defined for generic object-centric pose questions.Source: A Further Details on MVVBench · p. 15

B Labeling Tool

• Annotators use authenticated accounts and assigned video queues, with synchronized multi-view playback and adjustable speed controls.They select relevant viewpoints, typically two or more, then create a QA pair, assign its category, and mark the required temporal interval.Source: B Labeling Tool · p. 15
• Verifiers search annotations by video, category, or annotator, then validate or reject each QA pair with feedback required for rejection.Flagged items enter a pending-feedback queue accessible to both verifier and annotator, supporting iterative quality control.Source: B Labeling Tool · p. 15

C Benchmark Conversion Details

• Each annotation’s selected views, temporal window, question, free-form answer, and category are converted into a final question, option set, correct-option index, and category.Most items have four options; binary items that cannot be faithfully rewritten remain two-way, and model-based conversion uses the selected views and annotated window rather than the full video.Source: C Benchmark Conversion Details · p. 15
• QA pairs are classified as binary, counting, or descriptive from the question-answer text, solely to select a downstream conversion rule.This answer-type tag does not replace the original semantic category used in benchmark analysis.Source: C Benchmark Conversion Details · pp. 15, 19
• Binary questions are rewritten as counting or descriptive QA only when a faithful non-binary reformulation is available; otherwise, they remain two-option items.The procedure rewrote 93 binary QAs, accepting a rewrite only when it preserved intent and yielded a concise, specific answer.Source: C Benchmark Conversion Details · p. 19
• Counting distractors are the nearest distinct non-negative integers around the correct count, producing plausible alternatives without additional video access.The option text preserves the answer’s natural surface form, such as “4 people.”Source: C Benchmark Conversion Details · p. 19
• For descriptive questions, a VLM proposes three plausible but false distractors and a second VLM rejects sets with ambiguity, duplicates, unsupported options, or reduced difficulty.Rejected sets are regenerated until acceptable.Source: C Benchmark Conversion Details · p. 19
• A human verifier checks correctness, distractor plausibility, preservation of the cross-view challenge, and natural, unambiguous wording before release.Items that fail are edited or regenerated.Source: C Benchmark Conversion Details · p. 19

• The expert annotator used a custom interface with clipped multi-view segments, could pause and replay without a time limit, and had response times recorded per question.Source: D Human Performance Evaluation · p. 19
• 60.34 seconds was the average time humans took to answer a single question, suggesting the benchmark requires nontrivial reasoning even for humans.Source: D Human Performance Evaluation · p. 19
• Relative pose accuracy was 58.33% with 467.48 seconds average response time, versus 85.87% for attribute identification with 34.68 seconds.Relative distance with view index had the highest accuracy at 90.12%.Source: D Human Performance Evaluation · pp. 19–20, Table 5 · p. 21
• Correct answers took an average of 36.07 seconds, while incorrect answers took 252.02 seconds, approximately seven times longer.The authors suggest difficult questions resist resolution despite extended deliberation.Source: D Human Performance Evaluation · p. 20
• Binary questions had 86.63% accuracy and 34.76 seconds average response time; descriptive questions had 73.70% accuracy and 179.50 seconds.Counting questions fell between them at 77.13% accuracy and 41.71 seconds.Source: D Human Performance Evaluation · p. 21

• Table 6 lists prompts used for the experiments in §4.Source: Table 6 · p. 21

Design philosophy

• TADSG establishes cross-view identity, compresses relevant evidence into a structured intermediate representation, then answers from the graph rather than raw video.The design is motivated by the observation that forcing a low-confidence graph degrades performance relative to the baseline.Source: F Details in TADSG · p. 21

Cross-view identity establishment

• At an anchor moment chosen for joint person visibility, YOLOv8 detections are cropped, embedded with OSNet, and greedily clustered by cosine similarity into global identities.Source: F Details in TADSG · p. 22

Confidence gating

• Two gates require sufficient matched-person anchor coverage and reliable ReID similarity within the resolved target identity cluster; otherwise, TADSG falls back to prompting.Source: F Details in TADSG · p. 22
• 18 says full-benchmark gains come from a smaller number of high-confidence activations rather than uniformly applying a potentially noisy graph.Source: F Details in TADSG · p. 22

Dynamic scene graph construction

• For clips passing both gates, the query-conditioned target-centric graph has five blocks covering identity, anchor and temporal observations, cross-view relations, and question-relevant facts.The VLM builds it from identity montages, target-highlighted anchor frames, and full clips, guided by identity labels and correspondence hints.Source: F Details in TADSG · p. 22

Graph-grounded answering

• The VLM answers primarily from the dynamic scene graph, using target-focused anchor images only to resolve remaining minor ambiguities.The graph externalizes identity, spatial relations, and temporal dynamics into a structured text representation.Source: F Details in TADSG · p. 22

• Each training example uses two synchronized camera views trimmed to the annotated interval, with four sampled frames per view arranged by view and temporal order.The panel preserves cross-view correspondence and short-term temporal information while keeping RL training computationally manageable.Source: G RLVR post-training · p. 22
• Training uses trl for DAPO-based RL, LoRA adapter tuning, and bitsandbytes for 4-bit NF4 model loading.Source: G RLVR post-training · p. 22
• The LoRA setup uses rank 16, alpha 32, dropout 0.05, and learning rate 2 × 10^-6 across 1,200 update steps.The run also uses batch size 1, four generations per prompt, completion length 48, and KL coefficient β = 10^-3.Source: G RLVR post-training · pp. 22–23