PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

Yiming Zhang, Jiacheng Chen, Jiaqi Tan, Yongsen Mao, Wenhu Chen, Angel X. Chang

arXiv:2604.24300v2cs.CV

TL;DR

Existing VLM spatial evaluations can be invalid because noisy annotations and full-scene assumptions make some video-based questions incorrect, ambiguous, or unanswerable. ReVSI rebuilds the benchmark with video-aligned annotations, verified questions, and frame-budgeted protocols, revealing systematic model failure patterns and enabling more reliable, diagnostic assessment.

  • Problem

    Existing VLM spatial evaluations can be invalid because annotation-to-video drift and full-scene assumptions undermine the validity of video-based questions.

  • Method

    ReVSI re-annotates objects and geometry, regenerates and verifies questions, and provides answerable variants across 16/32/64/all-frame budgets.

  • Results

    Open-source models show accuracy drops of up to 40% on ReVSI, especially for counting, relative distance, and relative direction tasks.

  • Takeaways & Limitations

    ReVSI provides a more reliable and diagnostic assessment of spatial understanding by separating missing visual evidence from deficient spatial reasoning.

  • Takeaways & Limitations

    Expert-level human annotation is costly, limiting scalability to substantially larger datasets or training-scale supervision.

Abstract

from arXiv · show

Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate objects and geometry across 381 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.

1. Introduction

ReVSI argues that existing visual spatial intelligence benchmarks can be systematically invalid when point-cloud annotations and full-scene assumptions are applied to sparse-frame VLM video inputs. It rebuilds VSI-Bench around input-consistent annotations, QA generation, and frame-budgeted evaluation to produce more reliable and diagnostic assessments of 3D reasoning.

  • Evaluation validity: Existing VSI benchmarks can yield unreliable conclusions because their designs and evaluation pipelines are invalid under modern VLM video-input settings.The validity gap arises in realistic 3D spatial-intelligence evaluation for VLMs.
  • Evaluation validity: Point-cloud annotations can omit visible objects, misidentify objects, and corrupt geometry-dependent answers when reused as video ground truth.These failures stem from imperfect reconstructions and annotation artifacts created for traditional 3D perception.
  • ReVSI design: ReVSI rebuilds VSI-Bench so benchmark questions remain strictly consistent with what the model sees in its raw video inputs.The benchmark re-annotates object labels and scene geometry with professional 3D tools, regenerates QA pairs with bias controls and human verification, and redefines frame-budgeted evaluation.
  • Diagnostic evaluation: Frame-level consistency and fine-grained object visibility enable diagnostics that distinguish missing visual evidence from deficient spatial reasoning.These controls also support controlled experiments under different visibility conditions.
  • Key findings: Open-source models show accuracy drops of up to 40% on ReVSI, especially for object counting, relative distance, and relative direction, whereas proprietary models remain stable or improve.The benchmark is more challenging because its annotations have higher fidelity and diversity.

2. Related work

Visual-spatial intelligence benchmarks originated with questions grounded in 3D geometric data, particularly meshes, and later extended this paradigm to video-based and embodied VLM settings.

  • Benchmarks for visual-spatial intelligence: Early spatial-reasoning evaluations used 3D geometric data, with pioneering benchmarks grounding questions in meshes from datasets such as ScanNet.As VLMs became more capable, subsequent work adapted this protocol to video-based and embodied settings.

3. Validity pitfalls in VSI evaluation

VSI evaluations can be invalid when annotations and derived QA pairs drift from raw video evidence or when sampled frames omit queried objects. ReVSI therefore emphasizes explicit annotation verification and frame-budget-aware answerability, recommending at least 64 uniformly sampled frames for single-room environments.

  • Core validity requirement: Evaluation validity requires every QA pair to be answerable and correct under the model’s actual visual inputs.Prior evaluations can systematically violate this requirement because of annotation inaccuracies and limited frame sampling.
  • Annotation-to-video ground-truth drift: Annotations based on reconstructed meshes can drift from raw videos because of incomplete geometric reconstruction and annotation-workflow limitations.This drift produces errors in object identity, visibility, and geometry-dependent answers.
  • Annotation-to-video ground-truth drift: Explicit verification and correction are essential because prior benchmarks systematically lack verification, allowing errors to persist across datasets and task types.The paper identifies this as necessary when scanned indoor-scene annotations are reused for video-based evaluation.
  • Scene observability under frame budgets: Object visibility and question answerability vary with video frame-sampling rates, making frame budgets a distinct evaluation pitfall.Questions become unanswerable when queried objects are absent from sampled frames and incorrect when sampled-frame answers deviate from all-frame ground truth.
  • Scene observability under frame budgets: 64 uniformly sampled frames are recommended as the minimum for adequate scene-element visibility in predominantly single-room environments.The recommendation follows analysis of 16, 32, and 64 frame budgets.

4. Video-aligned annotation and QA curation

ReVSI rebuilds scene annotations and QA pairs to ensure accurate, video-aligned evaluation of VLM 3D reasoning. It combines manual object and geometry annotation with stricter, human-verified question generation and bias-mitigating task redesigns.

  • Scene and object annotation: ReVSI re-annotates object labels and 3D bounding boxes across five datasets using a dedicated 3D web interface.The annotations support open-vocabulary descriptions and correct object geometry for video-aligned evaluation.
  • Object counting: 62% accuracy is achieved by predicting “2” alone on VSI-Bench object-counting queries, motivating single-count queries, category balancing, and removal of ambiguous categories.The scene-level wording is also changed to match videos spanning multiple connected rooms or open-plan areas.
  • Object size estimation: ReVSI reduces category-prior shortcuts in size estimation by excluding near-fixed-size categories, sampling out-of-distribution instances, and manually annotating 3D boxes.This improves dimensions over sizes derived from noisy 3D segmentation masks.
  • Spatial relations and room size: ReVSI emphasizes harder spatial reasoning by removing sub-1m distance cases, adding long-range pairs, constraining relative-direction setups, and introducing farthest-object queries.Room areas are manually annotated from top-down boundary polygons, excluding scenes with ill-defined boundaries.

5. Frame-aware evaluation

ReVSI makes evaluation frame-aware by ensuring QA answerability and correctness under 16/32/64/all-frame inputs, while adding dummy-video stress tests that separate scene context from task-relevant visual evidence. These protocols support controlled diagnosis of whether VLM predictions are visually grounded or driven by priors and hallucination.

  • Frame-aware evaluation: Room Size Estimation and Route Planning are excluded from the 16-frame setting because insufficient global context frequently makes them unanswerable.
  • Frame-aware evaluation: ReVSI constructs QA pairs for 16/32/64/all-frame sampling and enforces answerability and ground-truth correctness under each configuration.Visibility is determined from projected objects, with manual annotation when pixel coverage does not exceed 5% of the frame area.
  • Frame-aware evaluation: Unlike VSI-Bench, ReVSI aligns annotation and inference frames, improving benchmark quality across different input budgets.VSI-Bench constructs data under full-frame videos but evaluates models on subsampled inputs.
  • Controlled diagnostics: Dummy videos remove every frame containing objects referenced by a question while preserving scene context and non-queried objects.The queried objects never appear, making the videos unanswerable for humans while retaining deterministic ground-truth answers such as zero for object counting.
  • Controlled diagnostics: These dummy videos test whether predictions are grounded in visual input or driven by memorized indoor-scene priors and hallucination.The diagnostic can expose systematic behavioral differences that remain hidden when models achieve similar accuracy on real videos.

6. Experiments

Experiments evaluate open-source, proprietary, and 3D-specialized VLMs on ReVSI using task-specific metrics and frame-adaptive ground truth. Results show that ReVSI reverses several conclusions from VSI-Bench, exposing weaker fine-tuning gains, data-quality bottlenecks, and model-specific reliance on visual evidence versus priors.

  • Evaluation setup: ReVSI evaluates representative open-source, proprietary, and 3D-specialized VLMs using native frame settings and frame-adaptive ground-truth answers.The evaluated models include Qwen3-VL-Instruct, InternVL-3.5, LLaVA-Video-Qwen2, GPT-5.2, Gemini 3, and specialized models fine-tuned for 3D spatial intelligence.
  • Evaluation metrics: MCQ tasks use exact-match Accuracy, while numerical tasks use Mean Relative Accuracy averaged across confidence thresholds C = {0.5, 0.55, . . . , 0.95}.MRA treats predictions as correct when relative error falls within a dynamic threshold and penalizes large deviations while rewarding relatively close predictions.
  • Benchmark discrepancies: ReVSI reverses VSI-Bench’s apparent ranking by showing proprietary models consistently outperform open-source models across numerical tasks, especially object counting.VSI-Bench systematically underestimates proprietary-model performance, making open-source models appear substantially stronger.
  • Specialized-model evaluation: Fine-tuned models show substantially smaller gains over base models on ReVSI, with SpaceR degrading across multiple tasks instead of exhibiting the large improvements reported on VSI-Bench.The discrepancy is attributed to noisy annotations that can teach incorrect answers and scene-data biases that encourage category-level priors or hallucinated cues.
  • Data scaling: ∼3% gains result when Spatial-MLLM training data increases from 135k to 820k samples, indicating data quality and supervision fidelity are larger bottlenecks than quantity alone.This result concerns the current data-construction paradigm.
  • Hallucination or perception?: Dummy-video tests reveal divergent grounding: Qwen3-VL has near-zero hallucination when queried objects are absent, whereas InternVL3.5 relies more heavily on scene priors, including on black videos.The pattern appears in object counting and object size estimation, where InternVL3.5 can retain high scores on black videos despite prompts to use visual evidence.

7. Conclusion

ReVSI addresses fundamental validity issues in VLM 3D spatial-intelligence evaluation through frame-budget-aware, visibility-consistent, human-verified benchmarking. Its expert annotation requirements limit scalability and motivate automated or semi-automated supervision pipelines.

  • ReVSI identifies uncontrolled frame sampling, ambiguous question construction, and noisy ground-truth annotations as fundamental validity issues in existing VLM spatial-intelligence benchmarks.
  • ReVSI enforces frame-budget-aware evaluation, visibility-consistent question generation, and systematic human verification across spatial reasoning tasks.
  • Limitations & future work: Expert-level human annotation limits the scalability of ReVSI’s high-quality 3D indoor spatial-intelligence dataset to substantially larger datasets or training-scale supervision.
  • Limitations & future work: Automated or semi-automated pipelines for generating high-quality spatial supervision remain an important direction for future work.

Impact Statement … B.7. Object selection and filtering

ReVSI targets reliable video-based 3D spatial reasoning evaluation by addressing annotation errors, frame-budget sensitivity, and ambiguous task construction. Its protocol combines manually verified annotations, controllable frame subsets, refined prompts, and task-specific object filtering.

  • Impact Statement: VLMs increasingly support visual information extraction, motivating accurate benchmarks for reasoning about 3D environments in videos.The paper frames reliable 3D spatial assessment as potentially important for applications using image- and video-understanding models.
  • A. Additional diagnostics on VSI-Bench: VSI-Bench contains annotation inaccuracies from noisy 3D reconstructions, while reduced frame budgets make many questions unanswerable or incorrect.These issues undermine ground-truth reliability and reveal that question validity degrades substantially under common sparse sampling settings.
  • B.1. Annotation tools: ReVSI provides web tools for object visibility, 3D boxes, room boundaries, and end-to-end verification using video-aligned reconstructions.Annotations are manually performed by authors with 3D-vision expertise, and the dataset contains no model-generated annotations.
  • B.2. GPT verification for object naming: Object names are human-determined after optional GPT-5.2 consistency checks, and objects are discarded when annotators cannot reach clear agreement.Annotators provide 1–3 minimally occluded representative frames and tight crops when requesting auxiliary verification.
  • B.3. Gravity-aligned 3D bounding box: ReVSI initializes gravity-aligned oriented 3D boxes and manually refines them, addressing inaccurate rotation, tilt, missed objects, and room geometry.Its interfaces support object-box correction, floor-boundary annotation, and comparisons between raw videos, renderings, camera trajectories, and sampled frames.
  • B.4. 2D bounding box guidance: Projected 2D boxes are generated by ray casting but serve only as auxiliary cues because camera-pose and mesh noise can misalign them; visibility labels remain manual.The visibility interface supports 16/32/64/All frame settings and object-level inspection.
  • B.5. Video sampling details: ReVSI supplies all-frame videos plus nested 64-, 32-, and 16-frame subsets spanning the same temporal duration, alongside dummy videos that remove visual evidence.The fixed subsets are constructed by recursive uniform sampling, while Query-Dropped, First-Frame Repeated, and Black videos probe reliance on visual evidence.
  • B.6. Prompt & QA templates: ReVSI largely preserves VSI-Bench prompts while refining robustness, requiring Arabic numerals for numerical answers and applying task-specific object filtering.It also adds variants that increase task difficulty without changing semantics, excludes ill-posed categories, and subsamples objects for size estimation.

B.8. QA construction

ReVSI follows VSI-Bench’s generation pipeline but refines QA construction through task-specific filtering, manual route re-annotation, and newly constructed navigation questions. These procedures remove trivial or ambiguous cases and improve answerability across spatial reasoning tasks.

  • Construction overview: Targeted refinements improve question quality and reduce ambiguity across several tasks while following the VSI-Bench data-generation pipeline.The construction process is designed as a refinement of the existing benchmark pipeline rather than a complete departure from it.
  • Object absolute distance: 80% and 50% subsampling rates are applied to object-distance questions in the 1–2 m and 2–3 m ranges, while distances below 1 m are removed.This targets trivial cases and promotes a more balanced answer distribution.
  • Object relative direction: Object-relative-direction questions exclude objects exceeding 1.4 m along x or y, inter-object distances below 1.5 m, and angles within 35° of boundary directions.The boundary directions are 0°, 90°, 135°, 180°, 225°, and 270°.
  • Object relative distance: Object-relative-distance questions remove query-anchor differences below 0.3 m and query-anchor distances below 1 m to avoid ambiguous or trivial comparisons.Both filters apply to comparisons between the query object and anchor object.
  • Route planning: Route-planning questions are manually re-annotated, newly constructed without VSI-Bench overlap, and augmented with auxiliary anchors such as walls.The task otherwise strictly follows VSI-Bench’s question templates.

C. ReVSI data statistics & examples … C.3. Room area annotation examples

ReVSI broadens object coverage and improves the accuracy and completeness of object and room annotations relative to VSI-Bench. These refinements provide more representative indoor-scene data for evaluating spatial reasoning beyond category-level priors.

  • C.1. Object label statistics: ReVSI features a substantially broader and more diverse object vocabulary than VSI-Bench.Its long-tailed label distribution covers more real-world objects and reduces reliance on frequent categories.
  • C.1. Object label statistics: The long-tailed ReVSI label distribution encourages evaluation beyond category-level priors.The broader coverage reduces dependence on a small set of frequent categories.
  • C.2. Object annotation examples: ReVSI provides more accurate and complete 3D object annotations than VSI-Bench.Figure 17 compares their 3D bounding box annotations.
  • C.2. Object annotation examples: ReVSI uses tighter bounding boxes, corrected orientations, and improved coverage of previously missing objects.These refinements make the annotations more representative of real-world indoor scenes.
  • C.2. Object annotation examples: High-quality ReVSI box annotations support spatial-reasoning evaluation beyond category-level priors.The passage identifies high-quality box annotations as foundational, though the supplied text is truncated after that statement.
  • C.3. Room area annotation examples: ReVSI replaces VSI-Bench’s automatically computed room mask with manually annotated room polygons.Figure 18 illustrates the difference between the two room representations.
  • C.3. Room area annotation examples: The room-mask difference indicates that ReVSI provides more accurate room-area definitions and measurements than VSI-Bench.The conclusion follows from the comparison of automatic masks and manual polygons.

D. ReVSI evaluation details

ReVSI evaluations use standardized inference and evaluation procedures for open and fine-tuned models, while proprietary-model experiments use reduced subsets and Gemini 3 follows FPS-based sampling. These choices prioritize reproducibility, default-setting consistency, and reduced evaluation cost.

  • Evaluation frameworks: Zero-shot models use ModelScope SWIFT and LMMs-Eval with greedy decoding for reproducible inference and evaluation.Fine-tuned models use their provided evaluation code and default settings.
  • Evaluation frameworks: Spatial-MLLM is evaluated with its non-SA sampling version for simplicity.
  • Proprietary-model evaluation: 1,093 samples, approximately 16% of ReVSI, are used for proprietary-model experiments to reduce evaluation cost.VSI-Bench experiments use its official tiny set.
  • Proprietary-model evaluation: Gemini 3 models use FPS-based video sampling instead of fixed frame counts, following VSI-Bench and Gemini’s default strategy.

E. Additional experiments

Additional experiments test ReVSI under reduced frame budgets and decompose performance across fine-grained question variants. They also show how ReVSI’s refined annotations improve object geometry, coverage, and room-area evaluation compared with prior annotations.

  • Results under different frame-samplings: ReVSI reports supplementary evaluations with 32 and 16 sampled frames, extending the main 64-frame experiments to test performance under varying visual evidence.These results are presented in Tables 12 and 13, complementing the 64-frame results in Table 3.
  • Fine-grained performance analysis: Fine-grained analysis reveals systematic performance asymmetries across single-versus-multiple counting, room-size, relative-distance, and relative-direction formulations.The breakdown distinguishes single versus multiple objects or rooms, closest versus farthest objects, and forward versus backward directions.
  • Annotation comparisons: ReVSI provides tighter, more complete 3D object boxes with more accurate scale and orientation than VSI-Bench annotations.The revised annotations correct missing geometry and misaligned extents.
  • Annotation comparisons: ReVSI expands object coverage with diverse open-vocabulary categories, better representing real-world indoor scenes for spatial reasoning evaluation.
  • Annotation comparisons: ReVSI manually annotates room polygons using 3D geometry and raw video and introduces templates that distinguish room areas in open-plan scenes.This contrasts with VSI-Bench’s Alpha Shape calculation on noisy reconstructed geometry and its ambiguous total-area queries.
  • Results under different frame-samplings: The 16-frame setting omits room size estimation and route planning because the available frames provide insufficient global scene context.
Loading 2604.24300v2…