PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?

Azmine Toushik Wasi, Wahid Faisal, Abdur Rahman, Mahfuz Ahmed Anik, Munem Shahriar, Mohsin Mahmud Topu, Sadia Tasnim Meem, Rahatun Nesa Priti, Sabrina Afroz Mitu, Md. Iqramul Hoque, Shahriyar Zaman Ridoy, Mohammed Eunus Ali, Majd Hawasly, Mohammad Raza, Md Rizwan Parvez

arXiv:2602.03916v3cs.CVcs.CEcs.CLcs.LG

TL;DR

Vision–language models remain fragile at spatial reasoning in realistic, visually complex settings, while prior benchmarks provide limited and narrow evidence. SPATIALAB introduces a diverse benchmark with complementary evaluation formats and compares contemporary models with humans. The study finds substantial model–human gaps and persistent failures across grounded spatial tasks, while showing trade-offs among improvement strategies.

  • Problem

    Prior spatial benchmarks rely heavily on synthetic or simplified settings and narrow task designs, limiting evaluation of realistic spatial reasoning.

  • Method

    SPATIALAB provides 1,400 visual question–answer pairs across six categories and thirty task types, evaluated in multiple-choice and open-ended formats across diverse VLMs and human baselines.

  • Results

    Leading models achieve roughly 50–55% multiple-choice accuracy versus over 85% for humans, while open-ended performance drops by another 10–25 percentage points.

  • Takeaways & Limitations

    SPATIALAB exposes persistent weaknesses in depth, occlusion, scale, navigation, and 3D geometry while supporting fine-grained diagnosis of spatial reasoning.

  • Takeaways & Limitations

    Prior benchmarks remain limited by synthetic-domain bias, narrow task scope, and restrictive evaluation formats, including formats that can inflate performance through option elimination.

Abstract

from arXiv · show

Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision-language models (VLMs). Prior work largely relied on synthetic or LLM-generated environments with limited task designs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce SpatiaLab, a comprehensive benchmark for evaluating VLMs' spatial reasoning in realistic, unconstrained contexts. SpatiaLab comprises 1,400 visual question-answer pairs across six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 questions, supporting both multiple-choice and open-ended evaluation. Experiments across diverse state-of-the-art VLMs, including open- and closed-source models, reasoning-focused, and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. In the open-ended setting, all models show a performance drop of around 10-25%, with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry. By providing a diverse, real-world evaluation framework, SpatiaLab exposes critical challenges and opportunities for advancing VLMs' spatial reasoning, offering a benchmark to guide future research toward robust, human-aligned spatial understanding. SpatiaLab is available at: https://spatialab-reasoning.github.io/.

1 INTRODUCTION

Spatial reasoning remains difficult for VLMs because existing benchmarks simplify visual conditions and task scope. SPATIALAB addresses this with realistic, diverse evaluation across spatial dimensions, response formats, model types, and human comparison.

  • Motivation: Existing benchmarks often use synthetic or simplified settings that reduce visual clutter and under-sample occlusion, scale consistency, and partially observable navigation.These setups can mask failures under distribution shift.
  • Motivation: Robust evaluation should cover relative position, depth and occlusion, orientation, size and scale, navigation, and 3D geometry in real-world scenes.It should also test both choice and generation against human performance.
  • Benchmark: SPATIALAB provides 1,400 visual question–answer pairs spanning six principal categories and thirty task types in realistic, unconstrained contexts.The benchmark supports both multiple-choice and open-ended evaluation with balanced category and subcategory coverage.
  • Evaluation: The study evaluates more than 25 diverse VLMs against human baselines and conducts diagnostic analyses of spatial reasoning failures.The evaluation includes open- and closed-source, reasoning-oriented, and spatially specialized models.
  • Findings: 50–55% accuracy is achieved by leading models in multiple-choice evaluation versus over 85% for human annotators, while open-ended performance drops by 10–25 percentage points.Failures concentrate in occlusion inference, scale consistency, and multi-step navigation.

2 PRELIMINARIES: SPATIAL REASONING IN VLMS

Visual–spatial reasoning requires VLMs to map images and linguistic queries to structured spatial relations under perceptual and embodied constraints. SPATIALAB operationalizes this challenge through a fine-grained taxonomy and complementary evaluation formats.

  • Definition: Visual–spatial reasoning is defined as perceiving, representing, and manipulating spatial relations among objects for navigation and action.The formal task maps an image and query to structured relations such as left-of, occludes, or supports.
  • Challenges: VLMs must integrate noisy visual perception with compositional linguistic abstraction under perspective, gravity, and physical-interaction constraints.These requirements make the mapping from images and queries to spatial relations challenging.
  • Prior approaches: Earlier approaches commonly rely on basic relations, narrow task designs, synthetic datasets, structured QA benchmarks, or puzzle-like simulated setups.These approaches provide useful foundations but do not cover the full range of spatial reasoning conditions.
  • Taxonomy: SPATIALAB organizes spatial reasoning into six principal categories linked to distinct cognitive faculties: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry.Each category is further decomposed into five subcategories, producing thirty task types.
  • Evaluation formats: Multiple-choice evaluation probes recognition and distractor discrimination, whereas open-ended evaluation measures generative reasoning and compositional description.Using both formats exposes different strengths and weaknesses in model capabilities.

3 SPATIALAB BENCHMARK

SPATIALAB is constructed from diverse real-world imagery and structured spatial annotations. Its corpus and quality-control process are designed to represent natural visual variability and higher-order spatial inference.

  • Benchmark scope: SPATIALAB comprises 1,400 visual question–answer pairs organized into six major categories, each divided into five subcategories.The categories include Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry.
  • Image collection: Images are collected through web crawling, targeted scene-specific retrieval, and manual snapshots from diverse indoor and outdoor environments.The multi-source pipeline is intended to diversify object categories, environmental conditions, and spatial configurations.
  • Image diversity: The corpus systematically varies lighting, texture, edge complexity, spatial relations, material type, and gravity constraints.This design targets in-the-wild scenarios rather than simplified or synthetic setups.
  • Complexity: 21.48 objects, 11.88 partially visible objects, and 3.23 depth layers occur on average per image, while spatial reference chains average 2.07 links.Additionally, 16.71% of questions require mental rotation and 56.07% require object specificity.
  • Annotation and review: Annotation pairs images with spatial QA tasks spanning perceptual grounding and higher-order inference, following targeted annotator training.A three-tier review checks semantic validity, question–answer correctness, ambiguity, and task clarity.

4 EXPERIMENTS AND EVALUATION

Experiments compare diverse VLM groups using accuracy on multiple-choice and open-ended tasks, alongside strategies intended to improve spatial reasoning. The evaluation framework supports comparisons across model families, formats, and categories.

  • Models: The evaluated models include proprietary, open-source, reasoning, and spatial-specialist systems.Color coding distinguishes these groups and model size in the results tables.
  • Metrics and reporting: The benchmark evaluates model performance by question category in SPATIALAB-MCQ.Table 2 reports multiple-choice accuracy percentages, with higher values indicating better performance.
  • Evaluation process: Accuracy is the primary metric for both multiple-choice and open-ended tasks.Multiple-choice answers are automatically checked, while open-ended responses are assessed by a large-language-model judge.
  • Improvement strategies: The study explores inherent model reasoning, Chain-of-Thought, self-reflection, supervised fine-tuning, and multi-agent systems as improvement strategies.Supervised fine-tuning uses stratified sampling on 40% of the dataset for Qwen-VL2.5-3B-Instruct.

5 FINDINGS AND ANALYSIS

SPATIALAB evaluations show substantial, uneven gaps between VLMs and humans, with open-ended generation harder than multiple choice and failures concentrated in grounding-intensive spatial tasks.

  • 87.57% human accuracy versus 54.93% for InternVL3.5-72B demonstrates substantial headroom in SPATIALAB-MCQ.
  • 40.93% for GPT-5-mini versus 64.9% for humans shows a large SPATIALAB-OPEN performance gap.Open-ended accuracy ranges from 9.64% to 40.93% across models.
  • Spatial navigation, depth and occlusion, and size and scale remain consistently difficult, with many open-ended subtask scores below 30%.Orientation and some relative-position items are comparatively stronger for top models.
  • 23.0% is the average MCQ-to-open-ended performance gap across 25 models.The gap averages 23.0% with σ = 5.5%, and specialist models show gaps of about 27%.
  • Error analysis identifies mislocalization, perspective and scale mistakes, occlusion failures, attribute confusion, and fluent but visually ungrounded rationalizations.Failures increase when models must fuse multiple cues, such as depth ordering and relative size.
  • Performance improvement approaches: Reasoning-enabled models improve both formats, but gains are uneven and largest in relational and orientation tasks.Relative Positioning improves by +13.1% in MCQ, while Size & Scale can decline.
  • Performance improvement approaches: CoT usually provides little benefit or reduces accuracy, whereas self-reflection yields modest MCQ gains but fails to generalize open-endedly.CoT gains are most consistent for orientation; self-reflection helps especially with geometry and depth.
  • Performance improvement approaches: SFT consistently improves MCQ accuracy across categories but offers little or negative transfer to open-ended tasks.The authors associate this divergence with overfitting to task-specific answer distributions.

6 DISCUSSION

SPATIALAB frames realistic spatial reasoning as a diagnostic challenge for VLMs and finds that current interventions improve selected tasks without establishing holistic competence.

  • SPATIALAB evaluates 1,400 visual question-answer pairs across six spatial categories and 30 subcategories in realistic conditions.It uses both multiple-choice and open-ended formats, with review and complexity checks for visual diversity.
  • Models show systematic failures in transparency, multi-step navigation, and occlusion, often relying on surface texture rather than internalized 3D geometry.
  • SFT, CoT, and agentic decompositions produce task-specific gains but can destabilize or degrade performance elsewhere.The reported trade-offs include open-ended instability, perceptual errors, hallucinations, and weaker holistic scene understanding.
  • The discussion concludes that scaling and reasoning scaffolds alone are insufficient for human-level spatial intelligence.It points toward geometric grounding and physics-aware representations as needed directions.

7 CONCLUDING REMARKS

SPATIALAB provides fine-grained MCQ and open-ended evaluation of real-world spatial reasoning, revealing persistent gaps between VLMs and humans and an additional open-ended performance drop.

  • SPATIALAB contains 1,400 diverse QA pairs across six spatial categories and 30 subcategories, evaluated through MCQ and open-ended formats.
  • VLMs remain substantially below human performance despite several performance-improving methods.Open-ended generation shows an additional 10–25% drop relative to multiple-choice evaluation.

ETHICS STATEMENT

The benchmark development includes privacy, licensing, and dataset-curation measures, while acknowledging that minor copyright-related oversights may remain possible.

  • Image metadata was removed, personally identifiable features were minimized or excluded, and images were curated with attention to sourcing and licensing.The authors note that minor oversights may occur despite best efforts.

REPRODUCIBILITY STATEMENT

SPATIALAB is positioned as a reproducible, broad spatial-reasoning benchmark that addresses limitations in synthetic, narrow, and restrictive prior evaluations. Its design combines realistic contexts, detailed task coverage, manual questions, and both closed- and open-ended formats.

  • Design choices: SPATIALAB covers thirty distinct spatial categories across geometry, occlusion, orientation, relative positioning, size and scale, and navigation.Its manually authored, puzzle-inspired questions aim to provide linguistic richness and reduce shortcut opportunities.
  • Evaluation and reproducibility: Combining high difficulty, broad coverage, manual annotation, and open-ended tasks enables correctness assessment alongside systematic error, causal, counterfactual, and calibration analyses.The benchmark also documents dataset construction, annotation, filtering, quality control, validation, evaluation, and configuration details for replication.
  • Prior limitations: Prior benchmarks are limited by synthetic or template-based settings, restricted domains, narrow taxonomies, and predominantly binary or multiple-choice evaluation.These constraints can reduce real-world complexity and make fine-grained diagnosis difficult.
  • Task design: The benchmark decomposes spatial reasoning into directional relations, proximity gradients, alignment patterns, and additional category-specific subtasks.These task descriptions connect spatial reasoning to scene understanding, navigation, sorting, inspection, and related real-world uses.

D.2.1 EVALUATION OF LLM-AS-A-JUDGE FOR OPEN-ENDED EVALUATION

The study evaluates Gemini-2.5-Flash as an automatic judge for open-ended answers against three human annotators using complementary agreement measures. The judge shows strong overall agreement, while human annotations are themselves substantially consistent.

  • Evaluation setup: Gemini-2.5-Flash was evaluated on 240 balanced samples spanning all categories, with three human annotators independently assigning binary correctness scores.The comparison used Cohen’s Kappa, majority-vote accuracy, and Fleiss’ Kappa.
  • Agreement metrics: Cohen’s kappa measures pairwise agreement between two raters while correcting for chance agreement.The study computed agreement between the LLM and each annotator and between the LLM and the annotator majority vote.
  • Agreement metrics: Majority-vote accuracy measures the proportion of identical labels without correcting for chance agreement.It compares the LLM’s decision for each sample with the annotator majority vote.
  • Agreement metrics: Fleiss’ kappa generalizes Cohen’s kappa to multiple raters and measures inter-annotator reliability beyond chance.Higher values indicate stronger reliability among the human annotators.
  • Results: Gemini-2.5-Flash achieved Cohen’s kappa of 0.738 with the majority vote, raw majority-vote accuracy of 0.880, and Fleiss’ kappa of 0.774 among humans.3D Geometry reached κ = 1.000 and accuracy of 1.000.

E.1 SUB-CATEGORY-WISE QUANTITATIVE ERROR ANALYSIS

Multiple-choice results show fragmented spatial competence across model families: systems achieve strong isolated subcategory scores but remain highly inconsistent across tasks. Closed-source models reach the highest peaks, while physical and geometric abstraction remain broadly difficult.

  • Open-source large models: 80.56% in Corner/Angle Positioning is achieved by InternVL3.5-72B, while llama-3.2-11b-vision-instruct scores 2.0% in Object Rotation.Large open-source models improve substantially but retain severe subcategory fragilities.
  • Open-source small models: 74.29% in Stacking Orientation is reached by qwen-2.5-vl-7b-instruct, whereas InternVL3.5-1B scores 12.0% in Cardinal Direction.Small open-source models also show modest performance on Pathway Existence and weak gravity-related reasoning.
  • Reasoning models: 71.79% in Complete Occlusion Inference is achieved by Gemini-2.5-Flash-Thinking, while reasoning-focused models remain inconsistent across subtasks.Other reported strengths include o4-mini at 65.12% in Alignment Patterns and 56.0% in Gravity Effects.
  • Specialized and overall patterns: Specialized models show targeted strengths, including 53.66% in Accessibility Constraints, but remain below 55% overall in the reported navigation and stability results.The broader discussion characterizes model competence as fragmented, with within-model spreads often exceeding 50 percentage points.
  • Persistent weaknesses: Gravity Effects, Stability Prediction, and Spatial Containment consistently score below 35–40% across model categories.The paper attributes these failures to unstable orientation and physics representations and reliance on surface-level correlations.

F.1 SUB-CATEGORY-WISE QUANTITATIVE ERROR ANALYSIS

Open-ended evaluation reveals fragmented competence across model families and persistent weaknesses in continuous, geometric, occlusion, and embodied reasoning. The gap from multiple-choice performance is substantial and varies by model family and spatial category.

  • Closed-source models: GPT-5-mini leads closed-source open-ended results, reaching 58.14% in Directional Relations and 55.81% in Alignment Patterns, while other models fall to 16.28% or 12.0% in difficult subtasks.The reported disparities highlight uneven performance on subtle, continuous visual cues.
  • Open-source large models: 60.0% in Depth and Occlusion is achieved by llama-3.2-90bvision-instruct, while qwen-2.5-vl-32b-instruct and qwen-2.5-vl-72b-instruct each score 9.3% in Proximity Gradients.Scaling improves some capabilities but does not guarantee coverage of difficult continuous-reasoning tasks.
  • Open-source small models: Open-source small models show localized strengths but catastrophic failures, including 0.0% for InternVL3.5-2B in Betweenness Relationships.Reported strengths include 26.0% in Relative Size Comparison and 24.0% in Spatial Containment.
  • Reasoning models: Gemini-2.5-flash-thinking records 75.0% in Tool Handedness and 57.14% in Complete Occlusion Inference, whereas Kimi-VL-A3B-Thinking-2506 scores 6.0% in Spatial Containment.Reasoning-oriented models therefore exhibit a sharp internal divide.
  • Spatial reasoning models: Spatial reasoning models record the lowest open-ended results overall, with several models tied at 16.0% in Gravity Effects, 8.0% in Shape Projection, and 14.29% in Stability Prediction.The paper links these results to limited capacity and narrowly focused training.
  • Cross-category failures: Depth and Occlusion has 72 zero-correct items, while Shape Projection has 19 and Stability Prediction has 10 zero-correct items.Orientation and Relative Positioning also show brittle performance in Stacking Orientation, Tool Handedness, Betweenness Relationships, and Proximity Gradients.
  • Evaluation-format gap: The average MCQ-to-open-ended performance gap is 23.0% across 25 models, with category means ranging from 22.89 for Spatial Navigation to 24.57 for 3D Geometry.Specialist models show the largest gaps, while reasoning-focused models show smaller gaps; navigation is the strongest predictor of disparity.

G.3 HYPOTHESES ON ROOT CAUSES

The results point to a persistent mismatch between structured-choice performance and unconstrained spatial reasoning, shaped by task format, model specialization, and the strength of perceptual grounding. Reasoning and reflection can stabilize some outputs, but they do not reliably overcome missing spatial and geometric representations.

  • Evaluation format: 23.0% is the average MCQ-to-open-ended gap, showing that constrained answer spaces systematically produce higher scores than free-form generation.The gap has σ = 5.5%, although misleading distractors can occasionally make open-ended performance higher.
  • Model specialization: 27.03% is the largest average gap for spatial reasoning models, with InternVL3.5-72B reaching 31.57% overall and especially large gaps in navigation and orientation.The reported gaps are 36.68% in spatial navigation and 34.44% in orientation.
  • Reasoning-oriented models: 19.11% is the approximate average gap for reasoning-oriented models, while Gemini-2-Flash-Thinking shows a smaller 13.0% overall gap.Its subtask gaps include 13.9% in 3D Geometry and 21.03% in Relative Positioning.
  • Spatial navigation: 0.99 is the Pearson correlation between spatial-navigation gaps and overall MCQ-to-open-ended disparity, linking sequential grounding demands to format sensitivity.SpaceQwen2.5-VL-3B-Instruct has a 35.87% navigation gap and a 29.78% overall gap.
  • Chain-of-thought prompting: Orientation is the only category where CoT prompting reliably improves performance, while most other categories stagnate or decline.The analysis attributes these failures to incorrect perceptual priors in depth, scale, and relative positioning, which can be reinforced across reasoning steps.
  • Root causes: Reasoning layers alone cannot compensate for missing spatial cognition, because robust performance requires geometric and perceptual structure rather than only additional reasoning steps.The benchmark therefore probes limitations in 3D and geospatial representations that surface-level reasoning does not resolve.
  • Self-reflection: Self-reflection produces model-dependent outcomes: Gemini-2.5-Flash gains 20.66% in MCQ, whereas InternvVL-3-78b declines 2.66% in MCQ and 8.67% open-ended.Localized gains, such as +12% in Size & Scale, are offset by degradations including −28% in relative-positioning open-ended performance.
  • Supervised fine-tuning: 36.59% is the MCQ accuracy after four training epochs, while open-ended accuracy falls from 34.40% to 12.62% before recovering to 35.48%.The contrasting trajectories indicate that supervised fine-tuning provides more stable optimization signals for constrained-choice evaluation than for open-ended generation.

H.4.3 SFT DYNAMICS ANALYSIS

SFT improves multiple-choice spatial performance but produces a collapse-and-recovery pattern in open-ended generation, yielding little net gain. The broader analyses show that current models remain brittle when spatial tasks require integrating multiple perceptual cues.

  • SFT dynamics: 35.74% average accuracy at Epoch 4, up from a 25.63% baseline, shows monotonic improvement on SPATIALAB-MCQ.The model increasingly maps visual features to constrained answer options.
  • SFT dynamics: 19.26% average accuracy at Epoch 1, down from 34.40%, marks an early collapse on SPATIALAB-OPEN before recovery to approximately 34.82% at Epoch 4.The final improvement over the zero-shot baseline is only +0.42%.
  • Interpretation: SFT improves answer selection but does not fundamentally strengthen the generative spatial representation needed for robust open-ended reasoning.The recovery phase reflects renewed articulation of spatial concepts without a significant gain beyond the initial baseline.
  • Transfer: +5.24% to +11.70% improvements across evaluated domains indicate that SPATIALAB fine-tuning transfers to other spatial benchmarks.The largest reported gain is +11.70% on MIND THE GAP, followed by +7.05% on OMNISPATIAL and +5.24% on SPACE.
  • Mitigation strategies: Agentic reasoning helps orientation but generally fails to improve other categories, including depth and occlusion, navigation, relative positioning, and 3D geometry.Reported changes include +8.00% MCQ and +36.00% open evaluation for orientation, versus -24.00% open evaluation for depth and occlusion and -12.00% for navigation.
  • Error analysis: Models handle coarse perception but systematically fail when spatial reasoning requires multiple cues, including object identity, scale, occlusion, attributes, and grounded explanations.Observed failure classes include spatial mislocalization, perspective and scale mistakes, occlusion and ordering errors, attribute confusion, and unsupported open-ended rationalization.

J REPRESENTATIVE BENCHMARK SAMPLES ACROSS ALL SUB-CATEGORIES

SPATIALAB presents representative samples spanning all 30 subcategories, covering diverse visual phenomena and requiring varied spatial reasoning across geometry, occlusion, orientation, scale, positioning, and navigation. The benchmark also includes examples of evaluation results and demonstrates that resampling robustness and run-to-run stability support interpreting performance differences.

  • Benchmark coverage: SPATIALAB covers all 30 subcategories across 3D geometry, occlusion, orientation, positioning, size and scale, and spatial navigation.Representative samples illustrate the breadth of visual phenomena and reasoning challenges included in the benchmark.
  • 3D Geometry: 3D geometry examples test gravity effects, shape projection, spatial containment, and volume comparison.The examples include overhead shape inference, whether one object fits inside another, and comparisons of internal volumes.
  • Spatial Navigation: Spatial navigation examples assess accessibility constraints, obstacle avoidance, pathway existence, viewpoint visibility, and spatial sequence.Questions include identifying blocking objects, checking direct paths, selecting safe walkways, explaining visibility, and choosing an easy route to a train.
  • Size and Scale: Size-and-scale examples probe scale consistency, shadow-size projection, perspective distortion, and relative-size comparison.The questions address wire alignment, projected shadows, space occupied by stacked pipes, and apparent or comparative object size.
  • Depth and Occlusion: Depth-and-occlusion examples test complete and partial occlusion, layering order, reflective surfaces, and transparency effects.Illustrative questions ask which object creates a striped shadow, which object is frontmost, and what lies behind an occluding device.
  • Orientation: Orientation examples cover cardinal direction, facing direction, and object rotation through route, shadow, portrait, and angle questions.These samples require determining travel direction, interpreting facing relative to a wall, and estimating a pocket-watch angle.

K.3.2 DATASET STABILITY AND INTERNAL CONSISTENCY

The stability analysis finds that SPATIALAB performance estimates remain reliable under repeated runs and reduced subcategory sample sizes. Multiple complementary tests indicate that reported fine-grained comparisons are not driven by stochastic variation or sampling noise.

  • Resampling study: 1,000 resampling trials found no statistically significant difference between accuracy estimates from subcategory samples of 20 and 25 items.The Wilcoxon signed-rank procedure consistently failed to reject the null hypothesis across trials.
  • Item-level consistency: Mean pairwise item-difficulty correlations above 0.94 show that question difficulty ordering is preserved across independent runs.The benchmark therefore functions as a unified and internally consistent measurement instrument.
  • Internal consistency: ICC and Cronbach’s α consistently exceeded 0.98, indicating near-perfect reliability and internal consistency for the evaluation protocol and dataset.These results support stable absolute scores and relative performance rankings across runs and sampling regimes.
  • Run-to-run stability: Accuracy standard deviations across runs were negligible, remaining below 0.6%, while resampling produced statistically indistinguishable estimates from the full set.The reported stability holds across models and evaluation formats, including MCQ and open-ended evaluation.
Loading 2602.03916v3…