PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do Vision Language Models Fail at 3D Reasoning?

Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu

arXiv:2610.39639v1cs.CVcs.AI

TL;DR

Vision-language models can answer individual questions about 3D objects while struggling to recover exact properties and use them consistently. BedBench evaluates these abilities across procedurally generated tasks and probes whether answers depend on visual evidence; its results show that individual accuracy does not ensure exact, consistent, image-dependent reasoning.

  • Problem

    Existing spatial and 3D reasoning scores may not reveal whether answers rely on visual evidence, language priors, or multiple-choice cues, nor whether individual judgments extend to exact, consistently used object properties.

    Source: §1 · p. 2

  • Method

    BedBench uses thirteen procedurally generated tasks with exact ground truth to compare individual judgments and composite answers, alongside consistency checks, interventions, image-removal controls, and visual-feature probes.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Models often answer individual judgments correctly while exact answers remain wrong and properties are used inconsistently; humans achieve 93.3% on T5 versus 7.2% for the best VLM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations of 3D reasoning should report exact answers with error sizes, test agreement among related answers, and check whether scores depend on images.

    Source: §6 · p. 9

    BedBench uses controlled procedural voxel objects and geometric primitives, so extending findings to complex real-world or embodied environments requires assessing noisier textures, lighting, and scene geometries.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench

1 Introduction

Vision-language models can answer individual questions about 3D objects yet struggle to recover exact object properties and use them consistently. BedBench tests this gap across controlled tasks, while probing whether answers rely on visual evidence.

1 Introduction

  • BedBench combines thirteen procedurally generated tasks on voxel objects and geometric scenes, with exact ground truth and generation-time filtering of samples detected as ambiguous.Its same-task family spans individual judgments and exact answers, enabling comparison under controlled conditions.Source: §1 · p. 2
  • The evaluation compares twelve open-weight and four frontier VLMs with 30 human participants, testing task simplifications, training, reasoning, visual cues, feature probes, and image removal.These comparisons target which exact answers fail and where the difficulty arises.Source: §1 · p. 2
  • Humans reach 93.3% on T5 versus the best VLM’s 7.2%; fine-tuning lifts T5 to 68.6% but T7 remains inconsistent with its counts.T7 accuracy is 35.0% from images versus 74.2% when the model receives its own counts as text.Source: §1 · p. 2
  • Thinking raises T4 by +13.9 pp on average, but T5 remains at 0.7–1.0%; larger reasoning budgets also lower T1 and T11.Visual misreadings often persist through reasoning.Source: §1 · p. 2
  • At Qwen3.5-27B’s language-model input, a probe recovers exact counts on 65.0% of objects versus 3.0% for the VLM, indicating difficulty using available visual information.Scaling and added visual cues produce selective gains but leave exact answers unreliable.Source: §1 · p. 2
  • Without images, models retain some scores through constant answers, and one model reaches 100% on a fixed-layout multiple-choice task by always choosing the third option.Individual judgments can be correct while exact answers remain wrong, and recovering a property does not ensure consistent use.Source: §1 · p. 2

2 Related Work

Prior work covers spatial skills, consistency checks, and visual-feature probing, but BedBench combines linked questions on shared objects with diagnostics aimed at locating failures. Its related-work discussion situates this design among work on failure patterns, probes, and interventions.

2 Related Work

  • Spatial benchmarks span scene understanding, mental rotation, and geometric reasoning; procedural generation can reduce memorization, while accuracy alone does not explain failures.BedBench uses questions whose answers entail one another on the same generated objects, including per-color counts that determine distinct-color and most-common-color answers.Source: §2 · p. 3
  • Prior consistency methods check agreement among entailed questions in visual question answering, while BedBench asks whether failure lies in property recovery, consistent use, or 3D perception.Prior examples include GQA, Ribeiro et al. (2019), and ConVQA.Source: §2 · p. 3
  • BedBench uniquely combines open-ended 3D questions, linked-question agreement checks, image-removal controls, and frozen-feature probes to locate where failures arise.The comparison table marks these benchmark properties and contrasts them with representative spatial and 3D reasoning benchmarks.Source: Table 1 · p. 3
  • Reported failure patterns include compositional reasoning, exact counting, reporting every image element, and multi-view spatial reasoning; BedBench adds diagnostics for exactness, consistency, and visual dependence.The cited examples are Li et al. (2025b), Pang et al. (2026), Zhang et al. (2026b), and Zhan et al. (2026).Source: §2 · p. 3
  • Prior probes recover counts and spatial properties from frozen visual features, while probe design can use selected-patch attention rather than averaging features.BedBench tests averaged patch features and an attention readout on exact per-color counts.Source: §2 · p. 3
  • Prior work reports task-dependent benefits from chain-of-thought prompting and targeted training, while questioning whether scale alone suffices for spatial reasoning.BedBench tests whether the gap between individual judgments and exact answers persists across reasoning budgets, scale, architecture, visual cues, and task-specific fine-tuning.Source: §2 · pp. 3–4

3 The BedBench Benchmark

BedBench compares varied questions about procedurally generated 3D objects and scenes, with tasks designed to be easier to verify than to answer. Its generation and rendering controls support fine-grained comparisons while filtering selected ambiguities.

3 The BedBench Benchmark

  • The thirteen tasks compare different questions about the same procedurally generated 3D objects and scenes, grouped by spatial demand with example inputs and ground-truth answers.The task design supports examining individual judgments and exact answers on shared objects.Source: §3 · p. 4
  • Tasks are chosen to be easier to check than to produce, requiring object imagination and operations such as mental rotation, decomposition, comparison, or querying.Source: §3 · p. 4
  • Objects contain 10–15 connected unit blocks rendered from two opposite views; generation rejects selected silhouette ambiguities and blocks whose removal barely changes both silhouettes.An independent audit found that excluding one object occluded in both views shifted model accuracies by at most 1.0 pp without changing findings; mixture-of-shapes scenes use visible, collision-free primitives.Source: §3 · p. 4
  • Matched visual variants add or remove color and coordinate-axis overlays; colored objects use 4–12 palette colors arranged in neighboring pairs with similar luminance.Only task-compatible variants are used.Source: §3 · p. 4

4 Experimental Setup

The experimental setup evaluates a broad model roster on a fixed set of instances, with human and random references and several complementary metrics. Additional interventions and visual-feature probes test how reasoning, presentation, and available representations relate to task performance.

4 Experimental Setup

  • The roster includes twelve open-weight models from four families and four frontier models, spanning dense and mixture-of-experts architectures.The listed open-weight families are Qwen3.5, Gemma4, GLM-4.6V-Flash, and Kimi-VL-A3B; frontier models include GPT-5, Grok-4.20, and Gemini variants.Source: §4 · p. 4
  • Evaluation uses 100 instances per task, except T11 and T13, which each use 300 questions covering three projection planes for 100 objects or scenes.Default T11 and T13 settings ask for the invalid candidate.Source: §4 · p. 4
  • Thirty volunteers each answer one randomly drawn instance per task; binary and multiple-choice tasks also use uniform random selection and a best-constant baseline.Human accuracy is reported with 95% Wilson confidence intervals.Source: §4 · p. 4
  • Structured task-specific answers are scored by exact match, with MAE and RMSE for numerical error size and precision, recall, and F1 for binary-task error composition.Source: §4 · p. 4
  • Open-weight models use greedy decoding in Non-Thinking and Thinking modes; Thinking adds reasoning mode and decomposition prompts, with a default budget of 1024 tokens.Each open-weight condition is repeated five times; frontier models are evaluated once in Thinking mode through provider APIs with provider-dependent budgets.Source: §4 · pp. 4–5
  • T5 controls compare twelve single-color queries, tiled 2D renderings of the same blocks, and LoRA fine-tuning of Qwen3.5-27B on 800 examples with its vision encoder frozen.The fine-tuned model is tested on T5 and related color tasks, including T7 with its own T5 counts supplied as text.Source: §4 · p. 5
  • Other diagnostic comparisons vary reasoning budget, model scale and architecture, color and axis overlays, image removal, and a five-option multiple-choice version of T1.Source: §4 · p. 5
  • Frozen-encoder probes predict targets from averaged patch features across 14 encoders, with object-grouped five-fold cross-validation and shuffled-label controls.For T5, probes use the fine-tuning objects and benchmark objects, comparing averaged features with an attention readout over patch tokens; T11–T13 use training-free candidate comparisons.Source: §4 · p. 5

5 Results and Diagnostic Analysis

Across these diagnostics, models can improve on individual judgments while exact answers remain unreliable. Probes and image-removal controls further test whether answers use available visual features and depend on image evidence.

5.1 Exact Counting and Consistent Use Remain Distinct Challenges

  • Humans reach 93.3% on T5: Color Distribution, versus 7.2% for the best VLM, the largest human–model gap across tasks.The best models exceed the human 95% confidence interval only on T2 and T6.Source: §5.1 · p. 5, Figure 2 · p. 5
  • Factorized queries leave average present-color exact-count accuracy near 33%, while Qwen3.5-27B rises from 3.0% to 52.4% EM when colors are shown as 2D tiles.Fine-tuning raises T5 to 68.6%, but T7 accuracy is 35.0% from images and 74.2% when given the model’s own counts as text.Source: §5.1 · p. 6

5.2 Test-time Reasoning Leaves Exact Answers Unreliable

  • Thinking raises T2 and T4 by up to 13.8 and 13.9 percentage points, respectively, but T5 remains at 0.7–1.0% and larger budgets can lower accuracy.In an exploratory audit, visual misinterpretations appeared in 42 of 44 incorrect traces.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

5.3 Scaling and Visual Support Leave Exact Answers Unreliable, Although Visual Features Hold the Counts

  • Scaling improves six Qwen3.5 tasks by 18–42 percentage points, while the other seven change by at most 6.4 points; visual overlays also leave T5 unreliable.On T5, coordinate axes reduce accuracy from 0.7% to 0.5%.Source: §5.3 · p. 7, Figure 5 · p. 8
  • With 800 training objects, probes recover Qwen3.5-27B per-color counts on 65.0% of objects at the language-model input, versus 3.0% for the VLM.Aggregate quantities are recoverable from frozen features, but the findings point to difficulty using those features consistently.Source: §5.3 · p. 8

5.4 Shortcut Controls Show Why Score Alone Is Insufficient

  • Without images, models still score 51% on T10 and 43% on T9 by always answering 0 and 3, respectively; a fixed third-option layout yields 100% for Gemma4-31B on T1.The controls show why scores need to be checked for image dependence.Source: §5.4 · pp. 8–9

6 Conclusion

BedBench finds that models can answer individual 3D judgments correctly while failing at exact answers and consistent use of recovered properties. The authors recommend evaluating exactness, consistency, and image dependence, while noting limits in scene realism, human coverage, and intervention scope.

6 Conclusion

  • The authors recommend measuring exact-answer correctness, agreement between related answers, and image dependence, and suggest training objectives that target visual information use and consistency.They also call for broader probes and evaluations in natural and embodied scenes.Source: §6 · p. 9

Limitations

  • BedBench uses synthetic voxel objects and geometric primitives, so extending the findings to real or embodied settings requires testing noisy textures, lighting variation, and complex geometry.The human reference comprises 30 participants; broader, more diverse cohorts could refine task-difficulty estimates and human variance.Source: Limitations · p. 9
  • Deeper probing and extensive fine-tuning were run on representative model families and selected tasks, not across the entire suite.The authors identify broader model families, training regimes, and embodied settings as targets for generalization.Source: Limitations · p. 9

Ethical Considerations

  • The human study used synthetic geometric renderings, collected coarse demographics and task responses, and released only anonymized verdicts and demographic fields.The sample is intended as a reference for task solvability, not a population estimate; the benchmark is for diagnosing behavior, not certifying deployment suitability.Source: Ethical Considerations · p. 9

Supplementary Material

The supplementary material provides task-specific appendices and an expanded related-work discussion. That discussion frames BedBench around task difficulty, answer consistency, visual dependence, and recovery of task-relevant quantities.

Supplementary Material

  • The supplementary material lists appendices for T5 through T13, including Color Distribution, Local Structure, Projection, Reconstruct, and Mixture-of-Shapes.The listed task appendices span pages 143–155.Source: Supplementary Material · p. 20

A Extended Related Work

  • The expanded related work connects benchmark design to four questions: which tasks remain difficult, whether related answers agree, whether answers depend on visual evidence, and which quantities visual features recover.It distinguishes human comparisons, consistency checks, shortcut controls, and probes as evidence for interpreting performance.Source: A Extended Related Work · p. 21

A.1 Benchmarks and Human References

BedBench situates its controlled geometry and diagnostic comparisons among spatial benchmarks, consistency checks, shortcut controls, representation probes, and training interventions. The supplied findings report strong average model performance relative to a constant-answer baseline but a large human–model gap on per-color counting.

A.1 Benchmarks and Human References

  • Geometric benchmarks test distinct operations, while broad spatial suites cover varied skills; BedBench separates total counts, per-color counts, part relations, and cross-view matching.Procedural generation provides matched cue variants and exact targets, but does not itself rule out answer priors or establish image use.Source: §A.1 · pp. 21–22

A.2 Consistency Across Related Questions

  • Related-question consistency checks reveal disagreements, while BedBench uses shared block-level ground truth to test whether per-color counts entail other answers and whether counts-as-text improves their use.Re-rendering the same blocks as 2D tiles tests whether the 3D presentation is the obstacle.Source: §A.2 · p. 22

A.3 Shortcut Controls and Answer Formats

  • Image-removal tests and matched answer formats help distinguish visual evidence from shortcut cues; the image-removal experiments cover three models, while the open-ended/MCQ comparison is limited to T1.The no-image MCQ result demonstrates that an apparent improvement can occur without counting depicted blocks.Source: §A.3 · pp. 22–23

A.4 Visual Representations and the Choice of Readout

  • Frozen-feature probes compare recoverable targets with end-to-end answers, but mean-pooled failures do not establish that patch tokens lack target information.On T5, attention and token-sum readouts recover exact counts that pooled probes miss; those readouts remain untested for other tasks.Source: §A.4 · p. 23

A.5 Reasoning, Scaling, and Training Interventions

  • Reasoning gains vary by task, matched visual cues yield modest color benefits and mostly negative axis effects, and targeted fine-tuning improves complete per-color answers while direct dominant-color answers often disagree.The fine-tuning result concerns one model trained on 800 in-distribution examples with its vision encoder frozen; it derives dominant colors correctly when given its own counts as text.Source: §A.5 · pp. 23–24

B.1 Human gap and per-color counting

  • The extended findings characterize per-color errors and compare factorized queries, 2D visibility, and task-specific fine-tuning, including whether direct answers agree with each model’s counts.The section’s framing is that individual judgments may be right while exact answers remain wrong and recovered properties may not be used consistently.Source: B Extended Findings and Detailed… · p. 25, §B.1 · pp. 25–26

B Extended Findings and Detailed Analyses

  • Humans average 70.0%, while the strongest model reaches 51.5%, 14.8 pp above the best-constant average of 36.7%; on T5, humans reach 93.3% and no model exceeds 7.2%.The extended findings also describe comparisons of human references, error magnitudes, and controls on T5.Source: Table 2 · p. 25, §B.1 · pp. 25–26

B.1.1 VLM-human gap concentrates on exact structure and cross-view tasks

The human–VLM gap is concentrated in tasks requiring full structure, especially per-color counting, rather than distributed uniformly across task difficulty. Error analyses further show that exact palette answers can fail even when average per-color errors appear modest.

B.1.1 VLM-human gap concentrates on exact structure and cross-view tasks

  • On T5, humans exceed the best OW cell by +86.1 pp and the best frontier cell by +87.3 pp, the largest separations; humans also lead on T1, T7, T11, and T12.Humans reach 93.3% on T5, compared with a best overall model score of 7.2%; VLMs exceed the human interval only on binary-decision tasks T2 and T6.Source: §B.1.1 · p. 26, Table 3 · p. 26

B.1.2 Floor accuracy hides two failure depths 

  • Human and VLM accuracy show no detectable task-level association, but uncertainty is wide; T5, T7, and T1 account for 59% of the total positive human–VLM gap.On present T5 colors, Non-Thinking MAE is 1.02 blocks against a mean true count of 1.48, yet only 32.5% are counted exactly; palette-vector exact match is 0.68%.Source: §B.1.1 · p. 26, §B.1.2 · pp. 26–27

B.1.3 Factorized querying does not consistently improve fine-grained color counting

Factorized color queries barely change present-color accuracy and can worsen exact full-palette answers by introducing false counts for absent colors. A 2D tile control yields much better counting than 3D renders, while a benchmark-matched example improves counts without making exact answers reliable.

B.1.3 Factorized querying does not consistently improve fine-grained color counting

  • Present-color accuracy changes only from 32.52% under joint prediction to 33.16% under factorized querying (+0.64 pp), with five models improving and seven declining.Model-level changes range from +26.06 pp for Qwen3.5-0.8B to −32.58 pp for Gemma4-E4B.Source: §B.1.3 · p. 27, Table 5 · p. 28
  • For present colors, subset EM falls from 31.88% (joint) and 32.72% (factorized) at k = 1 to 3.71% and 4.17% at k = 4.Over the full palette, EM falls from 48.98% to 8.35% (joint) and from 39.13% to 3.27% (factorized) between k = 1 and k = 4.Source: §B.1.3 · p. 27, Figure 6 · p. 28
  • At k = 12, full-palette EM is 0.683% for joint prediction versus 0.0% for factorized querying; factorization also cuts absent-color zero-count accuracy from 78.02% to 49.66%.Merged factorized vectors reach 0.0% full-distribution EM for every model.Source: §B.1.3 · p. 27, Figure 6 · p. 28
  • A benchmark-matched example raises Qwen3.5-27B present-color accuracy by +10.6 pp, but full-distribution EM reaches only 5.0%.The EM change is +2.0 pp, with a 95% CI that includes zero.Source: §B.1.3 · p. 27, Table 6 · p. 28

B.1.4 A 2D visibility control separates basic enumeration from 3D presentation

  • With every block shown as a separate tile, Qwen3.5-27B counts present colors exactly 91.0% of the time, versus 6.0% in 3D after merging gold and yellow diagnostically.The control changes occlusion, two-view integration, and visual layout together, and covers one model.Source: §B.1.4 · pp. 28–29, Table 7 · p. 29
  • Fine-tuning raises T5 exact match from 3.0% to 68.6% on the benchmark palette.The adapter uses 800 generated training instances and is evaluated on 100 benchmark instances.Source: §B.1.3 · p. 27, Table 8 · p. 29

B.1.5 Task-specific fine-tuning substantially improves counting but yields incomplete cross-task transfer

Task-specific fine-tuning substantially improves exact color counts, but direct answers about dominant colors often disagree with the model’s own counts. Across reasoning-mode comparisons and trace audits, Thinking helps selected tasks while perceptual misreadings often persist through further reasoning.

B.1.5 Task-specific fine-tuning substantially improves counting but yields incomplete cross-task transfer

  • Fine-tuning raises T5 exact match from 3.0% to 68.6% on the trained color setting and to 66.8% on color + axes, but the best cell remains below the human mean of 93.3%.Training uses 800 generated T5 instances, with the vision encoder and connector frozen; the probe on connector output reaches 65.0% exact match.Source: §B.1.5 · pp. 29–30
  • The fine-tuned model’s own T5 counts imply correct dominant colors 74.6% of the time, while its direct T7 answers are correct only 35.0% of the time.For objects with exactly correct T5 answers, direct T7 accuracy is 40.8%; gains on related tasks are partial.Source: §B.1.5 · p. 30, Table 10 · p. 30
  • Given its own counts as text, the model answers T7 correctly 74.2% of the time, versus 35.0% from images; with ground-truth counts, accuracy is 99.0%.When T5 is exactly correct, text-conditioned accuracy is 99.4% against 40.8% from images.Source: §B.1.5 · p. 30
  • Across 12 open-weight models, agreement between direct T7 answers and each model’s own count-implied dominant colors is 21.6%, versus 2.4% chance; direct T1 answers match count sums only 4.0%, below 5.4% chance.Even among 41 exactly correct open-weight T5 answers, a consistent T7 answer follows only 61.0% of the time.Source: §B.1.5 · pp. 30–31, Table 11 · p. 30

B.2 Thinking helps selected tasks but leaves exact answers unreliable

  • Thinking helps four tasks, hurts three, and ties six at the 2 pp threshold; its mean accuracy change across tasks is only +0.9 pp.The largest gains are on T2 and T4 (+13.8 and +13.9 pp), while the largest drops are on T1 and T9 (−8.9 and −7.4 pp).Source: Table 12 · p. 32, §B.2.1 · p. 32

B.2.1 When thinking helps and what the traces reveal

  • Good Perception appears in 16 of 21 correct traces but only 3 of 44 incorrect traces; on T2 and T4, Thinking traces include grounded intermediate steps.The audit covers 457 tagged spans across 65 traces.Source: §B.2.1 · p. 33
  • Wrong Perception appears in 42 of 44 incorrect traces, and open-weight traces often repeat earlier steps rather than re-inspecting the image.The audit is descriptive because exposed reasoning traces need not reflect the basis of an answer.Source: §B.2.1 · p. 33

B.2.2 Larger reasoning budgets do not recover the affected tasks

Larger reasoning budgets help T4: Color Count but do not recover T1: Volume or T11: Projection in the tested models. The results suggest that additional tokens do not reliably correct inaccurate scene estimates under the tested tasks and budgets.

B.2.2 Larger reasoning budgets do not recover the affected tasks

  • T4: Color Count rises from 49.0% at budget 0 to 59.2% at 2048 tokens (+10.2 pp), then levels off at 58.6% with 4096 tokens.Gemma4-31B continues improving from 52.8% to 60.0%.Source: §B.2.2 · p. 33
  • T1: Volume drops from 37.4% at budget 0 to 20.7% at 1024 tokens and remains near 20% through 4096 tokens (−17.6 pp).On T11: Projection, the two-model average falls −5.7 pp from budget 0 to 4096; budgets of 2048 or more leave T11 below budget 0 on average.Source: §B.2.2 · pp. 33–34
  • The budget findings are limited to two open-weight models, three selected tasks, and the tested token budgets.The authors suggest additional tokens do not reliably correct inaccurate scene estimates in this setting.Source: §B.2.2 · p. 34

B.2.3 Thinking shifts decision behavior on binary tasks 

Across model families, scale, and visual cues, some individual judgments improve while exact answers remain unreliable. Probes show that frozen visual features can contain recoverable information even when models fail to report complete answers.

B.2.3 Thinking shifts decision behavior on binary tasks 

  • On T8, Thinking lowers accuracy by 2.9 pp while raising precision from 0.383 to 0.543 and F1 from 0.398 to 0.430; recall stays essentially unchanged.On T2, T6, and T10, recall rises more than precision; these output changes do not establish recovery of 3D structure.Source: §B.2.3 · p. 34, Table 14 · p. 34

B.3.1 VLMs fail similarly across independently trained families 

  • Both independently trained families remain near floor on T3 and T5 and at best-constant levels on T8 and T13, despite strengths on different tasks.The families share isotropic, full-attention SigLIP 2-family vision encoders, although their language backbones and adapters differ.Source: §B.3.1 · pp. 36–37, Table 16 · p. 36

B.3.2 Scaling does not make exact answers reliable

  • For Qwen3.5, scaling raises accuracy by +42.0 pp on T6 and +39.2 pp on T2 and T4, while T3, T5, T8, and T13 stay within 3 pp of zero.The scaling split largely generalizes across Gemma4 and Thinking mode, with a ceiling effect on T6 and a reversal on T1.Source: §B.3.2 · p. 37, Figure 7 · p. 38

B.4.1 VLMs show modest gains from color add-ons 

  • Color add-ons raise accuracy on five tasks, most on T12: Reconstruct (+6.8 pp), while the T13 change is uncertain.The comparisons do not establish the mechanism behind the task-dependent gains.Source: §B.4.1 · p. 39, Table 18 · p. 39

B.4.2 VLMs do not reliably benefit from axes add-ons 

  • Axes lower mean accuracy on 10 of 11 tasks, most on T2: Packaging (−4.1 pp); only T11: Projection improves marginally (+0.9 pp).The proposed explanations—non-egocentric reference-frame difficulty and visual clutter—were not isolated by the comparisons.Source: Table 18 · p. 39, §B.4.2 · p. 40

B.5 Probing the visual front-end on ground truth

  • Probes of 14 frozen encoders recover several quantity and local-structure targets, including T1 Spearman ρ of 0.44–0.73 and T6 AUC 0.99.The pooled-feature probes use object-grouped cross-validation and shuffled-label controls.Source: §B.5.1 · p. 41, §B.5 · p. 41, Table 19 · p. 41

B.5.1 Pooled probes recover quantities but rarely exact decompositions

  • On T5, pooled probes trained on 100 objects decode per-color counts above the shuffled-label reference but rarely recover complete vectors (exact match ≤3.5%).Depth-Anything-V2-L improves T1 volume recovery and T11 candidate separation, but T5 exact match remains 0.0%.Source: §B.5.1 · pp. 42–43
  • Color raises cross-view retrieval in every encoder except Depth-Anything-V2-L, while axes depress retrieval in every encoder.Gains reach +43 pp, with Gemma4 top-1 retrieval at 0.77 against a 0.01 random baseline.Source: §B.5.1 · p. 43

B.5.2 With more training data, token-level probes recover exact per-color counts

With more training data and token-level readouts, probes recover exact per-color counts from frozen features, including at Qwen3.5-27B’s language-model input. The appendix’s image-removal and fixed-layout controls show that some apparent model accuracy can persist without visual evidence.

B.5.2 With more training data, token-level probes recover exact per-color counts

  • On Qwen3.5-27B, token-sum probes recover T5 exact counts on 77.0% of objects versus 3.0% for the VLM; at the connector, attention reaches 65.0%.Gemma4-31B attention reaches 44.7%, versus 4.0% for its VLM; the best tested readout differs by tower.Source: §B.5.2 · pp. 43–44
  • With 200, 400, and 800 training objects, attention exact match rises from 14.7% to 61.7% on Qwen’s tower and from 28.0% to 65.0% at its connector.All attention learning curves are still rising at 800 objects.Source: §B.5.2 · p. 44

B.5.3 Probe scores stay flat with tower size while VLM use improves 

  • Across both model families, pooled quantity-probe scores remain nearly flat with tower size, while Qwen’s task accuracy improves on several quantity tasks.For T11, every Gemma4 tower favors the task rule (+0.39), while every Qwen3.5 tower favors its reverse (−0.40 to −0.60); T12 and T13 rankings stay near chance.Source: §B.5.3 · pp. 44–45, Figure 9 · p. 44

B.6.1 VLMs retain scores without images by collapsing to constant predictions 

  • Without images, accuracy drops by 13–43 pp on several tasks, but T8 is unchanged for all three models and T3, T5, and T13 remain near their no-image scores.Image dependence varies by model, including GLM-4.6V-Flash gains on T1 open-ended volume and T12 reconstruction.Source: §B.6.1 · p. 46, Table 21 · p. 46
  • Many surviving no-image scores come from constant predictions, including fixed answers on every T9 and T10 instance that match frequent labels.Without images, Thinking changes many scores only slightly and lowers the T1 MCQ score by 25.7 pp.Source: §B.6.1 · pp. 46–47, Table 22 · p. 47, Table 23 · p. 48

B.6.2 VLMs may appear stronger under multiple choice 

  • On T1, fixed-layout multiple choice raises accuracy over open-ended answers by +39.0 pp on average, and Gemma4-31B reaches 100% without images by always choosing the third option.The correct volume is always the third of five options; the result reflects an answer-position cue rather than added visual information.Source: Figure 10 · p. 47, §B.6.2 · p. 48

C.1 Dataset Generation Details

BedBench builds controlled tasks from generated voxel objects and multi-object scenes, with filters and rendering variants designed to support interpretable 3D evaluations. A post-hoc audit identifies one nearly occluded block whose color counts cannot be recovered from the images, while excluding its object changes reported results only slightly.

C.1 Dataset Generation Details

  • Voxel objects contain 10–15 connected, supported unit blocks, and two oblique query views are selected with minimum footprint and aspect-ratio checks.The generator derives counts, extents, and spatial relations from the explicit geometry.Source: §C.1 · pp. 49–50
  • Generation rejects sampled two-view silhouette collisions when an alternative object matches both masks at IoU ≥0.995 but differs in canonical projections.A separate visibility filter rejects blocks whose removal barely changes both rendered masks.Source: §C.1 · p. 50
  • Multi-object scenes contain 6–8 primitives and pass visibility checks across query and canonical views before use.Projection distractors are generated through edits, while task-compatible render variants manipulate color and coordinate axes without changing geometry.Source: §C.1 · pp. 50–51
  • Excluding the audited object changes each of 768 open-weight accuracy cells by at most 0.97 pp and frontier accuracy on any affected task by at most 1.0 pp.The object contains a nearly occluded block whose per-color counts cannot be recovered from the images.Source: §C.1 · p. 53

C.2 General Experiment Setup

  • Open-weight conditions are repeated five times, while frontier models are evaluated once in Thinking mode; scoring uses exact-match accuracy and task-specific secondary metrics.Open-weight runs can vary because of inference nondeterminism, and paired item-level bootstrap intervals measure uncertainty over benchmark items.Source: §C.2 · pp. 53–54

C.3 Model Details

  • The evaluated roster contains 12 open-weight checkpoints and four provider-hosted models, with Kimi-VL-A3B using separate checkpoints for the two modes.For mixture-of-experts models, reported parameter metadata distinguishes total and active counts where available.Source: §C.3 · p. 54

C.4 Human Baseline Details

  • Thirty volunteers answered one randomly drawn question per task, yielding a mean accuracy of 70.0% across the 13 tasks.Task-level proportions use 95% Wilson confidence intervals; participants took a mean 1234.1 seconds overall.Source: §C.4 · p. 55

D Dataset Examples

The dataset examples define BedBench’s thirteen voxel and scene-reasoning tasks and show their default inputs and model answers, alongside instance-level comparisons across visual-cue, scaling, reasoning, and thinking conditions. Panels pair answers with reasoning traces or summaries, illustrating task-specific answer changes, unchanged or regressing responses, and traces that revise or conflict with interpretations of 3D structure.

D.1 T1: Volume; D.2 T2: Packaging

  • T1 counts occupied voxel positions across two views, while T2 tests whether an object fits a container after axis-aligned reorientation.T1 counts each block once, including blocks visible in only one view; T2 compares the object’s three extents with container dimensions under any axis permutation.Source: §D.1 · p. 57, §D.2 · p. 58

D.3 T3: Difference

  • T3 partitions two objects’ occupied positions into shared and object-unique blocks, with a color-aware variant requiring shared blocks to match in color.Correspondence uses identical generator-frame positions without independently translating or rotating the objects to maximize overlap.Source: §D.3 · p. 59

D.4 T4: Color Count; D.5 T5: Color Distribution

  • T4 counts distinct colors, whereas T5 requires a dictionary of every present color and its block count, making T5 the clearest VLM-specific failure case.T5 requires exact color keys and counts for exact match; omitted palette categories count as zero in its twelve-dimensional per-color error vectors.Source: §D.4 · p. 60, §D.5 · p. 61

D.6 T6: Local Structure; D.7 T7: Color Dominance

  • T6 asks whether any vertical column exceeds three blocks, while T7 identifies all colors tied for the greatest block count.T6 treats a column of exactly three blocks as not exceeding the threshold; T7 requires the exact set of dominant colors, regardless of output order.Source: §D.6 · p. 62, §D.7 · p. 63

D.8 T8: Silhouette Symmetry; D.9 T9: Visible Floors; D.10 T10: Color Connected

  • T8 tests mirror symmetry in canonical projections, T9 counts occupied height levels, and T10 asks whether any face-adjacent blocks share a color.T8 checks horizontal or vertical reflection in XY, YZ, or ZX projections; T10 excludes edge and corner contact.Source: §D.8 · p. 64, §D.9 · p. 65, §D.10 · p. 66

D.11 T11: Projection; D.12 T12: Reconstruct; D.13 T13: Mixture-of-Shapes

  • T11 selects a valid or invalid candidate projection, T12 selects the view rendered from supplied projections, and T13 selects a candidate projection for a multi-shape scene.T11 distractors edit projection cells, T12’s other view comes from an object with a block count differing by at most two, and T13 distractors modify 3D scenes.Source: §D.11 · p. 67, §D.12 · p. 68, §D.13 · p. 69

E.1 Visual-Cue Add-Ons: Instance-Level Evidence

  • Instance panels compare color absent versus present for T1, T3, T11, T12, and T13; their captions describe color as likely to help.The panels pair renderings of the same sample under the two conditions.Source: §E.1 · p. 70, Table 64 · p. 71, Table 65 · p. 71
  • Axes-add-on panels compare the same T1, T2, T4, and T5 instances with and without coordinate axes; their captions describe axes as likely to hurt.These panels provide qualitative instance-level evidence alongside the appendix’s aggregate visual-cue analyses.Source: §E.1 · p. 70, Table 69 · p. 73, Table 70 · p. 74

T5 Color Distribution

  • Axes-add-on qualitative panels compare the same sample with and without coordinate axes and suggest axes are likely to hurt across several tasks.The panels cover Local Structure, Color Dominance, Silhouette Symmetry, Visible Floors, Color Connected, and Projection.Source: Table 73 · p. 76, Table 74 · p. 76, Table 75 · p. 77

E.2 Parameter Scaling: Instance-Level Evidence

  • The panels classify all 13 tasks as scaling-responsive or scaling-resistant based on Qwen3.5 scaling curves, across both Qwen3.5 and Gemma4 model ladders.Responsive: T2, T4, T6, T7, T9, and T11; resistant: T1, T3, T5, T8, T10, T12, and T13.Source: §E.2 · p. 80

T2 Packaging – scaling-responsive

  • On scaling-responsive instances, both families reach the correct answers at larger checkpoints for T2, T4, T7, T9, and T11; Qwen3.5 flips to correct at 4B on T6.The T7 answer converges on {cyan, yellow}; T9 changes from 3 to 4; and T11 changes from candidate 2 to 1.Source: Table 80 · p. 81, Table 81 · p. 81, Table 82 · p. 82

T1 Volume – scaling-resistant for Qwen3.5, scaling-responsive for Gemma4

  • Qwen3.5’s T1 accuracy rises only +6.4pp from 0.8B to 27B, while Gemma4’s rises +26.8pp from E2B to 31B.On the instance, Qwen3.5 gives the same wrong value, 12, at five of six sizes; Gemma4 moves from wrong E2B = 14 to correct E4B = 11 and 31B = 11, with a wrong 26B-A4B = 13 dip.Source: T1 Volume – scaling-resistant for… · p. 83, Table 86 · p. 83

T3 Difference – scaling-resistant

  • Scaling does not recover the T3 difference triple, exact T5 color counts, or T8 symmetry: every tested size misses the T3 triple and T5 counts, while all answer False on symmetric T8.The T3 ground truth is (2, 8, 8), and every T8 response defaults to False without detecting the mirrored columns.Source: Table 87 · p. 84, Table 88 · p. 84, Table 89 · p. 84

T10 Color Connected – scaling-resistant

  • T10 and T12 remain wrong across the Qwen3.5 ladder or both families, while all sizes answer T13 correctly; T10 Gemma4 regresses at 31B.T12 always selects candidate 1 rather than correct 0; T13 consistently selects correct candidate 2.Source: Table 90 · p. 85, Table 91 · p. 85, Table 92 · p. 85

E.3 Thinking Budget: Instance-Level Evidence

  • Thinking-budget effects differ by task: T1 answers stay correct only without thinking, T4 reaches the exact count as budget grows, and T11 models move in opposite directions.At 4096 tokens, the T4 trace uses an explicit coordinate lattice; Qwen3.5 stays correct on T11 through 2048 tokens but becomes wrong at 4096, while Gemma4 becomes correct from 2048 onward.Source: §E.3 · p. 87, Table 93 · p. 88, Table 94 · p. 89

E.4 Thinking Mode Flips: Instance-Level Evidence

  • Thinking-mode outcomes also vary: T2 and T4 improve, T1 and T8 worsen, and T3 remains tied with both modes wrong.On the shown instances, T2 changes from 0 of 4 correct in Non-Thinking to 4 of 4 in Thinking; T4 changes from 0 of 4 to 3 of 4, while T1 changes from 3 of 4 to 0 of 4 and T8 from 4 of 4 to 2 of 4.Source: Table 96 · p. 92, Table 97 · p. 93, Table 98 · p. 93
  • For T12, Non-Thinking gets 3 of 4 answers correct versus Thinking’s 1; for T13, the counts are 4 versus 2, with both task verdicts ties favoring Non-Thinking slightly.Two models flip from correct to wrong in each panel.Source: Table 107 · p. 99, §E.4 · p. 99, Table 108 · p. 100

E.5 Thinking Traces: Instance-Level Evidence

  • The panels show one representative sample per task and all eight models’ parsed answers, alongside a subset of annotated traces or meta-reasoning summaries.Open-weight traces are raw token-level rollouts; frontier-model summaries were manually collected when endpoints did not expose traces.Source: §E.5 · p. 101

E.5.1 T1 Volume

  • In T1 Volume, the trace gives a 15-block reconstruction, then questions it and suggests 12 while struggling to settle on the requested answer format.Its account of the structure and total remains uncertain.Source: §E.5.1 · p. 103

E.5.2 T2 Packaging

  • In T2 Packaging, Gemini-3-Flash answers True, and the trace concludes dimensions [2, 4, 4] fit within boundary [2, 5, 6].It checks the sorted dimensions against the boundary constraints.Source: §E.5.2 · p. 105

E.5.3 T3 Difference

  • In T3 Difference, the trace repeatedly revises its coordinate assignments and axes while trying to reconstruct the first structure.The excerpts show reconsiderations rather than a settled comparison result.Source: §E.5.3 · p. 106

E.5.4 T4 Color Count

  • In T4 Color Count, the trace lists colors across the object but records wrong perceptions and color confusions, including in its descriptions of blue, yellow, green, and purple cubes.The supplied excerpts show revisions and inconsistent color identification rather than a final count.Source: §E.5.4 · pp. 106–107

E.5.5 T5 Color Distribution

  • In T5 Color Distribution, the trace predicts 15 blocks: red 2, gold 1, cyan 2, magenta 3, green 1, yellow 2, slategray 2, and teal 2.The trace also notes that its estimates varied across attempts.Source: §E.5.5 · p. 109

E.5.6 T6 Local Structure

  • In T6 Local Structure, Qwen3.5-27B answers True to whether any column exceeds height 3, while its trace describes a visible stack of three blocks.The trace pauses to recount the stack but does not describe one taller than three.Source: §E.5.6 · p. 109
  • The T6 trace misreads the front column’s height and reconstructs the object’s local shape inconsistently.It calls the front column three blocks high and later describes a plus- or T-shape atop a pillar; both are marked as wrong perception.Source: §E.5.6 · p. 110

E.5.7 T7 Color Dominance

  • The T7 trace names teal, slategray, orange, and orchid as tied dominant colors, each with a count of 2 blocks.Source: §E.5.7 · p. 111

E.5.8 T8 Silhouette Symmetry

  • The T8 trace answers false, judging both views’ silhouettes asymmetric, but its answer is marked as wrong perception.Source: §E.5.8 · p. 111

E.5.9 T9 Visible Floors

  • The T9 trace repeatedly settles on 4 visible layers by counting blocks in vertical stacks, with its layer judgments marked as wrong perception.The trace alternates between reconsidering the count and treating each distinct stack height as a layer.Source: §E.5.9 · p. 112

E.5.10 T10 Color Connected

This section documents the setup and evaluation materials for 3D block-reasoning tasks, including T10’s judgment of whether face-adjacent blocks share a color from two perspective views. It also covers reasoning traces, prompt variants, task procedures, ablations, and metric and reporting conventions.

E.5.10 T10 Color Connected

  • For T10 Color Connected, the trace concludes False after correcting an initial mistaken claim that same-color blocks were adjacent.It checks the apparent green and yellow pairs against their positions and concludes they are not adjacent.Source: §E.5.10 · pp. 113–114

E.5.11 T11 Projection

  • For T11 Projection, the trace selects candidate 0, reasoning that its 2×2 projection is too small for the object's extent.The passage also notes that the example answer is candidate 2, but does not establish that as the correct answer.Source: §E.5.11 · p. 115

E.5.12 T12 Reconstruct

  • For T12 Reconstruct, the trace selects candidate 1 because its structure matches the silhouettes, while candidate 0 does not.The cited reasoning points to a three-square-wide base and tall middle column as matching features.Source: §E.5.12 · p. 116

F Prompts

  • The prompt appendix says Thinking prompts make intermediate steps explicit, whereas Non-Thinking uses concise prompts.Response-format examples only illustrate output structure, not worked demonstrations; their values are randomly chosen, and predictions are scored against independently generated ground truth.Source: F Prompts · p. 118

F.3 T2: Packaging

  • The packaging task asks whether a 3D object fits inside a rectangular boundary in at least one orientation.It compares all permutations of the object’s dimensions with the boundary dimensions.Source: §F.3 · p. 120

F.4 T3: Difference

  • The difference task reconstructs two objects from two views each and counts shared blocks and blocks unique to either object.The prompts call for inferring occluded blocks and avoiding double-counting blocks visible in both views.Source: §F.4 · p. 121

F.5 T3: Difference – Color-Aware Variant

  • The color-aware difference variant counts blocks as shared only when both their structure and color match.It otherwise compares the two reconstructed objects’ block compositions as in the standard difference task.Source: §F.5 · p. 122

F.6 T4: Color Count

  • The color-count task asks for the number of distinct colors across two views, using a predefined palette and excluding shading or lighting variation.The prompt directs the model to account for colors visible in only one view because of occlusion.Source: §F.6 · p. 123

F.7 T5: Color Distribution

  • The color-distribution task requires counting blocks of each color in the reconstructed object, including inferred hidden blocks without double-counting.It uses the predefined color set and excludes shading or lighting variation as a separate color.Source: §F.7 · p. 124

F.8 T5: Color Distribution (Ablation prompts)

  • The color-distribution ablations ask models to count a specified color or report the full distribution from two object views, with shading and lighting variations excluded as separate colors.A separate tile-image prompt asks for the distribution across colored square tiles.Source: §F.8 · pp. 125–126

F.9 T6: Local Structure

  • The local-structure task asks whether any vertical column of blocks has height greater than a specified threshold.Both views are used to infer column heights, including occluded blocks.Source: §F.9 · p. 126

F.10 T7: Color Dominance

  • The color-dominance task identifies the color or colors with the highest block count, treating ties as jointly dominant.The prompts define both an image-based version that counts blocks per color and a version that supplies those counts directly.Source: §F.10 · pp. 127–128

F.11 T8: Silhouette Symmetry

  • T8 asks whether either silhouette is vertically or horizontally symmetric, checking mirrored block positions across both views.Only physically present blocks count; shading and lighting variations are excluded.Source: §F.11 · pp. 128–129

F.12 T9: Visible Floors

  • T9 asks for the number of occupied vertical layers, combining both views to account for occluded layers.Each distinct height with at least one block counts as a layer.Source: §F.12 · p. 129

F.13 T10: Color Connected

  • T10 asks whether any face-sharing blocks have the same color, using only predefined colors and excluding shading or lighting variations.The instructions call for reconstructing block positions from both views and checking adjacency along the X, Y, or Z direction, not diagonal contact.Source: §F.13 · p. 130

F.14 T11: Projection – Identify Invalid Orthographic Projection

  • T11 invalid-projection asks which of three candidate orthographic views contradicts the 3D geometry inferred from two perspective views.Candidates represent the XY, YZ, and ZX planes in randomized order and should be cross-checked for incompatible features or proportions.Source: §F.14 · p. 131

F.15 T11: Projection – Identify Valid Orthographic Projection

  • T11 valid-projection asks which of three candidate orthographic views matches the object reconstructed from two perspective views.The candidates correspond to the XY, YZ, and ZX planes in randomized order and must align with one another.Source: §F.15 · p. 132

F.16 T12: Reconstruct

  • T12 asks which of two candidate views matches an object represented by three silhouettes, checking consistency of geometry, features, and proportions.The other candidate depicts a different object.Source: §F.16 · p. 133

F.17 T13: Mixture-of-Shapes – Identify Invalid Silhouette

  • T13 invalid-silhouette asks which of three projections does not match a multi-object scene shown in two perspective views.The invalid candidate reflects a scene mutation, such as moving, swapping, deleting, or resizing an object.Source: §F.17 · p. 134

F.18 T13: Mixture-of-Shapes – Identify Valid Silhouette

  • T13 valid-silhouette asks which of three candidate silhouettes belongs to the multi-object scene shown in two perspective views.Only one candidate is valid; the other two do not belong to the scene.Source: §F.18 · p. 135
  • The valid-silhouette task presents two perspective views and three candidate orthographic silhouettes, asking the model to select the projection matching the exact scene rather than a scene with an altered object.The instructions direct models to infer the 3D layout and check object count, shape, size, positions, and spatial consistency across views.Source: §F.18 · p. 135

G Comprehensive Visual-Condition Tables

  • The comprehensive tables report accuracy, RMSE, and MAE for numeric tasks and accuracy, precision, recall, and F1 for other classification tasks; ± values are standard deviations across five inference runs, not item-level confidence intervals.Paired item-level bootstrap intervals for table comparisons are reported in Tables 18 and 12.Source: G Comprehensive Visual-Condition Tables · p. 137

G.1 T1: Volume (Open-Ended)

The appendix provides per-model metric tables across visual variants for BedBench tasks, followed by a disclosure of how AI assistants were used.

G.1 T1: Volume (Open-Ended)

  • Table 122 reports T1 results by visual variant, model, and inference mode, including accuracy, RMSE, and MAE.It gives five-run means ± standard deviations; frontier rows cover only the default visual variant in Thinking mode and report a single run.Source: §G.1 · p. 138, Table 122 · p. 139

G.2 T2: Packaging–G.7 T7: Color Dominance

  • Tables 123–128 report per-model results across visual variants for T2–T7, with five-run means ± standard deviations and single-run frontier rows for the default visual variant in Thinking mode.The reported metrics include accuracy; some tables also report RMSE and MAE, while others report precision, recall, and F1.Source: Table 123 · p. 140, Table 124 · p. 142, Table 125 · p. 143

G.8 T8: Silhouette Symmetry–G.13 T13: Mixture-of-Shapes

  • Tables 129–134 report per-model results across visual variants for T8–T13, with five-run means ± standard deviations and single-run frontier rows for the default visual variant in Thinking mode.The tables report accuracy, and some also include precision, recall, and F1.Source: Table 129 · p. 148, Table 130 · p. 149, Table 131 · p. 150

H Use of AI Assistants

  • The authors used AI assistants for coding, writing revisions, and literature searches, while retaining responsibility for study design, checks, and paper content.They state that every reported number was recomputed by scripts from saved model outputs.Source: H Use of AI Assistants · p. 158

Related papers

Loading 2610.39639v1…