Source-linked AI summary
PolyComp: A Polycube-based Benchmark for Compositional 3D Spatial Reasoning in Multimodal Models
Siddharth Patel
TL;DR
Multimodal models’ ability to perform compositional 3D spatial reasoning remains unclear. PolyComp introduces a procedurally generated, exactly verified polycube benchmark and compares three frontier models across multiple presentations, revealing substantial performance differences.
Problem
Whether multimodal models build internal representations supporting geometric perception, rigid transformations, and compositional 3D spatial reasoning remains unclear.
Method
PolyComp procedurally generates and exactly verifies polycube assembly problems, presenting target solids and four candidate component pairs across varied views.
Results
The paper compares three frontier multimodal models on 120 problems across three presentations, reporting accuracy and cost efficiency.
Takeaways & Limitations
PolyComp identifies a constrained gap in multimodal spatial intelligence and provides a method for generating verifiable problems involving polycube geometry, rotation, and composition.
Takeaways & Limitations
The study does not include a formal human-performance baseline, although every problem was verified as rendered.
Abstract
from arXiv · showhide
We introduce PolyComp, a procedurally generated and verified benchmark that stresses visual recognition and compositional spatial reasoning. In each problem, a model must identify which of four options shows a pair of polycube components that can be combined to form a target solid. The benchmark contains 120 problems across four geometry families, and each problem has three different presentation formats using either a single image or multiple images. The random guessing baseline is 25%. Across the three presentations (360 presented problems per model), GPT-5.6 Sol with max effort attains 50.0% accuracy (95% problem-cluster CI 43.3-56.7%) at a mean cost of \$0.951 per presented problem, Claude Fable 5 with max effort attains 39.4% (33.1-46.1%) at \$0.701, and Gemini 3.1 Pro Preview with thinking level high attains 27.5% (22.8-32.5%), near the 25% random guessing baseline, at \$0.350. The observed accuracy spread across geometry families is larger than across presentation formats. We present a problem development and evaluation protocol, cost and token accounting, and release the 120 problems.
1 Introduction
PolyComp addresses persistent difficulty in multimodal spatial reasoning by targeting geometric object perception, rigid-body transformation, and composition with discrete polycube geometry. The paper introduces a validated generation procedure, compares three frontier multimodal models on 120 problems across three presentations, and releases the problems.
- Motivation: Spatial intelligence spans perceptual abilities such as segmentation and identification and reasoning abilities such as viewpoint reconciliation and geometric transformation.It remains unclear whether LLM-based models can achieve human-level performance on visual tasks.
- Motivation: Modern vision-language models remain challenged by mental rotation and qualitative spatial reasoning despite strong performance on broad multimodal evaluations.
- Benchmark scope: PolyComp targets geometric object perception, rigid-body transformation, and composition using discrete polycube geometry.The task can be solved by identifying a Cartesian frame and counting blocks along three directions.
- Contributions: The paper presents a procedure to generate and validate a polycube decomposition problem.
- Contributions: The study compares three frontier multimodal models on 120 problems across three presentations, including accuracy and cost efficiency.
- Contributions: The paper releases the 120 problems themselves.
2 Related Work
PolyComp builds on multimodal benchmarks for deliberate visual reasoning, compositionality, spatial understanding, and explicit 3D scenes. It distinguishes itself through block geometry, multiple-choice responses, and three presentations of each problem to address robustness concerns.
- Multimodal reasoning benchmarks: PolyComp uses multiple-choice responses and block geometry, contrasting with MM-Vet’s integrated capabilities and open-ended grading.MMMU and MathVista evaluate deliberate reasoning over heterogeneous visual materials, while MMStar and MMMU-Pro address text-only shortcuts, data leakage, and presentation robustness.
- Multimodal reasoning benchmarks: Three presentations of the same problem motivate PolyComp’s evaluation of robustness to presentation changes.This design responds to concerns explicitly addressed by MMStar and MMMU-Pro.
- Spatial reasoning benchmarks: Prior spatial benchmarks span multi-view reasoning, 2D and 3D relations, spatial memory, instruction formats, interactive planning, and perception-to-...BLINK, CV-Bench, VSI-Bench, InternSpatial-Bench, and iVISPAR exemplify these directions.
- Synthetic, compositional, and 3D benchmarks: Synthetic and compositional benchmarks include CLEVR, GQA, and NLVR2, while SpatialSense, VSR, and Winoground expose persistent failures on relations, reference frames, and minimal compositional contrasts.ScanQA, SQA3D, and 3D-LLM ground questions and language models in explicit 3D scenes.
3 The PolyComp Benchmark
PolyComp is a verified benchmark of 120 polycube decomposition problems, requiring models to identify which option’s two components can form a target solid under rigid transformations. It varies geometry families and presentation formats, while using a selected challenge set rather than a representative sample.
- Problem task: Each problem presents a target polycube in two views and four candidate pairs, with exactly one pair whose union can match the target after independent rotations and translations.The model returns one label from A, B, C, or D.
- Generation and validation: Instances use integer-coordinate voxel sets, with incorrect options created by moving a cube or mirroring a component while preserving cube count and violating assembly.A verification routine enumerates the 24 orientation-preserving rigid transformations.
- Benchmark composition and presentation: 120 problems span 65 rectangular-loop, 24 toy-inspired, 14 block-cleavage, and 17 joinery cases, each rendered in three presentation formats.The formats are a single image, multi-image inputs with labels Image 1 through Image 5, and the same multi-image inputs with descriptive labels Target and Option A through Option D.
- Dataset limitation: The 120 problems are a selected challenge set consisting of 100 problems previously missed by GPT-5.5 and 20 difficult problems it answered correctly, rather than a representative sample.They were selected from 168 problems submitted in ChatGPT temporary chats using GPT-5.5 at Extra High.
4 Evaluation Protocol
The evaluation tests three multimodal models on 120 problems shown in three presentations, recording one answer per model and computing pass@1 accuracy. Because presentations share problem identifiers, confidence intervals use problem-level bootstrap resampling rather than treating 360 outcomes as independent.
- Uncertainty estimation: 95% confidence intervals resample the 120 problem identifiers while retaining all presentations for each sampled problem.This accounts for the dependence among presentations of the same problem.
- Uncertainty estimation: Presentation-specific intervals use the same problem bootstrap, whereas presentation contrasts use paired resampling of the two outcomes for each problem.The paired procedure preserves within-problem correspondence when comparing presentations.
5 Results
GPT-5.6 Sol performs best but remains well above the 25% random-guessing baseline only modestly, while accuracy varies more across geometry families than presentation formats. Shared successes and failures suggest that global silhouettes and complementary interfaces are easier than small cavities, hidden contacts, and multiple plausible alignments.
- Primary results: 50.0% accuracy: GPT-5.6 Sol leads Claude Fable 5 at 39.4% and Gemini 3.1 Pro Preview at 27.5%, against a 25% random-guessing baseline.Mean costs per response are $0.951, $0.701, and $0.350, respectively.
- Presentation formats: 53.3%: GPT-5.6 Sol rises from 45.0% with a single composite image to 53.3% with separate generic images, while Claude rises from 34.2% to 42.5%.Paired problem-bootstrap intervals include zero for every separated-minus-single contrast; Gemini remains between 26.7% and 28.3%.
- Geometry families: 81.0%: GPT-5.6 Sol reaches this accuracy on block cleavage, versus 74.5% on joinery and 34.4% on rectangular loop cases.Claude shows the same ordering at 69.0%, 56.9%, and 28.7%; the observed spread across geometry families is larger than across presentation formats.
- Shared outcomes: 24 of 360 problem-presentation cells: all three models answer correctly, while all three answer incorrectly on 91 cells.At the problem level, eight problems are missed in all presentations by all models.
- Shared outcomes: Diagnostic global silhouettes or large complementary interfaces characterize shared successes, whereas shared failures more often require tracking small cavities, hidden contact surfaces, or multiple locally plausible alignments.These interpretations are based on qualitative examination and remain tentative.
6 Release
PolyComp releases a public repository containing the 120 benchmark problems, validation materials, presentation builders, and model evaluation results. Figure 4 reports accuracy by model and presentation, with 95% problem-bootstrap intervals and a 25% random-guessing baseline.
- Repository contents: The public GitHub repository contains code and frozen manifests for all 120 problems.The repository is available at https://github.com/sidpatelgit/polycomp.
- Repository contents: The release includes exact image-hash and geometry validation for the benchmark problems.
- Repository contents: Payload builders cover all three presentation formats, alongside a table of results for all three evaluated models.
- Evaluation reporting: Figure 4 shows accuracy by model and presentation with pointwise 95% problem-bootstrap intervals over 120 problems and a 25% random-guessing baseline.It also reports weak evidence that multi-image presentations improved performance for GPT-5.6 Sol and Claude Fable 5.
7 Limitations •
The benchmark may challenge visual perception more than anticipated, and several potentially important evaluation and performance-enhancement factors were not tested. Human performance and run-to-run repeatability also remain unmeasured, although every problem was verified solvable as rendered.
- Perception: Perception may be harder than expected, and alternate line weights, colors, or image resolutions might improve performance.These rendering variations were suggested but not evaluated.
- Evaluation gaps: The study did not test whether prompt variations improved performance.
- Evaluation gaps: No formal human baseline was collected, although every problem was verified solvable as rendered by rejecting all incorrect options.Some problems nevertheless remain quite challenging.
- Evaluation gaps: Run-to-run repeatability was not evaluated for individual model responses or aggregate performance.
- Evaluation gaps: The study did not test whether test-time training or a harness would improve performance.
8 Conclusion and extensions
PolyComp exposes a gap in multimodal spatial intelligence for polycube geometry, rotation, and composition while providing a method for generating verifiable problems. Future work targets task difficulty, structured representations, broader spatial reasoning tests, diagnostic analysis, and smooth or rounded geometry.
- Conclusion: PolyComp identifies a gap in multimodal model spatial intelligence involving polycube geometry, rotation, and composition.It also presents a method for generating verifiable problems in this setting.
- Future directions: Future work could determine how few cubes still make the target-solid perception task challenging.This would further simplify the perception task while measuring the smallest difficult target solids.
- Future directions: Future work could scaffold perception by instructing models to create three-dimensional matrices for target and component solids.The matrices would represent the target solid and the component solids in each option.
- Future directions: Polycube geometry could test understanding of silhouettes, cross sections, interference-free assembly, and other three-dimensional spatial reasoning tasks.These tasks may involve internal representations of three-dimensional objects.
- Future directions: Future work could isolate which disqualifying features of wrong options models identify most easily and extend beyond polycubes to smooth, rounded geometry.These directions target diagnostic analysis and broader geometric forms.