Source-linked AI summary
LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kai Chen, Kaifeng Lyu
TL;DR
Current evidence for MLLMs’ multi-step spatial reasoning is limited despite its importance for sequential real-world tasks. LEGO-Puzzles evaluates this capability through staged LEGO-based VQA, planning, and image-generation tasks, finding substantial human–model gaps that worsen with longer horizons and visual output generation.
Problem
Few benchmarks evaluate MLLMs’ sequential spatial reasoning beyond one action, although many real-world applications require multi-step planning.
Method
LEGO-Puzzles evaluates spatial reasoning progressively through Elementary VQA tasks and Planning tasks based on open-source LEGO assembly projects.
Results
29 MLLMs lag humans by at least 20% on Elementary tasks, while planning accuracy approaches 0% with longer horizons and reaches zero for image generation at 3 steps.
Takeaways & Limitations
The benchmark reveals substantial limitations in current MLLMs’ spatial understanding, sequential reasoning, and generation of spatially coherent visual plans.
Takeaways & Limitations
The evaluation uses a zero-shot setting for fair comparison.
Abstract
from arXiv · showhide
Many real-world applications of spatial intelligence, such as robotic control, autonomous driving, and automated assembly, require spatial reasoning across multiple sequential steps. However, the extent to which current Multimodal Large Language Models (MLLMs) possess this capability remains largely unexplored. Inspired by LEGO construction, a recreational activity that critically relies on multi-step spatial reasoning, we introduce LEGO-Puzzles: a benchmark designed to systematically evaluate the spatial reasoning capabilities of MLLMs from basic spatial understanding to multi-step planning. LEGO-Puzzles contains two task sets. The Elementary set covers 11 visual question-answering (VQA) tasks with 1,100 carefully curated samples to test elementary spatial reasoning skills that are cruical for LEGO assembly. The Planning set directly requires the model to generate a step-by-step plan for assembling a target LEGO structure, where the tasks are organized into subsets with different planning horizons ranging up to 8. Our evaluation of 29 state-of-the-art MLLMs shows that even the strongest models struggle with elementary reasoning tasks in LEGO construction, falling at least 20% behind human performance. The planning accuracy also quickly drops to 0% as the number of planning steps increases, whereas our human participants solve all the tasks perfectly. Switching the output format from multiple choice to image generation degrades model performance even further, leading to zero accuracy even for planning 3 steps. Overall, LEGO-Puzzles reveals critical limitations in current MLLMs' spatial reasoning capabilities and highlights the need for substantial advances.
1. Introduction
LEGO-Puzzles evaluates MLLMs’ spatial reasoning from static spatial understanding through sequential reasoning and multi-step planning. Across 29 models, performance remains substantially below humans, especially as planning horizons increase and outputs require spatially coherent images.
- Spatial understanding covers recognizing 3D object attributes, spatial relations, and object counts from static scenes.
- Multi-step spatial reasoning requires imagining consequences of hypothetical actions across successive real or imagined scenes.
- The benchmark evaluates multi-step image generation because LEGO construction naturally requires sequential assembly and spatial reasoning.
- LEGO-Puzzles organizes progressively complex evaluation into Elementary and Planning sets based on open-source LEGO assembly projects.
- The Elementary set contains over 1,100 VQA pairs across 11 tasks, spanning basic spatial understanding, single-step reasoning, and multi-step reasoning.
- The Planning set tests explicit multi-step assembly plans across horizons up to 8 and also evaluates intermediate-state image generation over multiple turns.
- 29 MLLMs lag humans by at least 20% on Elementary tasks, while planning accuracy falls below 50% beyond 3 steps and approaches 0% at longer horizons.Humans achieve perfect accuracy on the planning tasks, whereas image-generation accuracy reaches zero even for 3-step planning.
2. Related Work
Prior benchmarks cover general multimodal understanding, static spatial reasoning, and some sequential tasks, but LEGO-Puzzles targets finer-grained multi-step spatial reasoning and multi-turn visual planning.
- General multimodal benchmarks evaluate image-centric tasks, bilingual multiple-choice understanding, image and video reasoning, and expert-level multimodal reasoning.
- Spatial benchmarks assess 3D relations and skills such as rotation, folding, penetration, and animation, but most focus on single-image reasoning.
- Sequential benchmarks examine visual-spatial intelligence, household procedural planning, and physical understanding, including some multi-step tasks.
- LEGO-Puzzles uses substantially more challenging planning tasks, where the strongest models fail completely despite higher performance reported on prior benchmarks.
- Unlike prior benchmarks, LEGO-Puzzles evaluates multi-turn image generation alongside fine-grained multi-step spatial reasoning.The paper connects this capability to applications such as robotic manipulation and automated assembly.
3. LEGO-Puzzles
LEGO-Puzzles combines a staged Elementary set with long-horizon Planning tasks to measure spatial understanding, sequential reasoning, and visual plan generation. Its data pipeline uses curated LEGO projects, task templates, and human-in-the-loop quality control.
- The benchmark contains Elementary VQA tasks and Planning tasks designed to assess increasingly complex spatial and sequential reasoning.
- Elementary Set: The Elementary set progresses from basic spatial understanding to single-step actions and multi-step reasoning across up to 7 intermediate stages.
- Elementary Set: Its 11 VQA tasks contain over 1,100 pairs derived from 407 LEGO instructions, with 100 samples per task for balanced evaluation.
- Planning Set: Plan-k-Step-VQA selects and orders correct intermediate LEGO states among 2k options containing correct and erroneous alternatives.
- Data Curation: The curation pipeline consists of data collection, question-answer generation, and quality control.
- Planning Set: Plan-k-Step-Generation generates intermediate assembly states turn by turn, conditioning each state on the history of prior generated states for k = 2, 3, 4, 5.
- Data Curation: Projects are filtered for moderate final size, while templates standardize instructions, image references, and ground-truth answers.
- Data Curation: Human-in-the-loop review removes duplicates and checks image quality, camera perspective, and LEGO part attributes.
4. Experiment on the Elementary Set
Evaluation on the Elementary set shows a substantial gap between MLLMs and humans, with performance worsening on harder spatial and sequential reasoning tasks.
- GPT-5 achieves 72.0% overall accuracy, while most open-source MLLMs perform only marginally above ↑Random.
- 93.6% human performance exceeds GPT-5 and GPT-o3 by over 20% across all Elementary-set tasks.The comparison uses LEGO-Puzzles-Lite, containing 220 questions solved by 30 human experts.
- 21 of 29 models perform below ↑Random on Height, often relying on 2D projected height instead of true 3D perspective.The resulting disagreement between 2D and 3D perspectives further exacerbates the task difficulty.
- 16 of 29 models fall below ↑Random on Rotation, indicating unreliable perception of object orientation changes.
- Increasing reasoning complexity from Next-Step to Ordering significantly reduces accuracy for most models.
- Almost half of the models fall below ↑Random on Ordering, while 12 of 20 open-source models do so on Backwards.InternVL2.5-8B and LLaVA-OneVision-7B fail completely on Ordering.
5. Experiment on the Planning Set
The Planning set evaluates long-horizon sequential reasoning with exact-match, set-match, and overlap metrics, revealing sharp degradation as planning steps increase. Models also fail at full-stage image-based planning, despite topology-aware evaluation across multiple valid assembly trajectories.
- The planning evaluation uses exact match, set match, and overlap ratio to assess whether predicted steps and their ordering match valid assembly sequences.
- GPT-5's exact match accuracy falls from 90% at k = 1 to 0% at k = 8, while all models drop below 50% for k > 2.
- Errors emerge during set selection because GPT-5's exact-match and set-match curves are nearly identical.
- At k = 7/8, proprietary models retain ∼65% overlap ratio despite 0% exact match, whereas open-source models approach random guessing for k > 4.
- Humans achieve 100% accuracy across all planning horizons, while MLLMs accumulate inconsistent predictions as spatial transformations compound.
- Topology expansion enumerates physically valid assembly trajectories, enabling automatic matching of generated intermediate states against complete legal sequences.
- Across k = 2, 3, 4, 5, no model produces a fully correct generative planning trajectory, with failures appearing even at short horizons.
- Full-stage generative planning remains unsuccessful because models produce unreasonable intermediate states and cannot maintain consistent multi-step visual reasoning.
6. Discussion
The discussion examines whether LEGO-Puzzles reflects real-world spatial reasoning and analyzes image-generation failures. Results support cross-dataset relevance but show that visual appearance does not guarantee precise instruction-following or spatially correct generation.
- 6.1. Correlation with Real-world Spatial Reasoning: Performance on LEGO-Puzzles correlates strongly with natural-image 3DSRBench results: PCC is 0.93 for Height and 0.98 for Adjacency, both with p < 0.01.
- 6.1. Correlation with Real-world Spatial Reasoning: These correlations suggest that the synthetic benchmark captures spatial reasoning patterns that generalize to natural images.
- 6.2. Image Generation Results: Figure 5 provides qualitative examples of generated intermediate states for both models at k = 5.
- 6.2. Image Generation Results: Human evaluation scores image outputs on Appearance and Instruction Following because standard fidelity metrics miss fine-grained LEGO alignment and orientation errors.
- 6.2. Image Generation Results: Nano Banana Pro achieves the highest reported scores, with App: 2.52 and IF: 1.99, followed by GPT-5 with App: 2.25 and IF: 1.88.
- 6.2. Image Generation Results: Despite strong appearance consistency, current models often fail to follow assembly instructions precisely, exposing a gap between visual understanding and spatially accurate generation.
7. Conclusion
LEGO-Puzzles benchmarks spatial understanding and sequential reasoning through Elementary and Planning task sets. Experiments with 29 MLLMs reveal substantial human-model gaps, especially in long-horizon planning and spatially coherent image generation.
- LEGO-Puzzles evaluates spatial understanding, single-step reasoning, and multi-step sequential reasoning in MLLMs through VQA and image-generation settings.
- The benchmark contains over 1,100 Elementary VQA samples across 11 tasks and a Planning set requiring step-by-step LEGO assembly plans.
- Experiments with 29 advanced MLLMs reveal substantial performance gaps relative to humans, particularly for long-horizon planning and spatially coherent visual outputs.
A. Error Cases in Height Task
Height-task errors arise from intentional 2D–3D ambiguities, especially when objects are separated or partly occluded. Careful inspection eliminates these mistakes, while the Elementary Set organizes related spatial and sequential reasoning tasks into three levels.
- Height-task error cases: 2D–3D ambiguities in the Height task can cause occasional human mistakes.Errors are more likely when blocks are nonadjacent, farther apart, or partially occluded.
- Height-task error cases: Reminding participants to inspect the images carefully raised their Height-task accuracy to 100%.
- Elementary Set organization: The Elementary Set contains three levels spanning spatial understanding, single-step sequential reasoning, and multi-step sequential reasoning.It contains over 1,100 VQA pairs from 407 LEGO building instructions, with 100 samples per task.
C.2. Human Evaluation in Image Generation
Human evaluation scores generated images along appearance similarity and instruction following. Five expert annotators independently score each sample on 0–3 task-specific scales, after calibration and discussion-based disagreement resolution.
- Evaluation protocol: Five expert annotators independently score each generated image, with final scores averaged across annotators.The evaluation uses two complementary dimensions: Appearance Similarity and Instruction Following.
- Instruction Following: Instruction Following measures whether the requested transformation is performed correctly using task-specific rubrics.For rotation, for example, the rubric considers direction and angle; other tasks assess viewpoint, part identity, placement, or resulting structure.
- Annotator calibration: Annotators are trained with labeled examples before evaluation, and disagreements are resolved through discussion.
D. Data Curation
LEGO-Puzzles uses a template-based pipeline to generate structured QA pairs across all benchmark tasks. Templates standardize instructions, image inputs, and response formats, while answers come from assembly metadata or human annotation.
- Question–answer generation: A fully template-based QA generation pipeline supports scalable and structured construction across LEGO-Puzzles tasks.
- Question templates: Each task-specific template defines the model’s role, introduces image tokens, and specifies the required output form.This creates a unified QA structure while preserving task-specific requirements.
- Ground-truth answers: Sequential-reasoning answers are automatically extracted from precise assembly metadata such as part IDs and step indices.For Next-Step, the ground-truth state is the image at assembly-step index k + 1.
- Ground-truth answers: Spatial-understanding answers are obtained through human annotation because geometric or perceptual judgments cannot be derived solely from metadata.
E. LLM-as-a-judge in Image Generation Evaluation
The study compares GPT-4o’s LLM-as-a-judge scores with human evaluation for generated LEGO images. The comparison indicates that automatic judging identifies severe failures but does not reproduce human judgment reliably for fine-grained spatial evaluation.
- Comparison setup: GPT-4o is compared with human experts on Appearance Similarity and Instruction Following scores for representative model outputs.
- Agreement pattern: GPT-4o and human annotators assign near-zero scores across both dimensions for severe failures.This indicates that current MLLMs can identify major failures in generated outputs.
- Limitation: Current LLM-as-a-judge approaches struggle with finer-grained judgments on spatially precise, instruction-grounded image-generation tasks.The paper therefore retains human evaluation as essential for these assessments.
- Qualitative evidence: Qualitative Plan-k-Step-Generation examples highlight inaccurate intermediate assembly states, particularly as the number of steps increases.
G. Cases of the Elementary Tasks in Image Generation
The section presents qualitative image-generation examples for three elementary spatial reasoning tasks—Rotation*, Multiview*, and Dependency*—to illustrate how models handle viewpoint changes, object rotations, and part-level reasoning. It also compares human and GPT-4o judgments of clearly failed generations.
- Qualitative overview: The examples highlight common success and failure modes across models in image-based spatial reasoning.The cases visualize task setups and generated responses, with questions simplified for clarity.
- Rotation*: Rotation* asks models to generate a LEGO object rotated clockwise by 60 degrees around its center from a top-down perspective.The task uses a reference image and requests a specific rotation.
- Multiview*: Multiview* evaluates whether models can generate a LEGO piece from a top-to-down perspective based on a reference image.The examples illustrate how models interpret viewpoint transformations.
- Dependency*: Dependency* assesses whether models can generate the part required to transition between a current and next LEGO assembly state.The input specifies the current assembly state and the desired next state.
- Evaluation: Human annotators and GPT-4o assign near-zero scores to clearly failed generations from GILL and Anole, but GPT-4o struggles with finer-grained judgments.The comparison indicates that LLM-based judging can identify severe failures while remaining less reliable for subtle quality differences.