Source-linked AI summary
MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft
Sewoong Lee, Risham Sidhu, Julia Hockenmaier, Yoonhwa Jung
TL;DR
LLM capabilities that succeed in text-based reasoning do not reliably transfer to embodied construction, motivating a controlled benchmark for spatial and engineering competence. MineCEraft provides expert-authored tasks with rule-based and physically grounded evaluation, and finds that state-of-the-art agents consistently fail reliable, safe construction requirements. The benchmark’s evaluation is intentionally conservative: satisfying its rubric does not certify real-world construction quality, and system-prompt design remains insufficiently studied.
Problem
It remains an open question how well LLM reasoning transfers to physical, spatial environments and whether agents can reliably perform basic construction tasks.
Method
MineCEraft combines natural-language instruction following, long-horizon embodied construction, and rule-based or physically grounded evaluation of accuracy, safety, and planning.
Results
State-of-the-art LLM-based agents consistently fail reliable spatial reasoning, planning, and fundamental engineering-level safety and stability constraints, with no model exceeding 40% overall accuracy across 723 problems.
Takeaways & Limitations
The results highlight fundamental limitations of current embodied language models and the need for more robust reasoning, planning, and physical grounding in construction engineering.
Takeaways & Limitations
Satisfying all rubric criteria cannot guarantee perfect real-world construction quality, and future work must study system-prompt design and optimization.
Abstract
from arXiv · showhide
We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft. The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs' ability to perform realistic construction engineering tasks. With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks.
1 Introduction
MineCEraft addresses whether language-model reasoning transfers reliably to embodied construction, where even simple tasks require spatial planning and engineering constraints. It introduces a benchmark designed to expose failures in instruction following, physical plausibility, and structural stability.
- Construction remains a difficult domain for AI despite progress in mathematics and science, illustrating Moravec’s paradox.
- MineCEraft asks whether LLM skills transfer to physical, spatial environments and whether agents can reliably perform basic construction in simplified Minecraft.
- LLM-generated constructions can contain shape and size inaccuracies as well as physically implausible and unstable structures.
- The benchmark evaluates embodied construction as an engineering-constrained task requiring physical plausibility, structural stability, and planning feasibility.
- MineCEraft combines natural-language instruction following, long-horizon construction, and programmatic evaluation of accuracy, safety, and planning.
- State-of-the-art LLM agents severely lack reliable spatial reasoning and planning, failing fundamental engineering-level safety and stability constraints.
2 Related Work
Prior benchmarks separately study language grounding, text-based construction knowledge, or physical simulation, but do not jointly evaluate embodied construction and engineering-aware planning. MineCEraft fills this gap with a unified benchmark for interpreting specifications, planning multi-step construction, and producing structurally valid outputs.
- Embodied AI benchmarks often focus on navigation or short-horizon manipulation, whereas construction requires long-horizon spatial structures satisfying geometric and engineering constraints.
- Minecraft construction research has studied instruction giving and following, but MineCEraft disallows floating blocks and permits the full range of Minecraft materials.
- MineCollab uses explicit coordinate-level blueprints for relatively basic structures, while MineCEraft evaluates realistic construction from underspecified natural-language instructions.
- Text-based construction benchmarks assess engineering knowledge without testing agents in embodied environments.
- Existing work does not jointly evaluate natural-language instruction following, embodied construction, long-horizon spatial planning, and engineering-aware structural evaluation.
- MineCEraft addresses this gap by unifying specification interpretation, multi-step planning, and structurally valid construction under engineering-inspired criteria.
3 Task Setup
MineCEraft defines construction tasks through underspecified natural-language requirements rather than single target geometries, then evaluates outputs with explicit rules and physics-based measures. Its task suite spans buildings, civil structures, resource constraints, and single- or multi-turn planning.
- The benchmark contains 723 domain-expert natural-language instructions spanning 17 distinct construction task types.
- Underspecified Instructions: Underspecified instructions allow multiple valid voxel configurations, evaluated against open-ended constraints such as shape, size, materials, and physical plausibility.
- Civil Engineering Structures: Arch-bridge evaluation combines geometric requirements with connectivity checks and von Mises stress, using a flat bridge at approximately 7.8 kPa as the reference threshold.
- Buildings: Building tasks cover material assignments, multiple rooms, multi-story constraints, accessibility, and creative geometric requirements.
- Civil Engineering Structures: Civil-engineering tasks include bridges, arch bridges, and domes with span, clearance, curvature, diameter, and height requirements.
- Planning: Planning tasks test material-limited optimization, sequence selection from single-turn instructions, and revisions across multiple turns.
4 Evaluation Criteria
MineCEraft evaluates construction quality through objectively verifiable accuracy, safety, and planning criteria, combining rule-based checks with physics-based stability analysis. These criteria assess final structures and the feasibility and efficiency of the construction process.
- Evaluation framework: The evaluation pipeline decomposes construction quality into accuracy, safety, and planning criteria that are checked programmatically.Rule-based algorithms verify structural properties such as ground connectivity and span separation.
- Accuracy: Accuracy represents structures as block sets B = {(x, y, z, material)} and maps them with task parameters to pass or fail.Task parameters include expected counts, materials, and heights.
- Accuracy: Material, shape, and size checks compare placed blocks with specified counts, ratios, geometric relationships, connectivity, symmetry, and dimensions.Shape checks include cluster connectivity, surface concavity, and symmetry; size checks use coordinate ranges and extrema.
- Safety: Safety analysis checks physical plausibility through ground connectivity and structural stability through maximum von Mises stress relative to a task-specific reference.Structures are penalized when σmax exceeds σref, reflecting suboptimal stability.
- Planning: Planning evaluation measures dependency and efficiency, requiring load-bearing elements to follow valid construction order while penalizing unnecessary movement.Efficiency uses the L1 path length of the ordered block-placement sequence, with an optimal adjacent-step path scoring 1.
5 Experiments
The experiments evaluate LLM-based Minecraft agents on MineCEraft using a planning-and-coding architecture. Models achieve moderate scores on individual criteria but perform poorly when all construction constraints must be satisfied simultaneously, with performance varying by prompt design and task structure.
- Experimental setup: MineCEraft experiments use MINDcraft as the primary agent architecture, combining language planning, code generation, Minecraft interaction, and a predefined skill library.The agent converts natural-language instructions into executable JavaScript commands through the Mineflayer API.
- Overall results: No model exceeds 40% overall accuracy across the 723 MineCEraft problems under the strict requirement that every applicable criterion be satisfied.This problem-level measure means even the strongest models fail to complete most construction tasks correctly.
- Overall results: Graduate-level construction-engineering participants achieve roughly 90% success on MineCEraft-Lite, compared with less than 40% for the strongest models.The comparison uses the same information for human participants and LLM agents, with human results reported on the Lite subset.
- Prompt effects: Adding a generic “step-by-step” prompt does not measurably improve construction performance, whereas prompts targeting block order and movement reduce inefficient trajectories.The reported gains occur when prompting explicitly addresses construction-specific constraints.
- Prompt effects: Models are relatively robust to construction-specific jargon but can respond differently to everyday verbs such as “build,” “create,” and “construct.”For certain models, “construct” produces higher material and shape accuracy than the alternatives.
- Task-dependent reasoning: LLMs may outperform human participants when spatial reasoning reduces to a closed-form optimization problem, such as maximizing a cabin from 56 oak planks.The cited example uses the volumetric relationship 4^3−2^3 = 56.
6 Conclusion
MineCEraft evaluates LLMs as construction engineers in a controlled Minecraft environment and finds that state-of-the-art models consistently fail to produce reliable, safe construction plans.
- MineCEraft jointly assesses natural-language instruction following, construction accuracy, safety, and long-horizon planning with rule-based and physically grounded metrics.The benchmark is designed for controlled evaluation of construction engineering behavior.
- State-of-the-art LLMs consistently fail to produce reliable and safe construction plans, even for seemingly simple tasks.The conclusion identifies reliability and safety as persistent weaknesses across the evaluated construction setting.
- The results highlight fundamental limitations of current embodied language models in construction engineering.
- The findings underscore the need for more robust reasoning, planning, and physical grounding.
Limitations
The benchmark detects objective requirement violations rather than certifying perfect construction quality, while its Minecraft abstraction, limited model and interaction coverage, and prompt sensitivity constrain interpretation.
- MineCEraft detects clear and objective violations of task requirements but cannot guarantee that a full-score construction is perfect in a real-world setting.As model capabilities improve, more refined benchmarks will be necessary.
- Reference-based comparisons are used for structural stability because globally optimal roof configuration is an NP-hard combinatorial optimization problem.The benchmark uses a balanced reference configuration rather than certifying global optimality.
- Minecraft is a controlled simulation abstraction that cannot represent certain structural types, such as suspension bridges, and uses elasticity-based stress analysis.Alternative physical assumptions and more realistic material models remain future-work directions.
- Computational constraints limit experiments to a subset of state-of-the-art LLMs and text-based Mineflayer interaction, excluding visual feedback.Egocentric visual perception and multimodal reasoning remain unevaluated directions.
- Performance is sensitive to system-prompt design, with removing the brevity constraint substantially changing several criteria, particularly dependency planning.The authors call for systematic study of prompt design and optimization.
- The 723-task set was expert-guided, varied across dimensions and wording, paired with executable checks, and reviewed against manual inspection.
B Toy Example: Arch Bridge Challenge
The Arch Bridge Challenge contrasts text-based construction with image generation: an LLM may produce a malformed or absent bridge in text despite generating a well-formed bridge image.
- B Toy Example: Arch Bridge Challenge: The Arch Bridge Challenge asks an agent to build a side-view arch bridge using stones and empty spaces.
- B Toy Example: Arch Bridge Challenge: A state-of-the-art LLM may generate a non-arch structure or no bridge at all for the text construction prompt.
- B Toy Example: Arch Bridge Challenge: A text-to-image model can produce a beautiful, well-formed bridge when prompted to draw an arch bridge as an image.
- B Toy Example: Arch Bridge Challenge: The modality contrast suggests that some knowledge within LLMs is not internally well-connected and that cross-modal generalization is limited.
- B Toy Example: Arch Bridge Challenge: Changing from image generation to text generation exposes a failure to generalize structural knowledge about arch bridges.
C Evaluation Algorithms
MineCEraft represents structures as coordinate-labeled block sets and evaluates construction through algorithms covering material, shape, size, physical, stability, efficiency, and dependency criteria.
- C Evaluation Algorithms: A structure is represented as a block set B whose blocks have integer coordinates and material labels; ordered placement sequences are recorded when construction order matters.
- C Evaluation Algorithms: Region selectors extract predefined block subsets for checks, while 6-neighbor adjacency handles 3D connectivity and 8-neighborhood connectivity handles planar flood-fill operations.
- C Evaluation Algorithms: Material accuracy is evaluated by Algorithms 1–3, shape accuracy by Algorithms 4–8, and size accuracy by Algorithms 9 and 10.
- C Evaluation Algorithms: Physical plausibility, structural stability, efficiency, and dependency are evaluated by Algorithm 11, Appendix G, Algorithm 12, and Algorithm 13, respectively.
D.1 Overall Problem Accuracy
MineCEraft evaluates construction at the problem level by requiring simultaneous satisfaction of accuracy criteria, then optionally adding physical plausibility and dependency satisfaction. These stricter intersections yield lower success rates and show that binary accuracy does not reliably imply efficiency or stability.
- Problem-level aggregation: Stricter problem-level aggregation produces substantially lower success rates than instruction-level accuracy, indicating persistent reliability challenges for LLM-based construction agents.
- Graded metrics: Binary accuracy does not necessarily improve efficiency, even when material, shape, and size constraints are satisfied.
- Graded metrics: Accurate constructions sometimes show slightly higher structural stability for gpt5-mini and gemini-3-pro.The paper distinguishes specification compliance from construction efficiency while observing modest stability gains for some models.
- Problem-level aggregation: Problem-level accuracy requires satisfying all applicable material, shape, and size criteria simultaneously.The stricter overall criterion additionally requires physical plausibility and dependency satisfaction.
- Prompting effects: Task-specific prompts about construction order or movement planning modestly improve planning-related metrics, whereas generic step-by-step prompting yields little to no measurable improvement.
- Prompt sensitivity: Changing only the instruction verb produces measurable performance differences across several models and evaluation criteria.The compared verbs include “build,” “create,” and “construct.”
- Prompt sensitivity: Removing the brevity constraint improves dependency planning by 20.0 to 40.0 percentage points across four representative models, but other criteria can worsen.Structural stability may improve, while shape, physical plausibility, or efficiency may decrease for some models.
- Model comparison: GPT-4o outperforms GPT-4 across all evaluation criteria, with particularly large gains in material accuracy and structural stability.The comparison does not directly evaluate visual perception because visual observations are not provided during evaluation.
E.1 Human Evaluation
The human evaluation compares two professional construction engineers with LLM agents under controlled task information and MineCEraft-Lite procedures. Human trials averaged three hours per participant, with randomized task order used to mitigate persistent human context across tasks.
- Controlled setup: Human subjects received only the specified guideline, while the LLM used the corresponding construction prompt and command documentation.
- Evaluation checks: MineCEraft evaluation includes checks for enclosed rooms, ground connectivity, reachability by stairs, bounding-box size, path efficiency, and material sequence order.
- Participants: Two voluntary participants were professional construction engineers with three years of U.S. field experience and prior Minecraft experience.
- Controlled setup: Unlike LLM evaluation, human context or memory could not be reset between tasks, so task order was randomized to alleviate this issue.
- MineCEraft-Lite: Across three random seeds, Lite results generally fall within the full-dataset score range, supporting it as a reasonable approximation of the full benchmark.
H Qualitative Analysis
Qualitative examples compare human and LLM construction behavior across representative MineCEraft tasks, including arch-bridge construction. Models can generate arch-like structures in image prompts but often produce irregular or structurally incorrect text-based layouts.
- Figure 7: Figure 7 presents representative MineCEraft cases highlighting behavioral differences between human participants and LLM-based agents.
- Compared systems: The qualitative examples include outputs from ChatGPT 5.4 Thinking, Claude Sonnet 4.6 Extended, and Gemini 3 Thinking in image and text settings.
- Arch bridges: Image-based prompts can elicit arch bridges from LLMs, while text-based prompts often yield irregular or structurally incorrect shapes.
- Arch bridges: Gemini 3 generates a relatively arch-like structure even in the text-based setting.