PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation

Yan Zheng, Florian Bordes

arXiv:2604.02580v1cs.LG

TL;DR

Code-generation benchmarks do not assess spatial reasoning over 3D structures, despite its relevance to applications such as robotics and embodied AI. The paper introduces VoxelCode and VoxelCodeBench, a code-execution platform and benchmark spanning symbolic, geometric, and artistic tasks. Models learn APIs robustly, but compositional geometric reasoning and creative spatial composition remain challenging.

  • Problem

    Existing code-generation benchmarks focus on algorithms and general programming rather than spatial reasoning over 3D structures.

    Source: §1 · p. 1

  • Method

    The paper builds VoxelCode and a 220-task VoxelCodeBench for API-driven voxel manipulation across symbolic, geometric, and artistic reasoning.

    Source: §1 · pp. 1–2

  • Results

    Models show robust API learning, but compositional geometric reasoning and creative spatial composition remain significant gaps.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The released platform and benchmark provide extensible infrastructure for studying code-based 3D generation and spatial reasoning.

    Source: §6 · p. 10

    Benchmark performance measures specific API-driven tasks and may not transfer to broader 3D reasoning scenarios.

    Source: Impact Statement · p. 11

Abstract

from arXiv · show

Evaluating code generation models for 3D spatial reasoning requires executing generated code in realistic environments and assessing outputs beyond surface-level correctness. We introduce a platform VoxelCode, for analyzing code generation capabilities for 3D understanding and environment creation. Our platform integrates natural language task specification, API-driven code execution in Unreal Engine, and a unified evaluation pipeline supporting both automated metrics and human assessment. To demonstrate its utility, we construct VoxelCodeBench, a benchmark of voxel manipulation tasks spanning three reasoning dimensions: symbolic interpretation, geometric construction, and artistic composition. Evaluating leading code generation models, we find that producing executable code is far easier than producing spatially correct outputs, with geometric construction and multi-object composition proving particularly challenging. By open-sourcing our platform and benchmark, we provide the community with extensible infrastructure for developing new 3D code generation benchmarks and probing spatial reasoning in future models.

1 Introduction

The paper addresses the lack of code-generation benchmarks for spatial reasoning, despite its relevance to robotics, embodied AI, and interactive content creation. It introduces VoxelCode and VoxelCodeBench, finding that API learning is robust while compositional geometric reasoning and creative spatial composition remain difficult.

1 Introduction

  • Existing code-generation benchmarks emphasize algorithms and general programming rather than spatial reasoning over 3D structures.The paper identifies this as a significant gap given spatial intelligence's relevance to robotics, embodied AI, and interactive content creation.Source: §1 · p. 1
  • Code-based 3D manipulation exposes symbolic reasoning for analysis and gives users more control than end-to-end neural approaches.The paper also argues that API-driven evaluation reflects software-engineering scenarios in which models learn domain-specific interfaces from documentation and examples.Source: §1 · p. 1
  • VoxelCode is a rendering API that takes code as input to produce real-time 3D renderings, letting users generate objects and capture multiple camera viewpoints to compare LLM coding performance.It is built with Unreal Engine and Voxel Plugin 2.0.Source: §1 · p. 1
  • VoxelCodeBench poses three research questions about API learning, capability across symbolic, geometric, and artistic reasoning, and general-purpose versus code-specialized models.The benchmark comprises 220 tasks across symbolic reasoning (80), geometric construction (50), and artistic composition (90).Source: §1 · pp. 1–2
  • Models show robust API learning, but compositional geometric reasoning and creative spatial composition remain significant gaps.The findings have implications for improving code-generation models and designing human-AI collaboration systems for 3D content creation.Source: §1 · p. 2

2 Related Work

Prior work covers code generation, 3D generation, spatial reasoning, and voxel representations, but the paper positions VoxelCodeBench as a code-based benchmark for forward language-to-shape tasks and compositional spatial reasoning.

Benchmarks for code generation

  • Code-generation benchmarks cover algorithms, general programming, domain-specific APIs, and scene synthesis, but do not assess spatial reasoning or 3D manipulation.SceneCraft and related work use language models to synthesize scenes with Blender code, while other work targets computer-aided design.Source: §2 · p. 3

3D generation and evaluation

  • Neural 3D generation evaluation often emphasizes single-object quality, while VoxelCodeBench evaluates compositional reasoning through interpretable code-based manipulation.T3Bench evaluates compositional text-to-3D generation with end-to-end neural systems rather than explicit code generation.Source: §2 · p. 3

CSG-based shape program synthesis

  • Shape-program synthesis work maps shapes to code and may require category-specific training data, whereas VoxelCodeBench tests zero-shot language-to-code-to-shape generalization.Source: §2 · p. 3

Spatial reasoning and voxel representations

  • Spatial-reasoning benchmarks use text, visual grounding, or interactive navigation, while code-generation studies address robotics and interactive environments.Voxel environments have also been used for embodied agents without systematic code-generation benchmarking; voxel grids provide discrete building blocks for programmatic construction and boolean operations.Source: §2 · p. 3

3 VoxelCode

VoxelCode is an Unreal Engine environment that executes Python code to manipulate voxels and render 3D scenes. Its API limits models to documented interfaces while supporting live visualization and varied example outputs.

3 VoxelCode

  • VoxelCode takes Python code as input and executes it to manipulate voxels in an Unreal Engine environment.The environment uses Voxel Plugin 2.0 for voxel manipulation.Source: §3 · pp. 3–4
  • Models receive a simple API with function definitions and Python examples, while the Voxel Plugin source remains inaccessible.The paper states that the plugin is not publicly available and requires a custom license, helping test generalization beyond training data.Source: §3 · p. 4
  • VoxelCode provides a web interface for real-time code visualization, and its outputs range from simple primitives to complex multi-component objects.Examples include characters, geometric shapes, animals, vehicles, and architectural structures.Source: §3 · p. 4, Figure 2 · p. 4

4 VoxelCodeBench Design

VoxelCodeBench combines documented tasks, code execution, and multidimensional assessment to evaluate spatial reasoning reproducibly. Its 220 tasks span symbolic, geometric, and artistic capabilities, with automated checks and human judgments of correctness and visual quality.

4 VoxelCodeBench Design

  • VoxelCodeBench combines task specifications, VoxelCode execution, and multidimensional assessment for reproducible evaluation of spatial reasoning through code generation.Figure 1 presents this end-to-end pipeline.Source: §4 · p. 5

4.1 Evaluation Framework

  • Tasks provide API documentation, minimal working examples, and natural-language specifications ranging from precise geometry to open-ended creative prompts.The API documentation covers 47 functions in primitive placement, geometric shapes, boolean operations, and surface utilities; examples test generalization from simpler problems.Source: §4.1 · p. 5

4.2 Task Taxonomy

  • The 220 tasks comprise 80 symbolic, 50 geometric-construction, and 90 artistic-composition tasks with distinct reasoning demands.Symbolic tasks test API comprehension and coordinate mapping; geometric tasks test procedural construction; artistic tasks test creative reasoning and aesthetic judgment.Source: §4.2 · p. 5

4.3 VoxelCode Environment

  • Generated code is validated, run with a 60-second timeout, rendered from multiple viewpoints, and exported for automated analysis.The pipeline initializes a clean scene and captures execution logs and errors to support reproducible evaluation and error analysis.Source: §4.3 · p. 6

4.4 Multi-Dimensional Assessment

  • Automated checks measure execution success and API compliance, while human annotators assess object presence, position, materials, shape, and visual quality.Shape Correctness is the primary success metric, and Visual Quality is secondary.Source: §4.4 · p. 6

5 Experiments and results

VoxelCodeBench evaluates eight models on API-driven 3D tasks using human ratings and finds substantial differences across models and reasoning dimensions. Geometric construction is especially difficult, while example expansion and error analysis reveal distinct model-specific behaviors.

5 Experiments and results

  • Eight models from the GPT, Claude, and Gemini families receive identical API documentation, working examples, and task specifications for comparison.The evaluation uses the models’ respective APIs under the same conditions.Source: §5 · p. 6

5.1 Evaluation Protocol

  • Human evaluation rates object presence, position, material, shape, and visual quality; Shape Correct is the primary metric.Generated scenes are rendered from four viewpoints before human assessment.Source: §5.1 · pp. 6–7

5.2 Results

  • GPT-5 leads overall with 87.9% shape correctness and a 5.71 visual quality score; Claude Sonnet 4.5 and GPT-5 Mini each reach 80.4% shape correctness.Table 1 reports model performance across visible-object, position, material, shape, and visual-quality measures.Source: §5.2 · p. 7, Table 1 · p. 8
  • Geometric construction is hardest for all models: GPT-5 reaches 66.7% shape correctness and Claude Sonnet 4.5 reaches 52.8%, while artistic tasks yield 97.5% and 89.5%, respectively.On geometric subcategories, GPT-5 scores 84.2% on geometry operations and 54.5% on parameterized primitives; Claude Sonnet 4.5 shows the reverse pattern, at 42.1% and 54.5%.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Human Evaluation Analysis

  • GPT-5 has higher visual quality than Claude Sonnet 4.5 (5.71 vs. 5.01), while Has Object and Shape Correct differ by less than 3% for most models.Gemini Pro’s 20.1% Has Object, 19.5% Shape, and 1.36 Look scores accompany frequent execution failures or empty outputs.Source: Figure 3 · p. 7, §5.3 · pp. 8–9

5.4 Ablation: Effect of Additional Working Examples

  • More examples improve low-performing Gemini Pro by 24.8 percentage points but reduce Claude 3.5 Sonnet by 34.8, GPT-5 Chat by 33.4, and Claude Sonnet 4.5 by 17.5.GPT-5 remains nearly stable at −1.4 percentage points; the comparison is on artistic-task shape correctness between 2–3 and 5+ examples.Source: §5.4 · p. 9, Table 3 · p. 10

5.5 Error Analysis

  • Invalid API usage causes nearly 40% of errors, while timeouts and rendering crashes account for about a quarter; GPT-5 has a 0.8% error rate versus Gemini Pro’s 5.6%.Claude Sonnet 4.5 has 10 of 19 errors in the timeout/crash category; Gemini Pro also has 14 basic syntax errors.Source: §5.5 · p. 10, Table 4 · p. 10

6 Conclusion

The paper introduces VoxelCodeBench for evaluating spatial reasoning through API-driven 3D voxel manipulation. Its experiments show that models can compose primitives into diverse 3D objects, while the benchmark and tools are released as infrastructure for further research.

6 Conclusion

  • VoxelCodeBench benchmarks spatial reasoning through API-driven 3D voxel manipulation on an Unreal Engine and Voxel Plugin 2.0 platform.VoxelCode supports real-time rendering of LLM-generated code and is released with evaluation tools.Source: §6 · p. 10
  • Experiments show that models compose primitives through boolean operations to generate diverse objects, suggesting pretrained world knowledge can generalize to 3D structure synthesis without domain-specific training.Source: §6 · p. 10
  • The released benchmark infrastructure is intended to support future research in code-based 3D generation and spatial reasoning.Source: §6 · p. 10

Impact Statement

VoxelCodeBench offers a standardized, open evaluation framework for code-based 3D generation, with interpretability advantages from inspectable generated programs. The paper cautions that benchmark performance is specific to API-driven tasks and may not transfer to broader 3D reasoning scenarios.

Impact Statement

  • The open-source tasks, rendering environment, and evaluation tools provide a standardized framework in an area described as lacking systematic benchmarks.Generated programs expose reasoning for analysis and debugging.Source: Impact Statement · p. 11
  • Benchmark performance should not be treated as evidence of general spatial intelligence because evaluation covers specific API-driven tasks that may not transfer to broader 3D reasoning.Source: Impact Statement · p. 11

A Task Examples

A.1 Symbolic Reasoning Tasks

The symbolic reasoning task examples test primitive creation and spatial arrangement through API calls, parameter choices, and object combination.

A.1 Symbolic Reasoning Tasks

  • A sphere task specifies radius 5 and center (10, 0, 10), illustrating coordinate-based primitive placement.The expected code creates a sphere and stamps it onto the surface.Source: §A.1 · p. 15
  • A second task places a torus above a cube, with both centered at the origin.The example uses a cube at (0, 0, 5) and a torus at (0, 0, 15).Source: §A.1 · p. 15
  • The torus and cube are combined with a union operation to form one composition.The example specifies zero smoothness for the union.Source: §A.1 · p. 15

A.2 Geometric Construction Tasks

The geometric construction examples cover boolean subtraction, repeated construction, and curved geometry, requiring models to translate shape descriptions into API-based operations.

A.2 Geometric Construction Tasks

  • A boolean-composition task asks for a cube with a central spherical cavity, using subtraction to remove the sphere.The cube has side length 10 and the sphere radius is 4.Source: §A.2 · p. 15
  • An iterative-construction task builds an eight-step spiral staircase by rotating each step 45 degrees and raising it by 2 units.The expected approach uses a loop and unions the generated steps.Source: §A.2 · p. 16
  • A curved-geometry task requires a pipe connecting two points with smooth bends, with a Hermite-curve approach specified.The expected API call is MakeHermitePipeNode.Source: §A.2 · p. 16

A.3 Artistic Composition Tasks

The section presents artistic composition as combining primitives into coherent scenes, then contrasts code-generated internal detail with surface-oriented neural 3D generation.

A.3 Artistic Composition Tasks

  • The castle task requires choosing dimensions, positions, and proportions for a central keep, four towers, walls, battlements, and roofs.Suggested primitives include houses, cylinders, cubes, and pyramids, combined with UnionShapes.Source: §A.3 · pp. 16–17
  • A tree task combines a cylinder trunk with spherical foliage to create a simple composed object.The example places the trunk at height 20 and foliage at height 50.Source: §A.3 · p. 17

B Structured Geometric Detail Generation

  • Code-based generation produces inspectable internal structures and fine detail, including ladders, cabin interiors, and distinct mall floors.The passage attributes this to logical composition of geometric primitives and boolean operations; surface-focused diffusion methods typically produce surface-only representations.Source: B Structured Geometric Detail Generation · p. 17

C Qualitative Analysis of Model Behavior

The qualitative analysis identifies uneven geometric understanding, difficult volumetric composition, inconsistent performance across similar prompts, and hallucinated API calls. These patterns point to brittle geometric reasoning and API adherence.

C Qualitative Analysis of Model Behavior

  • Only GPT-5 and Claude Sonnet 4.5 correctly interpret “frustum” (a truncated cone or pyramid), and only Claude Sonnet 4.5 correctly generates a “prism” shape.For “capsule” shapes, the passage reports diverse misinterpretations; geometric vocabulary understanding varies substantially across model families and does not consistently correlate with overall benchmark performance.Source: C Qualitative Analysis of Model Behavior · p. 17
  • For shapes absent from the API primitives, some models use MakeMeshNode while others approximate them by combining primitives.Even simple heart shapes repeatedly defeat models across attempts.Source: C Qualitative Analysis of Model Behavior · pp. 17–18
  • Only GPT-5 correctly handles cylinder intersections across multiple test cases and consistently joins cones and cylinders at their bases.The reported pattern identifies volumetric set operations and relationships among objects as significant challenges.Source: C Qualitative Analysis of Model Behavior · p. 18
  • Models can succeed on one instance of a shape yet fail on a similar instance with slightly different parameters or phrasing.Examples include crescent, pentagonal-prism, and dome tasks; this inconsistency may reflect prompt sensitivity rather than robust geometric understanding.Source: C Qualitative Analysis of Model Behavior · p. 18
  • API hallucinations include incorrect boolean-operation enum names, nonexistent shape functions, and invalid method calls.These errors suggest reliance on general programming conventions rather than faithful use of API documentation, and are especially pronounced in Gemini Pro.Source: C Qualitative Analysis of Model Behavior · p. 18
  • The qualitative findings characterize current LLM geometric reasoning as brittle, with task performance not reliably predicting performance on similar tasks.The passage identifies geometric concept grounding and robust API comprehension from documentation as areas for future work.Source: C Qualitative Analysis of Model Behavior · pp. 17–18

Related papers

Loading 2604.02580v1…