PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxProg: 3D Spatial Reasoning through Voxel Programming

Zhichao Zhai, Menglei Zhao, Guikun Chen, Zihao Yan, Kanzhi Wu, Wenguan Wang, Jun Xiao

arXiv:2610.30127v1cs.CV

TL;DR

Spatial instructions leave geometric references implicit, while existing interfaces and representations limit what programs can express or inspect. VoxProg uses executable programs over local voxel views to construct task-relevant surface geometry and compute placements. It outperforms relation-, keypoint-, and mesh-based baselines on VoxProg-Bench, with strong intent-alignment and physical-validity results.

  • Problem

    Spatial instructions leave scene-dependent geometric requirements implicit, while existing task interfaces and geometric representations limit adaptable computation and expose some local surface geometry only sparsely.

    Source: §1 · p. 1

  • Method

    VoxProg uses a language model to write evidence programs over local voxel views, then pose programs that measure selected geometry, with execution feedback supporting revision.

    Source: §1 · p. 1

  • Results

    83.10 overall IA Rank Score and 95.8% Physical Validity; VoxProg outperforms relation-, keypoint-, and mesh-based baselines on VoxProg-Bench.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Open-ended placement evaluation pairs perceptual intent alignment with physical checks because valid placements must satisfy both semantic intent and physical plausibility.

    Source: §1 · p. 2

    A 2D primary view alone cannot establish 3D physical validity: the pot lid appears seated from the front but is suspended above the rack in side view.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Language models' abilities to interpret tasks and generate programs offer a promising basis for 3D spatial reasoning. Yet following spatial instructions in 3D scenes requires programs to construct geometric references that language leaves implicit. In this work, we introduce VoxProg, which formulates 3D object placement through voxel programming: a language model translates task interpretations into executable programs over local voxel views of scene surfaces. Because these views index surface geometry on regular grids with explicit coordinates and neighborhood relations, an evidence program can isolate task-relevant regions and boundaries directly from scene meshes, and a downstream pose program can measure the resulting geometry to compute a target $SE(3)$ pose. Visual and numerical feedback from execution lets the model inspect intermediate selections and candidate placements to iteratively revise its code. To evaluate open-ended placement across functional use, geometric fit, and contextual arrangements, we introduce the 71-task VoxProg-Bench benchmark along with an evaluation suite pairing perceptual Intent Alignment and 3D Physical Validity. Across VoxProg-Bench and external benchmarks, VoxProg consistently outperforms relation-based, keypoint-based, and mesh-based baselines.

1 Introduction

VoxProg addresses open-ended placement by letting a language model construct task-relevant geometric references from voxelized scene surfaces, then compute object poses. On VoxProg-Bench, it outperforms relation-, keypoint-, and mesh-based baselines, especially on geometric tasks.

1 Introduction

  • Instructions such as placing a bottle in a holder or extending a cup row require inferring geometric references from the scene rather than receiving them explicitly in language.The needed references include the holder’s receiving structure or the row’s direction and spacing.Source: §1 · p. 1
  • Existing interfaces limit task coverage, while sparse keypoints can underspecify local surfaces and raw meshes leave programs to organize irregular geometry into useful regions.The paper frames open-ended reasoning as needing both instruction-adaptive computation and a representation that exposes geometry to code.Source: §1 · p. 1
  • VoxProg turns local voxel views into task-specific geometric references by indexing surface grids to isolate contact regions, boundaries, extent, and orientation.Regular grids expose coordinates and neighborhood relations as array operations.Source: §1 · p. 1
  • An evidence program selects task-relevant surface regions and spatial relations before a conditioned pose program measures them; visual and numerical feedback supports iterative revision.The evidence program can construct structures such as dish-rack dividers and support rails.Source: §1 · p. 2
  • Across VoxProg-Bench, VoxProg substantially outperforms relation-, keypoint-, and mesh-based baselines, with its largest margin on geometric tasks where contours govern fit and clearance.The 71 tasks depend on part-level affordances, 3D shape compatibility, and surrounding layouts.Source: §1 · p. 2
  • VoxProg-Bench contributes 71 language-guided tasks across functional, geometric, and contextual arrangements, evaluated with perceptual intent ranking and 3D physical verification.The paired evaluation uses IA Rank Score and Physical Validity checks for static support and object interpenetration.Source: §1 · p. 2

2 Related Work

Related work studies language-guided placement through spatial layouts, rearrangement, and fine-scale manipulation, while programmatic perception often hides geometry behind fixed interfaces. VoxProg instead makes instruction-dependent surface selection an explicit programming stage.

Spatial task specification

  • Prior work spans scene graphs and language-conditioned layouts (Feng et al., 2023), rearrangement priors or search (Liu et al., 2022; Chang et al., 2024), and manipulation over sparse keypoints and axes.The cited approaches represent inter-object relations, spatial configurations, multi-object structures, or fine-scale pose primitives.Source: §2 · pp. 2–3
  • Layout, surface-constraint, and keypoint formulations use boxes, planar patches, or sparse primitives, but open-ended tasks often depend on instruction-specific extended surfaces such as slots, ledges, rims, or recesses.VoxProg makes construction of these geometric references an explicit stage, retaining selected regions and source-mesh patches for measurement.Source: §2 · p. 3

Language models for spatial computation

  • Programmatic perception commonly returns centroids, boxes, or masks (Liang et al., 2023; Mu et al., 2024), or scores trajectories on coarse grids (Huang et al., 2023a), rather than exposing surfaces to code.VoxProg discretizes candidate objects into task-resolution voxel views while retaining correspondence to mesh triangles.Source: §2 · p. 3
  • Generated code can build custom surface witnesses from voxel views, verify them through visual and numerical feedback, and measure grid bounds alongside clipped mesh patches to compute placements.The voxel views use task-chosen resolutions and preserve exact correspondence with underlying mesh triangles.Source: §2 · p. 3

3 Method

VoxProg separates constructing task-relevant geometric references from computing a placement pose, using local voxel views and visual and numerical execution feedback to refine both programs.

3 Method

  • The pipeline grounds visible cues as 3D anchors, constructs task-relevant surface regions and relations, then uses the selected geometry to compute and apply a target pose.For the plate example, the evidence includes rack dividers, support rails, and a slot region.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Task Interpretation and Visual Grounding

  • An evidence graph routes visible cues to raycast surface anchors and sends references that cannot be reduced to a clickable pixel to voxel-based inspection.The latter include interior slots, recessed ledges, and occluded undersides.Source: §3.1 · p. 3

3.2 Voxel Programming

  • Local voxel views discretize mesh surfaces into regular grids, where programs select cells as witnesses using array operations and task-appropriate resolution.The grid-to-world transform maps selected indices and bounds back to scene coordinates; finer grids can resolve thin dividers or narrow slots that coarse grids merge.Source: §3.2 · p. 4

3.3 Evidence Program Generation and Refinement

  • The evidence program follows graph dependencies to construct witnesses and specify their spatial relations, using selection criteria, probes, rendered highlights, and checks to revise incomplete or leaking selections.The evidence record stores the witnesses, while the relation specification captures constraints such as a rim fitting between dividers and resting on support rails.Source: §3.3 · pp. 4–5

3.4 Pose Program Generation and Refinement

  • The pose program uses verified evidence and relations to compute a target rigid transformation from measured geometry, then updates the scene and revises the program using contact and clearance diagnostics.It combines voxel bounds with clipped source-mesh patches to derive placement geometry while keeping evidence fixed in the initial scene state.Source: §3.4 · p. 5

4 Experiments

Experiments assess VoxProg on 71 curated placement tasks spanning functional, geometric, and contextual requirements, then examine ablations, external benchmarks, and failure modes. VoxProg leads the reported task-suite metrics.

4.1 Experimental setup

  • VoxProg-Bench contains 71 curated language-guided placement tasks, grouped by primary requirement into functional (28), geometric (14), and contextual (29) cases.All methods receive scene meshes, initial poses, the instruction, and target-object identity; ground-truth poses and images are reserved for evaluation.Source: §4.1 · pp. 5–6

4.2 Results on the task suite

  • 83.10 IA Rank Score and 95.8% Physical Validity give VoxProg the overall lead, exceeding Direct Mesh (48.59 / 87.3%).VoxProg also leads each of the three task groups.Source: Table 1 · p. 6, §4.2 · p. 6
  • In qualitative cases, VoxProg matches the headphone arch to the stand’s inner support and aligns a sneaker’s sole with a sloped stand.The comparison also shows VoxProg positioning a VR headset over its stand with the headband behind the front shell.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3 Ablation and external evaluation

  • IA Rank Score more than doubles from 26.76 to 56.34 and Physical Validity rises from 87.3% to 95.8% from Direct Mesh to Mesh Evidence.Replacing direct mesh selection with voxel views raises IA Rank Score from 56.34 to 66.90 while maintaining 95.8% Physical Validity.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • On RoboCasa, VoxProg scores IA Rank Score 87.72 and Physical Validity 94.7%; on Open6DOR v2, it leads Direct Mesh 61.54 vs. 57.69 and 92.3% vs. 46.2%.The external evaluation covers 19 RoboCasa-adapted kitchen tasks and 13 Open6DOR v2 tabletop tasks.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Failure Analysis

  • Staged methods reduce physical rejections from 9 for Direct Mesh to 3; a single-view inspection can still mistake an unsupported pot lid for seating.The lid appears seated in the primary view but is suspended above the rack in the side view.Source: §4.4 · pp. 8–9

5 Conclusion

VoxProg treats task-relevant geometric references as intermediate surface structures constructed by executable code, then uses them to compute object poses. Across VoxProg-Bench and external tasks, it outperforms relation-, keypoint-, and mesh-based baselines in intent alignment and physical validity.

5 Conclusion

  • Figure 7 presents physical failure categories on VoxProg-Bench and a contact-check example in which a lid seated from the front floats when viewed from the side.Source: Figure 7 · p. 9, §5 · p. 9
  • Across VoxProg-Bench, RoboCasa, and Open6DOR v2, VoxProg consistently outperforms relation-based, keypoint-based, and mesh-based baselines in intent alignment and physical validity.The conclusion reports this pattern without giving quantitative values.Source: §5 · p. 9
  • VoxProg constructs instruction-relevant geometric references as intermediate surface structures rather than parameterizing tasks over fixed spatial relations or sparse keypoint primitives.It couples local voxel-grid selection with exact mesh surface normals and patches to isolate functional contact regions and compute physically valid SE(3) poses.Source: §5 · p. 9
  • The conclusion identifies coupling programmable surface witnesses with downstream motion generation as a direction for extending geometric reasoning to full manipulation execution.Source: §5 · p. 9

Related papers

Loading 2610.30127v1…