Source-linked AI summary
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei
TL;DR
Robot manipulation systems often depend on pre-defined motion primitives, limiting open-ended physical action from language. VoxPoser uses LLMs and VLMs to compose 3D value maps for model-based trajectory planning, achieving high success across everyday tasks and efficient dynamics learning from brief online interaction.
Problem
Existing language-driven robot manipulation approaches typically rely on pre-defined motion primitives, limiting fine-grained action generation for open-ended instructions and objects.
Method
VoxPoser combines LLM-inferred affordances and constraints with VLM-grounded 3D value maps to guide model-based synthesis of closed-loop robot trajectories.
Results
VoxPoser synthesizes trajectories with high average success across representative everyday manipulation tasks and learns effective contact-rich dynamics models with less than 3 minutes of online interaction.
Takeaways & Limitations
The framework supports zero-shot manipulation over open-set instructions and objects while using synthesized trajectories as exploration priors for efficient dynamics learning.
Takeaways & Limitations
The method assumes mostly static scenes and uses only a planar pushing model for object interactions, limiting its dynamics generality for contact-rich tasks.
Abstract
from arXiv · showhide
Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io
1 Introduction
The introduction frames grounding language instructions in physical robot actions as an unresolved challenge: existing methods commonly depend on predefined motion primitives. VoxPoser addresses this by using LLM-inferred affordances and constraints to compose dense 3D value maps for model-based robotic planning.
- LLMs encode generalizable world knowledge through language, but how to use that knowledge for physical action by embodied agents remains unresolved.
- Existing instruction-grounding methods use lexical parsing or language-model step decomposition, yet typically depend on predefined motion primitives for environmental interaction.
- Because direct textual control is impractical for high-frequency, high-dimensional actions, the approach uses LLMs to infer language-conditioned affordances and constraints.
- VOXPOSER composes dense 3D value maps in observation space by combining LLM reasoning, VLM visual grounding, and model-based planning to guide robot interactions.
2 Related Works
Related work spans language grounding, language-model-based robotics, and learning-based trajectory optimization. VoxPoser is positioned relative to prior costmap, potential-field, constrained-optimization, and embodied-language approaches.
- Grounding Language Instructions: Language grounding supports compositional goal specification, multimodal representation learning, and planning or reasoning across intelligent-agent and robotics settings.Prior methods include lexical analysis, formal logic, and language annotations for model, imitation, and reinforcement learning.
- Related Planning Methods: The approach also connects to potential-field path planning and constrained-optimization methods for manipulation planning.These connections are stated explicitly in the related-work discussion.
- Grounding Language Instructions: Sharma et al. optimize an end-to-end cost predictor that maps language instructions to 2D costmaps for collision-free motion planning.The passage identifies this work as most closely related and contrasts it with VoxPoser’s reliance on pre-trained language models for open-world knowledge.
- Language Models for Robotics: Language-model robotics research focuses on planning and reasoning, using textual scene descriptions, perception APIs, vision during decoding, or multimodal language-model inputs.These mechanisms are introduced to let language models perceive physical environments.
- Learning-based Trajectory Optimization: Learning-based trajectory optimization methods broadly learn models or learn costs, rewards, or constraints, typically from in-domain interactions.Related work also explores learning task specifications from large-scale offline data to improve generalization in the wild.
3 Method
VoxPoser decomposes free-form instructions into manipulation phases, uses LLM-generated code and VLMs to compose observation-grounded 3D value maps, and optimizes dense robot trajectories under dynamics and kinematics constraints. Closed-loop model-predictive control enables robust execution, while online dynamics learning focuses exploration around task-relevant trajectories.
- Instruction decomposition: Free-form instructions are decomposed into individually specified manipulation phases, with LLMs also handling sub-task decomposition and sequencing.The method focuses on phases such as grasping a drawer handle and pulling open the drawer rather than directly solving arbitrarily long-horizon or under-specified instructions.
- Trajectory optimization: For each phase, the method optimizes a dense sequence of 6-DoF end-effector waypoints while minimizing task cost and control costs subject to dynamics and kinematics constraints.Waypoints include desired end-effector pose, velocity, and gripper action, and solving each phase produces trajectories that collectively accomplish the full instruction.
- 3D value maps: VoxPoser uses LLM-written Python programs to invoke VLM perception APIs and compose voxel value maps that encode task-relevant affordances and constraints.The resulting map assigns real-valued costs to discretized 3D coordinates; sparse maps are densified with smoothing to encourage smoother planned trajectories.
- Closed-loop control: Zeroth-order trajectory sampling is implemented with model predictive control that replans at every step using the current observation, supporting execution under dynamic disturbances.The framework can use either a learned or known environment model for planning.
- Online dynamics learning: Online learning alternates environment data collection with dynamics-model training and biases action sampling toward task-relevant trajectories instead of the full action space.Small noise around the trajectory encourages local exploration, improving efficiency for contact-rich tasks such as opening a door.
4 Experiments and Analysis
Experiments evaluate VoxPoser’s implementation, real-world manipulation, simulated generalization, online dynamics learning, and component-level errors. Across these studies, VoxPoser demonstrates broad task applicability, robustness, and efficient improvement through experience.
- Experimental overview: The evaluation covers implementation details, real-world everyday manipulation, simulated generalization, online learning for challenging tasks, and error analysis.These studies correspond to Sections 4.1–4.4.
- Implementation: VoxPoser uses GPT-4-based language model programs, open-vocabulary perception, and composable voxel value maps to synthesize trajectories.Perception combines OWL-ViT, Segment Anything, XMEM, and RGB-D observations; value maps encode affordance, avoidance, velocity, rotation, and gripper action.
- Simulated generalization: In simulation, VoxPoser outperforms both baselines across 13 tasks from two categories on seen and unseen instructions and attributes while maintaining similar success rates.The benchmark contains 2766 unique instructions with highly randomized task attributes.
- Real-world manipulation: In real-world evaluation, VoxPoser performs five representative everyday manipulation tasks with high success and greater disturbance robustness than action primitives.The study targets open-set instructions and open-set objects, with quantitative results reported in Table 1.
- Online dynamics learning: Less than 3 minutes of online interactions can produce an effective dynamics model and high eventual success when zero-shot trajectories guide exploration.Without this prior, exploration exceeded the maximum 12-hour limit in every comparison.
- Error analysis: Component-level experiments isolate dynamics, perception, and specification errors, with VoxPoser significantly reducing specification error.The analysis uses simulation with ground-truth perception and dynamics models.
5 Conclusion, Limitations, & Future Works
VOXPOSER is presented as a general framework that extracts affordances and constraints from LLMs and VLMs, grounds them in 3D perceptual space, and supports everyday real-world manipulation. Its main stated limitation is reliance on external perception modules for tasks requiring holistic visual reasoning or fine-grained understanding.
- Conclusion: VOXPOSER extracts affordances and constraints from LLMs and VLMs and grounds them in 3D perceptual space for everyday real-world manipulation.The framework is designed for open-set instructions and objects.
- Conclusion: The framework offers significant generalization advantages for manipulation tasks involving open-set instructions and objects.
- Limitations: VOXPOSER relies on external perception modules, limiting tasks that require holistic visual reasoning or fine-grained understanding.
A Appendix · A.2 Emergent Behavioral Capabilities
VoxPoser exhibits embodied emergent capabilities inherited from pretrained LLM world knowledge, including commonsense interpretation, precise language-based correction, visual multi-step planning, and physical-property estimation. These capabilities are demonstrated through everyday manipulation tasks and can also expose reasoning limitations.
- A.2 Emergent Behavioral Capabilities: The appendix illustrates four capabilities: behavioral commonsense reasoning, fine-grained language correction, multi-step visual programs, and estimating physical properties.Figure 5 presents these capabilities as emergent behaviors inherited by VoxPoser from its language model.
- A.2 Emergent Behavioral Capabilities: VoxPoser studies embodied emergent capabilities driven by the rich world knowledge of pretrained language models.The appendix focuses on behavioral capabilities considered unique to VoxPoser.
- A.2 Emergent Behavioral Capabilities: Behavioral commonsense reasoning lets VoxPoser interpret “I am left-handed” and move the fork from the bowl’s right side to its left.This occurs during a table-setting task involving user preferences.
- A.2 Emergent Behavioral Capabilities: Fine-grained language correction enables VoxPoser to adjust its action when told it is off by 1cm while covering a teapot with its lid.The capability supports tasks requiring high precision.
- A.2 Emergent Behavioral Capabilities: Multi-step visual programming lets VoxPoser open a drawer fully, record handle displacement, and close it to the midpoint when object models are unavailable.The strategy uses visual feedback to satisfy the instruction to open the drawer precisely by half.
- A.2 Emergent Behavioral Capabilities: VoxPoser estimates unknown block masses by pushing both blocks down a ramp and selecting the one traveling farthest as heavier.The example involves conducting physics experiments with an existing ramp.
A.3 APIs for VoxPoser
VoxPoser exposes Python APIs that let the LLM detect objects, construct and manipulate value maps, convert between voxel and world coordinates, specify end-effector orientation, and execute planned trajectories. In model-predictive-control settings, detected entities and value maps can be re-evaluated from the latest observation.
- API interface: The LLM generates executable Python code with access to NumPy, Transforms3d, and VoxPoser environment APIs.These APIs are central to VoxPoser’s code-execution interface.
- Perception and execution: detect returns each matching object instance with its center position, occupancy grid, and mean normal vector.The returned dictionaries serve as entities of interest for downstream execution.
- Perception and execution: execute invokes the motion planner using a detected movable entity and optional affordance, avoidance, rotation, velocity, and gripper maps.In MPC settings, the movable entity and input value maps are functions that can be re-evaluated using the latest environment observation.
- Geometric utilities: cm2index and index2cm convert displacement between centimeter distances and voxel-coordinate indices along a direction.The two APIs are inverses for translating between world-coordinate distances and voxel displacements.
- Geometric utilities: pointat2quat converts a desired end-effector pointing direction into a target quaternion.The API provides orientation targets for the end-effector.
- Value-map utilities: Default maps support attraction, repulsion, orientation, gripper state, and velocity scaling, while set voxel by radius assigns values within a specified radius.Affordance maps attract at high values, avoidance maps repel, gripper maps use 1 for closed and 0 for open, and velocity maps encode scale factors.
A.4 Real-World Environment Setup
The real-world setup uses a Franka Panda with impedance-controlled operational-space control and two RGB-D cameras streaming observations at 20 Hz. Methods are evaluated with and without disturbances and compared against an action-primitive Code as Policies variant.
- Hardware and sensing: The setup uses a Franka Emika Panda robot with Deoxys impedance-based Operational Space Control and two Azure Kinect RGB-D cameras recording at 20 Hz.The cameras are mounted at opposite table corners and provide real-time RGB-D observations at the start of each rollout.
- Evaluation settings: Each method is evaluated both without disturbances and with pre-selected sequences of random forces, object displacements, and task-progress reversals.Examples include pulling a drawer open while the robot is closing it and displacing task-relevant or distractor objects.
- Baseline: The baseline is a Code as Policies variant using an LLM with move-to-position, quaternion-rotation, velocity-setting, and gripper-opening or closing primitives.Pick-and-place primitives are excluded because they would be tailored to a particular task suite.
A.4.1 Tasks
The tasks evaluate language-conditioned manipulation across object avoidance, table setting, drawer closing, bottle opening, and sweeping trash into a dustpan.
- Move & Avoid: Move & Avoid requires moving to the top of one randomized everyday object while staying away from another.Objects are selected from apple, banana, yellow bowl, headphones, mug, and wood block.
- Set Up Table: Set Up Table requires placing utensils for pasta.The instruction is: “Please set up the table by placing utensils for my pasta”.
- Close Drawer: Close Drawer requires closing either the top or bottom drawer, specified by deixis.The instruction uses “[deixis]” with the alternatives “top” or “bottom”.
- Open Bottle: Open Bottle requires turning open a vitamin bottle.The instruction is: “Turn open the vitamin bottle”.
- Sweep Trash: Sweep Trash requires sweeping paper trash into a blue dustpan.The instruction is: “Please sweep the paper trash into the blue dustpan”.
A.5 Simulated Environment Setup … A.6 Prompts
The simulated setup evaluates language-conditioned manipulation in a Franka Panda tabletop environment across 13 randomized tasks, while the prompts define modular components for composing and executing value-map-based behaviors. The appendix also documents the simulation results format and the separate simulation and real-world prompt variants.
- A.5 Simulated Environment Setup: A Franka Emika Panda in SAPIEN executes desired 6-DoF end-effector poses through inverse-kinematics interpolation and PD-controlled waypoint following.The environment contains 10 colored blocks, 10 colored lines, and an articulated cabinet with 3 drawers, initialized per task.
- A.5.1 Tasks: The custom suite contains 13 tasks with templated instructions, randomized object selections and attributes, feasible non-completed resets, and seen versus unseen attribute splits.The full attribute lists cover positions, objects, prepositions, deixis, regions, velocities, lines, and distances.
- A.5.2 Full Results on Simulated Environments: The simulated evaluation reports success rates averaged across 20 episodes for seen tasks and unseen tasks, with “SA” denoting seen attributes and “UA” denoting unseen attributes.These results are presented in Table 4.
- A.6 Prompts: The planner converts a user instruction L into sub-tasks ℓi for the composer, but simulation omits the planner because evaluated tasks contain a single manipulation phase.The composer takes each sub-task instruction and invokes the necessary value-map LMPs to compose affordance and constraint maps.
- A.6 Prompts: The parse-query-object component returns matching object or part instances with center position, occupancy grid, and mean normal vector.Separate simulation and real-world prompt files are provided for this component.
- A.6 Prompts: Value-map components convert composer-provided natural-language parameterizations into NumPy arrays for affordance, avoidance, rotation, gripper action, and end-effector velocity maps.Each map has separate simulation and real-world prompt files.