PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural MP: A Generalist Neural Motion Planner

Murtaza Dalal, Jiahui Yang, Russell Mendonca, Youssef Khaky, Ruslan Salakhutdinov, Deepak Pathak

arXiv:2409.05864v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Motion planning often solves each problem from scratch, while humans use prior experience to move quickly and reactively. Neural MP distills expert plans from diverse simulated scenes into a generalist policy and refines its paths at test time. In real-world tasks, it improves success over sampling-based planning and runs faster than AIT*.

  • Problem

    Motion planners often work from scratch, while scaling real-world data collection and diverse obstacle arrangements for learning remains costly and labor-intensive.

    Source: §I · pp. 1–2

  • Method

    Neural MP trains a reactive generalist policy on expert plans from diverse simulated scenes and applies lightweight test-time optimization to refine paths.

    Source: §I · p. 2, C. Deploying Neural Motion Planners · p. 5

  • Results

    Over 20% higher motion-planning success than sampling-based planning, while running 2.5x–20x faster than AIT*.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Neural MP achieves a 95.83% success rate across three tasks when test-time optimization is included.

    Source: §V · p. 6

    The model is susceptible to point-cloud quality and still does not handle tight spaces well.

    Source: §VI · p. 7

Abstract

from arXiv · show

The current paradigm for motion planning generates solutions from scratch for every new problem, which consumes significant amounts of time and computational resources. For complex, cluttered scenes, motion planning approaches can often take minutes to produce a solution, while humans are able to accurately and safely reach any goal in seconds by leveraging their prior experience. We seek to do the same by applying data-driven learning at scale to the problem of motion planning. Our approach builds a large number of complex scenes in simulation, collects expert data from a motion planner, then distills it into a reactive generalist policy. We then combine this with lightweight optimization to obtain a safe path for real world deployment. We perform a thorough evaluation of our method on 64 motion planning tasks across four diverse environments with randomized poses, scenes and obstacles, in the real world, demonstrating an improvement of 23%, 17% and 79% motion planning success rate over state of the art sampling, optimization and learning based planning methods. Video results available at mihdalal.github.io/neuralmotionplanner

I. INTRODUCTION

Neural MP addresses slow, from-scratch planning by distilling expert plans collected in diverse simulation scenes into a reactive policy, then refining plans at test time. The authors report higher real-world success rates than sampling-, optimization-, and learning-based methods across 64 tasks.

I. INTRODUCTION

  • Conventional planners often plan from scratch and reuse little prior information, making motion planning a speed, capability, and usability bottleneck in manipulation.Humans instead move reactively and draw on prior experience to handle varied obstacles and problem instances.Source: §I · p. 1
  • Real-world data collection requires substantial human time and effort, while diverse obstacle arrangements are costly and labor-intensive to construct.The authors note that existing real-world scene and object-arrangement diversity remains limited for learning obstacle avoidance that transfers to real-world settings.Source: §I · p. 2
  • Simulation supplies diverse expert-planner data for a reactive generalist policy, while lightweight test-time optimization uses predicted future states to seek a safe path.The policy is trained on data from 1 million scenes and is intended to generalize to novel obstacles and scene configurations.Source: §I · p. 2
  • Success rates improve by 23% over sampling-based, 17% over optimization-based, and 79% over neural motion planning methods across 64 real-world tasks.The evaluation spans four diverse environments with variation in poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal pairs.Source: §I · p. 2

II. RELATED WORK

Neural MP combines procedural simulation data generation and large-scale imitation learning with neural planning and test-time optimization. It emphasizes varied, realistic scenes and transfer to real-world tasks.

II. RELATED WORK

  • Neural MP combines procedural scene generation with large-scale imitation learning to build strong priors transferred to more than 64 real-world motion-planning problems.The related-work discussion positions this combination alongside large-scale imitation learning and sim2real reinforcement-learning approaches.Source: §II · p. 2
  • Compared with MotionBenchmaker’s primitive, sparse scenes, Neural MP uses articulated programmatic assets, up to five large obstacles, Objaverse meshes, and tightly packed obstacles.Its rule-based scene generation is designed specifically for neural motion-planning data rather than LLM-proposed tasks.Source: §II · p. 2
  • Neural MP extends neural-planner imitation with more diverse scenes, multimodal sequence modeling, and test-time optimization, distinguishing it from MPNet and MπNets.The training environments vary poses, procedural assets, and mesh configurations to support generalization.Source: §II · p. 2, Figure 2 · p. 3

III. NEURAL MOTION PLANNING

Neural MP uses large-scale simulation experience to train generalist motion-planning policies, then applies test-time optimization to improve their deployment performance.

III. NEURAL MOTION PLANNING

  • Large amounts of expert-planner data generated in simulation are the basis for training generalist neural motion-planning policies.The authors describe these policies as capable of generalizing to out-of-distribution settings through deep-learning architectures and diverse training data.Source: §III · p. 3
  • At deployment, test-time optimization selects the best path from multiple policy-generated options to improve performance.The passage presents this as an additional step beyond the base learned policy.Source: §III · p. 3
  • The approach is organized around simulation data generation, policy learning, and deployment-time optimization.The policy is intended to generalize beyond its training distribution.Source: §III · p. 3

A. Large-scale Data Generation

The data pipeline procedurally builds varied scenes, adds everyday-object meshes, and collects smoothed expert plans in obstacle-rich configurations.

A. Large-scale Data Generation

  • Procedural assets span six everyday categories and use parameterized cuboids whose constraints preserve category-specific structures and articulations.The categories are shelves, cubbies, microwaves, dishwashers, open boxes, and cabinets.Source: A. Large-scale Data Generation · p. 3
  • Objaverse meshes augment procedural assets with everyday objects sampled in task-relevant locations, such as inside cubbies or cabinets.This augmentation addresses objects outside the procedural categories, including bowls, utensils, and potted plants.Source: A. Large-scale Data Generation · p. 3
  • Scene construction adjusts colliding assets along an effective collision normal rather than relying on rejection sampling, which becomes costly and favors simpler scenes.The method iteratively adds assets while shifting their positions to resolve collisions.Source: A. Large-scale Data Generation · pp. 3–4
  • AIT* supplies expert plans, with tight-space configurations sampled 50% of the time to collect trajectories that maneuver around obstacles.Grasped objects are also randomized; planner waypoints are smoothed because far-apart waypoints create large action jumps and sparse learning data.Source: A. Large-scale Data Generation · p. 4

B. Generalist Neural Policies

Neural MP uses scene, joint-state, and goal observations with a sequence model to imitate expert plans. Its multimodal output predicts joint-angle changes for the next waypoint.

Observations

  • The policy observes a point cloud grounded in the robot base frame, current joint angles, and target joint angles.The point-cloud representation is view agnostic and described as largely consistent between simulation and real environments.Source: B. Generalist Neural Policies · p. 4

Network Architecture

  • An LSTM processes two-step histories of joint embeddings, with PointNet++ encoding segmented point clouds and MLPs encoding joint and goal vectors.The point cloud is segmented into robot, obstacle, and target-robot components before encoding.Source: B. Generalist Neural Policies · p. 4

Policy Output

  • A Gaussian mixture model predicts changes in joint angles for the next waypoint, and its negative log-likelihood objective outperforms PointMatch, diffusion, and action-chunking losses in the reported experiments.The GMM is used to represent the multimodal plans produced by sampling-based planners.Source: B. Generalist Neural Policies · p. 5

C. Deploying Neural Motion Planners

At inference, Neural MP samples policy trajectories and selects one with minimal predicted robot–scene intersection. Real-robot execution uses interpolated joint waypoints and vision-based collision checking.

Test time Optimization

  • Test-time optimization selects the trajectory with the least predicted robot–scene intersection from 100 policy samples.The procedure uses a linear forward model that assumes static obstacles and accurate waypoint tracking.Source: C. Deploying Neural Motion Planners · p. 5

Sim2real and Deployment

  • The robot executes predicted joint waypoints after linear interpolation, using a joint-space controller and point clouds from four calibrated cameras.The setup uses four Intel RealSense cameras positioned at the table corners.Source: C. Deploying Neural Motion Planners · p. 5
  • The real-world collision checker uses an SDF between the scene point cloud and a spherical robot representation, taking 0.01–0.02 seconds per query but potentially lacking precision in tight spaces.Source: C. Deploying Neural Motion Planners · p. 5

IV. EXPERIMENTAL SETUP

The evaluation tests motion planning across four real-world environments with randomized scenes and start–goal configurations. Methods are compared under an open-loop protocol using goal-reaching success, collision rate, and execution time.

Environment Design

  • Four environments—Bins, Shelf, Articulated, and in-hand—vary obstacle layouts, object poses, table height, and start–goal configurations, including free-space and tight-space cases.Each environment has four scenes and 12 large receptacles; two scenes have obstacles and two do not.Source: §IV · pp. 5–6

Evaluation Protocol

  • All methods are evaluated on open-loop plans executed on the robot, measuring goal success, collision rate, and time to reach the goal.Success requires reaching within 1 cm and 15 degrees of the target end-effector pose without collision.Source: §IV · p. 6

Comparisons

  • Real-world comparisons include AIT* with 80 seconds and 10 seconds of planning, Curobo, and the neural planner MπNets.AIT*-80s is the training expert; AIT*-3s cannot find a plan for any real-world task.Source: §IV · p. 6

V. EXPERIMENTAL RESULTS

Neural MP performs strongly across real-world free-hand and in-hand tasks, while also outperforming learning-based alternatives in simulation. Results additionally show gains from test-time optimization, diverse object data, and increased training data.

Free Hand Motion Planning

  • 95.83% success rate: Neural MP with test-time optimization achieves the best performance across the three free-hand tasks.The base policy alone achieves 66.67% versus 72.92% for AIT*-80s while using 1 second of planning time.Source: §V · p. 6, Table I · p. 6

In-Hand Motion Planning

  • 81% versus 31% and 44%: Neural MP performs better on in-hand tasks than the policy without test-time optimization and the model without Objaverse data.The method performs well on in-distribution objects, struggles with the oversized toy sword, and generalizes to bookcase insertion.Source: §V · pp. 6–7
  • 81.25% versus 43.75%: models trained with Objaverse data perform better than models trained only on cuboid-based parametric assets.The same experiments report that test-time optimization largely removes collisions on out-of-distribution in-hand tasks.Source: §V · p. 7

Dynamic Motion Planning

  • 63.33% versus 10%: Neural MP performs better than MπNets when avoiding introduced dynamic obstacles, though it struggles with the taller monitor.The evaluation uses closed-loop planning and single-step test-time optimization.Source: §V · p. 7

Comparisons to Learning-based Motion Planners

  • Neural MP outperforms EDMP and MπNets across the reported simulation test tasks, with results spanning 5,400 problems in the MπNets dataset.The authors attribute the performance to sequence modeling, multimodal GMM fitting, and test-time collision pruning.Source: §V · p. 7, Table II · p. 7

Data Scaling

  • 80% success rate: the model trained on 1 million trajectories succeeds on held-out tight-space shelf and bin configurations.Success rises from 15% with 1,000 trajectories to 50% with 10,000 and 65% with 100,000.Source: §V · p. 7

Ablations

  • GMM training exceeds L2, L1, and PointMatch losses by 7%, 12%, and 24%, respectively; omitting both joint and goal vectors causes a 75% performance drop.Using only the goal vector or only the joint vector causes drops of 62% and 65%, respectively.Source: §V · p. 7

VI. DISCUSSION AND LIMITATIONS

Neural MP distills procedural simulation data and sampling-planner demonstrations into a policy refined at test time. The authors report faster, higher-success real-world planning while noting limitations in point-cloud quality, tight spaces, and optimization time.

VI. DISCUSSION AND LIMITATIONS

  • 2.5x-20x faster than AIT* while improving motion-planning success rate by over 20%.The method is reported to generalize across a wide distribution of task instances.Source: §VI · p. 7
  • Neural MP scales procedural scene generation, distills sampling-based planning, and refines its policy at test time.The authors describe this as a data-driven policy for motion planning.Source: §VI · p. 7
  • The method is susceptible to point-cloud quality and still does not handle tight spaces well.The authors suggest improving 3D representations or RL fine-tuning as possible directions.Source: §VI · p. 7
  • Test-time optimization makes Neural MP slower than running the policy directly.The authors identify learned collision checking as a possible way to address this.Source: §VI · p. 7

A. Detailed Free Hand Motion Planning Results

The detailed results discuss free-hand planning speed and safety, and compare in-hand performance across methods and training variants. The reported tables state that Neural MP performs best across tasks in both settings, with larger improvements as task complexity or object challenge increases.

A. Detailed Free Hand Motion Planning Results

  • Neural methods have higher collision rates than sampling-based planners, but Neural MP's rate is significantly lower than MπNets'.The authors attribute Neural MP's lower collision rate to test-time optimization pruning bad trajectories; many remaining collisions are slight grazes.Source: A. Detailed Free Hand Motion Planning… · p. 10
  • Neural MP performs best across free-hand tasks, with greater improvement as task complexity grows.The method uses additional compute to roll out 100x more trajectories and select one using SDF-based collision checking.Source: Table III · p. 10, A. Detailed Free Hand Motion Planning… · p. 10

B. Detailed In-hand Motion Planning Results

C. Test-time Optimization Analysis

Test-time optimization filters sampled policy trajectories for collisions, while ablations examine design choices affecting success. The supplied methods and setup passages also describe scene and expert-data generation, network training, and real-world deployment.

C. Test-time Optimization Analysis

  • 25% of sampled trajectories are collision free, and the method selects from that subset; choosing the first among those with fewest expected collisions performed well.Collision estimates compare the robot SDF with the scene point-cloud; further selection for maximum distance from the scene was not found necessary.Source: Figure 5 · p. 10, C. Test-time Optimization Analysis · p. 10

IX. ABLATIONS

  • GMM outperforms L2 loss, L1 loss, and PointMatch Loss (MπNets) by 7%, 12%, and 24%, respectively.The authors attribute the result in part to GMMs fitting multimodal planner trajectories; PointMatch struggles to distinguish end-effector orientations flipped by 180 degrees.Source: §IX · p. 11, Figure 6 · p. 11
  • Removing either proprioceptive input q or goal input g, or both, severely harms performance compared with including them.The authors hypothesize that proprioception provides a richer signal for the correct delta action.Source: §IX · p. 11, Figure 6 · p. 11
  • History length 2 achieves 94% success, compared with 92.67%, 68%, and 14.67% for lengths 4, 8, and 16.The authors note that shorter history also speeds training, and suggest dense point-cloud observations make training-time partial observability fairly low.Source: §IX · p. 11, Figure 6 · p. 11
  • The 4M-parameter small encoder achieves 94%, versus 93% for medium and 92% for large, with no significant performance effect from encoder size.The authors use the small model in their experiments.Source: §IX · p. 11, Figure 6 · p. 11
  • Transformers and LSTMs perform similarly, while Transformers have 2x slower inference; MLP decoders perform significantly worse than either.ACT performs poorly, which the authors largely attribute to its L1 loss preventing effective handling of planner multimodality; they use LSTMs in their evaluation.Source: §IX · p. 11, Table V · p. 11
  • Policies trained on the Neural MP dataset generalize best to held-out scenes among the compared dataset-generation approaches.The comparison trains policies on Neural MP, MπNets, and MotionBenchMaker data and evaluates on held-out simulation tasks mirroring RealBins and RealShelf, without Objaverse meshes.Source: §IX · p. 11, Table VI · p. 11

X. PROCEDURAL SCENE GENERATION DETAILS; B. Motion Planner Experts; C. Data Pipeline Parameters and Compute; A. Real Robot Setup

B. Tasks

The evaluation tasks vary obstacle type, geometry, and configuration, including a task that requires planning with a grasped object. The perception analysis compares simulated and real point-clouds and reports policy operation despite noisier, partial real-world observations.

B. Tasks

  • The Bins, Shelf, Articulated, and In-Hand tasks test avoidance around bins, shelf rungs, a cabinet’s doors, drawer and cubbies, and a grasped object, respectively.Task setups randomize relevant scene properties, including bin and shelf poses, cabinet joints and cubby sizes, and in-hand objects and configurations.Source: B. Tasks · pp. 16–17

C. Perception Visualization and Analysis

  • Real-world point-clouds are noisier and often partial than simulated ones, yet the policy operates well in the Bins and Shelf scenes.The authors suggest PointNet++ and training on diverse boxes and shelves may enable generalization to partial real-world observations.Source: C. Perception Visualization and Analysis · p. 17, Figure 8 · p. 17
  • Fig. 9 presents images of the 16 evaluation scenes.The supplied passage identifies the scenes but does not specify additional visual encodings.Source: Figure 9 · p. 18

Related papers

Loading 2409.05864v1…