PaperlayerView exact revision on arXiv

Source-linked AI summary

LaMP: Learning Vision-Language-Action Policy with 3D Scene Flow as Latent Motion Prior

Xinkai Wang, Chenyi Wang, Yifu Xu, Mingzhe Ye, Fucheng Zhang, Jialin Tian, Xinyu Zhan, Lifeng Zhu, Cewu Lu, Lixin Yang

arXiv:2603.25399v2cs.CVcs.RO

TL;DR

VLA models often infer 3D physical interactions implicitly from 2D semantic features, a limitation for unfamiliar spatial dynamics. LaMP uses one-step partially denoised 3D scene-flow features to guide action prediction through gated cross-attention. It reports the highest average success rates across LIBERO, LIBERO-Plus, and SimplerEnv-WidowX, and improved robustness on LIBERO-Plus OOD perturbations.

  • Problem

    VLA models lack an explicit intermediate representation of 3D dynamic relationships and instead learn physical interactions implicitly from action supervision.

    Source: §1 · p. 2

  • Method

    LaMP aligns a flow-matching Motion Expert with an Action Expert, using gated cross-attention to condition action prediction on one-step partially denoised 3D motion states.

    Source: §1 · p. 3

  • Results

    LaMP achieves the highest average success rates across LIBERO, LIBERO-Plus, and SimplerEnv-WidowX, outperforming evaluated VLA baselines across the benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LaMP demonstrates dense 3D scene flow as an effective latent motion prior in pretrained VLA frameworks, with particular gains in long-horizon planning and cross-embodiment transfer.

    Source: §6 · p. 15

    The Motion Expert uses fixed spatial resolution and temporal horizon and depends on TraceForge-generated 3D scene-flow supervision.

    Source: §6 · p. 15

Abstract

from arXiv · show

We introduce \textbf{LaMP}, a dual-expert Vision-Language-Action framework that embeds dense 3D scene flow as a latent motion prior for robotic manipulation.Existing VLA models regress actions directly from 2D semantic visual features, forcing them to learn complex 3D physical interactions implicitly.This implicit learning strategy degrades under unfamiliar spatial dynamics.LaMP addresses this limitation by aligning a flow-matching \emph{Motion Expert} with a policy-predicting \emph{Action Expert} through gated cross-attention.Specifically, the Motion Expert generates a one-step partially denoised 3D scene flow, and its hidden states condition the Action Expert without full multi-step reconstruction.We evaluate LaMP on the LIBERO, LIBERO-Plus, and SimplerEnv-WidowX simulation benchmarks as well as real-world experiments.LaMP consistently outperforms evaluated VLA baselines across LIBERO, LIBERO-Plus, and SimplerEnv-WidowX benchmarks, achieving the highest reported average success rates under the same training budgets. On LIBERO-Plus OOD perturbations, LaMP shows improved robustness with an average 9.7\% gain over the strongest prior baseline.Our project page is available at https://summerwxk.github.io/lamp-project-page/.

1 Introduction

LaMP addresses the lack of explicit 3D dynamic representations in VLA policies, which otherwise infer physical interactions from action supervision. It integrates a latent 3D motion prior into action prediction and reports the highest average success rates across three benchmarks.

Motivation

  • VLA models rely on 2D-centric semantic features and implicitly learn 3D physical interactions, leaving them brittle under unfamiliar spatial dynamics.The paper highlights depth-sensitive tasks such as peg insertion and tight-clearance stacking under camera or layout shifts.Source: §1 · p. 2

Prior work

  • Prior methods use 2D flow, pixel-space futures, static 3D geometry, or standalone 3D pipelines, limiting their ability to provide continuous scene-level geometry within a unified language-conditioned policy.2D flow lacks explicit depth; pixel-space predictions conflate appearance with actionable kinematics.Source: §1 · pp. 2–3

Approach

  • LaMP aligns a flow-matching Motion Expert with an Action Expert, conditioning action prediction on one-step partially denoised motion states through gated cross-attention.The one-step strategy avoids full multi-step reconstruction, while the gate adaptively increases motion guidance when beneficial.Source: §1 · p. 3

Results

  • LaMP achieves the highest average success rates across LIBERO, LIBERO-Plus, and SimplerEnv-WidowX, outperforming general VLAs and geometry-aware models under identical data budgets.The paper also reports experiments on real-world tasks.Source: §1 · p. 3

2 Related Work

Prior VLA and visuomotor approaches encode dynamics implicitly or use intermediate representations that lack dense temporal-spatial geometry. LaMP incorporates future dense 3D scene flow as an internal prior in a pretrained VLA framework.

2.1 General Vision-Language-Action Models

  • VLA policies map observations to actions directly, while diffusion and point-cloud policies address multimodality or spatial ambiguity without modeling continuous temporal geometric dynamics.3D-VLA uses point-cloud inputs but is not treated as a direct baseline because it does not follow the SimplerEnv/LIBERO protocols.Source: §2.1 · pp. 3–4

2.2 World Model for Visuomotor Policy

  • World-model approaches predict future states, but pixel-space methods spend capacity on appearance and 3D motion-prior methods commonly operate as standalone, per-task pipelines.LaMP instead integrates 3D flow as an internal prior in a pretrained VLA; naive multi-layer fusion caused representational collapse, motivating gated cross-attention.Source: §2.2 · p. 4

2.3 Intermediate Representations for Manipulation

  • Optical flow, traces, depth supervision, and latent actions add geometric awareness but remain 2D or static; LaMP uses dense scene-level 3D flow for temporal-spatial dynamics.LaMP’s overview also describes motion pretraining, frozen-motion-expert features for action post-training, and curation of 1.6M observation-language-motion triplets.Source: §2.3 · p. 4, Figure 2 · p. 5

3 Method

LaMP factors action prediction through an explicit latent 3D motion representation. A flow-matching Motion Expert supplies partially denoised motion features to a gated Action Expert, trained and used without full motion reconstruction at inference.

3.1 Problem Formulation

  • LaMP factors action prediction into a Motion Expert modeling motion from VLM features and an Action Expert predicting actions from both perception and motion.The factorization does not assume conditional independence; the Action Expert jointly conditions on perceptual features and the motion prior.Source: §3.1 · p. 5

3.2 Motion Expert: Learning a Latent 3D Motion Prior

  • The Motion Expert represents motion as dense 3D scene flow over 400 keypoints and 32 future steps, and uses conditional flow matching to learn scene dynamics.At inference, one denoising step out of ten produces hidden motion states for action conditioning, avoiding full motion generation.Source: §3.2 · p. 6

3.3 Motion Guidance and Action Expert

  • Single-layer gated cross-attention fuses partially denoised motion features into the last-layer VLM features, while the Action Expert predicts actions from the guided representation.The learnable gate starts at zero and can increase motion guidance when beneficial; action denoising is conditioned on explicit 3D dynamics.Source: §3.3 · p. 7

3.4 Two-Stage Training and Inference

  • Training first learns motion from 1.6M observation-language-motion triplets, then freezes the Motion Expert and VLM while training the guidance module and Action Expert.Inference uses one motion-denoising step, gated fusion, and action-space denoising; it does not require full motion reconstruction.Source: §3.4 · pp. 7–8

4 Simulation Experiments

LaMP’s simulation evaluation reports leading average success on LIBERO and SimplerEnv-WidowX, with a further zero-shot advantage on LIBERO-Plus. Ablations compare motion-prior choices and fusion strategies, finding gains from 3D flow and gated integration.

4.1 Evaluation Setup

  • Evaluation covers language-conditioned manipulation, spatial reasoning, and long-horizon planning across LIBERO and LIBERO-Plus, plus visual-mismatch robustness on SimplerEnv-WidowX.LIBERO uses 500 expert demonstrations per task and evaluates each task over 50 episodes; the Motion Expert is pretrained, then frozen with the Qwen3-VL-4B-Instruct backbone while the Action Expert and guidance module are trained.Source: §4.1 · pp. 8–9
  • LaMP achieves 98.3% average success on LIBERO and 79.2% on SimplerEnv-WidowX, the best reported averages on both benchmarks.On SimplerEnv-WidowX, it exceeds FlowVLA’s 74.0%.Source: Table 1 · p. 9, §4.1 · pp. 9–10
  • On LIBERO-Plus zero-shot OOD evaluation, LaMP averages 79.3% versus OpenVLA-OFT’s 69.6%, a 9.7-percentage-point gain.All methods are trained on original LIBERO demonstrations and evaluated on seven perturbation dimensions without additional training data.Source: §4.1 · p. 9, Table 2 · p. 10
  • LaMP also transfers to bimanual RoboTwin 2.0, reaching 83.9/81.8 average success across 10 tasks under clean/randomized settings.The authors attribute SimplerEnv-WidowX transfer to reasoning in 3D camera coordinates rather than pixel space.Source: §4.1 · p. 10

4.2 Ablation Studies

  • Removing the Motion Expert reduces LIBERO average success from 98.3% to 92.4% and SimplerEnv-WidowX average success from 79.2% to 56.3%.On LIBERO-Long, success falls from 96.7% to 78.2%; on Stack Block, it falls from 75.0% to 25.0%.Source: Figure 3 · p. 11, §4.2 · p. 11
  • Compared with 2D flow, the 3D prior improves Stack Block by 16.7% and Put Spoon by 8.3%.On Put Eggplant, the methods are comparable: 91.7% for 2D flow and 95.8% for 3D flow.Source: Figure 3 · p. 11, §4.2 · p. 11
  • Gated cross-attention outperforms Add and Concat+MLP across all four SimplerEnv-WidowX tasks.On Stack Block, gated fusion reaches 75.0%, versus 33.3% for Add and 45.8% for Concat+MLP.Source: §4.2 · pp. 11–12

5 Real-World Experiments

In real-world experiments, LaMP outperforms π0 and 3D FDP across task categories, while its Motion Expert predicts physically meaningful trajectories before execution.

5.1 Setup

  • The evaluation uses a Flexiv Rizon 4 arm with a Robotiq gripper, RGB-D observations, and one unified policy trained across three tasks.Training uses 160 demonstrations; each method receives 10 trials per in-domain task and 5 trials per OOD condition, with task progress as the primary metric.Source: §5.1 · pp. 12–13, §5.2 · p. 14

5.2 Evaluation

  • The three in-domain tasks cover rigid-body pick-and-place, deformable towel folding, and long-horizon bread serving and egg pouring; OOD tests alter layouts, objects, or backgrounds.The baselines are π0 and 3D FDP, evaluated under identical data budgets.Source: §5.2 · pp. 13–14

5.3 Results

  • LaMP outperforms π0 and 3D FDP across all real-world task categories, including deformable and long-horizon tasks.On Pick-and-Place, LaMP reaches 80% average task progress versus 70% for π0 and 53% for 3D FDP; on Deformable manipulation, it reaches 50% versus 40% for both baselines.Source: Figure 6 · p. 14, §5.3 · pp. 14–15

5.4 Motion Foresight Visualizations

  • The Motion Expert’s predicted 3D trajectories anticipate the arm’s approach and manipulation across grasping, folding, and long-horizon tasks.The Action Expert conditions on one-step partially denoised hidden states at τ = 0.1; the full ODE rollout is shown only for visualization.Source: §5.4 · p. 15

6 Limitations and Future Work

LaMP’s motion expert has fixed spatial resolution and temporal horizon, relies on TraceForge-generated supervision, and injects motion only at the final VLM layer.

6 Limitations and Future Work

  • The Motion Expert is limited to fixed spatial resolution and temporal horizon, and depends on TraceForge-generated 3D scene flow supervision.The paper identifies adaptive resolution, longer horizons, and learning from unlabeled in-the-wild video as future directions.Source: §6 · p. 15
  • Motion is injected only at the last VLM layer to avoid representational collapse; multi-layer fusion ablations and broader real-world, multi-embodiment evaluation remain future work.Source: §6 · p. 15

A Training and Inference Algorithms

A.1 Stage-2 Motion-Guided Policy Training

Stage-2 training freezes the Motion Expert and trains the guidance module and Action Expert using partial motion states; inference uses the same guidance before action integration.

A.1 Stage-2 Motion-Guided Policy Training

  • Stage-2 training extracts a motion hidden state with one denoising step, applies single-layer gated cross-attention, then updates the Guidance module and Action Expert with the action flow-matching loss.The Motion Expert is frozen during this stage.Source: §A.1 · p. 20

A.2 LaMP Inference

  • Inference extracts VLM features, predicts a partial motion state, guides the features, and integrates the Action Expert’s velocity predictions over N = 10 steps to return a clean action.Source: §A.2 · pp. 20–21

B.1 Motion Expert Pretraining

  • The Motion Expert is pretrained on 1.6M observation-language-motion triplets generated by the TraceForge pipeline.The supplied hyperparameter table is identified as the Motion Expert pretraining configuration.Source: §B.1 · p. 21, Table 3 · p. 21

B.2 Action Expert Training

Table 4 summarizes the Action Expert training setup across benchmarks.

B.2 Action Expert Training

C Task Progress Metric Definition

Real-world task progress is the average percentage of stages completed per episode across evaluation trials. Pick-and-place, deformable, and long-horizon tasks use different stage counts and per-stage scores.

C Task Progress Metric Definition

  • Real-world task progress is the percentage of stages completed within an episode, averaged across evaluation trials for each task.The average task progress score is the primary real-world evaluation metric.Source: C Task Progress Metric Definition · p. 22
  • Pick-and-place tasks have two stages, scored 0.5 for completing grasping and 1.0 for completing placement and release.Source: C Task Progress Metric Definition · p. 22
  • Deformable tasks have four stages, each contributing 0.25 to the progress score.Stages cover folding two corners, adjusting the fold, and finalizing the target configuration.Source: C Task Progress Metric Definition · p. 22
  • Long-horizon tasks have three stages, each contributing approximately 0.33 to the progress score.The stages cover plating bread, preparing the egg, and placing it on the bread.Source: C Task Progress Metric Definition · p. 22

D Runtime and Efficiency Analysis

On Flexiv with an RTX 4090 and batch size 1, partial denoising adds modest runtime and memory overhead. With action chunking, LaMP meets the stated real-robot deployment requirements.

D Runtime and Efficiency Analysis

E Bimanual Evaluation on RoboTwin 2.0

LaMP attains the best average success under both clean and randomized settings across 10 RoboTwin 2.0 bimanual tasks. It also wins the majority of individual tasks, supporting transfer of its embodiment-agnostic motion prior to bimanual manipulation.

E Bimanual Evaluation on RoboTwin 2.0

  • LaMP achieves the best average success under both clean and randomized settings and wins the majority of individual tasks across 10 RoboTwin 2.0 tasks.The comparison is against π0.5 and X-VLA; randomized evaluation uses strong domain randomization.Source: E Bimanual Evaluation on RoboTwin 2.0 · p. 23, Table 6 · p. 23

Related papers

Loading 2603.25399v2…