PaperlayerView exact revision on arXiv

Source-linked AI summary

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng

arXiv:2610.13732v1cs.CVcs.AIcs.RO

TL;DR

Collecting embodied interaction data is costly, and human videos lack robot action labels while existing annotations can be noisy. LAPT-Scale learns continuous latent actions from video transitions and uses them to pre-train a VLA policy before robot embodiment alignment and task-specific post-training. As annotated human video scales from 1K to 5K hours, reported real-robot success rises from 63.3% to 85.0%.

  • Problem

    Embodied interaction data is costly to collect, while human videos lack robot action labels and explicit motion-estimation annotations can be noisy.

    Source: §1 · p. 1

  • Method

    LAPT-Scale learns continuous latent actions from human-video transitions and trains a VLA policy through video pre-training, target-robot alignment, and task-specific post-training.

    Source: §1 · p. 2

  • Results

    Real-robot success rises from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support latent actions as scalable human-video annotations for downstream manipulation, with reported robustness under tested position and appearance shifts.

    Source: §5 · p. 9

    Real-robot evaluation covers seven tasks on one target platform, with 20 trials per condition, limiting success-estimate precision and deployment-robustness claims.

    Source: §5 · p. 9

Abstract

from arXiv · show

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.

1 Introduction

Robot interaction data is costly to collect, while egocentric human videos offer broad interaction experience but lack robot action labels. LAPT-Scale converts those videos into continuous latent-action supervision and reports higher real-robot success as annotated video increases from 1K to 5K hours.

1 Introduction

  • Robot interaction data is costly to collect, and human videos lack robot action labels that map directly to robot control.Existing explicit motion-estimation approaches can produce noisy annotations, and simply adding more video does not necessarily improve policies.Source: §1 · p. 1
  • Discrete VQ latents can over-compress fine manipulation changes, while pixel reconstruction can underweight small hand-object interactions.Static regions may dominate pixel-wise losses, leaving transition-relevant structure weakly represented.Source: §1 · p. 2
  • LAPT-Scale-LAM learns continuous implicit actions with perceptual, semantic, and optical-flow alignment beyond pixel reconstruction.LAPT-Scale-Policy uses the annotations for human-video pre-training, target-robot embodiment alignment, and task-specific post-training.Source: §1 · p. 2
  • Success rises from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours.The scaling change comes from increasing Stage-1 human-video pre-training data.Source: §1 · p. 2

2 Related Work

Prior VLA and human-video approaches seek to reduce reliance on robot trajectories, but human observations and motion annotations do not directly specify robot-executable actions. LAPT-Scale instead uses continuous latent actions with multiple alignment signals to prepare human-video supervision before robot embodiment alignment.

Vision-language-action policy learning

  • VLA training remains constrained by action-labeled robot trajectories; egocentric approaches use explicit motion estimation or retargeting, while LAPT-Scale uses LAM-derived implicit actions.Simulation-mediated pipelines instead reconstruct demonstrations into physically checked robot-centered trajectories.Source: §2 · p. 2

Human video data for embodied learning

  • Human-video datasets span broad physical interactions, but human observations and motion annotations do not directly specify robot-executable actions.LAPT-Scale learns latent actions to bridge this missing-label gap before embodiment alignment.Source: §2 · pp. 2–3

Latent action models from videos

  • Many latent-action models use discrete or vector-quantized latents that can over-compress manipulation dynamics, and reconstruction-only training can be dominated by background regions.LAPT-Scale-LAM combines continuous latents with perceptual, self-supervised semantic, and optical-flow alignment signals.Source: §2 · p. 3

3 Method

The method derives continuous latent actions from human-video transitions, then trains a VLA policy through human-video pre-training, target-robot alignment, and task-specific post-training.

3.1 Problem Formulation & Data

  • The LAM is trained on 23.3K hours of egocentric video, while policy pre-training uses a cleaned 5.1K-hour subset selected for more reliable manipulation content.Filtering removes corrupted or blurry clips and segments dominated by camera motion or weak hand-object interaction.Source: §3.1 · p. 3

3.2 LAPT-Scale-LAM: Constructing Human Latent-Action Supervision

  • A spatio-temporal VAE encodes each adjacent-frame transition into a continuous low-dimensional latent action, while the decoder receives current-frame context to isolate change.The latent is sampled from a diagonal Gaussian posterior; the bottleneck can be as small as d = 32.Source: §3.2 · p. 4
  • Training combines reconstruction and perceptual, semantic, and optical-flow alignment to preserve visual fidelity, hand-object structure, and observed motion.Frozen DINOv2 features supply semantic alignment, and a frozen RAFT estimator supplies optical-flow alignment.Source: §3.2 · pp. 4–5

3.3 LAPT-Scale-Policy: Three-Stage VLA Policy Training

  • The same action expert trains first on LAM latent-action chunks, then robot-action chunks for embodiment alignment, and finally task-specific data, without a separate human-video head or reinitialization.At inference, the policy predicts executable robot actions; LAM-encoded actions serve only as Stage-1 supervision.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate downstream VLA performance in simulation and on real robots, scaling with annotated human video, and standalone LAM quality. They also examine how latent actions affect target-scene transitions.

4.1 Experimental Setup

  • The evaluation covers RoboTwin 2.0, RoboCasa-GR1, and seven real-robot tasks after 250 hours of target-robot embodiment alignment.The study reports policy success rates and uses human-video reconstruction metrics plus LARY linear-probe accuracy for LAM diagnostics.Source: §4.1 · p. 5, §4 · p. 5

4.2 Main Results

  • LAPT-Scale reaches 90.0% Easy and 89.3% Hard on RoboTwin 2.0, 64.8% average on RoboCasa-GR1, and 85.0% average across three original real-robot tasks.It slightly outperforms the strongest listed RoboTwin baselines and improves over π0.5 and JoyAI-RA on the original real-robot tasks.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 6

4.3 Scaling with Annotated Human Video

  • With robot data held fixed, LAPT-Scale-LAM improves real-robot success from 63.3% at 1K hours to 80.0% at 3K and 85.0% at 5K hours.Larger annotated datasets also reach lower Stage-1 pre-training losses.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Controlled Stage-1 and Alignment Ablations

  • Removing Stage 1 or using a compute-matched extended Stage 2 leaves both controls below the full model; removing flow causes the largest RoboTwin Hard drop, 11.5 points.The flow ablation has high PSNR despite its largest success-rate drop.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.5 Harder Real-Robot Tasks and Generalization

  • On four harder real-robot tasks, LAPT-Scale and π0.5 match at 85.0% in seen conditions, while reaching 70.0–75.0% and 40.0–50.0%, respectively, under tested shifts.The shifts include position, illumination, and background changes.Source: §4.5 · p. 8, Table 6 · p. 8

4.6 Human-Video LAM Evaluation

  • LAPT-Scale-LAM improves both transition reconstruction and latent-action semantics over VQ-LAM and MSE-only LAM.Qualitative reconstructions are most similar to ground truth around the moving hand and object.Source: Table 7 · p. 8, §4.6 · p. 8, Figure 5 · p. 8

4.7 Interpreting the Learned Latent Action Space

  • Replacing a target video’s latent action with a donor’s changes its short-horizon motion while preserving the target scene layout, suggesting transferable transition information.The clearest effects are changes in camera-relative hand-object motion.Source: §4.7 · p. 9, Figure 6 · p. 9

5 Conclusion and Limitations

LAPT-Scale turns egocentric human videos into supervision for embodied policy learning, with results supporting downstream manipulation gains and robustness under tested shifts. Its use depends on embodiment alignment, and the experiments have limited robot and task coverage.

5 Conclusion and Limitations

  • The approach uses LAM-based human-video supervision for manipulation and shows robustness under the tested position and appearance shifts.The conclusion describes increased usefulness as annotated human-video data grows.Source: §5 · p. 9
  • Visual latent actions are not guaranteed to align with independently controllable robot degrees of freedom, so target-robot embodiment alignment remains necessary.The real-robot study covers seven tasks on one platform, with 20 trials per condition, limiting success-estimate precision and warranting broader validation.Source: §5 · p. 9

A.1 Source Composition

  • The 23.3K-hour LAM corpus merges six egocentric sources, combining broad daily-activity coverage, dense manipulation, and target-relevant self-collected footage.The source mixture is designed to provide complementary scene, object, interaction, and viewpoint coverage.Source: §A.1 · p. 14

A.2 Filtering, Mixing, and Annotation

  • The pooled 23.3K hours favor coverage, while stricter visible-hand and object-interaction filtering yields a 5.1K-hour policy-pretraining subset.Segments dominated by blur, extreme ego-camera motion, missing hands, long static intervals, or severe occlusion are discarded.Source: §A.2 · p. 14
  • The 1K-, 3K-, and 5K-hour scaling subsets are nested when possible, and frozen-LAM transition annotations serve as deterministic Stage-1 policy targets.The stored annotation format supports human-video latent-action chunks in Stage 1 and executable robot action chunks in Stages 2–3.Source: §A.2 · pp. 14–15

B.1 LAM and Policy Training

The section describes a continuous-latent LAM training setup and a three-stage policy pipeline, then separates LAM training, offline annotation, and policy-training costs.

B.1 LAM and Policy Training

  • LAPT-Scale-LAM encodes each video transition as a continuous latent action and predicts the next frame; the full model adds LPIPS, DINO-feature, optical-flow, and KL terms to reconstruction.The MSE-only diagnostic omits the perceptual, semantic, and flow terms; the VQ-LAM baseline substitutes a discrete codebook.Source: §B.1 · p. 15
  • The LAM uses a 32-dimensional continuous Gaussian posterior, with MSE reconstruction, KL, LPIPS, DINO-feature, and optical-flow losses and specified weights.Training also uses paired augmentation with four repeats and resamples the latent action under augmentation.Source: §B.1 · p. 15
  • The policy uses human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training, with scaling experiments holding robot data, architecture, and schedule fixed.Stage 2 replaces the latent target with executable robot action chunks; Stage 3 adapts the same policy on task-specific robot data.Source: §B.1 · pp. 15–16

B.2 Compute and Hyperparameters

  • LAM training is a one-time cost; its frozen encoder annotates the cleaned 5.1K-hour subset and scaling subsets offline, and cached annotations feed Stage 1.Changing the annotated video pool is the stated exception that requires rerunning the encoder; Table 9 summarizes implementation, training, and compute details.Source: §B.2 · p. 16, Table 9 · p. 16

C Baseline Implementation Details

The section specifies how human-video supervision baselines are constructed and how robot data and task success are accounted for, enabling controlled comparisons of supervision forms.

C Baseline Implementation Details

  • Baseline variants share cleaned human-video subsets, Stage-1 policy architecture, 250-hour robot alignment data, and downstream post-training, isolating differences in human-video supervision.Table 12 summarizes implementation differences among the supervision baselines.Source: C Baseline Implementation Details · p. 17
  • The explicit hand-motion baseline uses reconstructed hand-pose trajectories as pseudo-action targets, with the same sampling, filtering, policy architecture, robot data, and training schedule as LAPT-Scale-LAM.The comparison is intended to isolate explicit hand-motion supervision versus LAM-derived implicit actions.Source: C Baseline Implementation Details · p. 17
  • Hand reconstruction can yield noisy pseudo-actions under occlusion, blur, unusual viewpoints, or contact-rich manipulation; LAM baselines instead learn transition variables directly from video.DreamDojo and MSE-only test reconstruction-driven latents, VQ-LAM tests a discrete bottleneck, and LAPT-Scale-LAM tests continuous latents with perceptual, semantic, and motion-aware alignment.Source: C Baseline Implementation Details · pp. 17–18

D Evaluation Protocol Details

  • Success requires the task goal within the evaluation horizon; simulation uses benchmark success detectors and termination rules, while real-robot success uses the final task state.Stage 2 uses 162,767 episodes and 26.7M frames; original tasks use 20 trials each, and four harder tasks use 320 trials per method across four domains.Source: D Evaluation Protocol Details · p. 18, §D.1 · p. 18

D.2 Simulation Benchmarks

The simulation results report average success on RoboTwin Easy and Hard and RoboCasa-GR1, while the real-robot subsection specifies the evaluated tasks and trial protocol.

D.2 Simulation Benchmarks

  • 90.0% Easy and 89.3% Hard are the main LAPT-Scale policy’s average RoboTwin 2.0 success rates.Hard adds stronger object-pose and scene-layout variation; compared methods share the task split, interfaces, and success definition.Source: §D.2 · p. 18
  • 64.8% is the main LAPT-Scale policy’s average success on RoboCasa-GR1 across 24 tabletop tasks and 50 rollouts per task.The reported value averages evaluated task instances rather than selecting the best task-specific checkpoint.Source: §D.2 · p. 18

D.3 Real-Robot Tasks

  • Real-robot evaluation uses an AgiBot G1 on Mouse Pad, Headphones, and Cup, with 20 trials per task and shared reset and success criteria.Table 13 summarizes downstream evaluation protocols.Source: §D.3 · p. 18, Table 13 · p. 18

E Additional Real-Robot Scaling Results

The additional-results section identifies average and per-task scaling tables and describes robot-only and compute-matched controls, but the supplied passages do not report their outcome values.

E Additional Real-Robot Scaling Results

  • Table 14 reports average success across three original real-robot tasks, with 20 trials per task and a shared 0-hour robot-only control.The supplied caption gives the table’s scope but not its success-rate values.Source: Table 14 · p. 19
  • Table 15 breaks down real-robot scaling by task and annotated human-video scale, with robot data held fixed and 20 trials per task.The table caption identifies the human-video scale as the amount used for VLA pre-training.Source: Table 15 · p. 19

E.1 Stage-1 Removal Control

  • The robot-only control removes Stage 1 while retaining the same initialization, robot data, and evaluation protocol; the compute-matched control extends Stage 2 to match training compute.Table 16 summarizes these Stage-1 removal and compute-matched controls.Source: §E.1 · p. 19, Table 16 · p. 19

F.1 Reconstruction Protocols

Reconstruction tests distinguish local transition fidelity from temporal stability, while cross-scene retrieval examines whether latent actions reflect motion rather than appearance. LAPT-Scale-LAM is more stable in autoregressive rollouts and groups visually different clips with similar short-horizon motion.

F.1 Reconstruction Protocols

  • Teacher-forced reconstruction uses the ground-truth current frame for each next-frame prediction; autoregressive rollout starts from the ground-truth first frame and then uses generated frames.The protocols assess local transition reconstruction and whether errors compound over multiple steps, respectively.Source: §F.1 · p. 19
  • Under teacher forcing, LAPT-Scale-LAM produces the sharpest reconstructions, while LAPA and VILLA-X show weaker local transition fidelity and MSE-only LAM smooths hand-object contacts.Each prediction uses a ground-truth previous frame, focusing the comparison on local transition reconstruction.Source: §F.1 · pp. 20–21, Figure 7 · p. 20
  • LAPT-Scale-LAM preserves visible hand structure and coherent short-horizon object motion longer in autoregressive rollouts than the compared latent-action models.In this protocol, models condition on their own generated frames, so errors can compound over time.Source: §F.1 · p. 21, Figure 8 · p. 21

F.2 Similar Motions Across Different Scenes

  • Latent-action replacement and cross-scene retrieval together support the interpretation that LAM tokens encode how a scene changes rather than only its appearance.Replacement changes target-sequence motion while leaving scene layout largely target-specific.Source: §F.2 · pp. 22–23
  • Clips with different backgrounds, objects, and hand appearances can be close in latent-action space when their short-horizon motion is similar.Nearest-neighbor retrieval excludes clips from the same source video; the clusters illustrate reusable transition patterns rather than near-duplicate frames.Source: §F.2 · pp. 22–23, Figure 9 · p. 22

G Stage-1 Latent Rollout Diagnostics

The diagnostics test whether LAM latents preserve human-video transitions and whether the Stage-1 VLA policy predicts latents that yield coherent motion. Both ground-truth and policy-predicted latents produce useful rollouts across short, fast and longer contact sequences.

G Stage-1 Latent Rollout Diagnostics

  • Ground-truth latent actions reconstruct future transitions with strong fidelity, and VLA-predicted latents achieve reasonable reconstruction quality.The results indicate that Stage-1 pre-training captures action-related latent structure, not only static appearance cues.Source: G Stage-1 Latent Rollout Diagnostics · p. 23, Table 18 · p. 23
  • In dice rolling and sandwich making, Stage-1 policy-predicted latents preserve the dominant motion pattern despite not being executable robot actions.The examples span short, fast hand-object motion and longer contact sequences with larger appearance changes.Source: G Stage-1 Latent Rollout Diagnostics · p. 23, Figure 10 · p. 23, Figure 11 · p. 24
  • The sandwich rollout remains useful as a diagnostic because it tests whether predicted latents encode transition tendency amid changing ingredients and hand contacts.The sequence is visually harder because multiple ingredients and contacts change across the clip.Source: G Stage-1 Latent Rollout Diagnostics · p. 24, Figure 11 · p. 24

H.1 Harder Real-Robot Tasks

On four harder real-robot tasks, LAPT-Scale is tied with π0.5 in-distribution and degrades less under shifts in object position, illumination, and background or tablecloth appearance. The tasks test precision grasping, disposal, and multi-stage stowing.

H.1 Harder Real-Robot Tasks

  • The four tasks add paper-ball disposal, slender-pen grasping, small eraser grasping under self-occlusion, and multi-stage book stowing.One policy is jointly trained with 200 demonstrations per task, and π0.5 is post-trained using the same downstream demonstrations.Source: §H.1 · p. 24

H.2 Generalization to Unseen Conditions

  • LAPT-Scale and π0.5 are tied in-distribution on four harder real-robot tasks, while LAPT-Scale degrades less under object-position, illumination, and background/tablecloth shifts.Each entry averages 20 trials per task and domain.Source: §H.2 · p. 24, Table 19 · p. 24, Figure 12 · p. 24

Related papers

Loading 2610.13732v1…