PaperlayerView exact revision on arXiv

Source-linked AI summary

ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation

Weisheng Dai, Kai Lan, Jianyi Zhou, Bo Zhao, Xiu Su, Junwen Tong, Weili Guan, Shuo Yang

arXiv:2602.00557v1cs.RO

TL;DR

ConLA addresses the difficulty of scaling diverse robot teleoperation data by learning latent actions from human videos, where prior reconstruction-based methods can capture visual appearance instead of motion. It uses contrastive disentanglement with action-category and temporal priors, and achieves strong benchmark performance, including gains over LAPA and real-robot-trajectory pretraining.

  • Problem

    Robot teleoperation datasets are costly and difficult to scale, while VQ-VAE-based video methods can learn entangled latent actions by reconstructing visual content rather than inter-frame dynamics.

  • Method

    ConLA uses contrastive disentanglement with action-category and temporal priors to isolate motion dynamics from visual content in human videos.

  • Results

    ConLA consistently outperforms previous methods across benchmarks, improving over LAPA by 12.5% on SimplerEnv and exceeding real-robot-trajectory pretraining by 1.1%.

  • Takeaways & Limitations

    Human videos can support scalable VLA pretraining when latent actions are made semantically consistent and more faithful to motion.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models achieve preliminary generalization through pretraining on large scale robot teleoperation datasets. However, acquiring datasets that comprehensively cover diverse tasks and environments is extremely costly and difficult to scale. In contrast, human demonstration videos offer a rich and scalable source of diverse scenes and manipulation behaviors, yet their lack of explicit action supervision hinders direct utilization. Prior work leverages VQ-VAE based frameworks to learn latent actions from human videos in an unsupervised manner. Nevertheless, since the training objective primarily focuses on reconstructing visual appearances rather than capturing inter-frame dynamics, the learned representations tend to rely on spurious visual cues, leading to shortcut learning and entangled latent representations that hinder transferability. To address this, we propose ConLA, an unsupervised pretraining framework for learning robotic policies from human videos. ConLA introduces a contrastive disentanglement mechanism that leverages action category priors and temporal cues to isolate motion dynamics from visual content, effectively mitigating shortcut learning. Extensive experiments show that ConLA achieves strong performance across diverse benchmarks. Notably, by pretraining solely on human videos, our method for the first time surpasses the performance obtained with real robot trajectory pretraining, highlighting its ability to extract pure and semantically consistent latent action representations for scalable robot learning.

1. Introduction

ConLA addresses the difficulty of scaling diverse robot teleoperation data by learning latent actions from human videos. Its contrastive framework uses action-category and temporal priors to reduce shortcut learning and supports strong benchmark performance.

  • Motivation: Robot teleoperation datasets are costly and difficult to collect across diverse environments and tasks, whereas human videos provide a rich, scalable source of demonstrations.Human videos offer diverse scenes and manipulation behaviors but lack explicit action supervision.
  • Problem: VQ-VAE-based latent-action methods can reconstruct future visual content instead of capturing meaningful inter-frame motion, producing entangled representations.Figure 2 illustrates that reconstruction based on a latent action from one frame pair fails on another because the model captures appearance rather than motion.
  • Approach: Action-category priors encourage same-category latent actions to cluster across environments and embodiments, while temporal cues help separate motion dynamics from relatively stable appearance.These priors provide semantic and temporal guidance for disentangling motion from visual content.
  • Approach: ConLA learns compact latent actions through contrastive disentanglement before quantizing them for latent-action pretraining of a vision-language model.The framework extracts representations from paired frames, applies contrastive learning, and uses the resulting tokens for policy learning.
  • Results: ConLA achieves state-of-the-art performance across multiple benchmarks and improves over the LAPA baseline.The supplied introduction reports broad benchmark gains, including a 12.5% improvement over LAPA on SimplerEnv and a 1.1% advantage over real-robot-trajectory pretraining.

2. Related Works

Related work develops several routes for transferring information from human videos to robot policies. These include explicit human-motion retargeting and unsupervised latent-action modeling, each with distinct scalability or expressivity constraints.

  • Vision-Language-Action Models: Vision-language-action models map visual observations and language instructions to robotic actions, with systems such as OpenVLA pretrained on large-scale teleoperation datasets.The related work frames VLA models as a foundation for manipulation policy learning.
  • Learning from Human Videos: Large-scale teleoperation data is difficult to scale, motivating human-video demonstrations as an alternative source for robot learning.Human videos have emerged as a promising paradigm for addressing teleoperation-data limitations.
  • Explicit Action Extraction: Explicit human-video approaches often use hand-pose estimation or motion capture to retarget actions, while some task-specific methods depend on paired human-robot data.Dependence on paired data can limit scalability and generalization.
  • Latent Action Learning: Latent-action methods use unsupervised inverse dynamics models to extract action priors from unlabeled videos for VLA policy training.LAPA uses VQ-VAE to extract motion priors from consecutive frames, while CLAM and COMO address expressivity limitations of discrete latent actions.

3. Methodology

ConLA uses a three-stage pipeline to extract semantically consistent latent actions, pretrain a policy on those tokens, and map them to executable robot controls. Its contrastive module separates action-related and visual information using category labels and temporal inversion.

  • Three-Stage Framework: The framework first extracts discretized latent actions using action-category information and temporal cues, then pretrains an autoregressive VLM-based policy and fine-tunes it on robot trajectories.The final fine-tuning stage establishes a mapping from latent actions to executable control signals.
  • Contrastive Latent Action Learning: Contrastive Latent Action Learning encodes current–future video pairs and splits latent embeddings into action-related and visual-related components.The action and visual components are processed by separate heads for disentanglement.
  • Action-Centric Contrastive Learning: Action-centric contrastive learning pulls representations sharing an action class together and pushes representations from different classes apart.This encourages compact, semantically coherent clustering and reduces sensitivity to visual distractions such as background variation.
  • Vision-Centric Contrastive Learning: Vision-centric contrastive learning uses reversed frame pairs because motion changes with temporal order while content and visual distractions remain relatively stable.The objective promotes content-consistent, motion-invariant visual features and separates appearance from dynamics.
  • Latent Action Quantization: The learned action-centric representation is quantized into codebook entries that serve as pseudo-labels for observation–instruction–action triplets.For each current–future frame pair, the nearest quantized action representation is retrieved before policy pretraining.
  • Robot Control Adaptation: Latent actions cannot directly execute downstream tasks, so a small set of ground-truth robot trajectories is used to fine-tune the policy into executable controls.This stage transfers video-derived motion priors into the robot action space.

4. Experiments

Experiments evaluate ConLA on simulation and real-world manipulation using robot and human video pretraining, then analyze its latent actions and scalability. ConLA outperforms baselines, transfers human motion priors effectively, and learns more motion-relevant, compact representations.

  • Experimental Setup: ConLA is evaluated in SimplerEnv and real-world tabletop manipulation, using BridgeV2 robot videos and Something-SomethingV2 human videos.The real-world setup uses a 7-DoF Franka Research 3 arm and three multi-instruction tasks.
  • SimplerEnv Results: ConLA pretrained solely on human videos surpasses ACTIONVLA pretrained on real robot trajectories by 1.1% average success rate on SimplerEnv.Table 1 reports average success rates across the SimplerEnv evaluation.
  • Real-World Results: 15.9%: ConLA surpasses LAPA in real-world success rate after human-video pretraining across three tasks and three generalization settings.The settings include unseen object combinations, unseen objects, and unseen instructions.
  • Latent Action Analysis: ConLA reconstructs motion-consistent futures across visual contexts, whereas LAPA’s latent actions are dominated by visual content in human videos.The analysis conditions future-frame prediction on the current frame and extracted latent actions.
  • Latent Action Analysis: ConLA produces compact, semantically coherent latent-action embeddings, with similar motions clustered despite differing visual appearances.The t-SNE analysis compares 100 sampled clips per action category against naive VQ-VAE representations.
  • Data Scalability: Performance scales positively from 10% to 100% of Something-SomethingV2, with ConLA using the data more efficiently than LAPA.This experiment evaluates how human-video dataset size affects performance.

5. Conclusion

ConLA extracts high-quality latent actions from human demonstration videos for VLA models. Its contrastive, action-centric representation learning mitigates shortcut learning and performs strongly even when trained solely on human videos.

  • ConLA extracts high-quality latent actions from human demonstration videos for Vision-Language-Action models.
  • Contrastive latent action learning uses action-category and temporal priors to build action-centric representations and mitigate shortcut learning.
  • ConLA consistently outperforms previous methods, including when pretrained solely on human video data.

A.1. Detailed Algorithm

The detailed algorithm first stabilizes latent representations with reconstruction training, then applies contrastive learning and uses the resulting encoder for latent-action policy pretraining and robot-action finetuning.

  • ConLA begins with a 5,000-step reconstruction-loss warm-up before applying contrastive latent action learning.The warm-up supplies preliminary latent representations for subsequent contrastive learning.
  • Contrastive latent action learning uses action-category and temporal priors to disentangle latent actions from mixed visual noise.
  • The pretrained encoder infers latent-action sequences from unlabeled videos, which serve as pseudo-labels for policy pretraining.
  • During finetuning, the policy learns from real robot trajectories to align latent actions with executed actions.

A.2. Pre-training Dataset Processing

The preprocessing pipeline converts instructions into discrete action labels and uses video pairs, class labels, and reconstruction-based latent-action learning to prepare data for policy training.

  • Natural-language instructions are processed into structured action signals because they encode executable action categories and motion semantics.
  • ConLA samples forward and reverse observation pairs to learn latent representations from unlabeled videos.
  • The reconstruction stage decodes a future observation from the current observation and the action latent.
  • Action-centric and visual contrastive losses are combined with reconstruction loss during latent-action learning.
  • The policy pretraining stage uses pseudo latent-action sequences inferred from unlabeled videos before real-action finetuning.
  • Instructions are normalized, reduced to verb-and-direction pairs, and cleaned to produce discrete action labels.
  • BridgeV2 receives 80 pseudo action classes, whereas Something-SomethingV2 provides 174 predefined action classes.

B.1. SimplerEnv

SimplerEnv evaluation follows a controlled finetuning and rollout protocol across four tasks, with randomized object locations and average success rate as the metric.

  • SimplerEnv finetuning uses 100 successful trajectories, comprising 25 trajectories for each of four SIMPLER tasks.
  • Evaluation performs 24 rollouts per task while randomizing initial object locations, and reports average success rate.
  • The four tasks involve stacking a green block, placing a carrot, placing a spoon, and putting an eggplant into a basket.
  • UniVLA performs comparably to LAPA on BridgeV2 but improves substantially more on human videos under complex environmental variation.

B.2. Real-World Robots

The real-world evaluation uses tabletop manipulation tasks with object-selection demands and graded task scores. ConLA achieves higher strict success than LAPA, supporting stronger transfer from human-video pretraining.

  • Experiment setup: The evaluation follows LAPA’s real-world tabletop setup, using three objects per scene and 150 teleoperation trajectories per task.The model must select the object specified by the task instruction.
  • Task design: Tasks include knocking down an object, covering it with a towel, and placing it in a box.Each task uses staged scoring that reflects partial task completion.
  • Results: ConLA achieves higher strict success than LAPA in the real-world evaluation.The section reports strict success as an additional evaluation measure.

C. More Visualization

The visualization analysis tests whether extracted latent actions transfer motion across starting frames rather than memorizing appearance. LAPA shows appearance leakage, whereas ConLA produces motion-centric representations; the supplied table captions identify complementary benchmark results.

  • Latent action consistency: The consistency analysis applies latent actions for left and right motion to new starting frames to reconstruct expected outcomes.This procedure tests whether the representation captures transferable motion rather than source-frame appearance.
  • Latent action consistency: LAPA’s reconstructions reproduce visual content from extraction frames, indicating that its latent actions encode appearance rather than motion.The paper identifies this behavior as a direct symptom of shortcut learning, especially on human videos.
  • Latent action consistency: ConLA extracts motion-centric latent actions in the same visualization analysis.The comparison contrasts ConLA with LAPA under the cross-frame reconstruction procedure.
  • Benchmark visualizations: The supplied captions cover BridgeV2 pretraining results, human-manipulation-video pretraining results, knocking and covering tasks, pick-and-place, and total success rates.The captions identify the reported metrics and task groupings but do not provide numerical values here.
Loading 2602.00557v1…