PaperlayerView exact revision on arXiv

Source-linked AI summary

CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos

Chubin Zhang, Jianan Wang, Zifeng Gao, Yue Su, Tianru Dai, Cai Zhou, Jiwen Lu, Yansong Tang

arXiv:2601.04061v2cs.ROcs.CV

TL;DR

CLAP addresses the scarcity of robotic data and the visual entanglement of prior latent-action methods by grounding human-video transitions in executable robot action tokens. It combines Act-VAE, contrastive alignment, CLAP-NTP, and CLAP-RF with Knowledge Matching, achieving strong benchmark performance while transferring human visual priors to robotic control. Its main limitation is that entirely new tasks and dexterous motions remain difficult without sufficient robotic grounding.

  • Problem

    Robotic data remain scarce, while existing latent-action methods can encode visual noise rather than manipulation skills when learning from human videos.

  • Method

    CLAP learns executable action tokens from robot trajectories with Act-VAE, contrastively aligns human visual transitions to them, and trains NTP and Rectified Flow policies with Knowledge Matching.

  • Results

    CLAP-RF achieves 97.2% average success on LIBERO, compared with 96.9% for π0.5 and 88.8% for SmolVLA, while real-world CLAP-RF reaches 62.7% mean success.

  • Takeaways & Limitations

    The framework transfers human visual priors into physically grounded robot control while supporting object generalization, semantic reasoning, and precise continuous action prediction.

  • Takeaways & Limitations

    Generalizing to entirely new tasks solely from human videos remains challenging, and dexterous human motions may not map directly to parallel-jaw gripper actions.

Abstract

from arXiv · show

Generalist Vision-Language-Action models remain constrained by the scarcity of robotic data relative to the abundance of human video demonstrations. Existing Latent Action Models attempt to use video data but often suffer from visual entanglement, encoding noise rather than manipulation skills. To address this limitation, we propose Contrastive Latent Action Pretraining (CLAP), a framework that first uses Act-VAE to learn an executable action-token vocabulary from robot trajectories and then aligns human visual transitions with this vocabulary through contrastive learning. This alignment maps unlabeled human videos into a physically grounded latent action space rather than reconstructing appearance. Building on the aligned tokens, we train CLAP-NTP as an autoregressive VLA using robot demonstrations and pseudo-labeled human videos, preserving instruction following and object generalization. For deployment and target-domain adaptation, we further introduce a post-training strategy that combines CLAP-RF, a Rectified Flow action head for low-latency continuous action chunk prediction, with Knowledge Matching regularization to preserve pretrained semantic knowledge during fine-tuning. Extensive experiments show that CLAP achieves strong performance against competitive baselines while enabling effective skill transfer from human videos to robotic execution.

I. INTRODUCTION

CLAP addresses the scarcity of robotic training data by grounding latent actions in executable robot trajectories and aligning them with human visual transitions. Its resulting VLA variants use robot and human-video data to support instruction following, object generalization, and low-latency continuous control.

  • VLA models face a primary bottleneck because robotic data remain limited relative to human data in scale, diversity, and semantic richness.
  • Contrastive learning aligns human-video visual transitions with robot-grounded tokens, reducing visual entanglement by prioritizing physical consequences over appearance changes.
  • CLAP-NTP trains an autoregressive VLA on robot demonstrations and pseudo-labeled human videos while preserving instruction following and supporting zero-shot object generalization.
  • CLAP-RF combines a Rectified Flow action head with Knowledge Matching to produce low-latency continuous action chunks while preserving pretrained semantic knowledge during fine-tuning.
  • CLAP learns a compact, executable action-token vocabulary from continuous robot trajectories using Act-VAE.The vocabulary establishes a physically grounded representation for subsequent alignment.
  • CLAP connects latent-action learning with compact action vocabularies by aligning human-video dynamics with robot-grounded action tokens for heterogeneous manipulation.

III. METHODOLOGY

CLAP unifies robot precision and human-video diversity by grounding visual transitions in a shared, discrete action space. Its methodology proceeds from action-token construction through cross-modal alignment and token-space or continuous-control policy learning.

  • Cross-Modal Alignment: CLAP establishes cross-modal alignment by grounding human visual state transitions in the quantized, physically executable robot action space.This bridges supervision between unlabeled human videos and labeled robot trajectories.
  • Token-Space Policy Learning: CLAP-NTP trains a token-space policy with next-token prediction on robot action tokens and human pseudo action tokens.The framework combines physical action labels from robots with semantic diversity from human videos.
  • Efficient Post-Training Strategy: CLAP-RF attaches a Rectified Flow action head for low-latency continuous control, while Knowledge Matching anchors fine-tuning to a trusted policy region.These components support post-training for target-domain adaptation.
  • Methodology: CLAP addresses modality mismatch by mapping human visual transitions and robot actions into a shared, discrete latent action space.Robot data provide explicit actions, whereas human videos provide visual state transitions without action labels.
  • Semantic Action Quantization: Act-VAE quantizes continuous robot trajectories into compact, executable action tokens using a VQ-VAE encoder, codebook, and decoder.The representation balances semantic compactness with the granularity needed for precise manipulation.

2) Cross-Modal Dynamics Alignment (VD-VAE):

VD-VAE infers action-related latent structure from visual transitions and aligns it with the frozen robot action vocabulary. It separates action-relevant content from nuisance variation while using contrastive, reconstruction, quantization, and sparsity objectives.

  • Cross-Modal Dynamics Alignment: VD-VAE maps visual transitions between frames to the pre-established Act-VAE action codebook without requiring human-video action labels.Robot transitions contain actions, whereas human transitions contain only observations.
  • Latent Decomposition: The inverse dynamics encoder decomposes each transition into action-relevant and action-irrelevant latent sequences.The action-relevant stream is quantized with the frozen action codebook, while the nuisance stream uses a separate environment codebook.
  • Contrastive Alignment: Contrastive learning aligns the vision-based latent sequence with Act-VAE action latents, using ground-truth robot actions or teacher-generated pseudo-positives for human videos.Robot positives use stop-gradient Act-VAE encodings; human positives come from an EMA teacher on clean transitions.
  • Disentanglement: L1 regularization sparsifies the action-irrelevant latent sequence so it captures nuisance information while leaving action-relevant information in the aligned stream.The total objective combines reconstruction, VQ, contrastive, and regularization terms.
  • Pseudo-Labeling: After training, VD-VAE produces discrete pseudo-action tokens from human video transitions for downstream policy learning.These pseudo-labels allow human videos to participate in token-space VLA training.

D. CLAP-NTP: Token-Space Policy Learning

CLAP-NTP is a token-space generalist policy that autoregressively predicts subtasks and discrete action tokens from observations and instructions. It trains jointly on robot demonstrations and human-video pseudo-actions while retaining the VLM’s reasoning and instruction-following capabilities.

  • Knowledge Matching: Knowledge Matching constrains the policy distribution during post-training, alongside continuous-action training for the downstream controller.The figure distinguishes discrete subtask and action-token representations from continuous actions.
  • Token-Space Policy Learning: CLAP-NTP predicts a joint sequence of subtasks and discrete action tokens autoregressively from current observations and instructions.The model uses next-token prediction over the VLM output sequence.
  • Mixed-Source Training: Training combines robot demonstrations with ground-truth action tokens and human videos with VD-VAE pseudo-action tokens.This unifies labeled robotic data and pseudo-labeled human data in one training formulation.
  • Capabilities: Because CLAP-NTP shares the base VLM’s training paradigm, it preserves reasoning faculties and supports direct robot control with instruction following.The aligned representation also provides the basis for learning from human videos.

E. Efficient Post-Training Strategy

The efficient post-training strategy combines a Rectified Flow action head with Knowledge Matching for target-domain adaptation. Together, they convert token-space representations into low-latency continuous action chunks while constraining updates against a trusted NTP reference.

  • CLAP-RF: CLAP-RF converts NTP representations into low-latency continuous action chunks through a Rectified Flow action head.This provides a continuous-control interface after token-space pretraining.
  • Knowledge Matching: Knowledge Matching regularizes the trainable token policy against a frozen NTP reference during target-domain adaptation.The strategy constrains policy updates within a trusted region during fine-tuning.

1) CLAP-RF: Flow Matching Head for Fast Inference:

CLAP-RF replaces slow autoregressive deployment with a Rectified Flow head that predicts continuous action chunks from CLAP-NTP representations. Knowledge Matching regularizes adaptation so the controller can tune to a target embodiment while retaining the trusted token policy’s semantic knowledge.

  • CLAP-RF: CLAP-RF attaches a DiT-based Rectified Flow action head to CLAP-NTP and predicts continuous action chunks through cross-attention to the backbone’s key-value cache.The head is trained with a rectified flow loss on robot trajectories.
  • CLAP-RF: The flow objective corrupts robot action chunks with Gaussian noise at flow time ρ and trains the model to predict the vector field from noise to the action chunk.The noised chunk is defined as ρa_t:t+H−1 + (1−ρ)ϵ, with ϵ sampled from N(0,I).
  • Knowledge Matching: Knowledge Matching addresses catastrophic forgetting by anchoring the adapted token-policy update within a trusted region around a frozen reference NTP policy.It operates on autoregressive token distributions using a reverse-KL regularizer.
  • Knowledge Matching: The RF controller adapts to the target embodiment while keeping the trainable token policy close to the trusted NTP reference.The RF head receives continuous robot-action supervision, while human-video pretraining influences it through the conditioning NTP representations.

3) Post-Training Procedure:

The post-training procedure first adapts CLAP-NTP on tokenized robot and human pseudo-action data, then performs robot-only RF post-training with Knowledge Matching. This preserves the influence of human videos through the adapted token policy and frozen reference while using continuous robot supervision for deployment.

  • Stage 1: NTP adaptation: Stage 1 adapts the pretrained NTP policy on all tokenized target data, including robot action tokens and human pseudo-action tokens.The adapted NTP policy is then frozen as the reference policy for the next stage.
  • Stage 2: RF post-training: Stage 2 initializes the RF action head and trainable policy branch from the adapted NTP model, then trains them only on target-domain robot trajectories.Continuous action chunks are available only for the robot dataset in this stage.
  • Information flow: Human videos influence CLAP-RF indirectly through NTP adaptation and the frozen Knowledge Matching reference used during robot-only RF training.This procedure separates human-video token supervision from continuous action supervision.
  • Stage 2: RF post-training: The post-training objective combines the Rectified Flow loss on continuous robot chunks with reverse-KL Knowledge Matching on the token-policy context.The algorithm updates the RF head and trainable policy branch using L_post = L_RF + αL_KL.
  • Data and platform: The experimental platform uses an Astribot S1 with dual 7-DoF arms and multicamera perception, while human demonstrations are collected through Meta Quest 3S VR teleoperation.The setup supplies robot and human demonstration sources for the procedure.

B. Cross-Modal Alignment via CLAP

Act-VAE is tuned to produce compact, executable action tokens without sacrificing the reconstruction fidelity needed for manipulation. The resulting token representation supports downstream VLA learning with robot and human-video-derived supervision, while RF post-training is reserved for later adaptation.

  • Act-VAE design: Act-VAE balances token expressiveness against downstream modeling difficulty because larger latent sequence lengths or codebooks improve reconstruction but increase sequence length or vocabulary size.The design treats the action representation as a compact physical language for the VLM.
  • Rate-distortion analysis: The rate-distortion analysis defines rate from latent sequence capacity relative to the information needed to represent the normalized continuous action chunk at the achieved precision.The rate uses Nq, K, Na, Da, and the normalized action range R.
  • Rate-distortion analysis: Hyperparameters are selected near the PSNR elbow point to preserve high reconstruction fidelity while keeping the latent representation compact for token-space policy learning.PSNR is reported as the reconstruction-fidelity metric against varying compression levels.
  • Human-video latent actions: VD-VAE uses DINOv3 patch-level feature losses and factorized attention to learn dynamics while reducing sensitivity to pixel-space noise and GPU memory demands.Spatiotemporal attention is used for inverse dynamics, while spatial attention is used for forward dynamics.
  • Token-space policy learning: CLAP-NTP is trained in token space with robot action tokens and VD-VAE pseudo tokens from human videos, while Rectified Flow and Knowledge Matching are introduced only during post-training.The implementation uses Qwen3VL-4B as the foundational VLM.

V. EVALUATION

CLAP is evaluated on real-world robotic manipulation and simulation, with tests covering execution precision, generalization, robustness, and cross-modal alignment. The evaluation uses a dual-arm platform, task-specific procedures, and comparisons against strong generalist and latent-action VLA baselines.

  • Evaluation objectives: The evaluation studies complex bimanual execution, high-precision control, generalization to unseen objects and embodiments, and alignment between human-video dynamics and robotic control.Experiments span a real-world robotic platform and LIBERO simulation environments.
  • Experimental setup: Real-world experiments use the Astribot S1 with its chassis and torso locked, controlling only the 14-DoF dual arms and grippers from head- and wrist-mounted RGB cameras.The setup matches the pretraining data distribution.
  • Additional evaluations: Evaluation also includes environmental-robustness, deployment-process, generalizability, and detailed real-world performance analyses.The referenced figures and tables organize these setup and robustness evaluations.
  • Real-world tasks: The task suite covers seen-object pick-and-place, novel-geometry and texture pick-and-place, long-horizon doll packing, bimanual T-shirt folding, and language-guided bouquet making.These tasks probe manipulation, OOD generalization, planning, deformable-object control, and semantic grounding.
  • Baselines: The baselines are π0, π0.5, and UniVLA, providing references from large-scale transfer learning and latent-action VLA modeling.UniVLA is used to help isolate the benefits of CLAP’s alignment and hierarchical control through component ablations.

4) Results and Analysis:

CLAP performs strongly in real-world manipulation and transfers semantic knowledge from human videos to robotic control. Its discrete and continuous variants complement one another across instruction following, precision, generalization, and robustness.

  • 62.7% mean success rate makes CLAP-RF the strongest evaluated real-world method, ahead of π0 at 54.0% and π0.5 at 60.0%.
  • CLAP-RF improves precision-demanding tasks, raising Pack the Doll pick-and-place from 80% to 90% and Fold T-shirt success from 30% to 40%.
  • CLAP-NTP achieves 85% success for both pick and place on OOD objects, while CLAP-RF reaches 70% on OOD placement.
  • Human-video fine-tuning improves CLAP-RF from 10% to 45% on Make Bouquets OOD and from 70% to 85% on PnP OOD.
  • CLAP-RF attains 70.0% mean robustness, exceeding π0.5 at 56.7% and UniVLA at 16.7% under environmental perturbations.

B. Simulation Results

On LIBERO, CLAP-RF performs competitively as a single generalist policy across four task suites. Ablations indicate that compact action representations balance reconstruction fidelity with policy learnability, while the full CLAP training mechanism drives substantial gains.

  • The generalist evaluation trains one policy across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long rather than separate suite-specific experts.
  • 97.2% average success makes CLAP-RF stronger than SmolVLA, π0, π0.5, FLOWER, and specialist OpenVLA-OFT, while remaining close to X-VLA at 98.1%.
  • Increasing latent sequence length Nq or codebook size K improves PSNR but also increases the token capacity and learning difficulty for the downstream VLM.
  • The selected Act-VAE configuration balances reconstruction quality with a compact token rate for effective VLM training.
  • CLAP-NTP ablations improve from 36.3% with the baseline setup to 40.0% with Qwen3VL, 45.8% with the data recipe, and 60.0% for the full model.

3) Contrastive Learning and Human Data:

Contrastive alignment makes human-video representations more semantically compact while human data improves object selection and instruction-conditioned generalization. The framework avoids relying on noisy hand-pose regression, but remains constrained by cross-embodiment and unseen-task limitations.

  • Removing pseudo-positive labels lowers performance from 60.0% to 58.8%, while removing contrastive alignment causes a larger drop to 53.8%.
  • Contrastive learning produces more compact same-action feature clusters than the non-contrastive variant in t-SNE analysis.
  • Estimated-pose supervision introduces localization noise and can degrade dexterous control, supporting visual-dynamics-based latent supervision instead.
  • CLAP-NTP has 382.0 ms inference latency because of sequential token generation, motivating CLAP-RF's lower-latency continuous action prediction.
  • CLAP currently may struggle to infer precise local dynamics for entirely new tasks without robotic grounding.
Loading 2601.04061v2…