PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild

Hao Luo, Ye Wang, Wanpeng Zhang, Haoqi Yuan, Yicheng Feng, Haiweng Xu, Sipeng Zheng, Zongqing Lu

arXiv:2602.21736v1cs.RO

TL;DR

VLAs face limited, heterogeneous robotic data, while human videos trade precise labels for broad behavioral diversity. JALA aligns predictive embeddings with inverse-dynamics latent actions and scales training with the 7.5M-sample UniHand-Mix corpus, improving hand-motion generation and robot manipulation across controlled and unconstrained settings.

  • Problem

    Robotic data is too scarce and heterogeneous for scalable VLA pretraining, while human datasets trade precise hand tracking for in-the-wild diversity.

  • Method

    JALA jointly aligns VLA predictive embeddings with inverse-dynamics latent actions and trains on a hybrid corpus of annotated and unlabeled human videos.

  • Results

    JALA improves hand-motion generation and robot manipulation performance across laboratory, wild, simulation, and real-world settings.

  • Takeaways & Limitations

    Jointly aligned latent actions provide a scalable pathway for VLA pretraining from heterogeneous human data.

  • Takeaways & Limitations

    Real-world failures remain due to perception-control mismatches, inadequate contact, and incorrect affordance reasoning.

Abstract

from arXiv · show

Despite progress, Vision-Language-Action models (VLAs) are limited by a scarcity of large-scale, diverse robot data. While human manipulation videos offer a rich alternative, existing methods are forced to choose between small, precisely-labeled datasets and vast in-the-wild footage with unreliable hand tracking labels. We present JALA, a pretraining framework that learns Jointly-Aligned Latent Actions. JALA bypasses full visual dynamic reconstruction, instead learns a predictive action embedding aligned with both inverse dynamics and real actions. This yields a transition-aware, behavior-centric latent space for learning from heterogeneous human data. We scale this approach with UniHand-Mix, a 7.5M video corpus (>2,000 hours) blending laboratory and in-the-wild footage. Experiments demonstrate that JALA generates more realistic hand motions in both controlled and unconstrained scenarios, significantly improving downstream robot manipulation performance in both simulation and real-world tasks. These results indicate that jointly-aligned latent actions offer a scalable pathway for VLA pretraining from human data.

1 Introduction

JALA addresses the scarcity and heterogeneity of robotic data by aligning predictive embeddings with inverse-dynamics latent actions, enabling joint learning from labeled and unlabeled human videos. UniHand-Mix scales this approach with hybrid data, improving hand-motion generation and robot manipulation, especially beyond curated laboratory settings.

  • Robotic datasets remain far smaller and less diverse than vision-language datasets, while cross-embodiment heterogeneity obstructs scalable VLA pretraining.
  • Lab videos provide precise 3D hand tracking but limited controlled scenarios, whereas in-the-wild videos provide diversity but unreliable action labels.
  • JALA jointly aligns VLA-context predictive embeddings with inverse-dynamics latent actions, avoiding pixel reconstruction while incorporating available ground-truth actions.
  • Transformer-based JALA uses intermediate hidden states as predictive embeddings and transfers them to robot control through a flow-matching head.
  • 7.5M samples combine 5M lab videos with 2.5M in-the-wild clips, supporting realistic wild hand motions and stronger manipulation performance.The model outperforms similar-size VLAs on Libero and RoboCasa and performs especially well on real-world out-of-distribution tasks.
  • JALA enables learning from both labeled and unlabeled human videos and improves generalization among similar-size models on robot tasks.

2 Related Work

Related work builds VLAs from robotics datasets and learns behavioral knowledge from videos through reconstruction, contrastive, predictive, or inverse-dynamics objectives. JALA follows the inverse-dynamics line while using tokenization for pretraining and flow matching for precise robot control.

  • Vision-Language-Action Model Pretraining: VLA pretraining commonly discretizes robot actions for autoregressive modeling or generates continuous actions with diffusion and flow matching.
  • Vision-Language-Action Model Pretraining: JALA uses tokenization to learn general action patterns during pretraining, then switches to flow matching for precise robot control.
  • Learning from Videos: Learning-from-video methods extract behavioral knowledge using masked auto-encoding, temporal contrastive learning, video prediction, or inverse dynamics.

3 Preliminaries

The pretraining setup represents human hand movements as tokenized MANO motion chunks conditioned on visual-text context. It trains jointly on annotated and unannotated videos, with auxiliary objectives enriching motion-token semantics before robot adaptation.

  • Human hand movements are represented with MANO parameters and discretized into fixed-length chunks, each tokenized into K motion tokens.
  • The VLA receives an instruction, an initial video frame, and tokenized motion chunks, then maximizes motion-token likelihood from preceding context.
  • Auxiliary tasks such as motion description generation and continuation enrich token semantics while preserving chunk-by-chunk motion generation.
  • Annotated data contains instructions, videos, and hand poses, while unannotated data contains only instructions and videos.
  • The hybrid training objective combines annotated and in-the-wild videos to leverage annotation precision and behavioral diversity.

4 Methodology

JALA learns predictive embeddings by jointly aligning VLA hidden states with inverse-dynamics latent actions, while combining labeled motion supervision with dynamics-only learning from unlabeled videos. A flow-matching head transfers the resulting latent action space to robot actions.

  • Joint Alignment with Latent Actions: JALA uses intermediate VLA hidden states as predictive embeddings aligned with latent actions inferred by an inverse-dynamics Perceiver.The alignment avoids reconstructing full video frames while retaining transition information.
  • Joint Alignment with Latent Actions: Masked Chunk Prediction models motion-token relationships within each chunk and supplies motion-label supervision to predictive embeddings.Motion tokens are masked jointly, allowing hidden states to encode chunk-level movement patterns.
  • Joint Alignment with Latent Actions: Latent Action Perceiver converts start and end frames into latent action vectors that capture chunk-level visual dynamics.These vectors are aligned with predictive embeddings rather than used to reconstruct full frames.
  • Joint Perceivers with Decoupled Updates: Asymmetric updates stabilize heterogeneous LAP and LSP representations by mapping visual features into the predictive-embedding space while action-grounding latent queries.The design keeps latent actions predictable from context and anchored with action cues.
  • Masked Decoding: Hybrid masking keeps preceding chunks intact, varies masking within the target chunk, and lightly masks later chunks to match training and inference contexts.This design enforces predictive consistency for sequential motion generation.
  • Two-Phase Training: JALA trains on unlabeled videos using LAP alignment alone, then transfers predictive embeddings to robot control through a flow-matching head.The hybrid objective activates MCP only when hand-tracking labels are available.

5 Hybrid Human Manipulation Dataset

UniHand-Mix combines precisely annotated laboratory manipulation data with diverse in-the-wild videos to provide scalable supervision across motion, language, context, and temporal horizons. The corpus contains over 7.5M instruction–video samples spanning complementary data sources and task types.

  • Dataset Overview: UniHand-Mix contains over 7.5M instruction–video samples, including 5M+ lab-annotated samples and 2.5M in-the-wild pairs.About 10% of the in-the-wild pairs include hand-pose annotations.
  • Lab-Annotated Subset: The lab subset pairs instructions, videos, and MANO-based hand-motion sequences across motion generation, description, and continuation tasks.Approximately 1,000 hours of lab videos yield 5M+ instruction–video–motion samples.
  • In-the-Wild Subset: The in-the-wild subset uses staged filtering, hand-activity validation, instruction generation, and optional pseudo-pose annotation to curate noisy Ego4D videos.The process yields approximately 2.5M instruction–video pairs, with about 10% receiving pseudo hand-pose annotations.
  • Statistics: Motion generation is the largest task category, while video-only samples broaden contextual diversity and description or continuation samples support language and temporal modeling.The task mix is intentionally heterogeneous rather than uniform.
  • Statistics: Short clips provide dense local interactions, whereas longer clips preserve multistep behaviors and temporal dependencies for complementary learning horizons.Their mixture supports both short-horizon motion learning and longer-horizon action continuation.
  • Statistics: Eight sources combine broad in-the-wild coverage with cleaner laboratory supervision distributed across multiple benchmarks and capture settings.The resulting dataset balances motion supervision, contextual diversity, and multiscale temporal coverage.

6 Experiments

The experiments evaluate whether JALA learns from in-the-wild human videos, transfers pretrained predictive embeddings to robot tasks, and benefits from its proposed alignment designs.

  • Experimental Questions: The study tests in-the-wild learning and scaling, downstream robot-task transfer, and the effectiveness of the proposed alignment designs.These questions cover both data scaling and methodological validation.

6.1 Implementation Details

The implementation pretrains a Transformer-based VLA with tokenized hand-motion chunks and then adapts it to robot tasks using a flow-matching policy head. Training uses the full UniHand-Mix corpus and evaluates hand-motion tasks on laboratory and wild splits.

  • Pre-training: JALA uses InternVL3-2B with DINOv3 or V-JEPA2 visual features and tokenizes each 15-frame motion chunk into wrist and finger tokens.Each chunk yields 128 motion tokens using separate 64-token wrist and finger representations.
  • Pre-training: Left- and right-hand chunks are interleaved temporally to preserve synchrony, while in-the-wild data is slowed by a factor of 0.5.Predictive embeddings are extracted from an intermediate attention layer.
  • Evaluation: Table 1 compares hand-motion generation and prediction tasks across laboratory and wild splits.The table is organized around performance differences between controlled and unconstrained data settings.
  • Pre-training: Pretraining runs for one epoch over the 7.5M-sample UniHand-Mix dataset using eight NVIDIA A800 GPUs.The reported implementation requires 68 hours on this hardware configuration.
  • Post-training: Post-training fine-tunes JALA on LIBERO, RoboCasa, and real-world tasks with a flow-matching Diffusion Transformer policy head.The DiT uses 16 layers of 32-head attention blocks with hidden dimension 2048.

6.2 Hand Motion Generation

JALA evaluates whether a unified latent action space can support hand-motion generation across precisely annotated lab data and unconstrained wild videos. It outperforms baselines on nearly all metrics, with especially strong gains and smaller degradation on the Wild split.

  • Task and evaluation: Hand motion generation predicts a motion chunk from an initial visual input and instruction, testing whether JALA learns transferable manipulation priors.The evaluation also probes whether one latent action space supports both annotated and unlabeled human videos.
  • Task and evaluation: The evaluation compares JALA-dino and JALA-vjepa with Being-H0, Being-H0+dino, JALA w/o align, and JALA w/o latent on Lab and Wild splits.The Lab split measures fidelity under precise supervision, while the Wild split measures generalization to unseen in-the-wild behaviors using MPJPE and related metrics.
  • Results: Both JALA variants outperform baselines on almost all metrics, with modest Lab gains but substantial improvements on the Wild split.All models degrade in the more variable Wild setting, but JALA variants show a much smaller performance drop.
  • Results: Removing latent actions performs noticeably worse than removing alignment, particularly on the Wild split, despite using more data.This comparison supports the importance of the latent-action design beyond the prediction paradigm alone.
  • Visualization: Predictive embeddings and latent actions cluster in closely aligned regions, while Wild samples largely expand the Lab manifold rather than forming a disjoint domain.Qualitative generations cover both lab and wild scenes with colored overlays denoting generated hand poses.

6.3 Simulation Results

JALA transfers its human-derived latent action space to robot manipulation through flow-matching adaptation across LIBERO, RoboCasa, and GR1. It consistently outperforms reconstruction-based baselines, including under single-view, few-shot, domain-shifted, and morphologically different settings.

  • Experimental setup: JALA is fine-tuned on LIBERO, RoboCasa, and GR1 using a flow-matching policy head after pretraining on hybrid human data.The simulation experiments compare joint alignment with reconstruction-based methods and additional data and alignment ablations.
  • LIBERO: JALA-dino reaches a 96.9% overall average success rate in two-view LIBERO without robotic-data pretraining.It performs on par with or better than stronger baselines despite being lighter.
  • LIBERO: LAPA scores 79.5% and LAPA† scores 83.5% on two-view LIBERO, while JALA exceeds LAPA† by over 13 average points.The controlled LAPA† comparison uses the same backbone and data, isolating the training-objective difference reported by the authors.
  • LIBERO: JALA-dino achieves a 92.3% average success rate in single-view LIBERO among models below 3B parameters and reaches 87.2% on the Long suite.The results also report larger degradation when joint alignment or latent actions are removed, while reconstruction-based models require substantially more computation and memory.
  • RoboCasa: JALA consistently outperforms reconstruction-based baselines and ablations on both RoboCasa splits, with a particularly pronounced advantage on synthetic data.JALA w/o dec. degrades sharply, and the LAPA† gap remains under identical data and backbone conditions.
  • GR1: JALA achieves a 26.33% success rate on GR1, exceeding GR00T N1.5 at 20.41%, LAPA at 11.42%, and Being-H0 at 12.91%.On the same action-available subset, JALA-act reaches 20.25% versus Being-H0’s 12.91% in a dexterous-hand setting.
  • Ablations: Adding more in-the-wild data from 0% to 100% consistently improves LIBERO downstream success rates.A separate hidden-state ablation finds that using the 19th backbone layer for flow matching gives the best transfer performance.
  • Cross-benchmark findings: Across three simulation benchmarks, JALA maintains an empirical advantage over reconstruction-based baselines, including under embodiment shift and larger domain gaps.The authors attribute the remaining controlled gap to the training objective, which aligns predictive embeddings with action-relevant dynamics.

6.4 Real-World Robot Experiments

JALA is evaluated on three multi-step real-world manipulation tasks, including unseen visual variants, and consistently outperforms baselines while remaining robust to distribution shifts. Qualitative analyses show adaptive correction, but failures remain in alignment, contact stability, and affordance reasoning.

  • Task design: JALA evaluates three multi-step tasks probing object placement, surface interaction, and affordance-based watering.Put-Three-Obj requires sequential fruit placement, Wipe-Board requires sustained planar contact, and Water-Plant requires oriented spray-bottle use.
  • Evaluation protocol: Each task uses 50 teleoperated demonstrations and 10 evaluation rollouts, with average subtask completion as the success metric.Put-Three-Obj and Wipe-Board additionally include unseen visual variations for robustness testing.
  • Quantitative results: JALA-dino achieves the best completion rate on every real-world task across seen and unseen settings.The reported advantage spans multi-step object placement, surface interaction, and Water-Plant rather than one favorable scenario.
  • Robustness to visual shifts: A 60.0% to 58.0% drop on unseen Put-Three-Obj indicates JALA-dino remains stable when tablecloth appearance changes.Baseline performance degrades sharply under visual shifts in Put-Three-Obj and Wipe-Board.
  • Successful rollout analysis: Successful rollouts show correction of initial misalignment and adaptive revisiting of residual ink under altered visual conditions.The policy retracts and repositions after a misaligned banana approach, while Wipe-Board behavior revisits remaining marks.
  • Failure analysis: Remaining failures involve spatial misalignment, incomplete or unstable contact, persistent residual ink, and incorrect bottle affordance reasoning.These failures expose practical limits in precise alignment, contact stability, closed-loop perception, and fine-grained affordance modeling.

7 Conclusion

The paper presents JALA as a reconstruction-light framework for scalable VLA pretraining from heterogeneous human videos, supported by the 7.5M-sample UniHand-Mix dataset. Experiments report improved hand-motion generation and downstream robot manipulation performance.

  • 7 Conclusion: JALA jointly aligns VLA predictive embeddings with IDM-derived latent actions to represent human motion without detailed visual reconstruction.The framework is designed to work across lab-annotated and in-the-wild human data.
  • 7 Conclusion: UniHand-Mix contains 7.5M samples spanning over 2,000 hours of annotated and unconstrained human videos.The hybrid resource combines reliable physical anchors with broader task diversity.
  • 7 Conclusion: JALA improves hand-motion generation and downstream robot manipulation benchmarks, surpassing existing VLA methods with comparable or larger model sizes.The conclusion characterizes predictive embeddings as a practical route for connecting human video data with robotic learning.
  • 7 Conclusion: The real-robot figures document successful seen and unseen rollouts alongside failure cases across Put-Three-Obj, Wipe-Board, and Water-Plant.These visualizations complement the conclusion’s reported downstream manipulation gains with qualitative task evidence and remaining errors.
Loading 2602.21736v1…