PaperlayerView exact revision on arXiv

Source-linked AI summary

LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment

Dujun Nie, Fengjiao Chen, Qi Lv, Jun Kuang, Xiaoyu Li, Xuezhi Cao, Xunliang Cai

arXiv:2604.11689v1cs.CVcs.RO

TL;DR

Large-scale human videos provide action information but lack ontology-independent latent representations, whose quality has not been rigorously evaluated for both semantics and control. LARYBench addresses this gap with a unified benchmark spanning semantic classification and robotic trajectory regression across a large, diverse corpus. Its experiments find that general visual foundation models outperform specialized embodied LAMs, while latent-based spaces align better with physical action space than pixel-based spaces.

  • Problem

    Latent-action quality lacks rigorous evaluation across both semantic action understanding and low-level robotic control, limiting assessment of representations derived from unlabeled videos.

  • Method

    LARYBench evaluates latent representations through semantic classification and trajectory regression using a curated corpus of videos, image pairs, and motion trajectories across diverse embodiments and environments.

  • Results

    General visual foundation models consistently outperform specialized embodied LAMs, and latent-based feature spaces achieve better robotic-control alignment than pixel-based spaces.

  • Takeaways & Limitations

    Large-scale visual pre-training can produce action-relevant representations without explicit action supervision, supporting general visual representations as a basis for VLA systems.

  • Takeaways & Limitations

    General LAM performance declines on robot actions, attributed to limited data scale and diversity.

Abstract

from arXiv · show

While the shortage of explicit action data limits Vision-Language-Action (VLA) models, human action videos offer a scalable yet unlabeled data source. A critical challenge in utilizing large-scale human video datasets lies in transforming visual signals into ontology-independent representations, known as latent actions. However, the capacity of latent action representation to derive robust control from visual observations has yet to be rigorously evaluated. We introduce the Latent Action Representation Yielding (LARY) Benchmark, a unified framework for evaluating latent action representations on both high-level semantic actions (what to do) and low-level robotic control (how to do). The comprehensively curated dataset encompasses over one million videos (1,000 hours) spanning 151 action categories, alongside 620K image pairs and 595K motion trajectories across diverse embodiments and environments. Our experiments reveal two crucial insights: (i) General visual foundation models, trained without any action supervision, consistently outperform specialized embodied latent action models. (ii) Latent-based visual space is fundamentally better aligned to physical action space than pixel-based space. These results suggest that general visual representations inherently encode action-relevant knowledge for physical control, and that semantic-level abstraction serves as a fundamentally more effective pathway from vision to action than pixel-level reconstruction.

1 Introduction

LARYBench addresses the lack of rigorous, representation-level evaluation for latent actions by measuring both semantic intent and low-level physical control. It introduces a standardized benchmark and broad data engine to compare representation paradigms across diverse agents and settings.

  • Motivation: Existing evaluations often conflate latent-action quality with downstream policy performance and do not test generalization across entities, tasks, and granularities.The effects of training architectures, strategies, and usage paradigms also remain underexplored.
  • Data and evaluation: 1.2M videos, 620K image pairs, and 595K trajectories span 151 action categories, 11 robotic embodiments, human and robotic agents, and simulated and real-world environments.An automated data engine re-segments and re-annotates the corpus to support standardized evaluation.
  • Data and evaluation: LARYBench compares Embodied LAMs, general vision encoders, and General LAMs across 11 models to assess action-relevant representations with and without explicit action supervision.The model families include semantic- and pixel-level general vision backbones and LAMs grafted onto frozen general vision backbones.
  • Benchmark scope: LARYBench directly evaluates latent representations along high-level semantic action encoding and low-level physical dynamics for robotic control.The two dimensions correspond to identifying what to do and reconstructing how to do it.
  • Findings: General visual pre-training can yield action-relevant features without explicit action supervision, while latent-based spaces tend to align with robotic control better than pixel-based spaces.These are reported as consistent findings from the systematic evaluation.

2 Related Work

Prior work extracts latent control signals from unlabeled videos and uses latent actions in generalist VLAs and world models. However, quantitative evaluation remains difficult because common metrics can miss action dynamics and distractor robustness.

  • Latent-action representations: Unsupervised latent-action methods use inverse dynamics models to extract control signals from unlabeled videos, with discrete and continuous paradigms.Discrete approaches commonly use vector quantization, which can lose fine-grained information.
  • Latent actions in generalist systems: Latent actions serve as a unifying interface in VLAs and condition future-frame prediction in world foundation models.In VLAs, they can decouple high-frequency control from low-frequency reasoning.
  • Evaluation challenges: Standard reconstruction metrics may fail to distinguish action dynamics from environmental noise, while existing alignment benchmarks use trajectory consistency or CCA.Diagnostic studies also report difficulties with distractor robustness.

3 The LARY Benchmark

LARYBench maps visual observations into latent actions and evaluates them through semantic classification and robotic trajectory regression. Its multi-embodiment dataset and automated curation pipeline support evaluation across semantic and kinematic granularities.

  • Benchmark formulation: LARYBench evaluates latent actions z through semantic decoding f_sem: Z → C and robotic-control construction f_dyn: Z → A.The two mappings correspond to classification and regression tasks.
  • Dataset: The dataset contains 1.2M annotated samples covering 151 actions, including atomic and composite behaviors across human and robotic data.The action inventory ranges from frequent actions such as pick and place to long-tail activities such as shovel and float.
  • Dataset: The benchmark spans 11 robotic embodiments, human egocentric interaction data, diverse objects, and simulated and real-world environments.This design provides morphological and environmental diversity for evaluation.
  • Evaluation pipeline: Figure 2’s pipeline constructs datasets, extracts continuous latent actions using multiple representation paradigms, and evaluates them with classification Accuracy and regression MSE.The architecture can integrate pre-trained general vision encoders within VQ-VAE training.
  • Semantic probing: Semantic probing targets multi-granularity understanding, including 28 atomic robot primitives and composite human and robot behaviors.The Atomic Robot task uses 25,940 image pairs with trajectories extracted from LIBERO demonstrations.
  • Physical execution mapping: Trajectory regression tests whether latent representations preserve physical detail by decoding continuous end-effector actions across heterogeneous hardware and action spaces.The protocol includes Franka, 10 RoboCOIN platforms, and the AgiBot G1.
  • Data curation: A VLM-driven curation process performs temporal video segmentation and semantic action alignment to transform raw videos into standardized evaluation tasks.The VLM provides spatio-temporal video understanding for the data engine.

4 Experiments

The experiments are presented as a systematic investigation organized around questions about the benchmark and latent action representations.

  • Experimental setup: The experiments are designed to answer a set of research questions about LARYBench and latent action representations.No specific experimental result is supplied in the provided passage.

3. What Constitutes an Effective Latent Action Model?

Effective latent action models combine broad visual pretraining with feature-level motion modeling and carefully chosen quantization settings. Across semantic classification and robotic regression, latent-based representations outperform pixel-based alternatives and specialized embodied models.

  • Model paradigms: The benchmark compares embodied latent action models, general semantic encoders, generative pixel encoders, and general latent action models under a unified evaluation protocol.Evaluation covers semantic classification and robotic control regression with standardized data splits and probing procedures.
  • Semantic action understanding: General vision foundation models achieve strong semantic action performance without explicit motion extraction, outperforming embodied latent action models that average 17.99%–20.90%.The result spans human and robot action recognition and is attributed in the passage to broad visual pretraining versus limited or low-level training constraints.
  • Semantic action understanding: 43.67% vs. 17.99%: LAPA-DINOv2 significantly outperforms UniVLA despite sharing the same vision foundation model.The comparison is associated with more diverse training data and fewer constraints in the General LAM setup.
  • Limitations: General LAM performance declines on robot actions because of limited data scale and diversity, leaving this issue for future work.The stated limitation concerns the gap between relatively high human-action accuracy and weaker robot-action performance.
  • Low-level control: Latent-based encoders such as DINOv3 and V-JEPA 2 achieve better robotic control regression than pixel-based Wan2.2 and FLUX.2-dev encoders.The reported comparison supports better alignment between latent-based visual space and robotic action space.
  • Quantization design: Moderate quantization settings improve latent action models: codebook size 64, sequence length 49, and latent dimension 256 balance capacity, utilization, and stability.Sequence length 16 yields 1.6% codebook utilization, whereas length 49 reaches 100%; overly large codebooks and intermediate latent dimensions can destabilize utilization.

5 Error Analysis

The error analysis examines how latent-action quality varies across sequence settings, class frequencies, and physical grounding. It highlights long-tail generalization differences and contrasting attention localization between general visual encoders and embodied LAMs.

  • 5.1 The Long-Tail Dilemma and Mid-Frequency Semantic Aliasing: Strong models show wider performance advantages as action-class frequency decreases, indicating better long-tail generalization.The analysis uses Composite Human classification performance across class frequencies to compare model robustness.
  • 5.1 The Long-Tail Dilemma and Mid-Frequency Semantic Aliasing: LAPA-DINOv3 closely mirrors the continuous DINOv3 encoder, inheriting its representational strengths and vulnerabilities.By contrast, baseline LAPA exhibits uniformly poor classification performance across the distribution.
  • 5.2 Spatiotemporal Grounding and Action-Centric Attention: V-JEPA 2 most accurately localizes interactions between both hands and the manipulated bowl, while DINOv3 maintains geometry-aware end-effector focus.Generative encoders produce dispersed attention, whereas standard LAPA produces broad, uninformative blobs.
  • 5.2 Spatiotemporal Grounding and Action-Centric Attention: LAPA-DINOv2 inherits strong localization from its visual backbone despite the coarse spatial resolution of latent-action representations.The comparison connects backbone properties with the spatial grounding behavior of hybrid latent-action models.
  • 5.3 Temporal Robustness: FLUX.2-dev reaches 0.04 MSE at stride=5 but fails catastrophically as temporal stride increases and regression dimensionality grows.The stride ablation tests robustness at strides 5, 15, and 30 on VLABench.

6 Conclusion

LARYBench concludes that general visual foundation models provide stronger latent-action representations than specialized embodied LAMs across semantic understanding and robotic regression. The benchmark combines large-scale curated videos, image pairs, and trajectories to evaluate these capabilities across action granularities.

  • 6 Conclusion: LARYBench evaluates latent-action representations across kinematic and semantic granularities.Its evaluation targets both high-level semantic intent and low-level physical execution.
  • 6 Conclusion: General visual foundation models consistently outperform specialized Embodied LAMs across semantic tasks and regression MSE.The conclusion reports this pattern as the benchmark’s central empirical finding.
  • 6 Conclusion: The benchmark contains over 1.2 million videos, more than 1,000 hours of footage, 620K image pairs, and 595K motion trajectories.These data span 151 curated action categories and support classification and regression tracks.
  • 6 Conclusion: The dataset combines human and robotic data from egocentric and exocentric settings to assess representations across diverse embodiments and environments.The composition is summarized in the benchmark overview and category-distribution figures.
  • 6 Conclusion: Classification covers 28 Atomic Robot classes, 123 Composite Human classes, and 54 Composite Robot classes with semantic overlap across embodiments.The subsets jointly represent discrete kinematic primitives and abstract composite behaviors.

B.2 Additional Data Curation Details

The curation pipeline standardizes heterogeneous embodied datasets through automated segmentation, strict video-label checks, verb consistency verification, and manual quality control. It produces clips and image-pair trajectories with more consistent temporal and semantic annotations.

  • B.2 Additional Data Curation Details: The data engine addresses disparities in temporal boundaries and semantic annotations across existing embodied datasets.It uses an automated, multi-stage processing pipeline to unify heterogeneous sources.
  • B.2 Additional Data Curation Details: Atomic classification extracts directional movements and gripper states by thresholding cumulative end-effector displacement.Start and end frames are selected when displacement crosses a predefined threshold, then labels are assigned from axis directions.
  • B.2 Additional Data Curation Details: Composite curation uses action segmentation, strict video-description matching, verb extraction, and a secondary verb-consistency check.The process filters clips whose visual content does not match the description or isolated action verb.
  • B.2 Additional Data Curation Details: Manual inspection removes abstract or kinematically ambiguous categories such as apply, arrange, and clean.This step preserves action categories with explicit physical and dynamic meanings.
  • B.2 Additional Data Curation Details: Sampling uses 5 FPS with at least 16 frames to capture action duration during video trimming and checking.The curation prompts also identify action timestamps and camera perspective.
  • B.2 Additional Data Curation Details: Regression pairs start and end observations around the continuous absolute kinematic actions occurring between them.The required observations and trajectories are sampled at a fixed temporal stride.

C Experimental Details of General LAMs

General LAMs replace LAPA’s pixel encoder with frozen pretrained visual backbones to test semantic and pixel-level visual priors for latent-action learning. Semantic variants reconstruct continuous feature embeddings, while the MAGVIT2 variant reconstructs pixels.

  • C Experimental Details of General LAMs: Four General LAM variants use frozen pretrained backbones: LAPA-DINOv2, LAPA-DINOv3, LAPA-SigLIP2, and LAPA-MAGVIT2.The variants systematically test different visual priors while keeping backbone weights fixed.
  • C Experimental Details of General LAMs: DINOv2, DINOv3, and SigLIP2 variants reconstruct continuous penultimate-layer feature embeddings instead of raw pixels.They substitute pixel-level inputs in LAPA’s VQ-VAE architecture with high-level visual features.
  • C Experimental Details of General LAMs: LAPA-MAGVIT2 retains a pixel-space reconstruction target as the generative-based General LAM variant.This provides a pixel-level counterpart to the understanding-oriented feature-reconstruction models.

C.2 Training Configurations and Hyperparameters

The evaluation examines representation behavior across human and robotic embodiments, including cross-domain performance on shared semantic actions. General LAMs raise overall F1 bounds but show a human-centric preference, with large gaps for several actions.

  • Cross-domain analysis: General LAMs raise overall F1 bounds compared with standard Embodied LAMs but favor human embodiments.The comparison covers shared semantic actions across human and robotic domains.
  • Cross-domain analysis: For roll, LAPA-DINOv3 achieves H: 0.90 versus R: 0.04.This example illustrates the human–robot performance discrepancy reported for General LAMs.
  • Representation analysis: The evaluation includes extended cross-attention visualizations to investigate representation behavior.These visualizations complement the cross-domain analysis.
  • Cross-domain analysis: The cross-domain analysis compares F1 discrepancies as ∆F1 = Robot − Human across shared action categories.Heatmap cells also report absolute Human and Robot F1 scores.
  • Action cases: The supplementary cases cover catch, compress, plug, and brew actions.The plug case reports that all models failed.
Loading 2604.11689v1…