PaperlayerView exact revision on arXiv

Source-linked AI summary

MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction

Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee

arXiv:2602.03668v3cs.ROcs.CV

TL;DR

Video-based VLA pretraining needs latent actions that remain informative about unavailable ground-truth actions. MVP-LAM learns discrete latent actions from multi-view videos using cross-viewpoint reconstruction, and these latents improve action-centricity and downstream VLA performance across reported benchmarks. Its VLA evaluation is limited to simulation, while training requires multi-view videos.

  • Problem

    Video latent actions are useful as pseudo-labels only when they preserve information about underlying actions despite unavailable action labels.

  • Method

    MVP-LAM learns discrete latent actions from multi-view videos by using a latent from one view to reconstruct a future observation in another view.

  • Results

    MVP-LAM produces more action-centric latents and improves VLA performance on SIMPLER and LIBERO, including with 3× less pretraining data than baselines.

  • Takeaways & Limitations

    Cross-viewpoint reconstruction provides action-centric pseudo-labels that support VLA pretraining across the reported manipulation benchmarks.

  • Takeaways & Limitations

    MVP-LAM requires multi-view videos for latent-action training, and its VLA evaluation is limited to simulation without real-world robot experiments.

Abstract

from arXiv · show

Latent actions learned from diverse human videos serve as pseudo-labels for vision-language-action (VLA) pretraining, but provide effective supervision only if they remain informative about the underlying ground-truth actions. For effective supervision, latent actions should contain information about the underlying actions even though they are inaccessible. We propose Multi-ViewPoint Latent Action Moel (MVP-LAM), which learns latent actions that are highly informative about ground-truth actions from multi-view videos. MVP-LAM trains latent actions with a cross-viewpoint reconstruction objective, so that a latent action from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on various benchmarks. The code and trained checkpoints are available at https://jmsnu.github.io.

1. Introduction

MVP-LAM addresses the lack of action labels in videos by learning action-centric latent actions through cross-viewpoint reconstruction. Its latents improve action-related evaluation and VLA pretraining performance over baselines, including with less pretraining data.

  • 1. Introduction: Video-based learning is motivated by the cost of collecting labeled robot trajectories, but human videos lack low-level action labels for standard supervised imitation learning.Latent actions provide compact frame-transition representations that can serve as pseudo-action labels.
  • 1. Introduction: Viewpoint variation can entangle camera motion with action-driven transitions, causing latent actions learned from human videos to overfit viewpoint-specific cues.MVP-LAM targets this exogenous noise through multi-view training rather than action supervision.
  • 1. Introduction: MVP-LAM uses a cross-viewpoint reconstruction objective to infer a latent action from one view and predict another view’s future observation.This discourages encoding viewpoint-specific cues and retains shared transition information.
  • 1. Introduction: 62% higher mutual information I(Z; A) than the prior LAM is reported for MVP-LAM latents, alongside more effective and data-efficient VLA pretraining.The overview attributes this improvement to retaining action-centric information across viewpoints.
  • 1. Introduction: VLA pretrained with MVP-LAM outperforms baselines using 3× larger pretraining datasets on SIMPLER and LIBERO.This demonstrates the usefulness of MVP-LAM latents as pseudo-labels for VLA pretraining.

2. Related Works

Related work studies video representations, latent actions, viewpoint diversity, and exogenous noise in robot learning. MVP-LAM differs by explicitly using multi-view cross-viewpoint reconstruction to learn action-centric latent actions without the additional components required by many viewpoint-robust observation methods.

  • Latent Action Learning from Video: Prior latent action methods use single-view video, whereas MVP-LAM explicitly uses multi-view video during LAM training.Its cross-viewpoint reconstruction objective constructs action-centric latent actions.
  • Latent Action Learning from Video: MVP-LAM swaps latent tokens across synchronized views while reconstructing each view’s future feature, encouraging latents to encode inherent transition information.This combines self-viewpoint and cross-viewpoint reconstruction in the training design.
  • Latent Action Learning from Video: The method architecture includes frozen DINOv2 features, vector-quantized discrete latent tokens, and a decoder for reconstructing future features.The supplied method fragments identify these components without specifying their full arrangement.
  • Learning from Videos with Diverse Viewpoints: Many viewpoint-diversity methods target observation representations and require camera calibration, dense multi-view coverage, or costly 3D reconstruction and neural rendering.Robustness to viewpoint variation at the latent-action level remains less explored.
  • Exogenous Noise in Latent Action Learning: Exogenous-noise approaches may depend on action labels, reliable object decomposition, or pretrained VLM quality, and are often evaluated only on controlled synthetic-distractor benchmarks.Their transfer to realistic noisy manipulation and multi-task or long-horizon settings remains open.

3. Method

MVP-LAM learns discrete latent actions from synchronized multi-view videos by combining self-viewpoint and cross-viewpoint reconstruction. The cross-viewpoint objective discourages viewpoint-specific information, aiming to preserve action-relevant information under a capacity bottleneck.

  • Multi-Viewpoint Latent Action Model: MVP-LAM learns discrete latent actions from time-synchronized multi-view videos, using one view’s latent action to predict the future observation in another view.Its full objective combines self-viewpoint reconstruction, cross-viewpoint reconstruction, vector-quantization, and commitment losses.
  • Latent action model: Standard LAM encodes consecutive observation features into a vector-quantized latent action, then decodes the next observation from the current observation and that latent.The discrete representation can serve as a pseudo-action label and supports cross-entropy pretraining of a vision-language model.
  • Action-centricity: Action-centricity is defined by how informative latent actions are about underlying actions, measured through mutual information and action prediction from latent actions.Viewpoint variation can consume limited latent capacity and reduce I(Z_t; A_t), motivating objectives that suppress viewpoint-dependent variation.
  • Action-centricity: Cross-viewpoint reconstruction discourages latent actions from encoding viewpoint-specific cues, reducing viewpoint dependence and preserving capacity for action-relevant dynamics.The decoder is not conditioned on the latent action’s viewpoint, so viewpoint-specific factors increase cross-view reconstruction loss.
  • Evaluation: Figure 3 estimates I(Z; A) with KSG, BA, and MINE on Bridge V2, while Figure 4 measures action-prediction NMSE in-distribution and on OOD LIBERO suites.Higher mutual information and lower NMSE indicate latent actions retain more information about actions.

4. Experiments

Experiments evaluate MVP-LAM’s action-centricity, downstream VLA manipulation performance, viewpoint robustness, ablations, and scaling across simulated benchmarks.

  • Are MVP-LAM latent actions more action-centric?: MVP-LAM achieves the highest estimated mutual information across estimators and lower probing NMSE on Bridge V2 and OOD LIBERO suites, with a small LIBERO-Goal drop versus UniVLA.
  • Is MVP-LAM Effective for Manipulation?: 60.4% average success rate improves over UniVLA’s 39.6% on SIMPLER, with gains on all four tasks and the best average performance overall.
  • Is MVP-LAM Effective for Manipulation?: 94.1% average success rate on LIBERO improves over UniVLA under the same Bridge V2 pretraining, while MVP-LAM also outperforms π0 on LIBERO-Long.
  • Does MVP-LAM Preserve Transition Information Under Viewpoint Perturbation?: MVP-LAM concentrates attention on grippers and manipulated objects and remains stable under viewpoint perturbation, unlike UniVLA’s more diffuse, shifting attention.
  • Ablation Study: Including human videos improves Bridge V2 action-centricity over robot-only training, while synthetic synchronization lags leave performance consistent.
  • Ablation Study: Cross-viewpoint reconstruction is associated with higher action-centricity: removing Lcross lowers mutual information and increases linear-probe NMSE.

5. Conclusion and Limitations

MVP-LAM uses cross-viewpoint reconstruction to reduce viewpoint-specific information while learning action-centric latent actions. Its scope is bounded by multi-view training requirements and simulation-only VLA evaluation.

  • The method requires multi-view videos, adding instrumentation compared with single-view data, and its VLA evaluation excludes real-world robot experiments.The authors identify weakly synchronized or pseudo-paired videos as a future direction for relaxing synchronization requirements.
  • MVP-LAM learns discrete latent actions through cross-viewpoint reconstruction, using one view’s latent action to predict another view’s future observation.This design is intended to discourage encoding viewpoint-specific cues.
  • MVP-LAM bounds action mutual information by latent entropy minus viewpoint information and conditional state uncertainty.Under fixed capacity, reducing viewpoint information helps provided latent entropy does not substantially collapse.

B. Action-centricity Estimation Details

The evaluation converts variable-horizon normalized action sequences into standardized seven-dimensional net-action targets. This preserves comparable output dimensionality and cumulative-control semantics across horizons.

  • Actions are first de-normalized with dataset-specific means and standard deviations before net-action aggregation.Elementwise multiplication with the standard deviation restores the original action scale.
  • Net actions sum the first six continuous control dimensions over time and use the final gripper command as the seventh dimension.The resulting target is a single 7D vector rather than a flattened H-step sequence.
  • The aggregated net action is re-normalized with horizon-aware statistics for use in mutual-information estimation and linear probing.Elementwise division and a small numerical-stability constant are used in the final normalization.
  • Summation preserves cumulative-control semantics and avoids horizon-dependent rescaling, unlike averaging.Keeping the target dimension fixed also avoids changing network capacity across horizons.

B.1. Mutual Information

Mutual information between latent actions and ground-truth actions is estimated from paired samples using complementary non-parametric and neural estimators. The protocol freezes the latent action model and reports held-out, multi-seed estimates.

  • Latent-action MI is evaluated on Bridge V2 using paired latent-action and ground-truth-action samples.The reported protocol uses disjoint training, validation, and test splits with mean and standard deviation across random seeds.
  • MI is estimated with KSG, Barber–Agakov, and MINE, while random pairing serves as a dependence-mismatch sanity check.Neural estimators are trained without updating the frozen LAM.
  • KSG standardizes latent actions and actions, then applies a Gaussian random projection because nearest-neighbor estimation is unstable in high dimensions.The projection uses W sampled from a standard normal distribution and k = 5.
  • Projected KSG mutual information is a lower bound on true mutual information because random projection discards information.This makes the estimate conservative relative to the original latent space.
  • The BA estimator fits a conditional diagonal Gaussian whose mean is predicted by an MLP, whereas MINE uses a Donsker–Varadhan critic with shuffled actions for product-of-marginals samples.Both neural estimates are evaluated on held-out test data after validation-based checkpoint selection.

B.2. Details of Linear Probing

Linear probing evaluates how well latent actions predict normalized robot actions, with results reported in-distribution and across LIBERO out-of-distribution suites. MVP-LAM leads on Bridge V2 but not uniformly on OOD suites.

  • The probe freezes the LAM and trains a simple linear layer by minimizing mean-squared error on latent-action and action pairs.NMSE is reported with standard deviations over four random seeds.
  • MVP-LAM achieves the lowest NMSE on Bridge V2 among all compared methods, including LAPA and Moto.This indicates the strongest step-level action encoding on the target training distribution.
  • On LIBERO OOD, LAPA is best on Spatial, Object, and Long, while Moto is best on Goal; MVP-LAM is second-best on the first three and underperforms on Goal.The results indicate that OOD action predictability depends on factors beyond viewpoint robustness alone.
  • The authors hypothesize that MVP-LAM’s OOD weakness reflects smaller training scale, lower bottleneck capacity, and viewpoint-distribution mismatch.They leave scaling to larger multi-view datasets and larger codebooks for future work.
  • Inverse probing predicts latent actions from actions as a proxy for minimality, and MVP-LAM has lower NMSE on Bridge V2 and LIBERO-Long but not LIBERO-Goal.The authors interpret this result with Figure 4 as evidence of minimal action-centric latents.
  • Linear-probe NMSE and MI are necessary but insufficient indicators of downstream VLA performance because backbone and other factors also affect results.Jointly optimizing latent-action quality with downstream VLA performance is identified as future work.

C.1. MVP-LAM training details

MVP-LAM is trained on synchronized multi-view robot and human manipulation videos without proprioceptive inputs or action labels. Its latent-action tokens are used to pretrain VLAs and decode continuous actions during downstream finetuning.

  • MVP-LAM training: MVP-LAM combines selected multi-view Open X-Embodiment robot trajectories with synchronized multi-view EgoExo4D human videos.Robot data requires single-arm end-effector control and time-synchronized views; EgoExo4D contains approximately 5k in-the-wild videos.
  • MVP-LAM training: Training excludes proprioceptive inputs and robot action labels, using visual observations and latent-action pseudo-labels instead.The same supervision restriction applies when MVP-LAM tokens are used for VLA pretraining.
  • MVP-LAM training: The training mixture and architecture are specified through dataset sampling weights and MVP-LAM hyperparameters.Table 8 covers the data mixture, while Table 9 covers training and architecture details.
  • VLA pretraining: VLA pretraining uses a Prismatic-7B VLM with cross-entropy prediction of MVP-LAM latent-action tokens, restricted to Bridge V2.Pretraining follows the UniVLA recipe and uses 4× H200 GPUs for 45 GPU-hours.
  • VLA finetuning: During finetuning, learnable vision and action queries attend to VLM embeddings, while L1 action regression and token cross-entropy losses are optimized.The resulting decoder maps latent-action representations to continuous robot actions; SIMPLER and LIBERO use separate hyperparameter configurations.
  • Latent-action configurations: The compared latent-action models differ in codebook size, token count, and embedding dimension, with Moto using K=128, L=8, and d=32.Table 13 defines K, L, and d as codebook size, tokens per transition, and token embedding dimension.

D.2. Implementation details of baselines

Baseline VLAs are finetuned with their official or standard training procedures, using different parameter-update strategies and observation inputs.

  • Octo: Octo-base and Octo-small are fully finetuned for 10k steps with batch size 32 and learning rate 3 × 10^-4.
  • π0: π0 is finetuned with LoRA for 10k steps using batch size 16 and learning rate 5 × 10^-5.For SIMPLER, π0 receives one RGB image and the language instruction, excluding wrist-view images and proprioceptive inputs.

E.1. Latent action examples

MVP-LAM assigns discrete tokens to recurring transition dynamics and uses cross-viewpoint reconstruction to focus attention on manipulation-relevant regions.

  • Latent action examples: Similar motion patterns activate similar MVP-LAM codes across sources, clustering transition dynamics in a shared token space without action supervision.Figure 10 pairs discrete codes with before-and-after observations for representative transitions.
  • Cross-viewpoint attention: Cross-viewpoint reconstruction makes attention concentrate on manipulation-relevant regions, unlike standard single-view reconstruction.The comparison uses attention maps trained with and without Lcross across two viewpoints of the same state.

E.2. Details of novel view synthesis in Bridge V2

Viewpoint robustness is evaluated by synthesizing perturbed camera views for Bridge V2 transitions and comparing reconstruction quality under perturbed latent-action inference.

  • Perturbation setup: The evaluation samples 100 Bridge V2 trajectories and synthesizes five viewpoint-perturbed images per step, yielding 3.7k perturbed transitions.The perturbations are generated with a zero-shot NVS model finetuned from DROID.
  • Perturbation setup: Camera perturbations combine axis–angle rotations and 3D translations, using σθ=0.075 rad and σp=0.03 m unless otherwise specified.The perturbed pose is formed by translating the camera position and composing a quaternion rotation.
  • Perturbation setup: Viewpoints are randomly sampled within a range whose LPIPS similarity is below 0.5.Figure 12 illustrates the resulting viewpoint-perturbed trajectories.
  • Evaluation: Both LAPA and Moto substantially degrade under viewpoint perturbations, with blurred or distorted predictions that often lose scene structure.Pixel-level results support that the degradation is not solely caused by re-embedding predictions into DINOv2 space.
  • Evaluation: MVP-LAM cannot be compared symmetrically with pixel-level PSNR because it does not decode pixels, so DINOv2-space error remains the common metric.
  • Evaluation: Pixel-level PSNR and perturbed-input PSNR assess reconstruction quality on unperturbed and viewpoint-perturbed transitions, respectively.Table 15 reports mean±std over three random seeds; qualitative results compare LAPA and Moto predictions across the two settings.
Loading 2602.03668v3…