PaperlayerView exact revision on arXiv

Source-linked AI summary

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Manish Kumar Govind, Dominick Reilly, Pu Wang, Srijan Das

arXiv:2602.20231v2cs.ROcs.CV

TL;DR

RGB-derived latent actions capture appearance-driven dynamics but lack 3D geometric structure important for precise manipulation. UniLARN learns unified and modality-specific RGB/depth latent actions for UniLACT’s depth-aware latent pretraining. The paper reports gains over RGB-based latent baselines in simulation and real-world settings.

  • Problem

    RGB-based latent actions capture appearance-driven dynamics but lack explicit 3D geometry important for contact-rich manipulation, while prior depth-aware VLAs use depth as pixel- or feature-level input and still need substantial robot-annotated trajectories.

    Source: §I · p. 1

  • Method

    UniLARN uses inverse and forward dynamics to learn shared-space unified and modality-specific RGB/depth latent actions, which serve as pseudo-labels for UniLACT’s depth-aware latent pretraining.

    Source: §I · pp. 1–2

  • Results

    UNILACT achieves a 29.2% relative improvement over a purely RGB-based latent baseline [10] on simulation benchmarks, with real-world experiments reporting more stable grasping and improved collision avoidance.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Experiments in simulation and the real world support the benefit of integrating depth into latent action representations for downstream robot policy learning.

    Source: §V · p. 7

Abstract

from arXiv · show

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/

I. INTRODUCTION

UNILACT addresses the lack of geometric structure in RGB-based latent actions by learning from RGB-D observations and using depth-aware latent pretraining. Simulation and real-world experiments report improvements over RGB-based approaches.

I. INTRODUCTION

  • Unlabeled internet-scale videos offer a route to pretraining VLA models while reducing dependence on costly, difficult-to-scale teleoperated robot data.The approach transfers video-derived priors to downstream robot tasks with limited supervision.Source: §I · p. 1
  • RGB-based latent actions capture appearance-driven dynamics but lack 3D geometry needed for contact-rich tasks such as grasping, placement, and collision avoidance.Without depth, a policy may have difficulty judging reachability or an impending collision.Source: §I · p. 1
  • Prior depth-aware VLAs use depth as pixel- or feature-level input and still require substantial robot-annotated trajectories, leaving depth within latent action representations under-explored.The stated research question is whether geometric structure can be embedded in RGB latent space during unsupervised pretraining.Source: §I · p. 1
  • UNILACT achieves a 29.2% relative improvement over a purely RGB-based latent baseline [10] on simulation benchmarks, while real-world experiments report more stable grasping and improved collision avoidance.The paper attributes these gains to incorporating geometric structure via depth during pretraining.Source: §I · pp. 1–2
  • UNILARN uses inverse and forward dynamics to learn unified and modality-specific RGB/depth latent actions in a shared space, capturing visual semantics and 3D geometry.These representations serve as pseudo action labels for UNILACT pretraining; learned policies are fine-tuned with limited action supervision, and depth is used only during training.Source: §I · pp. 1–2
  • UNILACT is a cross-modally trained VLA that uses unified and modality-specific latent representations to improve policy learning.The authors report improved 3D spatial understanding over RGB-based latent approaches in simulation and real-world experiments.Source: §I · p. 2

II. RELATED WORK

Related work explores VLA action prediction and latent actions learned from videos, as well as the use of depth for spatial reasoning. UNILACT differs by integrating RGB and depth into unified and modality-specific latent action representations.

A. Vision-Language-Action Models.

  • VLA approaches predict actions with language-model tokens or dedicated action heads, and some use multimodal reasoning or hierarchical planning; large-scale low-level control still often needs costly action-labeled robot trajectories.These data requirements can limit scalability across embodiments and environments.Source: A. Vision-Language-Action Models · p. 2

B. Latent Action Representations.

  • Latent action methods learn discrete control dynamics from videos without explicit action supervision, but most use a single modality, typically RGB, and thus capture mainly appearance-driven dynamics.UNILACT instead learns jointly from RGB and depth to add 3D geometry to latent actions for manipulation.Source: B. Latent Action Representations · p. 2

C. Enhancing VLAs with Depth.

  • Depth-aware VLAs commonly use depth as pixel- or feature-level input or auxiliary supervision, while UniSkill learns generic skills in a joint embedding space.UNILACT integrates RGB and depth and then disentangles them into unified and modality-specific latent actions for cross-modal pretraining.Source: C. Enhancing VLAs with Depth · p. 2

III. PROPOSED METHOD

UNILACT has three training stages: UNILARN learns latent actions from RGB-D observations, UNILACT predicts those latents, and action fine-tuning maps predictions to executable robot control. UNILARN’s shared-space pipeline combines modality-specific and unified representations.

III. PROPOSED METHOD

  • UNILACT proceeds from unified latent action learning, to latent pretraining, to action fine-tuning on robot demonstrations.The stages learn RGB/depth latent actions, train predictions without explicit action annotations, and map unified latent predictions to an executable policy.Source: §III · pp. 2–3, Figure 1 · p. 3

A. UNILARN: Unified Latent Action Learning

  • For paired RGB and depth observations, modality-specific inverse dynamics models produce continuous embeddings that are discretized with vector quantization using a codebook shared across modalities.The observations correspond to current and future frames, with H denoting their temporal offset.Source: A. UNILARN: Unified Latent Action… · p. 3
  • UNILARN concatenates modality-specific codebook embeddings, linearly projects them, and applies a second vector quantizer to produce a unified latent action.The resulting unified representation conditions modality-specific forward dynamics models that reconstruct future observations.Source: A. UNILARN: Unified Latent Action… · p. 3
  • The disentangled reconstruction objective uses modality-specific forward dynamics to make the unified representation capture complementary dynamics across RGB and depth.The complete parameterized RGB-and-depth model for learning unified and modality-specific representations is named UNILARN.Source: A. UNILARN: Unified Latent Action… · pp. 3–4

B. Unified Latent Pretraining.

UNILACT uses UNILARN-derived modality-specific and unified latent actions as action-free supervision. Conditioned on visual observations, task instructions, and a unified latent, it autoregressively predicts RGB, depth, or unified latent tokens.

B. Unified Latent Pretraining.

  • UNILARN extracts modality-specific and unified latent action tokens from RGB-D videos to provide action-free supervision for VLA pretraining.UNILACT predicts a target latent-token sequence from visual observations, task instructions, and a unified latent representation.Source: B. Unified Latent Pretraining · p. 4
  • The target latent tokens may represent RGB, depth, or unified latents, and UNILACT predicts them with an autoregressive transformer.The model conditions on the unified latent representation to encourage complementary semantic and geometric cues.Source: B. Unified Latent Pretraining · p. 4
  • Cross-modal latent prediction promotes alignment between modality-specific and unified latent spaces while retaining modality-specific structure.The pretraining objective is standard autoregressive next-token prediction.Source: B. Unified Latent Pretraining · p. 4

C. Action Fine-Tuning

Action fine-tuning combines unified latent-action prediction with prediction of executable robot actions, while retaining the pretrained latent structure. At inference, the model uses RGB observations and task instructions without depth.

C. Action Fine-Tuning

  • UNILACT appends action-query tokens and maps their transformer outputs through a lightweight decoder to continuous robot actions.The decoder predicts 7-DoF end-effector deltas and a binary gripper command.Source: C. Action Fine-Tuning · p. 4
  • Fine-tuning uses a convex combination of unified latent-action prediction and actual-action prediction to preserve pretrained latent structure.The unified-latent term is a next-token prediction loss for latent-action prediction.Source: C. Action Fine-Tuning · p. 4
  • At inference, UNILACT requires only RGB observations and task instructions; depth is used during training, not at test time.Source: C. Action Fine-Tuning · p. 4

IV. EXPERIMENTS

The experiments assess policy performance in simulation and real-world environments, then examine runtime complexity and design choices through ablations.

IV. EXPERIMENTS

  • The evaluation covers policy performance in both simulation and real-world environments.Source: §IV · p. 4
  • The experiments include an analysis of the model's run-time complexity.Source: §IV · p. 4
  • Ablations examine key design choices in the model.Source: §IV · p. 4

A. Implementation details

UNILARN learns latent actions with modality-specific dynamics modules, while UNILACT uses a causal transformer to predict latent and robot actions from language and visual inputs.

A. Implementation details

  • UNILARN uses modality-specific IDMs with frozen ViT-L encoders and spatio-temporal transformers to capture changes between frames.Learnable query embeddings are appended to patch features; a VQ-VAE objective discretizes the resulting features into latent-action tokens.Source: A. Implementation details · p. 4
  • Each UNILARN forward dynamics model is a ViT-B-style decoder trained with mean-squared-error reconstruction loss.Source: A. Implementation details · p. 4
  • UNILACT uses a GPT-2 causal transformer, frozen T5 instruction encoder, and ViT-L visual encoder, with separate latent and action decoders.Source: A. Implementation details · p. 4
  • The action decoder uses a two-layer MLP and separate output layers for continuous arm actions and a binary gripper command.Source: A. Implementation details · p. 4

B. CALVIN Experiments

On CALVIN ABC→D, UNILACT outperforms the RGB latent baseline Moto under out-of-domain pretraining and consistently surpasses Moto in-domain. Task-level comparisons favor RGB latents on some appearance-driven tasks and unified RGB+depth latents on some geometry-centric tasks.

B. CALVIN Experiments

  • On CALVIN, UNILACT is evaluated in ABC→D, trained on environments A, B, and C and tested on unseen D to assess generalization; each episode requires a sequence of five consecutive tasks.The benchmark includes four environments and 34 manipulation tasks; performance is reported using per-task success rates and average sequence length.Source: B. CALVIN Experiments · pp. 4–5
  • Under in-domain pretraining, UNILACT consistently outperforms Moto and surpasses baselines with comparable input modalities.GR-1 reports the strongest overall performance but uses gripper-view RGB images and proprioceptive state signals, unlike UNILACT's static RGB observations.Source: B. CALVIN Experiments · p. 5
  • +29.2% in average sequence length: UNILACT outperforms Moto with out-of-domain pretraining on OXE.Source: B. CALVIN Experiments · p. 5
  • RGB latents perform better on some appearance-driven tasks, while unified RGB+depth latents perform better on some geometry-centric tasks.Examples include stacking blocks and rotating a blue block for RGB, and moving a slider and turning on a light bulb for unified latents.Source: Figure 2 · p. 5, B. CALVIN Experiments · p. 5

C. Real-World Experiments

UNILACT is evaluated on four real-world tabletop tasks, including two training-seen tasks and two unseen tasks for zero-shot generalization. It outperforms the RGB baseline overall and completes the illustrated carrot-placement task that Moto fails.

C. Real-World Experiments

  • UNILACT achieves higher success rates on 3 of 4 real-world tasks and a 10% overall improvement over Moto.The evaluation uses 10 trials per task with randomized initial object configurations.Source: C. Real-World Experiments · p. 5, Table II · p. 6
  • The four tasks use 30 teleoperated demonstrations per task; T1 and T2 are seen during training, while T3 and T4 test zero-shot generalization.Training combines an OXE subset with real-world demonstrations, then fine-tunes using only real-world demonstrations; missing OXE depth maps are generated from RGB.Source: C. Real-World Experiments · p. 5
  • For T1, Moto fails to place the carrot and pushes the bowl away, whereas UNILACT completes the placement.The text attributes Moto's collision to inaccurate depth prediction and describes UNILACT positioning the carrot above the bowl without collision.Source: C. Real-World Experiments · p. 5, Figure 4 · p. 6

D. Computational Analysis

UNILACT retains Moto's model size and inference runtime because depth is used during training, not inference. Ablations favor aligned unified and modality-specific latents, cross-modal pretraining targets with unified-only fine-tuning, and single-task objectives.

D. Computational Analysis

  • UNILACT maintains the same model size and per-step inference runtime as Moto because depth is required only during training, not inference.Table III reports model parameters and per-step latency.Source: D. Computational Analysis · p. 6, Table III · p. 6

E. Ablation studies

  • RGB and depth latent actions each outperform the no-latent baseline, but UNILACT's aligned unified-plus-modality-specific latents perform best.Independently learned RGB and depth latents underperform individual modalities, which the authors attribute to representation inconsistency.Source: Table IV · p. 6, E. Ablation studies · p. 7
  • Cross-modal pretraining targets followed by unified-only fine-tuning yield the best results among the tested output modalities.The authors suggest cross-modal supervision enriches pretraining representations, while a compact unified target improves fine-tuning action prediction.Source: E. Ablation studies · p. 7, Table V · p. 7
  • Single-task training achieves higher performance than jointly predicting RGB, depth, and unified latents.The authors suggest joint optimization may introduce task interference.Source: E. Ablation studies · p. 7, Table VI · p. 7

V. CONCLUSION

The conclusion presents UNILARN as jointly learning from RGB and depth, and UNILACT as using cross-modal latent prediction with unified and modality-specific actions. Simulation and real-world experiments support the benefits of integrating depth into latent actions.

V. CONCLUSION

  • UNILARN jointly learns representations from RGB and depth observations.It is the unified latent action learning framework underlying the depth-aware approach.Source: §V · p. 7
  • UNILACT is pretrained through cross-modal latent action prediction using both unified and modality-specific latent actions.Source: §V · p. 7
  • Simulation and real-world experiments validate depth integration in latent actions, with UNILACT consistently outperforming representative baselines.The conclusion links a unified space capturing semantic and geometric cues with improved downstream robot policy learning.Source: §V · p. 7

Related papers

Loading 2602.20231v2…