PaperlayerView exact revision on arXiv

Source-linked AI summary

GLaD: Geometric Latent Distillation for Vision-Language-Action Models

Minghao Guo, Meng Cao, Jiachen Tao, Rongtao Xu, Yan Yan, Xiaodan Liang, Ivan Laptev, Xiaojun Chang

arXiv:2512.09619v1cs.RO

TL;DR

VLA models commonly rely on 2D visual features that lack explicit geometric information relevant to manipulation. GLaD distills frozen VGGT geometry features into LLM visual-token states, and achieves 94.1% average success on LIBERO against UniVLA’s 92.5%.

  • Problem

    Many VLA models rely on 2D encoders that capture semantic correspondences but do not encode 3D positions, depth, or object relations relevant to manipulation.

    Source: §I · p. 1

  • Method

    GLaD uses frozen VGGT as a teacher and aligns its geometric features with LLM hidden states for visual tokens during pretraining.

    Source: §I · pp. 1–2

  • Results

    94.1% average success on LIBERO exceeds UniVLA’s 92.5% with identical pretraining and posttraining data, and GLaD shows stronger resilience to appearance perturbations.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The results support geometry-aware pretraining as a promising way to improve policy generalization beyond superficial visual pattern matching.

    Source: §VI · p. 8

    Spatial-layout generalization remains limited, including limited improvement on position perturbations.

    Source: §VI · p. 8, C. Alternative Approaches and… · p. 8

Abstract

from arXiv · show

Most existing Vision-Language-Action (VLA) models rely primarily on RGB information, while ignoring geometric cues crucial for spatial reasoning and manipulation. In this work, we introduce GLaD, a geometry-aware VLA framework that incorporates 3D geometric priors during pretraining through knowledge distillation. Rather than distilling geometric features solely into the vision encoder, we align the LLM's hidden states corresponding to visual tokens with features from a frozen geometry-aware vision transformer (VGGT), ensuring that geometric understanding is deeply integrated into the multimodal representations that drive action prediction. Pretrained on the Bridge dataset with this geometry distillation mechanism, GLaD achieves 94.1% average success rate across four LIBERO task suites, outperforming UniVLA (92.5%) which uses identical pretraining data. These results validate that geometry-aware pretraining enhances spatial reasoning and policy generalization without requiring explicit depth sensors or 3D annotations.

I. INTRODUCTION

The introduction frames geometric understanding as a gap in 2D-focused VLA models and proposes distilling 3D priors into multimodal representations. GLaD is evaluated on LIBERO and LIBERO-PRO.

I. INTRODUCTION

  • 2D-focused VLA encoders capture semantic correspondences but omit 3D positions and object relations needed for spatially precise manipulation.The introduction illustrates the issue with attention maps for cloth-moving and bowl-placement tasks.Source: §I · p. 1, Figure 1 · p. 1
  • GLaD uses frozen VGGT as a teacher to align visual-token LLM hidden states with geometric features, integrating geometry into action-driving multimodal representations.The approach does not require depth sensors or explicit 3D annotations.Source: §I · pp. 1–2
  • A combined objective trains latent action prediction alongside geometry alignment to support task-specific visuomotor skills and generalizable geometric reasoning.The paper evaluates on LIBERO and LIBERO-PRO.Source: §I · p. 2
  • 94.1% average success on LIBERO exceeds UniVLA’s 92.5%, while GLaD scores 81% on LIBERO-GOAL versus 62% under object-appearance perturbations.UniVLA uses identical pretraining and posttraining data; the appearance perturbations change color, texture, and size while preserving semantic equivalence.Source: §I · p. 2

II. RELATED WORKS

Related work covers 3D-aware VLA approaches and geometry representation learning, while noting the challenge of integrating geometry with vision and language. GLaD builds on the UniVLA backbone and aligns its LLM representations with a geometry-aware teacher.

II. RELATED WORKS

  • Earlier VLA work largely relies on 2D observations, while later methods incorporate RGB-D, point clouds, or inferred geometry; consistent alignment of 3D, visual, and language representations remains challenging.Geometry-aware vision models such as VGGT provide latent 3D priors that can serve as teachers.Source: §II · p. 2
  • Geometry distillation work spans generation, representation alignment, and self-bootstrapping, with recent methods transferring 3D cues into multimodal models.Examples include Spatial Forcing, Vid-LLM [83], and 3D-Aware VLMs with Geometric Distillation [84].Source: §II · pp. 2–3

III. METHODOLOGY

  • GLaD combines an LLM backbone, vision encoder, action head, and geometry-distillation module to extract geometric information from images and predict latent actions.The method section introduces the distillation module and training strategy.Source: §III · p. 3

A. Geometry Distillation

  • GLaD follows UniVLA’s Prismatic encoder, projector, LLaMA-2-7B backbone, and action decoder, adding alignment between LLM visual representations and pretrained geometry-teacher features.The geometry-distillation module has two subcomponents.Source: A. Geometry Distillation · p. 3

1) VGGT Feature Extractor:

The geometry-distillation module extracts frozen VGGT features, aligns them with final-layer LLM image-token states, and trains geometry alignment alongside latent action prediction.

1) VGGT Feature Extractor:

  • VGGT is kept frozen and supplies 3D features; GLaD uses a single historical frame and pools and aggregates features to match visual-patch dimensions.The stated VGGT input sequence spans 32 historical frames, although GLaD uses one frame for simplicity.Source: 1) VGGT Feature Extractor · p. 3

2) Feature Alignment Network:

  • A two-layer MLP maps final-layer LLM hidden states at image-token positions into the VGGT feature space, integrating geometry into fused multimodal representations.The hidden states are extracted from the LLM rather than the vision encoder.Source: 2) Feature Alignment Network · pp. 3–4

3) Training Objective:

  • The pretraining objective combines latent-action cross-entropy with geometry-alignment MSE, with λ balancing the two losses.The VGGT teacher remains frozen throughout training.Source: 3) Training Objective · p. 4

B. Training Strategy

GLaD is trained in two stages: large-scale pretraining with geometry distillation, followed by task-specific posttraining.

B. Training Strategy

  • During pretraining on a large-scale robotic manipulation dataset, GLaD learns to predict latent actions while simultaneously aligning its internal visual representations with VGGT geometry features.Training ran for 45 epochs with AdamW at a 5e-7 learning rate on 8×A100 GPUs for approximately 9 days.Source: 1) Stage 1: Pretraining with Geometry… · p. 4
  • The model starts from a pretrained UniVLA checkpoint with a learnable alignment network, while the VGGT teacher stays frozen during pretraining.The distillation-loss weight is tuned on validation performance to balance action accuracy and geometric alignment.Source: 1) Stage 1: Pretraining with Geometry… · p. 4
  • Task-specific posttraining uses LoRA to adapt the VLA backbone while fully training the action decoder and feature-alignment module.The VGGT remains frozen; posttraining runs for 60k steps at a 3.5e-5 learning rate on 8×A100 GPUs.Source: 2) Stage 2: Posttraining on Downstream… · p. 4

IV. EXPERIMENTS

The experiments pretrain GLaD on Bridge, then evaluate it on LIBERO and LIBERO-PRO. LIBERO covers 130 language-conditioned manipulation tasks across four suites.

IV. EXPERIMENTS

  • The experiments proceed from Bridge pretraining to LIBERO post-training and evaluation on standard LIBERO and LIBERO-PRO.The paper introduces the datasets, training protocol, standard-benchmark results, robustness evaluation, and ablations in that sequence.Source: §IV · p. 4

1) Bridge Dataset:

  • Bridge supplies diverse manipulation demonstrations for large-scale pretraining, selected as a computationally efficient dataset with sufficient diversity and scale.Source: 1) Bridge Dataset · p. 4

2) LIBERO Benchmark and LIBERO Dataset:

3) LIBERO-PRO Benchmark:

LIBERO-PRO tests whether manipulation policies remain robust under controlled changes that distinguish task understanding from memorization. It perturbs object appearance, position, language, and task composition.

3) LIBERO-PRO Benchmark:

  • LIBERO-PRO tests generalization beyond standard LIBERO, where high success can coexist with failure under minor perturbations.The benchmark varies objects, positions, semantics, and task composition to distinguish memorization from genuine generalization.Source: 3) LIBERO-PRO Benchmark · pp. 4–5
  • Object perturbations change color, texture, or size; position perturbations vary initial object placements and spatial arrangements.Source: 3) LIBERO-PRO Benchmark · p. 4
  • Semantic perturbations rephrase instructions, while task perturbations recombine known objects and actions into novel task configurations.Source: 3) LIBERO-PRO Benchmark · pp. 4–5
  • The evaluation focuses on perturbation types consistently available across all suites in the benchmark release.Source: 3) LIBERO-PRO Benchmark · p. 5

B. Training Protocol

GLaD is pretrained on Bridge and then post-trained and evaluated on LIBERO. It averages 94.1% success on LIBERO, exceeding UniVLA’s 92.5% with the same pretraining data.

1) Pretraining Phase:

  • Bridge pretraining runs for 45 epochs, and checkpoints from epochs 27 and 45 initialize subsequent post-training experiments.The run uses 8×A100 GPUs for 9 days at learning rate 5e-7.Source: 1) Pretraining Phase · p. 5

2) Post-training Phase:

  • LIBERO post-training adapts the model to the benchmark’s task distribution for 48k steps, with checkpoints selected by validation performance.Training uses learning rate 3.5e-5 and 8×A100 GPUs.Source: 2) Post-training Phase · p. 5

C. Evaluation on LIBERO Benchmark

  • Evaluation covers all four LIBERO suites, reporting success rates averaged over 50 episodes per task.The training pipeline excludes failure demonstrations.Source: C. Evaluation on LIBERO Benchmark · p. 5
  • Under object perturbations, GLaD scores 81% on GOAL versus UniVLA’s 62%, and 54% on Long versus 47%.Source: Figure 3 · p. 5
  • 94.1% average success on LIBERO exceeds UniVLA’s 92.5%, with both models using the same pretraining data.GLaD reaches 97.4% on LIBERO-OBJECT, 94.4% on LIBERO-GOAL, and 89.4% on LIBERO-LONG.Source: C. Evaluation on LIBERO Benchmark · p. 5

D. Robustness Analysis on LIBERO-PRO

GLaD’s robustness advantage is strongest under object-appearance changes, while position and task-composition perturbations remain difficult. Ablations associate performance with the geometry encoder, alignment layer, and fusion strategy.

D. Robustness Analysis on LIBERO-PRO

E. Ablation Studies

  • On LIBERO-SPATIAL, VGGT scores 95.0% versus PI3’s 65.2%, the largest encoder-choice gap among suites.The comparison evaluates alternative geometry encoders.Source: E. Ablation Studies · pp. 6–7
  • Attention maps show focused target-object attention in GLaD, while variants display scattered attention or greater focus on the gripper.These qualitative patterns support the ablation results; the authors also report that the three design choices work synergistically.Source: Figure 4 · p. 7, E. Ablation Studies · p. 7
  • Layer-24 alignment reaches 92.6% average success and 90.6% on LIBERO-OBJECT, below the full model’s 97.4% there.The authors attribute the drop to geometric signals being diluted through subsequent transformer layers.Source: E. Ablation Studies · p. 7
  • Early weighted fusion scores 84.0% average success, including 80.8% on LIBERO-OBJECT and 76.0% on LIBERO-LONG.The authors contrast fixed early weighting with late fusion, which lets the LLM integrate geometric and semantic cues task-adaptively.Source: E. Ablation Studies · p. 7

V. DISCUSSION

GLaD develops sharper attention to manipulation-relevant objects than baselines, while aligning geometric features to LLM hidden states helps combine object semantics with 3D spatial structure.

A. Why Geometric Understanding Matters

B. Design Choices and Robustness

GLaD's late-stage geometry alignment outperforms early fusion and achieves 94.1% average LIBERO success, but spatial-layout robustness remains limited.

B. Design Choices and Robustness

  • 94.1% vs. 84.0%: late-stage alignment to LLM hidden states substantially outperforms early fusion.LIBERO-PRO evaluation shows strong robustness to object-appearance changes but limited improvement under position perturbations.Source: B. Design Choices and Robustness · p. 8

C. Alternative Approaches and Limitations

VI. CONCLUSION

  • GLaD distills frozen VGGT features into LLM hidden states corresponding to visual tokens, integrating geometric and semantic cues.Source: §VI · p. 8
  • 94.1% average LIBERO success rate: GLaD outperforms UniVLA (92.5%) trained on identical data.Source: §VI · p. 8
  • Spatial-layout generalization remains limited, although the results support geometric priors as a promising direction for robotic-manipulation VLAs.Source: §VI · p. 8

APPENDIX

The appendix provides detailed LIBERO-PRO results, including per-task object-generalization performance and perturbation-specific success rates.

A. Detailed LIBERO-PRO Results

  • The appendix reports detailed LIBERO-PRO results by task and perturbation type, complementing suite-level averages in the main text.Source: A. Detailed LIBERO-PRO Results · p. 11
  • LIBERO-Object results cover object-generalization tasks placing 10 different objects with varied visual properties, averaged over 50 episodes per task.Source: Table VII · p. 12
Loading 2512.09619v1…