PaperlayerView exact revision on arXiv

Source-linked AI summary

V-Link: Recovering Lost Visual Representations in Action DiT for Vision-Language-Action Models

Yehao Lu, Jiarui Yang, Yuning Su, Yufeng Xie, Yu Zhong, Yazhou Zhang, Haiyu Lan, Kaixiang Lu, Peiwen Lin, Chuang Wang, Zequn Qin, Enyu Li, Xi Li

arXiv:2608.25308v1cs.CV

TL;DR

VLA action experts can lose access to geometric and semantic information during VL-to-A transfer, limiting perceptual grounding for fine-grained manipulation. V-Link recovers these representations with complementary Spatial and Semantic Queries injected asymmetrically into Action DiT, improving results across simulation and real-world tasks. The method adds only 1.58 ms of inference latency while consistently outperforming GR00T N1.6.

  • Problem

    VL-to-A transfer limits Action DiT’s access to 3D geometric and 2D semantic information encoded in VLM features.

  • Method

    V-Link learns complementary Spatial and Semantic Query representations within the VLM and injects them asymmetrically into Action DiT.

  • Results

    +31.2%, +18.8%, and +1.9% average success-rate gains over GR00T N1.6 are reported on LIBERO-Plus, RoboTwin 2.0, and LIBERO, respectively.

  • Takeaways & Limitations

    V-Link establishes representation recovery at the VL-to-A interface as an effective approach for precise and efficient robot manipulation.

Abstract

from arXiv · show

Vision-language-action (VLA) models provide a scalable path toward generalist robotic manipulation by integrating visual perception, language understanding, and continuous action control. However, we reveal a critical limitation of VLA architectures: the action expert has limited access to the 3D geometric and 2D semantic information available in VLM features. This accessibility gap weakens perceptual grounding and limits performance on fine-grained robotic manipulation. To address this issue, we propose V-Link, which explicitly recovers visual representations during the vision-language (VL) to action (A) feature transfer. Specifically, V-Link learns complementary Spatial and Semantic Query representations within the VLM and injects them into Action DiT through asymmetric pathways. Semantic Queries complement the original VLM image tokens, whereas Spatial Queries provide dedicated geometric conditioning for spatially grounded action generation. Across LIBERO, LIBERO-Plus, and RoboTwin 2.0, our V-Link improves the average success rate over base model GR00T N1.6 by +1.9%, +31.2%, and +18.8%, respectively. On the AGIBOT A3 Ultra, V-Link further achieves gains of +20% and +24% on two real-world humanoid tasks.

I. INTRODUCTION

The paper identifies a visual-representation accessibility bottleneck in VL-to-A transfer, where Action DiT receives limited 3D geometric and 2D semantic information. V-Link addresses this gap with complementary query representations and reports improvements across simulation and real-world manipulation.

  • Method: V-Link’s training-only diagnostic heads specialize the recovered representations for depth estimation and semantic segmentation.The diagnostic protocol freezes VLM and Action DiT features and trains only lightweight task heads under identical supervision.
  • Motivation: GR00T N1.6 Action DiT features show a pronounced accessibility gap relative to VLM features, especially for 3D geometry and 2D semantics.With frozen features, depth MAE increases from 0.015 to 0.071 and segmentation mIoU decreases from 0.665 to 0.290.
  • Results: +31.2% average success-rate gain over GR00T N1.6 is achieved on LIBERO-Plus, alongside gains of +18.8% on RoboTwin 2.0 and +1.9% on LIBERO.These results are reported across three simulation benchmarks.
  • Results: +20% and +24% gains over GR00T N1.6 are achieved on AGIBOT A3 Ultra power-on and power-off tasks, respectively.The improvements add 1.58 ms of inference latency, increasing latency from 43.21 ms to 44.79 ms.
  • Method: V-Link learns complementary Spatial and Semantic Queries within the VLM and injects them into Action DiT through asymmetric pathways.The framework targets geometric and semantic information during VL-to-A transfer.

A. VLA Architectures

The paper situates V-Link within dual-system VLA architectures that separate multimodal reasoning from action control. Unlike approaches focused mainly on VLM enhancement or architectural connectivity, V-Link targets perceptual accessibility inside Action DiT.

  • VLA Architectures: Dual-system VLAs pair a VLM for scene understanding and task reasoning with a separate lightweight action expert for robot control.GR00T N1.6 is presented as a representative dual-system model.
  • V-Link: V-Link examines and recovers 3D geometric and 2D semantic representations within the action expert during VL-to-A transfer.This focus distinguishes it from work centered on architectural integration or execution efficiency.
  • Related Approaches: Existing perception methods primarily enhance VLM representations, while VLA-Adapter focuses on efficient VLM–Action DiT connections.These approaches do not explicitly address whether transferred visual representations remain accessible within the action expert.
  • V-Link: V-Link introduces learnable Spatial and Semantic Queries into the VLM and injects them asymmetrically into Action DiT.The design uses separate pathways to provide complementary perceptual information for action generation.
  • V-Link: Training-only depth-estimation and semantic-segmentation heads specialize the two query sets for geometry and semantics, respectively.A causal attention mask isolates the query sets while preserving native image-text computation.

B. VLM Representations Decoupling

V-Link decouples geometric and semantic information from VLM representations using dedicated query tokens and task-specific supervision. Lightweight depth and segmentation heads train these queries to capture complementary visual cues for downstream use.

  • Query representations: V-Link introduces Spatial and Semantic Query tokens to separately capture geometric and semantic information from VLM inputs.The queries are designed to recover complementary visual representations rather than relying solely on entangled image and text features.
  • Query representations: Under a tailored causal attention mask, both query sets independently attend to VLM image and text tokens and progressively aggregate visual cues.The mask preserves native image-text computation while isolating the two query sets.
  • Geometric supervision: The Spatial Query is trained through a lightweight depth head that predicts dense multi-view depth maps, encouraging depth information to remain in the query representation.The depth head has approximately 2.3M parameters and uses smooth-L1, absolute-relative, and hard-patch losses.
  • Semantic supervision: The Semantic Query is trained through a semantic-segmentation head that predicts logits across image height, width, semantic classes, and camera views.Its loss combines pixel-wise cross-entropy with soft dice loss to address region overlap, background dominance, and class imbalance.
  • Training and inference: At inference, V-Link removes the auxiliary depth and segmentation heads, requiring no depth or segmentation annotations.The auxiliary heads are used only during training after a warm-up stage for the VLM LoRA parameters, query sets, and task heads.

C. Visual Representation Recovery in Action DiT

V-Link recovers complementary semantic and geometric representations and injects them into Action DiT through an asymmetric attention design. Semantic enrichment precedes dedicated spatial conditioning within selected image cross-attention layers.

  • Visual Representation Recovery in Action DiT: Query representations are injected only at layers satisfying ℓ=4k+2, preserving the original alternating-attention schedule.These layers correspond to image cross-attention in the 4-layer cycle.
  • Visual Representation Recovery in Action DiT: V-Link injects Spatial and Semantic Query representations into Action DiT through asymmetric pathways.The method builds on GR00T N1.6 and separates semantic enrichment from geometric conditioning.
  • Visual Representation Recovery in Action DiT: Semantic Queries and VLM image tokens provide keys and values in parallel cross-attention operations over the action features.The outputs are concatenated, projected, and added residually to form a semantically enriched representation.
  • Visual Representation Recovery in Action DiT: Spatial Queries subsequently provide keys and values for spatial cross-attention, explicitly conditioning the action features on 3D geometry.A learnable gating parameter controls the spatial conditioning contribution.
  • Visual Representation Recovery in Action DiT: The overall training objective includes the action-generation flow-matching loss.The supplied passage introduces the objective and identifies Lact as the flow-matching loss for action generation.

A. Benchmarks and Evaluation Metrics

V-Link is evaluated on three simulation benchmarks covering standard and shifted-distribution manipulation settings, as well as diverse contact interactions.

  • Benchmarks and Evaluation Metrics: V-Link is evaluated on LIBERO, LIBERO-Plus, and RoboTwin 2.0 simulation benchmarks.LIBERO and LIBERO-Plus follow official protocols, while models trained on standard LIBERO are evaluated on LIBERO-Plus without fine-tuning.
  • Benchmarks and Evaluation Metrics: RoboTwin 2.0 evaluation uses six challenging tasks spanning move, place, and contact interactions.The joint training set contains 300 demonstrations, with 50 demonstrations per task.

B. Implementation Details

Experiments initialize from GR00T N1.6 3B, freeze pretrained language and vision backbones, and train V-Link with LoRA-based VLM adaptation and benchmark-specific action chunks.

  • Implementation Details: All models are initialized from GR00T N1.6 3B and trained in BF16 on 4 NVIDIA H100 GPUs.Training uses DeepSpeed ZeRO-2, while pretrained language and vision backbones remain frozen.
  • Implementation Details: The VLM is adapted using rank-128 LoRA with AdamW, an initial learning rate of 1 × 10^-4, weight decay of 1 × 10^-5, cosine decay, and 5% warm-up.These settings are reported in the implementation configuration.
  • Implementation Details: On LIBERO, training uses 80K steps, global batch size 160, external and wrist camera views, 2×5×5 tokens per query set, and 16-step action chunks.The supplied implementation passages specify these LIBERO settings.
  • Implementation Details: On RoboTwin 2.0, training uses 120K steps, global batch size 128, 3×5×5 tokens per query set, and 15-step dual-arm action chunks.The action chunks are dual-arm predictions for the RoboTwin setting.

C. Comparisons with State-of-the-art Methods

V-Link consistently outperforms GR00T N1.6 across LIBERO, LIBERO-Plus, and RoboTwin 2.0, with gains under standard evaluation and distribution shifts.

  • Comparisons with State-of-the-art Methods: +1.9% average success rate over GR00T N1.6 on LIBERO, where V-Link ranks first across all 4 suites.V-Link achieves an average success rate of 99.3% and gains +5.1% on LIBERO-Long.
  • Comparisons with State-of-the-art Methods: +31.2% over GR00T N1.6 on LIBERO-Plus, where V-Link achieves a 75.0% overall success rate under 7 distribution shifts.V-Link ranks first in six settings and surpasses the previous best method by +5.4%.
  • Comparisons with State-of-the-art Methods: +18.8% over GR00T N1.6 on RoboTwin 2.0, with a 56.8% average success rate across 6 tasks.Improvements are reported consistently across all tasks.

D. Ablation Study

The ablations show that Spatial and Semantic Queries contribute complementary benefits, while task supervision and query injection are jointly important. Query capacity improves performance with diminishing returns, making 3 × 5 × 5 the best accuracy–efficiency trade-off.

  • Auxiliary Queries: Combining Spatial and Semantic Queries reaches 56.8% average success, exceeding Spatial Queries alone at 51.2% and Semantic Queries alone at 47.8%.Spatial Queries provide the largest reported gains on Move and Place, while Semantic Queries show the largest gain on Contact.
  • Task Supervision and Injection: Combining task supervision and query injection reaches 56.8% average success, outperforming removing task supervision or query injection by +16.5% and +17.0%, respectively.Removing both components separately produces only 39.8% and 40.3% average success.
  • Query Capacity: Increasing the query grid from 3 × 1×1 to 3×2×2 raises average success from 37.8% to 51.8%, indicating that sufficient query capacity matters.The reported performance improves with larger grids but shows diminishing returns.
  • Representation Recovery: Qualitative comparison shows that GR00T N1.6 Action DiT retains little 3D depth and incomplete 2D semantics, while V-Link recovers both representations.This visual comparison aligns with the ablation emphasis on spatial and semantic query contributions.
  • Query Capacity: The 3 × 5 × 5 grid achieves 56.8% average success and +18.8% over GR00T N1.6 with 1.58 ms additional latency, whereas 3 × 6 × 6 falls slightly to 56.5%.The larger grid also incurs additional latency, so 3 × 5 × 5 is selected as the best accuracy–efficiency trade-off.

E. Real-Robot Experiments

Real-robot experiments on the AGIBOT A3 Ultra evaluate V-Link over 50 consecutive trials per task. V-Link substantially outperforms GR00T N1.6 on both power tasks without auxiliary heads or pseudo-labels at inference.

  • AGIBOT A3 Ultra: V-Link achieves 98%/94% success on power-on/off, compared with 78%/70% for GR00T N1.6.Training uses completed RGB-D depth and pseudo-ground-truth segmentation, but auxiliary heads and pseudo-GT are absent at inference.
  • Conclusion: Across real-world humanoid manipulation, V-Link demonstrates consistent gains with minimal inference overhead and no auxiliary heads or annotations at inference.The conclusion characterizes representation recovery at the VL-to-A interface as effective for precise and efficient robot manipulation.
Loading 2608.25308v1…