PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Jianman Lin, Shailesh Shailesh, Zhongyi Luo, Jiafei Duan

arXiv:2609.12641v1cs.RO

TL;DR

Visual distribution shifts can expose vision–action shortcuts in robot foundation models, where task-irrelevant cues influence action generation. LIT learns an image-free spatial-goal-conditioned action prior and then constrains visual conditioning through a pose-supervised latent interface. Across four architectures and real-world visual shifts, LIT improves reported success while preserving in-distribution performance.

  • Problem

    Vision–action shortcuts can make action generation rely on task-irrelevant visual cues that become unreliable under visual distribution shifts.

  • Method

    LIT uses two stages: image-free spatial-goal-conditioned action pretraining followed by visual conditioning exclusively through a latent interface supervised to reconstruct the terminal pose.

  • Results

    LIT improves overall LIBERO-Plus success by 3.87%–10.70% across four architectures while preserving or improving average LIBERO success, with real-world gains of 13.3–16.7 percentage points.

  • Takeaways & Limitations

    LIT supports improved zero-shot generalization under visual variations while retaining the spatial information needed for action generation.

  • Takeaways & Limitations

    Evaluation at larger real-world scales remains open, with future work needed on larger, more diverse datasets and broader tasks and environments.

Abstract

from arXiv · show

Robot foundation models achieve strong in-distribution performance but often degrade under visual distribution shifts. When learning to generate actions from pretrained visual representations, models may exploit task-irrelevant visual cues that correlate with demonstrated actions within the training distribution. Such vision-action shortcuts can undermine generalization when these correlations change under distribution shifts. Mitigating these shortcuts requires constraining how visual information is used for action generation while preserving task-relevant spatial information. We propose Latent Interface Training (LIT), a framework-agnostic two-stage strategy that first establishes a spatial-goal-conditioned action prior without images, then constrains visual conditioning through a pose-supervised latent interface. Stage 1 trains the action expert to generate action chunks conditioned on language, robot state, and each demonstrated chunk's terminal SE(3) end-effector pose, learning goal-directed action generation independently of visual cues. Stage 2 introduces a latent interface that aggregates visual and semantic representations and serves as the pretrained action expert's only visual conditioning pathway. The interface is supervised to reconstruct the terminal pose previously used to condition Stage 1, encouraging it to retain the goal-relevant spatial information needed for action generation. Across four vision-language-action and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, and ImageWAM), LIT improves overall LIBERO-Plus success by 3.87-10.70 percentage points while preserving or improving average LIBERO success. Real-world evaluations show 13.30-16.70 percentage-point gains in success aggregated across three tasks under unseen camera configurations, lighting variations, and distractors.

I. INTRODUCTION

Robot foundation models can perform well in-distribution yet rely on task-irrelevant visual cues that fail under visual shifts. LIT addresses this by learning a spatial-goal-conditioned action prior without images and routing later visual conditioning through a pose-supervised latent interface.

  • I. INTRODUCTION: Vision–action shortcuts use task-irrelevant visual cues correlated with demonstrated actions, undermining robustness to viewpoint, appearance, and sensing changes.The challenge is to reduce sensitivity to nuisance variation while preserving spatial information needed for different actions.
  • I. INTRODUCTION: Existing representation-enhancement and image-free pretraining methods do not explicitly constrain how visual information is used during subsequent action generation.They can enrich inputs or learn priors, but leave visual conditioning susceptible to shortcut-prone correlations.
  • I. INTRODUCTION: LIT first learns a spatial-goal-conditioned action prior without images, then introduces visual conditioning through a pose-supervised latent interface.The two stages connect goal-directed action learning with constrained visual conditioning.
  • I. INTRODUCTION: Across four VLA and WAM architectures, LIT improves overall LIBERO-Plus success by 3.87%–10.70% while preserving or improving average LIBERO success.The reported gains are accompanied by real-world improvements under unseen camera configurations, lighting variations, and distractors.
  • I. INTRODUCTION: Real-world evaluations show 13.3–16.7 percentage-point success gains aggregated across three manipulation tasks under unseen visual conditions.These conditions include camera configurations, lighting variations, and distractors.

A. Visual Conditioning in Robot Foundation Models

Robot foundation models expose action experts to rich visual representations through diverse architectural pathways, enabling strong control but leaving them vulnerable to scene-specific visual correlations. Prior work enriches, filters, or pretrains these representations, yet does not jointly constrain visual use while preserving task-relevant spatial information.

  • A. Visual Conditioning in Robot Foundation Models: Autoregressive, diffusion, flow-based, and world–action architectures condition action generation on pretrained visual or vision-language representations through different pathways.Examples include shared visual–action token prediction, conditional denoising, joint attention, layer-wise cross-attention, and learned visual representations.
  • A. Visual Conditioning in Robot Foundation Models: Rich visual conditioning can encourage reliance on scene-specific correlations when robot demonstrations have limited visual diversity.Such correlations may not remain reliable outside the training distribution.
  • A. Visual Conditioning in Robot Foundation Models: Representation-enhancement methods add transferable, spatial, temporal, semantic, or trajectory information to improve policy generalization.Examples include video-pretrained features, motion traces, 3D position information, reasoning, and predicted end-effector trajectories.
  • A. Visual Conditioning in Robot Foundation Models: Information-filtering methods discourage nuisance cues or learn task-conditioned bottlenecks, but do not ensure retention of task-relevant spatial information.Filtering and richer representations alone do not guarantee shortcut-resistant visual conditioning.
  • A. Visual Conditioning in Robot Foundation Models: Image-free action-prior pretraining facilitates downstream policy learning, but prior approaches do not explicitly constrain later visual conditioning.LIT adds terminal-pose guidance during pretraining and a pose-supervised latent interface afterward.

III. METHOD

LIT is a model-agnostic two-stage strategy for reducing vision–action shortcuts while preserving task-relevant spatial information. It first establishes an image-free action prior and then routes visual conditioning exclusively through a pose-supervised latent interface.

  • III. METHOD: Stage 1 trains an action expert without images, while Stage 2 routes visual conditioning through a pose-supervised latent interface.This differs from standard architectures, where backbone visual representations directly condition the action expert.
  • III. METHOD: The policy receives visual observations, language, and robot state, and predicts an action chunk over horizon H.The setup combines a pretrained backbone with an embodiment-specific action expert.
  • III. METHOD: Each demonstrated action chunk uses its terminal robot state as a spatial goal for training.The goal includes end-effector position, axis-angle orientation, and gripper joint positions, and is not required at inference.

B. Stage 1: Spatial-Goal-Conditioned Action Pretraining

Stage 1 trains the action expert from scratch to generate action chunks from language, robot state, and terminal end-effector pose, establishing a spatial-goal-conditioned prior without images. The learned expert then initializes Stage 2, where visual information is introduced through the later interface.

  • B. Stage 1: Spatial-Goal-Conditioned Action Pretraining: Stage 1 learns a spatial-goal-conditioned action prior by generating demonstrated action chunks from language, robot state, and terminal pose without images.This separates goal-directed action learning from visual cues during pretraining.
  • B. Stage 1: Spatial-Goal-Conditioned Action Pretraining: A trainable MLP maps the terminal pose to goal tokens, which are concatenated with frozen backbone language and state representations at each coupling layer.The resulting representations condition the corresponding action-expert layers through each architecture’s native mechanism.
  • B. Stage 1: Spatial-Goal-Conditioned Action Pretraining: The action expert retains each framework’s native action-generation objective, including flow matching with sampled time and noise variables.The expert predicts the target velocity for the noisy action chunk.
  • B. Stage 1: Spatial-Goal-Conditioned Action Pretraining: Only the action expert and SE(3) encoder are updated in Stage 1; the frozen backbone remains fixed and the learned expert initializes Stage 2.This preserves the pretrained backbone while transferring the action prior to visual interface learning.

C. Stage 2: Vision–Action Interface Learning

Stage 2 introduces an exclusive, pose-supervised latent interface that aggregates backbone information for action generation while reconstructing the Stage 1 terminal goal state.

  • Stage 2 introduces visual conditioning exclusively through a latent interface initialized from the Stage 1 action expert.
  • The interface updates learnable latent tokens using self-attention, semantic cross-attention, and visual cross-attention from backbone representations.The latent tokens provide queries, while backbone representations provide keys and values.
  • Updated latent tokens condition corresponding action-expert layers through each architecture’s native conditioning mechanism.The action loss replaces the original conditioning representations with the latent tokens.
  • An MLP decoder reconstructs the Stage 1 terminal goal state from final latent tokens, encouraging retention of goal-relevant information.The reconstruction loss is averaged over valid targets, with λ_pose = 0.3, and Stage 2 jointly optimizes the backbone, action expert, interface, tokens, and decoder.

D. Framework Integration and Inference

LIT inserts a latent interface between pretrained backbones and action experts without changing the surrounding model architectures or action-generation procedures.

  • The latent interface applies across VLA and WAM architectures while retaining their backbones, action experts, action representations, prediction horizons, and native objectives.It provides layer-wise conditioning through each architecture’s native mechanism.
  • At inference, only visual observations, language, and robot state are required because the Stage 1 pose encoder and Stage 2 pose decoder are omitted.The latent interface remains active and the policy follows each framework’s native sampling and execution procedure.

IV. EXPERIMENTS

The experiments evaluate LIT across heterogeneous VLA and WAM architectures on LIBERO and LIBERO-Plus, using task success under in-distribution and perturbed conditions.

  • The experimental suite examines in-distribution preservation and zero-shot generalization under changes in viewpoints, sensing, lighting, backgrounds, states, layouts, and language instructions.All models train only on original LIBERO demonstrations and are evaluated on LIBERO-Plus without adaptation.
  • The study tests LIT across π0.5, MolmoAct2, FAST-WAM, and ImageWAM, covering two VLA and two WAM designs.The architectures differ in how visual or world-model representations condition their action experts.
  • Evaluation uses task success on 40 LIBERO tasks with 50 rollouts per task and all 10,030 LIBERO-Plus perturbed instances.LIBERO-Plus includes seven task-preserving perturbation dimensions, and overall success is their unweighted mean.
  • LIT routes visual conditioning through the pose-supervised interface while retaining each architecture’s native backbone, action expert, action representation, and objective.This comparison is made against architecture-matched baselines.

B. Performance Preservation Across Tasks

LIT preserves or improves in-distribution LIBERO performance and consistently improves zero-shot LIBERO-Plus success across VLA and WAM architectures.

  • B. Performance Preservation Across Tasks: LIT preserves or improves average LIBERO success for all four architectures, rising from 87.75% to 91.80% for π0.5 and improving the other three averages as well.MolmoAct2 rises from 93.50% to 94.10%, FAST-WAM from 97.60% to 98.10%, and ImageWAM from 98.10% to 98.40%.
  • C. Generalization Across VLA and WAM Architectures: LIT consistently improves overall LIBERO-Plus success across every evaluated VLA and WAM architecture.Table II reports overall success as the mean across seven perturbations.
  • C. Generalization Across VLA and WAM Architectures: The real-robot evaluation compares three manipulation tasks across in-distribution, lighting-OOD, camera-OOD, and distractor-OOD conditions.Camera OOD uses only the top-camera view.
  • C. Generalization Across VLA and WAM Architectures: MolmoAct2 improves from 63.62% to 71.92% overall, while π0.5 improves from 68.97% to 79.67% overall on LIBERO-Plus.The largest cited gains include +21.74 points for MolmoAct2 under Sensor Noise and +22.01 points for π0.5 under Camera Viewpoints.
  • C. Generalization Across VLA and WAM Architectures: FAST-WAM improves from 51.44% to 60.63% and ImageWAM from 83.02% to 86.89% overall on LIBERO-Plus.Across the evaluated architectures, LIT improves 26 of 28 architecture–perturbation comparisons, with no decrease exceeding 2.11 percentage points.

D. Real-world Evaluations

Real-robot evaluations show that LIT improves task success under both in-distribution and unseen lighting, camera, and distractor conditions across three manipulation tasks.

  • 16.7 points higher success under Camera OOD, rising from 30.0% to 46.7% across the three tasks.Aggregated success also rises from 53.3% to 70.0% under Lighting OOD and from 50.0% to 63.3% under Distractors OOD.
  • LIT improves aggregated ID success from 74.7% to 88.0% across Keep LEGOs, Wipe trash, and Transfer egg.Each policy is evaluated under ID and three OOD visual conditions, including changed lighting, camera configuration, and added distractors.
  • Transfer egg shows especially large gains, with ID success increasing from 52.0% to 92.0% and Lighting OOD success from 30.0% to 90.0%.Under Distractors OOD, Transfer egg success also increases from 20.0% to 90.0%.
  • On Wipe trash, LIT matches the baseline under ID conditions while improving Camera OOD success from 10.0% to 80.0%.
  • Across tasks, improvement magnitude varies by perturbation, but LIT achieves higher success under ID and every tested OOD condition.The physical results complement simulation evidence for robustness to visual distribution shifts.

1) Behavioral Analysis:

Behavioral and ablation analyses indicate that LIT stabilizes task-relevant visual responses, preserves goal sensitivity, and benefits from combining action pretraining, pose supervision, and restricted visual conditioning.

  • Behavioral Analysis:: Baseline attention shifts across visual perturbations, whereas LIT remains concentrated on task-relevant robot–object regions.The comparison composes action-to-latent and latent-to-visual attention for LIT against direct action-to-visual attention for the baseline.
  • Behavioral Analysis:: LIT trajectories stay close under distractors or blurred images but redirect when the instructed goal changes while the scene stays fixed.These counterfactuals test stability to task-preserving visual changes and responsiveness to goal changes.
  • Component Ablations:: Removing Stage 1 lowers OOD success from 71.92% to 68.23%, while removing pose supervision lowers it to 68.86%.The corresponding decreases are 3.69 and 3.06 percentage points, respectively, on MolmoAct2.
  • Component Ablations:: Removing exclusive routing through the pose-supervised latent interface produces a 4.18-point OOD decrease, supporting the restricted visual pathway.
  • Component Ablations:: Latent-token aggregation alone reaches 65.70% OOD success, 2.08 points above baseline but 6.22 points below LIT.This supports combining aggregation with action pretraining and spatial-goal supervision.
  • Component Ablations:: The tested staged-training alternative raises OOD success from 63.62% to 65.46% but remains 6.46 points below LIT.Pose supervision alone similarly raises success to 65.45%, remaining 6.47 points below LIT.

V. LIMITATIONS AND CONCLUSION

The paper concludes that LIT combines image-free spatial-goal-conditioned action pretraining with a pose-supervised latent interface to improve robustness while preserving LIBERO performance.

  • Larger-scale real-world evaluation remains open, motivating future tests on more diverse datasets, tasks, and environments.
  • LIT preserves in-distribution LIBERO performance while improving zero-shot LIBERO-Plus generalization across four VLA and WAM architectures.The strategy is presented as framework-agnostic and uses the latent interface as the action expert’s only visual conditioning pathway.
Loading 2609.12641v1…