PaperlayerView exact revision on arXiv

Source-linked AI summary

OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel

arXiv:2503.03734v4cs.ROcs.CV

TL;DR

VLA models often require fine-tuning that can disrupt pre-trained vision-language alignment, limiting generalization to unseen robotic tasks. OTTER instead freezes the encoders and extracts instruction-aligned visual features for the policy. It significantly improves generalization across manipulation tasks, while remaining limited on some robot morphologies and insufficiently studied for long-horizon, complex-scene tasks.

  • Problem

    Existing VLA approaches can disrupt pre-trained vision-language features through fine-tuning, while independently encoded modalities leave the policy to connect vision and language for unseen-task control.

  • Method

    OTTER freezes pre-trained vision-language encoders and selectively extracts visual features semantically aligned with language instructions for policy prediction.

  • Results

    OTTER achieves significantly better generalization than existing VLA models across manipulation tasks, maintaining higher success rates on unseen objects and environments.

  • Takeaways & Limitations

    Performance scales with larger pre-trained vision-language encoders, increased policy capacity, and pre-training on larger robot datasets.

  • Takeaways & Limitations

    OTTER’s adaptability is constrained across robot morphologies that cannot be easily parameterized by SE(3) transforms, and long-horizon tasks and complex scenes remain insufficiently explored.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained visionlanguage models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zeroshot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https://ottervla.github.io/.

1. Introduction

Existing VLA policies often pass visual and language features independently, leaving the policy to connect them and potentially weakening generalization when pre-trained VLMs are fine-tuned. OTTER freezes the encoders and extracts task-relevant visual features aligned with the instruction, improving zero-shot performance on unseen tasks.

  • Independent visual and language tokens force the policy network to connect modalities for precise control, especially in unseen environments.
  • Fine-tuning pre-trained vision-language encoders can interfere with their semantic features and reduce performance on unseen objects or environments.
  • OTTER freezes pre-trained encoders and selectively extracts visual features semantically aligned with task descriptions before policy prediction.
  • OTTER separates task planning through relevant-feature selection from robot action planning while incorporating proprioceptive data.
  • OTTER significantly outperforms state-of-the-art VLA models on unseen manipulation tasks through zero-shot generalization to novel objects and environments.
  • Performance scales with larger pre-trained vision-language encoders, greater policy capacity, and pre-training on larger robot datasets.

2. Related Work

Related VLA approaches commonly pass separately encoded language, vision, and proprioception features into a transformer, while fine-tuning or adding layers to CLIP can weaken its reasoning and alignment. OTTER instead retrieves language-relevant visual features from pre-trained CLIP representations before policy prediction.

  • Fine-tuning or adding layers on top of CLIP under scarce data can produce weaker reasoning than using vanilla CLIP features.
  • Most VLA models separately encode language, vision, and proprioception before feeding them into a single transformer policy.
  • OTTER combines vision and language before the policy by extracting visual features aligned with the text instruction.
  • FiLM-based policies learn language-vision alignment from robotic data rather than using alignment already present in pre-trained models such as CLIP.
  • OTTER retrieves CLIP visual patch features corresponding to the language task using cosine similarity, then combines them with language and proprioceptive features.

3. Method

OTTER uses frozen CLIP representations to construct visual tokens aligned with each language token, compresses them with language and embodiment features, and feeds the result to a transformer policy. The model predicts robot actions from these compact task-relevant representations.

  • Text-Aware Visual Feature Extraction: OTTER uses text-aware visual feature extraction from a pre-trained VLM to construct representations for robot manipulation.
  • Text-Aware Visual Feature Extraction: The method uses CLIP attention features because they contain cleaner semantic information than CLIP output features for selecting task-relevant visual content.
  • Text-Aware Visual Feature Extraction: Similarity scores between normalized language and visual features select and combine patches aligned with each task instruction.
  • Model Architecture: Separate attention pooling compresses text-aware visual features and language features, while an FFN encodes proprioception into an embodiment representation.
  • Model Architecture: The compressed perception and embodiment features are concatenated into a policy token used for action prediction.
  • Model Architecture: OTTER uses a four-layer, eight-head transformer with hidden dimension 512 and predicts the next 12 actions from a 12-step context.
  • Model Architecture: The proprioceptive representation has 10 dimensions, including end-effector translation, a 6DoF rotation vector, and gripper state.
  • Model Architecture: Actions are parameterized as delta end-effector poses with an appended continuous gripper position.

4. Experiments

OTTER is evaluated on language-conditioned multi-task learning and zero-shot generalization using simulation and real-robot tasks, with comparisons against several VLA baselines. Physical pick-and-place experiments show similar performance on training and unseen tasks while significantly outperforming the baselines.

  • Evaluation Settings: The evaluation covers language-conditioned multi-task learning and zero-shot generalization to unseen tasks or environments.The latter provides language descriptions of unseen tasks, while the former requires selecting the correct task from an instruction.
  • Evaluation Settings: Simulation evaluation uses LIBERO task suites spanning diverse objects, scene layouts, and language instructions, with 50 demonstrations per task.The suites include LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-90.
  • Evaluation Settings: Real-robot evaluation covers pick-and-place, poking, pouring, and drawer opening or closing using Franka-robot datasets collected across multi-task scenes.The datasets include 724 demonstrations for pick-and-place and 1,185 demonstrations across the four primitives.
  • Evaluation Settings: The real-robot study evaluates 19 in-distribution and 15 unseen tasks across four primitives, using ten trials per task with varied target locations and distractors.Unseen tasks use novel target objects or combinations of target objects and instructions.
  • Baselines: OTTER is compared with Octo, OpenVLA, π0-Fast-Droid, and Direct Feature Passing OTTER, which passes independently pooled text and vision tokens.The DFP-OTTER variant isolates the importance of text-aware visual feature extraction.
  • Real-World Results: OTTER achieves similar success rates on in-distribution and unseen physical pick-and-place tasks while significantly outperforming the baselines.The physical comparison uses 100 trials on training tasks and 70 trials on unseen tasks.

5. Results

Across real-world and simulation experiments, OTTER improves zero-shot generalization by selecting text-aware visual features while keeping the pretrained vision-language encoder frozen. Its performance also scales with larger encoders, greater policy capacity, and larger robot-data pretraining.

  • Real-world experiments: OTTER achieves 60-93% success rates on unseen tasks across four manipulation primitives, while baselines particularly struggle on pouring.The evaluation uses 1,185 tele-operated trajectories and 150 trials of completely unseen tasks.
  • Simulation experiments: OTTER outperforms baselines by a large margin on unseen LIBERO tasks, while models perform similarly on training tasks.The unseen tasks vary objects and distractors to test generalization to novel scenarios.
  • Ablations: Removing embodiment features reduces performance by 28% on training tasks and 33% on unseen tasks.The results indicate that embodiment features contribute to physical grounding for decision-making.
  • Ablations: Removing language features lowers performance by around 10% on both training and unseen tasks.The language features provide complementary information for more nuanced task understanding.
  • Ablations: Removing CLIP vision causes performance to drop by more than 50% on physical and simulation unseen tasks.The comparison supports using pretrained VLM representations rather than training a vision encoder from scratch.
  • Ablations: Fine-tuning the pretrained vision encoder degrades generalization, whereas effective text-aware extraction and a frozen encoder support generalizable VLA performance.The training-versus-unseen discrepancy appears in OpenVLA and OTTER with fine-tuned CLIP.
  • Scaling: Scaling the CLIP encoder from ViT-B/32 to ViT-L/14 improves success by 27.5% on training and 39.3% on unseen pick-and-place tasks.OTTER also scales with increased policy capacity and pretraining on larger robotic datasets.
  • Scaling: Pretraining OTTER on human videos improves performance despite differences in camera views, wrist images, and proprioceptive coordinate frames.The human and robot datasets differ substantially in observation and state representations.

6. Limitations and Conclusions

OTTER achieves stronger generalization across manipulation tasks, but its adaptability remains constrained by robot morphology and underexplored task complexity. The study concludes that preserving pre-trained vision-language alignment is beneficial for capable, generalizable robot learning systems.

  • Scaling OTTER across robot morphologies is challenging when morphologies cannot be easily parameterized by SE(3) transforms, such as multi-finger hands.
  • This morphology limitation restricts adaptability to a wider range of robotic platforms and task types.
  • The study has not extensively explored scaling to long-horizon tasks and more complex scenes.
  • OTTER achieves significantly better generalization than existing VLA models across manipulation tasks, including unseen objects and environments.
  • The method scales through larger vision-language encoders, increased policy capacity, and pre-training on larger robot datasets.

A.2. Real-world Tasks

The real-world evaluation uses manipulation tasks with randomized scenes, explicit success scoring, and model-specific time limits. These procedures assess instruction following and task completion across varied object and environment configurations.

  • The real-world task list is provided in Table 7.
  • Each poking and pouring trial varies the target location and adds 2 or 3 random distractor objects, while drawer trials vary drawer location.
  • Pick-and-place trials score 0.5 for correct pickup and 1 for correct pickup plus placement, while failures score 0.
  • Models other than OpenVLA have 30 seconds per trial, whereas OpenVLA has 60 seconds because of its lower inference speed.

B.1. Model Architecture for OTTER and Baselines

OTTER and its baselines use shared model configurations with targeted architectural ablations, action chunking for OpenVLA, and cached CLIP features during inference. Table 8 specifies the OTTER architecture and larger real-world variant.

  • All baselines share OTTER’s hyperparameters, while ablations remove corresponding attention pooling layers or replace the CLIP vision encoder.
  • OpenVLA is modified to predict the next 16 actions, while Octo uses an official checkpoint comparable in model size.
  • Caching CLIP feature outputs enables inference with the ViT-L/14 OTTER model.
  • Table 8 reports OTTER architecture hyperparameters and the settings for a larger and wider real-world model.

B.2. Training Hyper-parameters

Training uses AdamW with cosine learning-rate decay and linear warm-up, with shared hyperparameters across real-world and simulation experiments.

  • OTTER and the compared models use AdamW, cosine learning-rate decay, and linear learning-rate warm-up.
  • The real-world and simulation experiments share the listed hyperparameters and use 4 NVIDIA A100 80GB GPUs.

C. Vision-Language Attention Visualization

The visualization compares fine-tuned CLIP, frozen CLIP output features, and frozen CLIP attention features, showing that Xattn better preserves text-aligned object localization. This supports OTTER’s use of frozen, text-aware features for downstream generalization.

  • Attention comparisons: Xattn preserves stronger text–visual alignment than fine-tuned CLIP or frozen CLIP output features in the visualization.The comparison evaluates cosine similarity between CLIP visual features and per-token text features across the three settings.
  • Attention comparisons: Frozen CLIP attention features highlight the object associated with the text query rather than irrelevant background regions.The attention maps show this behavior for both the illustrated examples and Open-X observations.
  • Attention comparisons: Fine-tuning and final output processing introduce noise that makes correct object localization harder.Residual connections and the final feed-forward network worsen alignment, while fine-tuning can overfit to foreground-background separation.
  • Downstream effect: Frozen CLIP with Xattn achieves higher task success than fine-tuned CLIP on both training and unseen tasks.Reported success rates are 68% versus 26% for training tasks and 62% versus 15% for unseen tasks.

D. More Ablations

The ablations compare alternatives to OTTER’s text-aware visual feature extraction. Both alternatives fail to generalize to unseen tasks, supporting the specific feature-selection design.

  • Ablation variants: DFP-OTTER (CLS) replaces text-aware extraction with CLIP’s CLS token.This variant is trained on DS-PnP.
  • Ablation variants: OTTER (xattn) replaces OTTER’s text-aware extraction with standard cross-attention pooling between text and vision tokens.The resulting fused vision-language features differ from those produced by OTTER’s extraction in Equation 4.
  • Ablation results: Both DFP-OTTER (CLS) and OTTER (xattn) fail to generalize to unseen tasks.The comparison uses physical results from 70 trials on unseen pick-up-and-place tasks.
Loading 2503.03734v4…