PaperlayerView exact revision on arXiv

Source-linked AI summary

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation

Sanghwan Jang, Minjin Jeon, Minsoo Kim, Seongjin Choi, Dongha Kim, Hwanjo Yu

arXiv:2608.25585v1cs.RO

TL;DR

VLAs are brittle on unseen manipulation tasks, and existing ICIL methods struggle to convert expert demonstrations into precise actions. RA-VLA combines behavior-aligned retrieval with context-grounded execution and scalable segment caching. It reports 17.60% absolute success-rate improvement on LIBERO and 20.83% in UR5e over existing state-of-the-art baselines, with nearly constant inference latency as retrieved segments increase.

  • Problem

    Existing VLAs are brittle on unseen manipulation tasks, while ICIL frameworks have an adaptation bottleneck in translating in-context guidance into precise actions.

  • Method

    RA-VLA uses behavior-aligned action-aware retrieval, context-grounded action generation, and independently pre-encoded cached segments for scalable in-context adaptation.

  • Results

    17.60% absolute success-rate improvement on LIBERO and 20.83% in UR5e are reported over existing state-of-the-art baselines, with nearly constant inference latency as retrieved segments increase.

  • Takeaways & Limitations

    RA-VLA adapts to unseen tasks through in-context guidance without weight updates while bypassing the latency penalty of existing ICIL approaches.

  • Takeaways & Limitations

    RA-VLA's demonstration buffer can be vulnerable to adversarial data injection, motivating verification of demonstration integrity and quality before real-world control.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models provide a versatile foundation for general robotic manipulation, yet they exhibit significant brittleness when confronted with novel task distributions. While In-Context Imitation Learning (ICIL) offers a training-free alternative, existing frameworks suffer from an adaptation bottleneck that hinders the effective translation of expert context to executable actions. This failure originates from superficial retrieval mechanisms and an inherent behavioral inertia that anchors the policy to its pre-trained priors. To address these limitations, we present RA-VLA, a retrieval-augmented VLA framework that integrates behavior-aligned context retrieval with a grounded execution pipeline. By enforcing faithful adherence to functional cues within a scalable architecture, RA-VLA facilitates seamless task adaptation while preserving inference efficiency. Our empirical evaluations across the LIBERO benchmark and a real-world UR5e environment demonstrate that RA-VLA achieves superior success rates and computational efficiency, establishing a robust framework for training-free robotic adaptation.

1. Introduction

VLAs are brittle on unseen manipulation tasks, while ICIL avoids weight updates but struggles to convert retrieved demonstrations into precise actions. RA-VLA addresses these bottlenecks through behavior-aligned retrieval, context-grounded execution, and scalable segment encoding.

  • VLAs often revert to familiar training-distribution behaviors or produce erratic motions on novel instructions, making retraining necessary for dynamic deployment.The paper characterizes this retraining as expensive and non-scalable.
  • ICIL uses a few expert demonstrations as in-context cues to steer policies toward new objectives without explicit weight updates.Demonstrations are incorporated directly into the input prompt.
  • Existing ICIL methods retrieve contexts by superficial visual similarity, fail to leverage guidance effectively, and incur increasing computational overhead as contexts grow.The adaptation bottleneck includes behavioral inconsistency, behavioral inertia, and poor latency scaling.
  • RA-VLA combines action-aware retrieval with context-grounded execution and independently pre-encodes and caches segments to reduce expert-guidance overhead.Behavioral alignment loss trains retrieval around similar expert behavior, while contextual adherence loss encourages action generation to use retrieved context.
  • 17.60% absolute success-rate improvement on LIBERO and 20.83% in UR5e are reported for RA-VLA over existing state-of-the-art baselines.Inference latency remains nearly constant as the number of retrieved segments changes.

2. Related Work

Related work develops VLA models from multimodal priors and robotic datasets, while ICIL pursues test-time adaptation using expert demonstrations. Existing approaches remain limited by modality constraints or incompatibility with modern VLA models.

  • Vision-Language-Action (VLA) Models: VLA models combine Vision-Language Model priors with diverse robotic datasets for generalist robotic control.Earlier systems discretize actions, while newer approaches use diffusion or flow-matching action heads.
  • Vision-Language-Action (VLA) Models: Recent VLA studies report persistent fragility when these models are deployed on novel manipulation tasks.The supplied passage introduces this fragility as an unresolved issue in the related work.
  • In-Context Imitation Learning (ICIL): ICIL enables test-time adaptation without expensive weight updates by using expert demonstrations as few-shot guidance for robotic policies.Early one-shot imitation methods established this demonstration-guidance approach.
  • In-Context Imitation Learning (ICIL): Earlier ICIL-related methods are constrained by non-RGB modalities or lack compatibility with modern VLA models.The passage specifically cites depth and point-cloud inputs as examples of non-RGB modalities.

3. Preliminaries

The paper formulates language-conditioned manipulation as mapping visual, linguistic, and proprioceptive inputs to action chunks. Its flow-matching VLA uses multimodal features and a generative action head to produce temporally consistent control.

  • Robotic Manipulation: At each timestep, the policy receives visual observation V_t, language instruction L, and proprioceptive state s_t.Visual input may come from static or eye-in-hand RGB cameras.
  • Robotic Manipulation: Action chunking makes the policy predict A_t ∈ R^{h×c}, where h is the action horizon and c is the action dimension.The chunking paradigm is used to promote temporal consistency.
  • Robotic Manipulation: Behavioral cloning trains the policy by minimizing discrepancy between predicted and expert action chunks.
  • Flow-Matching-Based VLA Architecture: The VLA encodes visual observations and language with a VLM f_ϕ to produce multimodal features H_t.These features bridge semantic perception and low-level generative control.
  • Flow-Matching-Based VLA Architecture: A Diffusion Transformer action head refines noise into the target action chunk and regresses a vector field conditioned on H_t and s_t.The vector field guides the denoising process.
  • Flow-Matching-Based VLA Architecture: The decoupled architecture grounds generative control in semantic features to support precise motor control.

4. Method

RA-VLA adapts VLA policies to novel tasks by retrieving behaviorally aligned expert segments and grounding action generation in that context. Its design combines efficient segment processing, action-aware retrieval, and contextual adherence objectives.

  • Overall Framework: RA-VLA targets training-free adaptation by using sparse expert demonstrations while keeping training and evaluation tasks strictly separated.The framework stores demonstrations in a buffer and uses them for in-context guidance without weight updates during adaptation.
  • Context Buffer Construction: The framework slices long-horizon demonstrations into functional segments and encodes each independently to avoid expanding the input sequence length.Segments are formed with a sliding window and stride, then processed as separate encoding units.
  • Expert Segment Retrieval: A lightweight two-layer Transformer retrieves top-K expert segments by cosine similarity between query and segment embeddings.The query combines visual observations and language instructions, whose multimodal tokens are average-pooled into an embedding.
  • Grounded Action Generation: Pairwise grounding concatenates current observation features with one retrieved expert feature per cross-attention layer, distributing segments progressively across layers.Higher layers receive more relevant segments for fine-grained action refinement while preserving minimal computational overhead.
  • Learning Action-Aware Retrieval: Behavioral alignment loss trains retrieval to group segments with similar action patterns despite minor visual variations.Behaviorally similar pairs are obtained by aligning action sequences from same-task demonstrations with Dynamic Time Warping, then optimized contrastively.
  • Contextual Adherence: Contextual adherence loss compares relevant- and irrelevant-context MSE losses, penalizing policies that ignore retrieved guidance while adding zero inference overhead.The policy is fine-tuned with a weighted combination of relevant-context regression and adherence losses; retrieved mean action chunks also initialize denoising.

5. Experiments

RA-VLA is evaluated for training-free adaptation to entirely unseen tasks in simulation and on a real UR5e robot, using expert demonstrations and strict train–evaluation separation. Across adaptation, contextual grounding, retrieval quality, and scalability analyses, it improves task success while maintaining efficient inference.

  • Experimental Setups: Experiments use LIBERO and a real-world UR5e environment, with evaluation tasks strictly separated from training tasks.LIBERO results cover four task suites, while held-out task suites provide unseen-domain evaluation; the UR5e setup also evaluates novel tasks.
  • Novel Task Adaptation: RA-VLA achieves the highest LIBERO success rates across task suites, improving average success from 20.9% to 38.5%.The comparison uses three expert demonstrations per held-out task and reports mean success rates across task suites.
  • Novel Task Adaptation: 56.3% success on UR5e surpasses the 35.4% state-of-the-art baseline in real-world novel-task adaptation.The qualitative comparison attributes baseline failures to training-distribution behavior or imprecise control, whereas RA-VLA completes the unseen box-stacking instruction.
  • Contextual Sensitivity Analysis: RA-VLA exhibits higher Relative Contextual Sensitivity than existing ICIL baselines, and removing contextual adherence loss lowers both Sctx and success rate.Sctx measures action-prediction change when retrieved context is replaced with random context; higher values indicate tighter coupling to expert guidance.
  • Retriever Analysis: Replacing off-the-shelf retrieval with action-aware retrieval raises RA-VLA success from 10.2% to 53.2% on LIBERO-Goal.Behavioral alignment targets functional dynamics rather than superficial visual similarity, filtering behaviorally irrelevant distractors.
  • Inference Scalability Analysis: RA-VLA maintains near-constant inference time as retrieved-segment count K increases, unlike RICL’s steep latency growth.Its independent segment encoding decouples inference latency from retrieved-context scale, reducing the context-window overhead of direct prompt concatenation.

6. Conclusion

RA-VLA addresses ICIL’s bottleneck by combining behavior-aligned retrieval with an efficient, grounded execution pipeline. Experiments show adaptation to unseen tasks without weight updates and without the latency penalty of existing ICIL approaches.

  • RA-VLA integrates behavior-aligned retrieval into an efficient, grounded execution pipeline for transferring expert knowledge to novel task execution.

Impact Statement

RA-VLA targets unsafe or unpredictable behavior caused by limited generalization to out-of-distribution robotic scenarios. Its training-free adaptation uses corrective demonstrations, but the demonstration buffer is vulnerable to adversarial data injection requiring verification protocols.

  • Limited generalization to out-of-distribution scenarios can produce unpredictable or unsafe robotic behavior in critical environments.
  • Training-free in-context adaptation lets human operators quickly adjust a robot’s policy through corrective demonstrations.
  • Adversarial data injection into the demonstration buffer could introduce deceptive or harmful guidance into execution.The paper recommends verifying demonstration-source integrity and quality before real-world control.

B. Additional Details on RA-VLA

The additional details describe how demonstrations are segmented, behaviorally aligned, stored, and grounded for execution, alongside baseline conditioning and the real-world platform configuration.

  • Demonstration Slicing: Expert demonstrations are sliced into functional segments at strided timesteps, with each segment containing visual, language, state, and action-chunk information.
  • Retrieval Efficiency: Retrieval stores compact key vectors, pre-computed multimodal features, and action chunks in a GPU-managed buffer.A 128-step demonstration produces 15 segments with 1KB retrieval keys, 2MB feature tensors, and 1KB action-chunk tensors per segment.
  • Pairwise Grounding: Retrieved segments are assigned across eight cross-attention layers to distribute behavioral guidance gradually and evenly.For K = 2, the assignment pattern is [2, 2, 2, 2, 1, 1, 1, 1].
  • Behavioral Pairs: Dynamic time warping aligns action sequences to create positive segment pairs, while unaligned pairs become negatives for behavioral training.
  • Real-World Platform: The real-world system uses a UR5e arm with a Robotiq gripper and two RealSense cameras for global and wrist-mounted observations.The setup includes a static third-person camera and a wrist-mounted egocentric camera.
  • ICIL Baselines: ICIL baselines combine retrieved expert action chunks with generative policy outputs using distance-dependent weighting.RICL formulates the final action chunk as a weighted ensemble of direct imitation and generative prediction.

E. Additional Implementation Details

The implementation specifies RA-VLA’s retrieval and loss hyperparameters and identifies the VLA-adapted RAEA architecture used in the described baseline implementation.

  • RA-VLA uses temperature τ = 0.01, stride s = 4, margin m = 0.01, and loss weight λ = 0.1.
  • The retrieval encoder has hidden size 512, eight attention heads, intermediate size 1024, and two hidden layers.
  • The described RAEA architecture is a VLA-adapted implementation extending its original principles to high-capacity VLA models.

F. Limitations

The framework’s in-context adaptability depends on the expert demonstrations available for unseen tasks, while out-of-distribution evaluation remains low and sensitive to hyperparameters. Broader validation is still needed across scales, architectures, embodiments, and task complexity.

  • Expert demonstration quality and diversity directly constrain in-context adaptability because the buffer is the sole guidance source for unseen tasks.The experiments use the same buffer to ensure fair comparisons because stored demonstrations affect performance.
  • Absolute success rates remain low relative to in-distribution performance, with high sensitivity to hyperparameters under out-of-distribution evaluation.
  • The findings require validation across larger training scales, non-flow-matching architectures, and more complex manipulation tasks.At larger training scales, independent encoding may constrain performance.
  • Future extensions include cross-embodiment adaptation and more diverse expert guidance, including unstructured human videos.

H. Additional Experimental Results

Additional rollouts illustrate how RA-VLA behaves across unseen manipulation tasks and ablations. The comparisons emphasize task-specific execution quality, contextual adherence, and the integration of multiple expert segments during inference.

  • Additional Experimental Results: Additional policy rollouts cover diverse experimental configurations, contextual adherence loss ablations, and full inference scalability results.
  • LIBERO-Goal: RA-VLA precisely executes the unseen LIBERO-Goal instruction to place a wine bottle on a cabinet, whereas baselines revert to priors or lack control precision.
  • Real-World Tasks: RA-VLA precisely executes the unseen real-robot Throw Trash instruction, while baselines revert to pre-trained priors or lack control precision.
  • Close Drawer: The vanilla VLA fails to close the open drawer in one rollout, while another rollout shows it closing the drawer successfully.
  • Real-World Tasks: RA-VLA precisely executes the unseen real-robot Press Pedal instruction, unlike baselines that revert to priors or lack control precision.
  • Contextual Adherence: Without contextual adherence loss Ladhere, the policy either reverts to pre-trained priors or produces erratic actions instead of aligning with expert guidance.
  • Inference Scalability: RA-VLA enables seamless integration of multiple expert segments, with latency averaged over 1,000 inference runs using four denoising steps for one action chunk.
Loading 2608.25585v1…