PaperlayerView exact revision on arXiv

Source-linked AI summary

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation

Sanghwan Jang, Minjin Jeon, Minsoo Kim, Seongjin Choi, Dongha Kim, Hwanjo Yu

arXiv:2608.25585v1cs.RO

TL;DR

Existing in-context imitation learning can struggle to translate expert demonstrations into precise actions for novel tasks. RA-VLA combines behavior-aligned retrieval with context-grounded action generation. Across simulation and real-world evaluations, it outperforms state-of-the-art baselines, with a 17.60% absolute success-rate gain on LIBERO.

  • Problem

    Existing ICIL frameworks leave unresolved how to translate in-context expert guidance into precise actions for novel tasks.

    Source: §1 · p. 2

  • Method

    RA-VLA combines behavior-aligned retrieval with a context-grounded execution pipeline for in-context adaptation.

    Source: §1 · p. 2

  • Results

    Across simulation and real-world evaluations, RA-VLA outperforms state-of-the-art baselines, with a 17.60% absolute success-rate improvement on the LIBERO benchmark.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RA-VLA adapts to unseen tasks through in-context guidance without weight updates or the latency penalty reported for existing ICIL approaches.

    Source: §1 · p. 2, §6 · p. 8

    Its reliance on a demonstration buffer creates vulnerability to adversarial injection of deceptive or harmful expert demonstrations.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models provide a versatile foundation for general robotic manipulation, yet they exhibit significant brittleness when confronted with novel task distributions. While In-Context Imitation Learning (ICIL) offers a training-free alternative, existing frameworks suffer from an adaptation bottleneck that hinders the effective translation of expert context to executable actions. This failure originates from superficial retrieval mechanisms and an inherent behavioral inertia that anchors the policy to its pre-trained priors. To address these limitations, we present RA-VLA, a retrieval-augmented VLA framework that integrates behavior-aligned context retrieval with a grounded execution pipeline. By enforcing faithful adherence to functional cues within a scalable architecture, RA-VLA facilitates seamless task adaptation while preserving inference efficiency. Our empirical evaluations across the LIBERO benchmark and a real-world UR5e environment demonstrate that RA-VLA achieves superior success rates and computational efficiency, establishing a robust framework for training-free robotic adaptation.

1. Introduction

VLA models struggle to adapt to unseen manipulation tasks, while ICIL uses expert demonstrations to guide test-time adaptation without weight updates. RA-VLA addresses ICIL’s adaptation bottleneck with behavior-aligned retrieval and context-grounded execution, and reports gains on LIBERO and UR5e.

Introduction

  • VLAs can revert to familiar training behaviors or produce erratic motions on novel instructions, making retraining an expensive, non-scalable route to continued functionality.The paper frames this brittleness as a challenge to deployment in dynamic environments.Source: §1 · p. 1
  • ICIL (Yoo et al., 2025; Zhu et al., 2024; Sridhar et al., 2025) uses a few expert demonstrations as in-context cues to guide novel tasks without weight updates.It aims to reconcile pre-trained knowledge with unseen task demands through demonstrations placed in the input prompt.Source: §1 · p. 1
  • Existing ICIL retrieves by superficial visual similarity, underuses retrieved cues due to behavioral inertia, and incurs latency that scales poorly with more segments.In the stove-task failure, guidance is behaviorally inconsistent and the policy remains anchored to its bowl-picking prior.Source: Figure 1 · p. 1, §1 · p. 2
  • RA-VLA trains retrieval with behavioral alignment loss, grounds actions using contextual adherence loss, and pre-encodes and caches segments independently to limit inference overhead.The adherence loss imposes an action regression margin between relevant and irrelevant expert segments to counter pre-trained behavioral inertia.Source: §1 · p. 2
  • RA-VLA yields absolute success-rate improvements of 17.60% on LIBERO and 20.83% in the UR5e environment, with inference latency nearly constant as retrieved segments increase.The evaluation tests tasks excluded from training and requires no weight updates.Source: §1 · p. 2

2. Related Work

VLA models have advanced from discrete action tokens to generative action heads, yet remain fragile on novel tasks. ICIL enables test-time adaptation without weight updates, but existing approaches struggle to turn expert demonstrations into actionable guidance.

Vision-Language-Action (VLA) Models

  • VLA research progressed from discretized action tokens to generative diffusion or flow-matching heads, but models remain fragile on novel tasks and minor perturbations.This fragility motivates dynamic in-context adaptation to address uncertainty in unseen tasks.Source: §2 · p. 2

In-Context Imitation Learning (ICIL)

  • ICIL offers test-time adaptation without weight updates, but earlier methods rely on non-RGB inputs or lack compatibility with modern VLA models.Later generalist frameworks add in-context adaptability but still fail to use expert demonstrations effectively as actionable guidance.Source: §2 · p. 2

3. Preliminaries

The preliminaries define language-conditioned manipulation as predicting action chunks from visual, linguistic, and proprioceptive inputs, then describe a flow-matching VLA architecture for grounded generative control.

3.1. Robotic Manipulation

  • At each timestep, the policy maps visual observations, a language instruction, and proprioceptive state to an action chunk, and behavioral cloning trains it to match expert chunks.The action chunk has horizon h and action dimension c.Source: §3.1 · p. 3

3.2. Flow-Matching-Based VLA Architecture

  • A flow-matching VLA pairs a VLM encoding of vision and language with a Diffusion Transformer action head conditioned on those features and proprioception, grounding generative control in semantic features (Peebles & Xie, 2023).The action head refines a noise distribution into the target action chunk.Source: §3.2 · p. 3

4. Method

RA-VLA adapts a VLA policy to novel manipulation tasks by retrieving expert demonstrations and grounding action generation in their context. Its design combines behavior-aligned retrieval with contextual adherence training while targeting negligible computational overhead.

4. Method

  • The method evaluates training-free adaptability with strict separation between training and evaluation tasks; here, training means endowing the policy with in-context adaptability.RA-VLA targets novel manipulation tasks with negligible computational overhead.Source: §4 · p. 3

4.1. Overall Framework

  • RA-VLA uses a sparse buffer of 1–5 expert demonstrations per task, sliced into functional segments whose features are independently encoded and cached for retrieval and action guidance.Each segment is stored as a retrieval key with cached features, state, and actions as its value.Source: §4.1 · p. 3, Figure 2 · p. 4
  • A lightweight two-layer Transformer retrieves top-K segments by cosine similarity, and pairwise grounding distributes them across cross-attention layers, assigning the most relevant segments to higher layers.This design uses retrieved context for fine-grained action refinement while maintaining minimal computational overhead.Source: §4.1 · p. 3

4.2. Learning Action-Aware Retrieval

  • DTW-aligned action trajectories form behavioral pairs for contrastive training, teaching the retriever to match motion profiles from observation inputs and generalize behaviorally aligned retrieval to unseen tasks.Positive pairs follow DTW alignments; negatives include other unaligned segments and segments from different tasks.Source: §4.2 · p. 4

4.3. Learning Context-Grounded Action Generation

  • To counter behavioral inertia, a contextual adherence loss with a regression margin penalizes similar errors on relevant and irrelevant retrieved context, encouraging grounded predictions with zero inference overhead.The loss reuses multimodal features, adding only minor training cost.Source: §4.3 · pp. 4–5
  • Initializing the noisy action chunk with the retrieved segments’ empirical mean action chunk accelerates training convergence and stabilizes denoising on unseen tasks.This replaces sampling the initial action chunk from a standard normal distribution.Source: §4.3 · p. 5

5. Experiments

RA-VLA adapts to unseen manipulation tasks by retrieving behaviorally relevant expert context and grounding execution in that guidance. Experiments report stronger adaptation, greater contextual sensitivity, and near-constant inference time as more expert segments are used.

5.1. Experimental Setups

  • Across four LIBERO suites and four UR5e tasks, evaluation holds out tasks and supplies three or four expert demonstrations, respectively.LIBERO success is averaged over 50 trials; UR5e success is measured over 12 trials per task. All frameworks use GR00t N1.5, dual-view RGB, a 16-step action horizon, and four denoising steps; the default uses one retrieved segment.Source: §5.1 · p. 5

5.2. Novel Task Adaptation

  • RA-VLA raises LIBERO average success from 20.9% to 38.5% and UR5e success from 35.4% to 56.3% on held-out tasks.On the real-robot Stack Box task, RAEA picks up trash instead of the small box, while RICLR lifts the box but fails to transport it accurately; RA-VLA completes the task.Source: Table 1 · p. 6, §5.2 · p. 6, Table 2 · p. 6

5.3. Contextual Sensitivity Analysis

  • RA-VLA has higher Relative Contextual Sensitivity than ICIL baselines, and removing its adherence loss lowers both sensitivity and success rate.Sctx measures the relative change in action predictions when retrieved expert context is replaced by random context; higher sensitivity indicates tighter coupling to expert guidance.Source: §5.3 · pp. 6–7, Table 3 · p. 7

5.4. Retriever Analysis

  • On LIBERO-Goal, the action-aware retriever raises RA-VLA success from 10.2% to 53.2%, avoiding behaviorally irrelevant segments retrieved by off-the-shelf models.The authors attribute the improvement to representations encoding action dynamics through behavioral alignment loss.Source: §5.4 · p. 7, Figure 4 · p. 8

5.5. Inference Scalability Analysis

  • As retrieval size increases, RA-VLA maintains near-constant inference time while RICL latency rises steeply, because RA-VLA encodes each retrieved segment independently.Appending segments to a multimodal prompt expands the context window and increases LLM-backbone overhead with retrieval size.Source: Figure 5 · p. 8, §5.5 · p. 8

5.6. Analysis of Buffer and Retrieval Size

  • On LIBERO-Goal, success increases consistently with buffer size N, while increasing retrieved segments K yields only marginal gains.A larger buffer gives the retriever a richer search space; the authors note that scaling K may help in complex or ambiguous scenarios.Source: Table 4 · p. 8, §5.6 · p. 8

6. Conclusion

RA-VLA addresses ICIL’s failure to transfer collected expert knowledge to novel task execution through behavior-aligned retrieval and an efficient, grounded execution pipeline. Experiments in simulation and real-world environments show adaptation to unseen tasks without the latency penalty of existing ICIL approaches.

6. Conclusion

  • RA-VLA adapts to unseen tasks in simulation and real-world environments while bypassing the latency penalty of existing ICIL approaches.The conclusion reports extensive experiments across both environment types.Source: §6 · p. 8
  • The work identifies ICIL’s systematic inability to transfer collected expert knowledge into novel task execution as a primary bottleneck.Source: §6 · p. 8
  • RA-VLA combines behavior-aligned retrieval with an efficient, grounded execution pipeline to address that bottleneck.Source: §6 · p. 8

Impact Statement · A. Extended Related Work

RA-VLA targets the safety risks of robotic agents’ limited out-of-distribution generalization through training-free adaptation from corrective demonstrations, while its demonstration buffer creates an adversarial-data risk. The related work discusses why existing context-adherence approaches are ill-suited or insufficient for large-scale robotic manipulation.

Impact Statement

  • Limited generalization to out-of-distribution scenarios can make robotic agents unpredictable or unsafe in critical environments.The paper identifies this as a primary bottleneck to real-world deployment.Source: Impact Statement · p. 9
  • Training-free in-context adaptation lets human operators quickly adjust a robot’s policy through corrective demonstrations.RA-VLA provides this mechanism without requiring further training.Source: Impact Statement · p. 9
  • A demonstration buffer exposes RA-VLA to potential adversarial data injection, which could introduce deceptive or harmful guidance into execution.The passage describes malicious actors placing such demonstrations in the retrieval buffer.Source: Impact Statement · p. 9
  • Future deployments should verify demonstration-source integrity and quality before using buffer contents for real-world robotic control.The paper presents robust verification protocols as a mitigation for adversarial demonstrations.Source: Impact Statement · p. 9

A. Extended Related Work

  • Contrastive decoding (Shi et al., 2024; Zhao et al., 2024) assumes baseline context sensitivity and requires multiple forward passes, making it ill-suited for large-scale VLAs.It amplifies contextual adherence by leveraging prediction differences with and without context.Source: A. Extended Related Work · p. 12
  • Context-mandatory training scenarios (An et al., 2024) are difficult to establish in in-distribution robotic manipulation because observations usually contain the cues needed for action generation.The passage says it is practically unfeasible to force policy reliance on retrieved context for accurate action prediction.Source: A. Extended Related Work · p. 12
  • Fixed relative loss weighting with and without context (Zhang et al., 2025) struggles to sustain context-differentiation optimization throughout training.The paper describes this approach as the closest parallel to its work.Source: A. Extended Related Work · p. 12

B. Additional Details on RA-VLA · C. Real-World Experimental Setup · D. Additional Details on ICIL Baselines

RA-VLA’s additional details describe how demonstrations are sliced, aligned, retrieved, and grounded, while the real-world setup and ICIL baselines specify the platform and alternative ways to condition or combine actions.

B. Additional Details on RA-VLA

  • Demonstrations are sliced at intervals s into functional segments containing the observation, language, state, and action chunk.Each chunk spans actions from timestep t through t+h−1.Source: B. Additional Details on RA-VLA · p. 12
  • 0.18 ms retrieval latency on an NVIDIA A100 is 0.36% of total inference latency for N = 107 stored keys.A typical 128-step demonstration yields 15 segments; each uses a 1KB key, a 2MB precomputed multimodal feature tensor, and a 1KB action-chunk tensor. Retrieval uses query-key matrix multiplication and Top-K selection.Source: B. Additional Details on RA-VLA · p. 12
  • Pairwise grounding assigns retrieved segments across eight cross-attention layers; for K = 2, the pattern is [2, 2, 2, 2, 1, 1, 1, 1].The implementation supports K ≤ 8 and could dynamically adjust assignments according to flow-matching denoising steps.Source: B. Additional Details on RA-VLA · p. 12
  • RA-VLA aligns expert demonstration pairs with DTW on action sequences, treating warping-path matches as positives and unaligned pairs as negatives.The warping path minimizes cumulative L2-distance matching cost.Source: B. Additional Details on RA-VLA · p. 12
  • A difference-based contextual-adherence margin loss replaces ratio-based variants, which become unstable and provide a diminishing optimization signal as MSE approaches zero.The difference-based loss is reported to provide a robust, stable signal throughout training.Source: B. Additional Details on RA-VLA · p. 13

C. Real-World Experimental Setup

  • The real-world platform uses a UR5e arm with a Robotiq 2F-85 gripper and static third-person and wrist-mounted Intel RealSense D435 cameras.Demonstrations use GELLO teleoperation (Wu et al., 2024); data and commands run at 10 Hz via ur rtde, while the VLA model runs on a single NVIDIA A100 GPU.Source: C. Real-World Experimental Setup · p. 13

D. Additional Details on ICIL Baselines

E. Additional Implementation Details

The implementation specifies ICIL baseline weights and RA-VLA hyperparameters, along with the retrieval encoder’s architecture.

E. Additional Implementation Details

F. Limitations · G. Future Directions

The framework’s adaptability depends on the quality and diversity of its demonstration buffer, while out-of-distribution evaluation yields low, hyperparameter-sensitive success. Broader validation and extensions to other embodiments and guidance sources remain future directions.

F. Limitations

  • Buffer quality and diversity constrain in-context adaptation because stored expert demonstrations are the policy’s sole guidance for unseen tasks.The demonstrations notably affect performance.Source: F. Limitations · p. 14
  • Experiments use the same demonstration buffer to ensure fair comparisons across methods.Source: F. Limitations · p. 14
  • The absolute success rate remains low relative to in-distribution performance.Source: F. Limitations · p. 14
  • Out-of-distribution evaluation makes performance highly sensitive to hyperparameters.Source: F. Limitations · p. 14
  • Findings warrant validation across large-scale training, non-flow-matching architectures, and more complex manipulation tasks.Source: F. Limitations · p. 14
  • At larger training scales, an independent encoding strategy may constrain performance.Source: F. Limitations · p. 14

G. Future Directions

  • Extending the grounding mechanism to cross-embodiment adaptation could broaden the framework’s applicability.Source: G. Future Directions · p. 14
  • Incorporating diverse expert guidance, including unstructured human videos, is a future direction toward general-purpose robotic intelligence.Source: G. Future Directions · p. 14

H. Additional Experimental Results

Additional experiments compare policy rollouts across LIBERO and real-robot tasks, test the contextual adherence loss, and report inference scalability. They show RA-VLA’s precise execution of unseen instructions, while also revealing a task where vanilla VLA adheres to the instruction.

Related papers

Loading 2608.25585v1…