PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen, Duc-Duy Nguyen, Hung T. Ho, Linh Dang Le, Trong-Bao Ho, Thien-Loc Ha, Tan Q. Nguyen, Tuan Dam, Vu N. Duong, Daniel Sonntag, Ngan Le, Khoa D. Doan, Jan Peters, An Thai Le, Duy M. H. Nguyen, Vien Anh Ngo

arXiv:2610.37215v1cs.ROcs.AI

TL;DR

VLA policies can execute poorly under visual shifts even when task semantics remain unchanged, and existing test-time methods overlook kinematic priors in training demonstrations. R2A retrieves compatible geometric motion from those demonstrations and fuses it with frozen-policy actions; it improves robustness across simulated and real-world evaluations.

  • Problem

    VLA policies can lose track of safe execution under OOD visual shifts despite unchanged task semantics, while existing test-time approaches overlook motion priors in training demonstrations.

    Source: §1 · pp. 1–2

  • Method

    R2A organizes successful demonstrations into a spatiotemporal kinematic graph, uses a query-dependent GNN to retrieve compatible future motion, and fuses it with frozen VLA actions.

    Source: §1 · p. 2

  • Results

    R2A consistently improves three VLA architectures, with success-rate gains of up to +12% in LIBERO-Plus and up to 40% on a real-world Aloha task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion reports that kinematic retrieval can improve execution robustness under visual shifts, while its pose-action improvement conditions do not guarantee task success.

    Source: §6 · p. 10

    The retrieval index lacks explicit scene geometry, so changes such as newly introduced obstacles must be conveyed through the policy output.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, $\pi_{0.5}$, and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/

1 INTRODUCTION

VLA policies can understand novel instructions yet fail to execute safely when visual conditions shift, while existing robustness approaches overlook motion knowledge in training demonstrations. R2A retrieves geometric motion priors from that experience and improves robustness across simulation and real-world evaluations.

1 INTRODUCTION

  • VLA policies can fail under visual shifts despite unchanged task semantics, and finetuning on the long tail of environmental variations is costly and practically impossible.Existing test-time methods address visual noise or steer away from failure, but overlook successful motion patterns embedded in training demonstrations.Source: §1 · pp. 1–2
  • Flat retrieval based on visual embeddings misses trajectory sequence geometry and confounds locating current execution with selecting the next motion primitive.R2A instead aligns ongoing geometric context with prior demonstrations independently of visual noise.Source: §1 · p. 2
  • R2A organizes successful demonstrations into a motion-primitive graph, localizes ongoing execution with a query-dependent GNN, and fuses retrieved future priors with frozen VLA actions.It also uses diffusion-based rectification to project OOD initial robot states back into the training manifold.Source: §1 · p. 2
  • R2A improves success rates by up to +12% in LIBERO-Plus and +40% on a real-world Aloha task across evaluations of three VLA architectures.On RoboTwin2.0-Plus, π0.5 success rises from 56.8% to 66.2%.Source: §1 · p. 2

2 RELATED WORK

Prior test-time robustness methods emphasize visual cues or steer action generation, while retrieval methods reuse experience but often treat it as static matching. R2A instead retrieves future continuations by localizing ongoing execution in a graph of pure kinematic states.

2 RELATED WORK

  • Test-time approaches suppress visual regions or steer actions, but can require external visual modules and neglect behavioral priors in training data.Examples include BYOVLA, VLS, and SDN; ProbeAct uses 3D tracking for grasp-failure recovery.Source: §2 · p. 2
  • Earlier retrieval methods reuse affordance memories or trajectories, but predominantly match independent observations rather than structured temporal executions.R2A maps the current trajectory into successful rollouts, then retrieves a compatible future continuation.Source: §2 · p. 3
  • Unlike visual-flow retrieval, contrastive motion pretraining, or motion-derived auxiliary rewards, R2A uses a query-conditioned GNN to localize execution in a visual-independent kinematic graph.Figure 1 depicts graph construction, initial-state rectification, and retrieval of geometric priors during execution.Source: §2 · p. 3, Figure 1 · p. 3

3 METHOD

R2A frames recovery from visual OOD shifts as geometry-grounded retrieval, using successful training demonstrations to guide the VLA’s spatial predictions. Its pipeline rectifies initial states, retrieves compatible motion primitives through a query-dependent GNN, and produces a reference motion for action fusion.

3.1 PRELIMINARIES & PROBLEM FORMULATION

  • R2A formulates visual OOD recovery as geometry-grounded retrieval of training motion primitives to correct spatial predictions that diverge from successful trajectories.The VLA predicts action chunks from visual observations and a language instruction; the correction uses historical motion primitives from the training corpus.Source: §3.1 · pp. 3–4

3.2 R2A OVERVIEW

  • The overview pipeline indexes successful demonstrations into a graph, retrieves geometrically relevant primitives during execution, and harmonizes them with the policy’s prediction.A diffusion model rectifies OOD initial states, while a query-dependent GNN produces the reference motion used for adaptive fusion.Source: §3.2 · p. 4

3.3 MOTION PRIMITIVE GRAPH CONSTRUCTION

  • The offline graph represents demonstration segments as kinematic nodes and links them with temporal and spatial edges to capture sequential dynamics and cross-task geometric similarities.Node attributes include joint configuration, displacement, gripper state, temporal position, task label, and estimated progress.Source: §3.3 · p. 4

3.4 INITIAL STATE RECTIFICATION VIA DIFFUSION

  • R2A rectifies OOD initial robot states with a conditional diffusion model before execution, grounding subsequent kinematic queries within the motion primitive graph.At inference, the model samples an in-distribution starting configuration conditioned on the task instruction, then computes a corrective trajectory from the OOD state.Source: §3.4 · p. 5

3.5 QUERY-DEPENDENT GNN RETRIEVAL

  • R2A uses a query-dependent GNN to retrieve relevant historical experiences and fuse their reference motion with the VLA prediction, correcting trajectory deviations while preserving local reactivity.The query combines the VLA’s current pose prediction with recent robot-state and executed-action history; retrieval marginalizes task and progress context before selecting top-K nodes.Source: §3.5 · p. 5

4 ACTION HARMONIZATION

R2A harmonizes retrieved geometric reference motion with the VLA’s prediction using a learned dimension-wise gate. The gate selectively corrects spatial poses while leaving gripper commands unchanged, and its MLP is trained with the pretrained VLA and GNN frozen.

4 ACTION HARMONIZATION

  • R2A applies a learned dimension-wise gate as a residual correction to spatial poses, leaving discrete gripper commands unaltered and restoring motion primitives when spatial actions deviate from the geometric prior.The gate is conditioned on the discrepancy between predicted and reference motions and is trained to rely more on the retrieved prior when simulated noise compromises the VLA prediction.Source: §4 · pp. 6–7
  • The harmonization MLP is trained against ground-truth demonstration actions under simulated query noise, while the pretrained VLA and query-dependent GNN remain frozen.R2A trains sequentially: first the base VLA, then the GNN, and finally the harmonization MLP.Source: §4 · p. 7

5 EXPERIMENTS

The experiments evaluate R2A across VLA policies, perturbation categories, and simulation and real-world settings. They also examine comparisons with OOD methods, component contributions, and inference efficiency.

Q1. Does R2A improve the robustness of existing VLA policies?

  • Average success rises from 77.1% to 89.0% for GR00T N1.7, 78.1% to 89.5% for StableVLA, and 85.9% to 90.4% for π0.5 on LIBERO-Plus.R2A-enhanced policies also outperform recent strong VLA policies.Source: Table 1 · p. 7, §5 · p. 7

Q2. How does R2A compare with existing OOD methods?

  • R2A outperforms OOD baselines on average, with gains of 11.2% over SDN for GR00T N1.7 and 2.5% for π0.5 on LIBERO-10.SDN is less effective under camera-viewpoint and robot-initial-state shifts, while R2A retrieves context-relevant positive action guidance.Source: Table 3 · p. 8, §5 · pp. 8–9

Q1. Does R2A improve the robustness of existing VLA policies?

  • R2A improves π0.5 in six of seven LIBERO-Plus perturbation categories and StableVLA in all seven; on RoboTwin2.0-Plus, π0.5 average success rises from 56.8% to 66.2%.Robot Initial States yield the largest gains: 49.1% for GR00T N1.7, 48.0% for StableVLA, and 21.9% for π0.5; on RoboTwin2.0-Plus, success rises from 34.2% to 75.9% in that category.Source: §5 · p. 8, Table 2 · p. 8

Q3. Does R2A improve robustness under real-world perturbations?

  • Under real-world robot initial-state perturbations, R2A largely preserves in-distribution Banana2Pot performance and raises Cube2Drawer success from 10% to 45%, exceeding SDN by 15%; Table 4 uses GROOT N1.7.Source: Table 4 · p. 9, §5 · p. 9

Ablation Study.

  • Ablations show that replacing GNN retrieval reduces success from 77.1% to 63.8%, replaying retrieved actions yields 1.2%, and initial-state rectification raises success from 49.9% to 87.5%.The rectification comparison is under Robot Initial States perturbations.Source: §5 · p. 9, Table 5 · p. 9

Inference Efficiency

  • R2A reaches 159ms average per-step latency versus 138ms for the base policy, while VLS exceeds 800ms; R2A has the highest reported success rate.The authors describe R2A's latency overhead as modest relative to the base policy.Source: Figure 4 · p. 9, §5 · p. 9

6 CONCLUSION AND DISCUSSION

R2A uses a kinematic graph to address visual OOD shifts through pose rectification, retrieval of aligned motion primitives, and adaptive action harmonization. The paper identifies limits in scene geometry, error recovery, and the distribution used to train harmonization.

6 CONCLUSION AND DISCUSSION

  • R2A combines initial-pose rectification, geometrically aligned motion retrieval, and adaptive harmonization; its analysis gives sufficient conditions for pose-action improvement, not task success.The analysis separates localization stability, reference compatibility, and correction strength.Source: §6 · p. 10
  • The retrieval index lacks explicit scene geometry, error recovery for failures absent from the demonstration graph remains unclear, and real-world shifts outside simulated training may defeat harmonization.The last limitation concerns visual shifts that induce VLA prediction errors outside the training distribution.Source: §6 · p. 10
  • Future work could extend evaluation across architectures, add memory for non-Markovian transitions, ground retrieval with multi-view inputs, and align geometric priors with human reference motions.The passage presents these as directions for investigation, not established capabilities.Source: §6 · p. 10

AI USE STATEMENT

The authors used LLMs to improve manuscript grammar, clarity, and readability, and retain responsibility for the paper’s content.

AI USE STATEMENT

  • LLMs were used only for manuscript grammar, clarity, and readability; the authors say they developed and reviewed the research content.The authors state they take full responsibility for the paper’s content.Source: AI USE STATEMENT · p. 10

A QUANTITATIVE RESULTS

Paired evaluation of GR00T N1.7 across ten LIBERO-Plus Long tasks finds gains across all seven perturbation axes, with faster solving and resilience as perturbations strengthen. Gains vary by axis and task, and a few task–axis combinations regress.

Paired outcomes

  • +56.0 points under robot initialization, +26.5 under sensor noise, and +21.0 under camera perturbations; R2A fixed 229, 133, and 101 episodes on those axes.Across all seven axes, gains were driven by more fixed than broken episodes.Source: A QUANTITATIVE RESULTS · p. 15, Figure 5 · p. 15

Per-task breakdown

  • Only 5 of 70 task–axis combinations regress; under robot initialization, nine of ten tasks improve by at least 45 points.The largest regressions are soup + cheese →basket under language (−27) and layout (−15) perturbations.Source: A QUANTITATIVE RESULTS · p. 15

Time to success

  • 59.7% to 80.9%: the final fraction of solved episodes rises with R2A, while median solve step falls by 26 steps across axes.Median solve-step reductions range from 18 steps for layout to 34 for robot initialization.Source: A QUANTITATIVE RESULTS · p. 15, Figure 6 · p. 16

Sensitivity to perturbation strength

Camera and sensor perturbations

  • +12 to +31 points: R2A improves across tested camera perturbations, from zoom to orbit with a 15° tilt.For sensor noise, all tested conditions improve; gains include +43 points under zoom blur and +37 under glass blur.Source: A QUANTITATIVE RESULTS · p. 16

B QUALITATIVE RESULTS

The appendix illustrates R2A retrieval during LIBERO-Plus rollouts and describes the setup for real-world experiments using a fine-tuned GR00T N1.7 baseline.

B.1 LIBERO-PLUS

  • LIBERO-Plus examples show two perturbation categories, with rollout rows distinguishing methods and columns marking task stages; a third row details an R2A retrieval and action step.The query uses the last 16 motion steps, including joint angles, velocities, and gripper state, plus the task instruction; the displayed initial-state query follows pose correction.Source: §B.1 · p. 17

B.2 REAL-WORLD SETUP

  • The real-world setup uses Mobile ALOHA with two 6-DoF arms and two 640 × 480 RGB cameras; the baseline is GR00T N1.7 fine-tuned on 150 demonstrations per task.The same fine-tuned policy is used with R2A; the tasks are Banana2Pot and Cube2Drawer, with objects initialized in a 40 × 60 cm workspace.Source: §B.2 · p. 17
  • For Robot Initial States perturbations, the arm starts outside demonstrated initial-pose ranges while objects remain in place; task-specific demonstrations train the base policy and build the graph.Source: §B.2 · p. 17

C.1 IMPLEMENTATION DETAILS

R2A is attached to a frozen VLA policy and is built from unit-specific demonstrations and replayed policy actions, without using evaluation episodes. Its graph, auxiliary networks, training, and benchmark-specific execution settings are configured without finetuning the base policy.

C.1 IMPLEMENTATION DETAILS

  • For each evaluation unit, R2A builds its motion graph and auxiliary networks from that unit’s demonstrations and frozen-policy replayed actions, and no module sees evaluation episodes.R2A is attached to a frozen VLA policy.Source: §C.1 · p. 17
  • Every policy is used as released without finetuning; LIBERO-Plus action chunks are 8 steps for GR00T N1.7 and StableVLA and 5 for π0.5.On RoboTwin2.0-Plus, π0.5 executes the full 50-step joint-target chunk before the next query.Source: §C.1 · p. 17, Table 6 · p. 19
  • The graph coarsens demonstrations to one node per 5 frames, connects nodes 1, 2, 4, 8, and 16 steps ahead, and retrieves 8 segments from a shortlist of 24.Each segment spans 16 frames; every evaluation unit is built from 500 demonstrations per task, yielding 13k–28k LIBERO-Plus nodes and 8k–10k nodes per arm on RoboTwin2.0-Plus.Source: §C.1 · pp. 17, 19
  • Figure 8 shows R2A completing two Level-5 LIBERO-Long rollouts where the policy alone fails to return for cream cheese or approaches a microwave with an empty gripper.In the camera-viewpoint case, the retrieved segment executes a 1.2 cm release-and-rise motion versus 0.2 cm from the policy.Source: Figure 8 · p. 18
  • Figure 9 presents recorded Banana2Pot and Cube2Drawer episodes, with three episode samples per task shown from initial to final scene.Frames use the third-person camera and are cropped to reduce background.Source: Figure 9 · p. 19
  • R2A adds 0.37M trainable parameters per evaluation unit; graph and start-state alignment networks are shared across policies, while locator, retriever, and harmonization modules are trained per policy.Training uses cached demonstrations and frozen-policy replayed actions and requires no robot rollouts.Source: §C.1 · p. 19, Table 7 · p. 20
  • RoboTwin2.0-Plus corrects each arm against its own graph, retrieves paired segments from the same demonstration, and uses benchmark-specific pause and gripper-feedback rules.After a gripper closes, the policy is queried every 10 action steps for the next 80 steps; otherwise, the execution horizon is 50 steps.Source: §C.1 · p. 20

C.2 EVALUATION PROTOCOL

The evaluation protocol defines distinct units and trial coverage for LIBERO-Plus and RoboTwin2.0-Plus. LIBERO-Plus comparisons with OOD methods and ablations use LIBERO-Long.

C.2 EVALUATION PROTOCOL

  • On LIBERO-Plus, one graph is built per suite from all its task demonstrations, allowing localization and retrieval across tasks; each released perturbed episode is run once in each of seven categories.The number of episodes varies by perturbation category, and OOD-method comparisons and ablations use LIBERO-Long.Source: §C.2 · p. 20
  • On RoboTwin2.0-Plus, the evaluation unit is a group of ten bimanual tasks, with each task evaluated for 10 trials per perturbation category.The groups are the same five alphabetical groups used by the RoboTwin 2.0 leaderboard.Source: §C.2 · p. 20

Related papers

Loading 2610.37215v1…