PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation

Yibo Qiu, Haoliang Ye, Shu'ang Sun, Zan Huang, Ronald X Xu, Mingzhai Sun

arXiv:2608.27079v1cs.RO

TL;DR

Fine-grained biomedical manipulation requires learning task-relevant local visual grounding from limited real-robot interaction, while online VLA adaptation must remain computationally efficient. GRAFT uses training-time region supervision for view-specific anchors, cached frozen-prefix states, and single-step action generation; across four real-robot tasks, it improves final-stage success by 25 percentage points under the same 45-minute budget.

  • Problem

    Fine-grained manipulation depends on subtle view-dependent local cues, but demonstrations and scalar task rewards provide little spatial guidance for learning which regions matter during online adaptation.

    Source: §I · p. 1

  • Method

    GRAFT learns view-specific visual anchors with training-only identity-free multi-region supervision and combines frozen-prefix KV caching with single-step action-chunk prediction for efficient online adaptation.

    Source: §I · pp. 1–2

  • Results

    25 percentage points: GRAFT improves final-stage success across four real-robot tasks under the same 45-minute adaptation budget, while prefix caching speeds learner updates by up to 9.94×.

    Source: §V · p. 7

  • Takeaways & Limitations

    Learned anchors attend to task-relevant regions while region proposals remain unnecessary at deployment, supporting efficient grounded adaptation within a fixed wall-clock budget.

    Source: §V · p. 7

    The reactive policy does not explicitly track previous actions or task progress, which may limit longer-horizon tasks where the relevant visual target changes over time.

    Source: §V · p. 7

Abstract

from arXiv · show

Pretrained vision-language-action (VLA) policies provide strong priors for robot manipulation, yet adapting them online to fine-grained biomedical tasks remains challenging. Task success often hinges on subtle, view-dependent visual cues, while task-level rewards provide little guidance about which regions matter, making it difficult to learn task-relevant visual grounding from limited real-robot interaction. Online adaptation is further constrained by the computational cost of VLA inference and replay-based updates. We introduce GRAFT (Grounded Reinforcement Adaptation for Fast Task Learning), a framework for efficient online VLA adaptation through grounded perception. GRAFT uses region-level supervision to learn view-specific visual anchors that focus perception on task-relevant local cues without requiring region proposals at deployment. It further combines single-step action generation with cached visual-language prefix reuse to accelerate online learning. Across four biomedical manipulation tasks, GRAFT improves success rates by 25 percentage points under matched adaptation budgets, while reducing the computational overhead of online policy updates.

I. INTRODUCTION

GRAFT addresses the difficulty of learning fine-grained, task-relevant visual grounding during online VLA adaptation while keeping acting and replay updates efficient. It combines view-specific multi-region supervision, cached prefix reuse, and single-step action generation, then evaluates the approach on four biomedical manipulation tasks.

Motivation

  • Fine-grained laboratory manipulation depends on precise local contact geometry and small task-relevant regions across multiple camera views.Scalar rewards and demonstrations provide little explicit spatial guidance about which cues determine success or failure.Source: §I · p. 1

Grounded adaptation

  • GRAFT learns view-specific visual anchors using identity-free multi-region supervision during adaptation, without requiring masks or proposal generators at deployment.The supervision preserves distinct plausible regions rather than collapsing them into a single union target.Source: §I · pp. 1–2

Efficient online learning

  • Frozen visual-language prefix states are cached across replay updates, while the actor predicts complete action chunks in a single forward pass.The learner recomputes only trainable anchor-conditioned action pathways during replay optimization.Source: §I · pp. 1–2

Evaluation

  • GRAFT is evaluated under matched 45-minute adaptation budgets across four fine-grained biomedical manipulation tasks.The evaluation measures online success, grounding effects, learner-update efficiency, and learned anchor structure.Source: §I · pp. 1–2

II. RELATED WORK

Related work improves VLA spatial reasoning, grounding, reinforcement adaptation, and computational efficiency through complementary strategies. GRAFT combines these directions around online acquisition of task-relevant visual representations from multiple robot cameras, using training-time rather than deployment-time proposals.

Spatial reasoning

  • Prior VLA methods improve spatial reasoning with geometric representations, spatial features, visual traces, and adaptive viewpoint or resolution selection.Multi-view approaches combine observations through explicit geometric reasoning or cross-view aggregation.Source: B. Grounded and Multi-View Perception… · p. 2

Visual grounding

  • Explicit grounding methods provide masks, regions, or crops at execution time, while others use gaze, affordances, alignment, or auxiliary grounding objectives during training.GRAFT is most closely related to training-time grounding approaches but focuses on online adaptation from multiple robot cameras.Source: B. Grounded and Multi-View Perception… · p. 2

Reinforcement adaptation

  • Real-world reinforcement adaptation combines demonstrations or interventions with supervised and reinforcement learning, offline-to-online fine-tuning, chunk-level RL, and improved value estimation.GRAFT builds on this line without proposing a new reinforcement-learning objective.Source: C. Real-World Reinforcement Adaptation… · pp. 2–3

Efficiency

  • Efficiency methods accelerate VLA execution through parallel decoding, action chunking, or reuse of representations across observations and control steps.GRAFT instead reuses frozen visual-language prefix states on the learner side during replay updates.Source: C. Real-World Reinforcement Adaptation… · p. 3, Figure 2 · p. 3

III. METHOD

GRAFT combines a frozen visual-language prefix with view-specific anchors and training-only region supervision to produce action chunks from camera observations, language, and adapter state. Proposal inputs guide adaptation but are excluded from the deployed policy.

Problem setup and deployment boundary

  • At time t, the policy receives side- and wrist-view RGB observations, a language instruction, and adapter state, then predicts an H = 10 action chunk.Each low-level action contains translation, rotation, and gripper commands.Source: A. Problem setup and deployment boundary · p. 3
  • A frozen VLM maps the inputs to view tokens, pooled context, and prefix key–value states for the side and wrist views.These representations form the frozen visual-language prefix used by the downstream policy.Source: A. Problem setup and deployment boundary · p. 3
  • Training observations may include offline-generated patch-aligned masks, but neither masks nor proposal generators are policy inputs.Region proposals therefore provide auxiliary supervision during training without creating a deployment dependency.Source: A. Problem setup and deployment boundary · p. 3, §III · p. 3

B. Efficient replay adaptation

GRAFT makes replay adaptation more efficient by reusing frozen prefix representations and by generating action chunks in one consistency-policy pass. Separate view-specific anchors process their assigned camera views while sharing context and cross-attention projections.

Efficient replay adaptation

  • For replayed observations, GRAFT caches frozen prefix states, pooled summaries, and visual tokens, then recomputes trainable anchor, action, and critic pathways during learner updates.Live observations always receive a fresh prefix forward pass.Source: B. Efficient replay adaptation · p. 3
  • A single-step consistency policy replaces iterative flow-matching generation and produces an action chunk in one forward pass while retaining stochastic exploration.The action head is trained on both demonstration and online-replay chunks.Source: B. Efficient replay adaptation · p. 3

View-specialized visual anchors

  • GRAFT maintains separate sets of N_v = 8 anchor templates for the side and wrist views while sharing context and cross-attention projections.Each anchor reads only its assigned view and has no fixed proposal identity.Source: C. View-specialized visual anchors · p. 3
  • Each observation-dependent visual anchor is derived from view-specific attention, with global context used to reweight anchors within each view.A centered-tanh adapter with scale β_g is applied before suffix fusion.Source: C. View-specialized visual anchors · p. 3

D. Training-only multi-region guidance

GRAFT uses proposal masks during training to guide view-specific anchor attention across multiple plausible regions, while rebuilding matching per frame and view.

E. Online adaptation and proposal-free inference

GRAFT’s online process uses asynchronous actor–learner adaptation with replayed windows and twin-critic reinforcement learning components.

2) Actor update:

The actor is trained with behavior cloning and value maximization within an actor–learner schedule that periodically applies grounding updates and publishes the adapted policy.

Actor update

  • The actor combines behavior cloning and value maximization.Source: 2) Actor update · p. 4
  • Each iteration performs two critic updates and one actor update, using λbc = 0.5 and λq = 1.Source: 2) Actor update · p. 4
  • Every ten iterations, proposal-valid demonstrations provide an actor-only grounding update, and the updated policy is periodically published to the actor.Source: 2) Actor update · p. 4
  • The procedure initializes GRAFT from a pretrained VLA with twin critics, a prefix-KV cache, and action horizon H.Source: 2) Actor update · p. 4
  • At replanning, the actor samples an H-step action sequence and executes its first e actions.Source: 2) Actor update · p. 4
  • Once online replay contains 100 valid windows, the learner samples equally from online and demonstration data and reuses frozen-prefix states.Source: 2) Actor update · p. 4

IV. EXPERIMENTS

The experiments evaluate GRAFT on four fine-grained biomedical manipulation tasks using matched adaptation budgets, multiple camera views, and task-specific success criteria.

Experiments

B. Baselines and controlled ablations

Controlled comparisons show that GRAFT improves final-stage adaptation through identity-free multi-region grounding, with gains that are strong overall but not uniformly faster across tasks.

D. Learner-update efficiency

The efficiency evaluation compares learner-update throughput and online trajectories, showing that prefix reuse and single-step generation provide complementary computational gains.

E. Grounding diagnostics and deployment

Grounding diagnostics show that GRAFT concentrates attention on manipulation-relevant regions with complementary roles across views, while deployment removes proposal dependence.

Related papers

Loading 2608.27079v1…