Source-linked AI summary
GRAFT: Grounded and Efficient Online Reinforcement Adaptation for Fine-Grained Robot Manipulation
Yibo Qiu, Haoliang Ye, Shu'ang Sun, Zan Huang, Ronald X Xu, Mingzhai Sun
TL;DR
Fine-grained biomedical manipulation requires learning task-relevant local visual grounding from limited real-robot interaction, while online VLA adaptation must remain computationally efficient. GRAFT uses training-time region supervision for view-specific anchors, cached frozen-prefix states, and single-step action generation; across four real-robot tasks, it improves final-stage success by 25 percentage points under the same 45-minute budget.
Problem
Fine-grained manipulation depends on subtle view-dependent local cues, but demonstrations and scalar task rewards provide little spatial guidance for learning which regions matter during online adaptation.
Method
GRAFT learns view-specific visual anchors with training-only identity-free multi-region supervision and combines frozen-prefix KV caching with single-step action-chunk prediction for efficient online adaptation.
Results
25 percentage points: GRAFT improves final-stage success across four real-robot tasks under the same 45-minute adaptation budget, while prefix caching speeds learner updates by up to 9.94×.
Takeaways & Limitations
Learned anchors attend to task-relevant regions while region proposals remain unnecessary at deployment, supporting efficient grounded adaptation within a fixed wall-clock budget.
Takeaways & Limitations
The reactive policy does not explicitly track previous actions or task progress, which may limit longer-horizon tasks where the relevant visual target changes over time.
Abstract
from arXiv · showhide
Pretrained vision-language-action (VLA) policies provide strong priors for robot manipulation, yet adapting them online to fine-grained biomedical tasks remains challenging. Task success often hinges on subtle, view-dependent visual cues, while task-level rewards provide little guidance about which regions matter, making it difficult to learn task-relevant visual grounding from limited real-robot interaction. Online adaptation is further constrained by the computational cost of VLA inference and replay-based updates. We introduce GRAFT (Grounded Reinforcement Adaptation for Fast Task Learning), a framework for efficient online VLA adaptation through grounded perception. GRAFT uses region-level supervision to learn view-specific visual anchors that focus perception on task-relevant local cues without requiring region proposals at deployment. It further combines single-step action generation with cached visual-language prefix reuse to accelerate online learning. Across four biomedical manipulation tasks, GRAFT improves success rates by 25 percentage points under matched adaptation budgets, while reducing the computational overhead of online policy updates.
I. INTRODUCTION
GRAFT addresses the difficulty of learning fine-grained, task-relevant visual grounding during online VLA adaptation while keeping acting and replay updates efficient. It combines view-specific multi-region supervision, cached prefix reuse, and single-step action generation, then evaluates the approach on four biomedical manipulation tasks.
- Motivation: Fine-grained laboratory manipulation depends on precise local contact geometry and small task-relevant regions across multiple camera views.Scalar rewards and demonstrations provide little explicit spatial guidance about which cues determine success or failure.
- Grounded adaptation: GRAFT learns view-specific visual anchors using identity-free multi-region supervision during adaptation, without requiring masks or proposal generators at deployment.The supervision preserves distinct plausible regions rather than collapsing them into a single union target.
- Efficient online learning: Frozen visual-language prefix states are cached across replay updates, while the actor predicts complete action chunks in a single forward pass.The learner recomputes only trainable anchor-conditioned action pathways during replay optimization.
- Evaluation: GRAFT is evaluated under matched 45-minute adaptation budgets across four fine-grained biomedical manipulation tasks.The evaluation measures online success, grounding effects, learner-update efficiency, and learned anchor structure.
II. RELATED WORK
Related work improves VLA spatial reasoning, grounding, reinforcement adaptation, and computational efficiency through complementary strategies. GRAFT combines these directions around online acquisition of task-relevant visual representations from multiple robot cameras, using training-time rather than deployment-time proposals.
- Spatial reasoning: Prior VLA methods improve spatial reasoning with geometric representations, spatial features, visual traces, and adaptive viewpoint or resolution selection.Multi-view approaches combine observations through explicit geometric reasoning or cross-view aggregation.
- Visual grounding: Explicit grounding methods provide masks, regions, or crops at execution time, while others use gaze, affordances, alignment, or auxiliary grounding objectives during training.GRAFT is most closely related to training-time grounding approaches but focuses on online adaptation from multiple robot cameras.
- Reinforcement adaptation: Real-world reinforcement adaptation combines demonstrations or interventions with supervised and reinforcement learning, offline-to-online fine-tuning, chunk-level RL, and improved value estimation.GRAFT builds on this line without proposing a new reinforcement-learning objective.
- Efficiency: Efficiency methods accelerate VLA execution through parallel decoding, action chunking, or reuse of representations across observations and control steps.GRAFT instead reuses frozen visual-language prefix states on the learner side during replay updates.
III. METHOD
GRAFT combines a frozen visual-language prefix with view-specific anchors and training-only region supervision to produce action chunks from camera observations, language, and adapter state. Proposal inputs guide adaptation but are excluded from the deployed policy.
- Problem setup and deployment boundary: At time t, the policy receives side- and wrist-view RGB observations, a language instruction, and adapter state, then predicts an H = 10 action chunk.Each low-level action contains translation, rotation, and gripper commands.
- Problem setup and deployment boundary: A frozen VLM maps the inputs to view tokens, pooled context, and prefix key–value states for the side and wrist views.These representations form the frozen visual-language prefix used by the downstream policy.
- Problem setup and deployment boundary: Training observations may include offline-generated patch-aligned masks, but neither masks nor proposal generators are policy inputs.Region proposals therefore provide auxiliary supervision during training without creating a deployment dependency.
B. Efficient replay adaptation
GRAFT makes replay adaptation more efficient by reusing frozen prefix representations and by generating action chunks in one consistency-policy pass. Separate view-specific anchors process their assigned camera views while sharing context and cross-attention projections.
- Efficient replay adaptation: For replayed observations, GRAFT caches frozen prefix states, pooled summaries, and visual tokens, then recomputes trainable anchor, action, and critic pathways during learner updates.Live observations always receive a fresh prefix forward pass.
- Efficient replay adaptation: A single-step consistency policy replaces iterative flow-matching generation and produces an action chunk in one forward pass while retaining stochastic exploration.The action head is trained on both demonstration and online-replay chunks.
- View-specialized visual anchors: GRAFT maintains separate sets of N_v = 8 anchor templates for the side and wrist views while sharing context and cross-attention projections.Each anchor reads only its assigned view and has no fixed proposal identity.
- View-specialized visual anchors: Each observation-dependent visual anchor is derived from view-specific attention, with global context used to reweight anchors within each view.A centered-tanh adapter with scale β_g is applied before suffix fusion.
D. Training-only multi-region guidance
GRAFT uses proposal masks during training to guide view-specific anchor attention across multiple plausible regions, while rebuilding matching per frame and view.
- Proposal masks supervise raw anchor attention without persistent anchor–proposal assignments.
- FreeDice matches proposals to anchors and anchors to valid proposals.
- The two matching terms encourage proposal coverage and prevent anchors from ignoring all valid regions.
- Matching is rebuilt for every frame and view, supporting view-specific grounding without fixed assignments.
- Views without valid proposals contribute zero, using one fixed configuration across tasks.
E. Online adaptation and proposal-free inference
GRAFT’s online process uses asynchronous actor–learner adaptation with replayed windows and twin-critic reinforcement learning components.
- GRAFT runs asynchronous actor and learner processes for online adaptation.
- Replay transitions include an executed action window, return, next state, and bootstrap mask.
- Twin critics provide the target action and Bellman target used by the learner.
2) Actor update:
The actor is trained with behavior cloning and value maximization within an actor–learner schedule that periodically applies grounding updates and publishes the adapted policy.
- Actor update: The actor combines behavior cloning and value maximization.
- Actor update: Each iteration performs two critic updates and one actor update, using λbc = 0.5 and λq = 1.
- Actor update: Every ten iterations, proposal-valid demonstrations provide an actor-only grounding update, and the updated policy is periodically published to the actor.
- Actor update: The procedure initializes GRAFT from a pretrained VLA with twin critics, a prefix-KV cache, and action horizon H.
- Actor update: At replanning, the actor samples an H-step action sequence and executes its first e actions.
- Actor update: Once online replay contains 100 valid windows, the learner samples equally from online and demonstration data and reuses frozen-prefix states.
IV. EXPERIMENTS
The experiments evaluate GRAFT on four fine-grained biomedical manipulation tasks using matched adaptation budgets, multiple camera views, and task-specific success criteria.
- Experiments: The evaluation asks whether multi-region supervision improves adaptation, efficiency, grounding, and proposal-free deployment.
- Experiments: The four tasks are Petri Dish De-lidding, Centrifuge Tube Loading, Precision Liquid Transfer, and Pipette Tip Attachment.
- Experiments: Policies receive side-view and wrist-view RGB observations with proprioception, using action horizon H = 10.
- Experiments: Each task uses 10 teleoperated demonstrations and 45 minutes of online adaptation per method.
- Experiments: Figure 3 depicts training-only view-specific anchors guided independently by multiple offline proposal masks, with the proposal branch removed at deployment.
- Experiments: Table I reports rolling success over the final 10 episodes of each available trace.
- Experiments: Objects are randomly perturbed within ±30 mm, and predefined task-specific criteria determine success.
- Experiments: Common failures include inaccurate alignment, premature release, low-height contact, collision, and insertion errors.
B. Baselines and controlled ablations
Controlled comparisons show that GRAFT improves final-stage adaptation through identity-free multi-region grounding, with gains that are strong overall but not uniformly faster across tasks.
- GRAFT achieves 33/40 successes (82.5%) versus 23/40 (57.5%) for GRAFT-Union under the same 45-minute budget, a 25-percentage-point improvement.The comparison isolates the grounding supervision objective because the variants differ only in grounding supervision.
- The largest gain occurs on Centrifuge Tube Loading, where success rises from 2/10 to 9/10.
- GRAFT also recovers to 90% success on Petri Dish De-lidding and achieves stronger late-stage performance on Pipette Tip Attachment.
- On Precision Liquid Transfer, RL-CP reaches 9/10 while GRAFT reaches 8/10, so GRAFT does not converge faster on every task.
- GRAFT approaches near-zero intervention on three tasks and maintains a low intervention rate on Pipette Tip Attachment.
- Across all four tasks, GRAFT achieves at least 7/10 successes, with the largest gain on the task requiring coordination of multiple spatial cues.
D. Learner-update efficiency
The efficiency evaluation compares learner-update throughput and online trajectories, showing that prefix reuse and single-step generation provide complementary computational gains.
- Prefix reuse increases RL-FM throughput from 1.99 to 6.30 learner steps/s, a 3.17× speedup, and RL-CP throughput from 2.21 to 21.96 steps/s, a 9.94× speedup.
- With caching enabled, RL-CP is 3.49× faster than RL-FM.
- Single-step action generation reduces action-generation cost, while prefix reuse avoids repeated computation over replayed observation prefixes.
- Their combination yields the highest learner-update throughput.
- Figure 5 tracks rolling success rate, intervention rate, and episode length against wall-clock training time for four adaptation methods.
E. Grounding diagnostics and deployment
Grounding diagnostics show that GRAFT concentrates attention on manipulation-relevant regions with complementary roles across views, while deployment removes proposal dependence.
- GRAFT-Union and GRAFT visualize the same anchor-to-patch attention quantity, enabling a controlled qualitative comparison of grounding objectives.
- GRAFT produces more spatially concentrated responses around pipette tips, centrifuge tubes, and liquid-transfer targets than GRAFT-Union.
- Side-view anchors emphasize global approach and object alignment, whereas wrist-view anchors focus on local contact and insertion regions.
- Proposal generators and region masks are used only during training and are removed entirely at policy inference.
- Across four real-robot tasks, GRAFT improves final-stage success by 25 percentage points under the same 45-minute adaptation budget.
- The policy is reactive and does not explicitly track previous actions or task progress, limiting its scope for longer-horizon tasks with changing targets.