PaperlayerView exact revision on arXiv

Source-linked AI summary

GRASP: Global-Local Representation Alignment and Spatial-Physical Grounding for Bimanual Robotic Manipulation

Zeyuan Ren, Guanxun Zhao, Qi Li, Hao Wang, Wenlong Liao, Keyi Luo, Xijing Zhang, Zhiyuan Zhang, Jingye Cui, Qibin Gu, Zhichen Wang, Yucheng Hu, Liqiao Xia, Nan Bu, Zhengde Ma, Rankun Li, Xue Yang, Wende Xie

arXiv:2610.23123v1cs.RO

TL;DR

Joint-space VLA policies avoid deployment-time inverse kinematics, but their losses may not reflect the physical consequences of joint errors or explicitly guide attention to hand-object interactions. The paper adds physical and visual priors during training, then evaluates the approach in simulation and on real robots. The full model performs best in clean scenes, while the query-only variant is stronger under domain randomization.

  • Problem

    Joint-space action losses can misalign with physical task outcomes, and action-driven visual conditioning may not consistently focus on hand-object interaction regions.

    Source: §I · pp. 1–2

  • Method

    The framework adds differentiable-FK geometric supervision of clean-action estimates and global–local queries distilled from frozen visual teachers during training.

    Source: §I · p. 2

  • Results

    The full model reaches 79.22% success on clean scenes, 15.1 percentage points above baseline, while under randomization it trails the query-only variant by 12.3 points.

    Source: C. Simulation Results · p. 7

  • Takeaways & Limitations

    The two priors are complementary and regime-dependent: they are super-additive in-distribution, while geometric supervision trades some randomization robustness for in-distribution precision.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models that directly regress joint-space actions avoid the deployment-time cost of inverse kinematics. However, their training objectives remain misaligned with task success, as joint-space losses ignore the unequal physical consequences of joint errors and visual conditioning lacks explicit focus on hand-object interaction regions. To address these issues, we propose a training-phase prior injection framework for fine-grained bimanual manipulation. First, a physics-grounded flow-matching loss maps the implicit clean-action estimate at every noise level through differentiable forward kinematics (FK) and supervises the resulting Tool Center Point (TCP) pose and inter-arm relative pose, with a bounded noise-level weighting that compensates gradient decay. Second, we introduce a global prediction query group distilled from a frozen DINOv3 teacher through future-frame feature prediction, together with a local interaction query supervised by Grounding DINO at the attention level. Both teachers are removed at inference, and the deployment pipeline is that of a standard joint-space flow policy, adding only the learned query pathway. Simulation and real-robot experiments on long-horizon bimanual tasks show that the two priors are complementary across different deployment settings. The full model excels in in-distribution deployment, while the distilled query branch is preferable under strong visual randomization.

I. INTRODUCTION

The paper targets mismatches between joint-space losses and physical task outcomes, alongside visual conditioning that may not focus on hand-object interactions. It injects physical and visual priors during training and evaluates them in simulation and on real robots.

I. INTRODUCTION

  • The motivation is that joint-space losses ignore configuration-dependent physical consequences, while action-driven visual conditioning may not consistently focus on fine-grained hand-object interactions.Similar joint-space errors can produce substantially different task-space deviations after forward-kinematic propagation.Source: §I · pp. 1–2
  • Differentiable forward kinematics maps estimated clean actions at each flow noise level to TCP pose and inter-arm relative pose for geometric supervision, while inference still outputs joint commands directly.The supervision is training-only and ties penalties to their physical consequences.Source: §I · p. 2
  • Global prediction queries learn from frozen DINOv3 future-frame features, while local interaction queries use Grounding DINO attention-level supervision to guide focus toward hand-object regions.Both teachers are removed at inference.Source: §I · p. 2
  • On 50 bimanual RoboTwin 2.0 tasks, the full model improves average success over π0.5-SFT by 15.1 percentage points in clean settings, while π0.5-Global+Local Query improves it by 29.0 points under domain randomization.Source: §I · p. 2
  • On four long-horizon real-world tasks, the method improves average success rate by 14.2 percentage points over the baseline.The tasks include watermelon juice making, soda bottle opening, beverage mixing, and clothes folding.Source: §I · p. 2
  • The geometric supervision reduces TCP position error compared with joint-space-only training, aligning the objective more closely with physical execution.The paper describes this as confirmation that the geometric loss better aligns optimization with physical execution.Source: §I · p. 2

A. Kinematic constraints in the control of dual-arm robots

The paper contrasts task-space policies that require inverse kinematics with joint-space policies whose losses do not reflect the unequal physical effects of joint errors. It proposes combining differentiable kinematic supervision with global and local visual-query supervision during training, while retaining joint-space action generation at inference.

A. Kinematic constraints in the control of dual-arm robots

B. Visual Query and Distillation Supervision in VLA models

  • The global query group predicts future-frame features from frozen DINOv3, while the local interaction query receives attention-level supervision from Grounding DINO regions.Both teachers are removed at inference.Source: B. Visual Query and Distillation… · p. 2

III. METHODOLOGY

  • The framework combines physics-grounded flow-matching supervision with global and local queries that learn future semantics and interaction-region awareness.The priors are integrated into the pretrained π0.5 backbone.Source: §III · pp. 2–3

A. Differentiable Kinematic Flow Matching Supervision for Bimanual Physical Interaction

  • Differentiable forward kinematics supervises TCP and inter-arm relative-pose errors, extending physical-space constraints to the implicit clean-action estimate at every flow noise level.The geometric loss is applied during training; the policy still outputs joint commands directly at inference.Source: A. Differentiable Kinematic Flow… · p. 3, Figure 2 · p. 3

B. Interaction-Guided Global–Local Query Distillation

The method adds global and local queries to π0.5: global queries learn future-scene semantics, while task-state-conditioned local queries focus on interaction regions. Frozen-teacher supervision trains both branches, which are designed to predict future visual representations and align local attention and features.

B. Interaction-Guided Global–Local Query Distillation

  • Global queries provide future-scene semantic supervision, while local manipulation queries use interaction-region priors to emphasize objects and grippers relevant to the current action.The design targets the limited image area occupied by grippers and relevant objects, including in long-horizon tasks with distractors.Source: B. Interaction-Guided Global–Local… · p. 4
  • The global branch partitions shared learnable sampler latents into equal-sized groups and averages each group to form global queries.The full latent pool also serves as the auxiliary Perceiver's query set.Source: B. Interaction-Guided Global–Local… · p. 5
  • The global sampler predicts future visual-semantic features from current observations using targets from a frozen DINOv3 teacher, resampled to match its prediction slots.The future image is used for training targets, not at inference; the implementation uses 576 prediction slots and a 30-frame future offset.Source: B. Interaction-Guided Global–Local… · p. 5
  • Local prefix tokens combine learnable slot vectors with instruction and robot-state features, shared across slots through learned projections.The resulting task-state-conditioned local queries are concatenated with the other prefix tokens before processing by the π0.5 backbone.Source: B. Interaction-Guided Global–Local… · p. 5
  • For local feature extraction, local tokens query a Perceiver whose key-value context contains image and global-query hidden states.The image and global-query representations are extracted from the vision-language backbone's output.Source: B. Interaction-Guided Global–Local… · p. 5
  • Grounding DINO-derived attention targets supervise local attention, while the local objective also preserves the visual content pooled under that attention.Invalid targets contribute zero local loss, and local prefix tokens and task-state projection layers also receive gradients from the action objective.Source: B. Interaction-Guided Global–Local… · p. 5

IV. EXPERIMENTS

The experiments assess geometric supervision, query distillation, and their combination against a baseline VLA model.

IV. EXPERIMENTS

  • The experiments assess geometric supervision by its effect on action accuracy.The evaluation also tests query distillation, robustness, and the combined model against the baseline.Source: §IV · p. 6
  • The experiments assess query distillation by its effect on robustness under randomized scenes and unseen language.Source: §IV · p. 6
  • The experiments compare the combination of geometric supervision and query distillation against the baseline VLA model.Source: §IV · p. 6

A. Simulation Setup

Simulation evaluation uses RoboTwin 2.0 manipulation tasks under clean and randomized conditions, with separate fine-tuned policies and task-level success-rate reporting.

A. Simulation Setup

  • The RoboTwin 2.0 evaluation covers 50 manipulation tasks under clean scenes with seen-language instructions and randomized scenes with unseen-language instructions.The randomized condition includes visual and environmental changes such as distractors, lighting variation, background changes, and scene perturbations.Source: A. Simulation Setup · p. 6
  • Each simulation task uses an Aloha AgileX dual-arm policy fine-tuned from π0.5 on 50 clean demonstrations.Separate policies are trained per task using the benchmark pipeline and protocol; the baseline uses RGB observations, batch size 256, and 20,000 training steps.Source: A. Simulation Setup · p. 6
  • Evaluation uses 100 episodes per task and condition, replacing initialization failures from the candidate list.Success follows each task's RoboTwin 2.0 criterion, and results report average success rate over evaluated tasks.Source: A. Simulation Setup · p. 6

B. Baselines and Implementation Details

The implementation compares five π0.5-based variants, controlling their initialization and fine-tuning data while setting geometric and query-loss weights for training.

B. Baselines and Implementation Details

  • Five variants compare the RGB π0.5 baseline, global-query-only, global-plus-local-query, geometry-only, and the complete geometry-plus-query model.Source: B. Baselines and Implementation Details · p. 6
  • All variants start from the same pretrained π0.5 checkpoint and use the same demonstrations and number of training steps.Position error is measured in meters, and rotation error is the geodesic angle on SO(3), measured in radians.Source: B. Baselines and Implementation Details · p. 6
  • The geometric auxiliary loss has weight λgeo=0.15, with the relative-pose term weighted at βrel=0.1 of the TCP term; global-feature loss uses λD=0.08 and local attention KL uses λP=1.The geometric weight warms up over 2,000 steps; the action horizon is 50 steps, and inference flow integration uses 10 steps.Source: B. Baselines and Implementation Details · p. 6

C. Simulation Results

On RoboTwin 2.0, the full model performs best in clean scenes, while the query-only variant is preferable under strong visual randomization. Geometric supervision improves clean-task performance and reduces TCP trajectory deviation.

C. Simulation Results

  • The π0.5-SFT baseline reaches 64.13% success on clean tasks and 36.15% on randomized tasks, surpassing the benchmark-reported RDT and π0 averages under corresponding conditions.The cited benchmark averages are 34.5%/13.7% for RDT and 46.4%/16.3% for π0 under Easy/Hard settings corresponding to Clean/Randomized conditions.Source: C. Simulation Results · p. 6
  • Geometric supervision raises clean-task success from 64.13% to 67.74% and reduces mean and maximum TCP deviation from 5.3 to 4.6 mm and 7.7 to 6.8 mm.The paper reports that precise-positioning tasks such as Move Pillbottle Pad and Pick Dual Bottles benefit substantially.Source: C. Simulation Results · p. 6
  • Adding the global query raises randomized-task success from 36.15% to 55.29%; adding the local query raises it to 65.15%, with clean success at 68.30%.The paper hypothesizes that appearance-invariant features and hand–object attention help under scene variation and unseen-language instructions.Source: C. Simulation Results · pp. 6–7
  • 79.22% clean-scene success is 15.1 points above the baseline and 10.9 above query-only; 52.84% randomized-scene success is 16.7 points above the baseline.The full model exceeds the query-only variant in clean scenes but trails it by 12.3 points under domain randomization.Source: C. Simulation Results · p. 7, Table I · p. 7

1) Experimental Setup:

The real-world evaluation uses a dual-arm PiPER-X robot and RGB-only policy observations. The study evaluates four tasks using demonstrations, multiple training seeds, and repeated trials.

1) Experimental Setup:

  • The dual-arm PiPER-X uses three Intel RealSense D435i cameras, with only RGB images used for policy inference at 30 Hz.The watermelon-juice task uses 3D-printed grooved gripper attachments to improve knife and spatula grasp stability.Source: 1) Experimental Setup · p. 7
  • Each task has about 150 demonstrations; each method is trained with five seeds and each checkpoint is evaluated over 30 trials.Reported success rates are means across seeds with one-standard-deviation error bars.Source: 1) Experimental Setup · p. 7
  • A trial counts as successful only when the entire task sequence is completed without human intervention.Source: 1) Experimental Setup · p. 7

2) Long-Horizon Manipulation Tasks:

The real-world suite covers four long-horizon bimanual tasks involving sequential subtasks, tool use, liquid manipulation, or coordinated object interactions.

2) Long-Horizon Manipulation Tasks:

3) Real-World Results:

On four real-world tasks, the method improves success over the π0.5-SFT baseline, with the full model achieving the highest success on each. The conclusion identifies regime-dependent benefits and limits the evaluation to one simulation benchmark and one real dual-arm platform.

3) Real-World Results:

  • The method improves success on all four real-world tasks; watermelon-juice success rises from 56.7% to 83.3%.Figure 4 summarizes the real-world deployment results.Source: 3) Real-World Results · p. 7, Figure 4 · p. 8
  • Both ablations consistently outperform the π0.5-SFT baseline, while the full method achieves the highest success rate on all four tasks.Source: 3) Real-World Results · pp. 7–8

V. CONCLUSIONS

  • The priors are complementary across deployment regimes, but evaluation is limited to one simulation benchmark and one real dual-arm platform.The conclusion also identifies static loss weights and dependence on offline detector quality as limitations.Source: §V · p. 8

Related papers

Loading 2610.23123v1…