Source-linked AI summary
GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
Yunfei Li, Xiao Ma, Jiafeng Xu, Yu Cui, Zhongren Cui, Zhigang Han, Liqun Huang, Tao Kong, Yuxiao Liu, Hao Niu, Wanli Peng, Jingchao Qiao, Zeyu Ren, Haixin Shi, Zhi Su, Jiawen Tian, Yuyang Xiao, Shenyu Zhang, Liwei Zheng, Hang Li, Yonghui Wu
TL;DR
GR-RL addresses the challenge that human demonstrations can be noisy and suboptimal for long-horizon, precise dexterous manipulation. It filters demonstrations with learned task progress, augments them through morphological symmetry, and applies latent-space online reinforcement learning, achieving 83.3% success on autonomous multi-eyelet shoe lacing. The authors report behavior drift as a remaining limitation during online RL.
Problem
General VLA policies remain unreliable for long-horizon manipulation requiring millimeter-level precision and dexterous interaction with deformable objects.
Method
GR-RL filters demonstrations with an offline-RL task-progress evaluator, augments bimanual data using morphological symmetry, and aligns deployment behavior through latent-space online reinforcement learning.
Results
83.3% success was achieved on autonomous shoe lacing through multiple eyelets, which the authors describe as the first such learning-based result.
Takeaways & Limitations
The framework demonstrates that a generalist VLA policy can be specialized for long-horizon, dexterous, high-precision shoe lacing.
Takeaways & Limitations
Online RL can produce unstable behavior under sparse, noisy rewards, potentially because of limited noise-predictor capacity or difficult credit assignment in the latent action space.
Abstract
from arXiv · showhide
We present GR-RL, a robotic learning framework that turns a generalist vision-language-action (VLA) policy into a highly capable specialist for long-horizon dexterous manipulation. Assuming the optimality of human demonstrations is core to existing VLA policies. However, we claim that in highly dexterous and precise manipulation tasks, human demonstrations are noisy and suboptimal. GR-RL proposes a multi-stage training pipeline that filters, augments, and reinforces the demonstrations by reinforcement learning. First, GR-RL learns a vision-language-conditioned task progress, filters the demonstration trajectories, and only keeps the transitions that contribute positively to the progress. Specifically, we show that by directly applying offline RL with sparse reward, the resulting $Q$-values can be treated as a robust progress function. Next, we introduce morphological symmetry augmentation that greatly improves the generalization and performance of GR-RL. Lastly, to better align the VLA policy with its deployment behaviors for high-precision control, we perform online RL by learning a latent space noise predictor. With this pipeline, GR-RL is, to our knowledge, the first learning-based policy that can autonomously lace up a shoe by threading shoelaces through multiple eyelets with an 83.3% success rate, a task requiring long-horizon reasoning, millimeter-level precision, and compliant soft-body interaction. We hope GR-RL provides a step toward enabling generalist robot foundation models to specialize into reliable real-world experts.
1 Introduction
GR-RL addresses the gap between general VLA capability and reliable long-horizon, precise dexterous manipulation by filtering, augmenting, and reinforcing human demonstrations. It achieves 83.3% success on autonomous shoe lacing through multiple eyelets.
- GR-RL is built to specialize a generalist VLA policy for challenging real-world manipulation rather than relying on demonstrations as optimal.The starting policy is GR-3, whose generalization does not suffice when precision, dexterity, and long-horizon robustness matter.
- Current VLA policies remain unreliable for millimeter-level deformable-object control and long-horizon execution, where errors accumulate across steps.Shoelacing requires dexterity, precision, and robustness across diverse and unexpected scenarios.
- GR-RL targets suboptimal human demonstrations and mismatch between offline training actions and deployment-time control behavior.Human demonstrators may hesitate or introduce noisy actions, while smoothing and receding-horizon control alter executed trajectories.
- The framework uses learned task progress to filter demonstrations, action augmentation, and online reinforcement learning in a multi-stage pipeline.The pipeline is presented as offline filtered behavior cloning, augmentation, and online reinforcement learning.
- 83.3% overall success makes GR-RL, to the authors’ knowledge, the first learning-based model to thread shoelaces through multiple eyelets.The task combines long-horizon manipulation, millimeter-level precision, and deformable-object interaction.
2 The GR-RL Model
The GR-RL model combines a VLA policy and a distributional multi-task critic within a Mixture-of-Transformer architecture. The critic evaluates action chunks and provides uncertainty-aware task-progress estimates under sparse rewards.
- GR-RL uses a Mixture-of-Transformer architecture containing a VLA policy and a multi-task critic with 5B total parameters.The policy generates action chunks for a bi-manual mobile robot conditioned on language, observations, and robot state.
- The critic evaluates action chunks with distributional reinforcement learning, representing values as bounded discrete distributions rather than unbounded regressions.The distributional formulation captures uncertainty and is reported to be more robust under sparse rewards.
- The policy and critic are trained over robot vision-language-action trajectories and temporal-difference errors using flow matching and distributional reinforcement learning.These objectives are represented as co-training components of the GR-RL model.
3 Training Recipe
GR-RL’s training recipe filters noisy demonstrations with an offline-RL progress evaluator, augments bimanual data through symmetry, and uses latent-space online RL to align deployment behavior. These stages address suboptimal demonstrations and the mismatch between training and execution.
- 3.3 Online Steering for Policy Deployment Alignment: The recipe supports closed-loop trial-and-error adaptation while avoiding teleoperated trajectories in the online buffer because their dynamics mismatch policy rollouts.The online stage combines off-policy and on-policy data for sample-efficient adaptation.
- 3.1 Data Filtering with a Learned Task Progress Evaluator: GR-RL filters demonstrations by removing transitions whose predicted task progress drops beyond a threshold, then behavior-clones the remaining data.The progress signal comes from the mean of a distributional critic trained on successful and hindsight-generated failed trajectories.
- 3.1 Data Filtering with a Learned Task Progress Evaluator: The offline-RL critic detects subtle failures because progress can drop sharply when a teleoperator makes a mistake, including millimeter-scale insertion errors.This avoids directly imitating all noisy, multimodal demonstration fragments.
- 3.2 Morphological Symmetry Augmentation: Morphological symmetry augmentation flips images, swaps left- and right-wrist views, and transforms proprioception to exploit symmetry in bimanual manipulation.The method is reported to enhance policy performance.
- 3.3 Online Steering for Policy Deployment Alignment: Latent-space online reinforcement learning uses a noise predictor to steer the flow policy toward high-return behavior while adapting to deployment-time action mismatch.Structured exploration is used because direct wrist-pose or joint-position noise rarely succeeds in millimeter-precision tasks.
4 Robot & System
ByteMini-v2 is the wheeled mobile manipulation robot used to verify GR-RL, with 7-DoF dual arms and a wrist spherical joint design. Its updates increase load capacity, improve confined-space mobility, and enhance usability.
- ByteMini-v2 is a wheeled mobile manipulation robot with 7-DoF dual arms and a wrist spherical joint design.
- Figure 4 presents ByteMini-v2 specifications in terms of sensors, DoFs, and electronic devices.
- Higher Load for Manipulation: 35 Nm elbow actuators increase each 7-DoF arm’s peak load from 1.4 kg to 3.15 kg.
- Enhanced Mobility in Confined Spaces: The chassis shrinks from 500 mm x 720 mm to 450 mm x 650 mm, while redesigned servo steering wheels support synchronous yaw-and-pitch adjustment.
- Higher Polished Robot Design and Enhanced Usability: A refined chassis encapsulates exposed wiring, and the monitor moves to the shoulder for better user experience.
5 Experiments
GR-RL improves long-horizon shoelace manipulation through filtered demonstrations, symmetry augmentation, and online reinforcement learning. Experiments show gains in overall success, intermediate-stage completion, progress estimation, and recovery behaviors.
- Main Results: 45.7% success rises to 61.6% after data filtering and 72.7% after symmetry augmentation.The results isolate the contribution of filtering and augmentation before online reinforcement learning.
- Main Results: 83.3% success rate is achieved by the evaluated checkpoint after online reinforcement learning, exceeding the offline-trained policy's performance.The offline-to-online transition initially reduces success, then recovery exceeds 90% during training; the 500-step checkpoint is used for evaluation.
- Intermediate-Stage Analysis: Data filtering and online reinforcement learning largely reduce failures during threading, while augmentation improves every evaluated intermediate stage.Stages include picking up the correct shoelace, threading the eyelet, handing over to another gripper, and pulling tight.
- Progress Evaluator Ablation: The RL-based progress evaluator captures subtle failures and long-term effects that a regression-based progress predictor smooths over.GR-RL shows a significant value change when the robot puts down the shoelace to adjust its grasp, whereas regression remains nearly flat.
- Progress Evaluator Ablation: Distributional critics produce bounded value predictions that align more robustly with temporal progress than non-distributional critics under sparse rewards.The comparison concerns progress prediction on a successful trajectory, where non-distributional critics suffer severe overestimation earlier in the horizon.
- Learned Behaviors: GR-RL handles varied shoe configurations and recovers from dropped or missed shoelaces by retrying and actively adjusting the scene.Observed behaviors include regrasping, reorienting the shoe, pulling it closer, and selecting the correct crossed shoelace end.
6 Related Work
GR-RL builds on generalist VLA policies and combines filtered data, action augmentation, and online real-world RL to improve long-horizon dexterous manipulation.
- Generalist Robotic Foundation Policy: Generalist VLA policies adapt vision-language models pretrained on web-scale data to robotic actions and aim to generalize across novel scenes and tasks.
- Generalist Robotic Foundation Policy: GR-RL improves the generalist GR-3 policy by filtering high-quality data, augmenting actions, and applying online real-world RL.The framework targets long-horizon dexterous and precise manipulation.
- Real-World Reinforcement Learning: Real-world RL addresses imitation learning's susceptibility to compounding errors and its inability to exceed demonstration performance.
- Real-World Reinforcement Learning: GR-RL uses distributional critics to learn task progress, filtered behavior cloning for offline improvement, and online exploration from a stronger base policy.The stronger offline policy helps reduce the search space during online exploration.
7 Limitations & Conclusions
GR-RL demonstrates strong performance on long-horizon, high-precision dexterous tasks, but online reinforcement learning can produce unstable behavior under sparse and noisy rewards.
- Limitations: Online RL remains limited by behavior drift, potentially caused by a lightweight noise predictor or difficult credit assignment in the large latent action space.The authors identify policy distillation into the base VLA as a possible future direction.
- Conclusion: GR-RL is presented as the first learning-based policy capable of lacing up shoes and as a framework for specialist VLA policies.
- Conclusion: The framework aligns rollout behavior with training signals through online RL after filtering transitions with an RL-based task-progress evaluator.
Contributions and Acknowledgments
The paper credits its contributors, supervisors, and collaborators for research, data curation, and hardware-system support.
- Contributors: Yunfei Li, Xiao Ma, and Jiafeng Xu are identified as core contributors.
- Contributors: Hang Li and Yonghui Wu are identified as supervisors, alongside a larger contributor group.
- Acknowledgments: The acknowledgments thank collaborators and data annotators for discussions and data curation, and others for hardware-system development and maintenance.