Source-linked AI summary
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Zhanyi Sun, Shuran Song
TL;DR
Pretrained generative robot policies can produce broad but imperfect behaviors, while sparse-reward, long-horizon manipulation makes unconstrained RL exploration costly and unstable. DICE-RL uses residual off-policy RL with selective behavior regularization and value-guided action selection to contract the policy toward successful behaviors. It achieves strong, stable, sample-efficient performance across tasks and observation modalities, including real-robot manipulation.
Problem
Sparse-reward, long-horizon manipulation requires improving imperfect generative BC policies without relying on costly, unconstrained online exploration.
Method
DICE-RL freezes a pretrained generative BC policy and learns residual corrections using selective behavior regularization, value-guided action selection, and off-policy RL.
Results
DICE-RL achieves the highest final performance with greater stability and sample efficiency across benchmark tasks, reaching ≥90% success on Tool Hang from state or pixel inputs using 50 demonstrations.
Takeaways & Limitations
RL finetuning can sharpen and contract generative policy distributions around high-value behaviors while retaining controlled deviations for manipulation skill improvement.
Takeaways & Limitations
The resulting policies may inherit dataset biases or exhibit unsafe behavior under distribution shift, requiring safeguards before safety-critical deployment.
Abstract
from arXiv · showhide
We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: https://zhanyisun.github.io/dice.rl.2026/.
1. Introduction
DICE-RL treats RL as a distribution contractor that refines a pretrained generative behavior prior into a more successful policy. It addresses sparse-reward, long-horizon manipulation through controllable exploration, residual corrections, selective regularization, and value-guided action selection.
- RL reweights a pretrained generative BC policy toward high-success behaviors while suppressing failure-prone ones.
- Robotics makes this difficult because actions are continuous, verification is costly, rewards are delayed, and horizons are long.
- A diffusion-based BC policy pretrained on offline demonstrations provides diverse, physically plausible action proposals for structured exploration.
- DICE-RL uses a lightweight residual over the frozen BC prior, preserving expressiveness while reducing the effective search space.
- Selective behavior regularization and value-guided action selection support stable, sample-efficient contraction toward successful behaviors.
- The paper contributes a practical framework, empirical evaluation in simulation and on a real robot, and analyses of generative-policy post-training.
2. Related Work
Prior work uses offline or online methods to improve pretrained BC policies, while DICE-RL combines regularization, steering, and residual correction for controlled refinement. Its design targets sharpened action distributions without abandoning the pretrained policy’s support.
- Offline BC Pretraining: RL post-training is motivated by BC covariate shift and compounding errors, while diffusion and flow policies offer expressive action-distribution modeling.
- RL Finetuning of Generative BC Policies: Existing RL finetuning methods trade off stability, efficiency, and the ability to correct pretrained-policy failures.
- DICE-RL combines BC regularization, value-guided action selection, and residual action learning to balance stability, efficiency, and flexibility.
3. Preliminaries
The paper formulates sparse-reward manipulation as an MDP with offline demonstrations and online replay, then models actions using diffusion- or flow-based generative BC policies. These policies generate stochastic actions by transforming latent noise conditioned on the state.
- The setting is a sparse-reward MDP in which success is observed only at episode termination.
- Training uses an offline demonstration dataset and an online replay buffer collected during finetuning.
- Diffusion and flow-matching BC policies transform latent noise into state-conditioned actions, producing expressive generative behavior.
- The pretrained policy is deterministic given state and latent noise, while DICE-RL applies to both flow-matching and diffusion implementations.
4. DICE-RL
DICE-RL freezes a pretrained generative policy and learns a latent-conditioned residual that corrects its action chunks. Selective regularization, multi-sample training, best-of-N selection, action chunking, and adaptive data mixing make exploration more controlled and finetuning more stable.
- Residual Policy: The pretrained flow-matching policy remains fixed while RL learns a lightweight residual correction instead of differentiating through the generative model.
- Residual Policy: The residual conditions on the same latent noise as the base policy, making each correction explicitly aware of its proposed action chunk.
- Residual Policy: Residual action chunks combine the pretrained action with a correction, preserving stochastic generation while enabling controlled deviations.
- Selective Behavior Regularization: The actor combines critic-value maximization with BC-style regularization that encourages exploration within the pretrained policy’s support.
- Training Procedure: Action chunking reduces decision frequency and improves temporal consistency, while adaptive offline-to-online mixing shifts learning toward online experience after warm-start.
- Exploration and Training: Multi-sample expectation training averages objectives over latent-induced candidates, and best-of-N selection executes the highest-valued sampled action.
- Selective Behavior Regularization: The BC penalty is selectively relaxed when value improvement is reliable, while critic consistency guards against overestimation-driven drift.
5. Experiments
DICE-RL is evaluated against prior RL finetuning methods, across simulation and real-robot manipulation settings, with analyses of finetunability, distribution sharpening, contraction, robustness, and design choices. It achieves strong, stable, sample-efficient finetuning while improving performance on challenging tasks.
- 5.1. Comparison to RL Finetuning Algorithms: DICE-RL attains the highest final performance across Robomimic tasks while remaining more stable and sample efficient across difficulty levels.The comparison includes Can, Square, Transport, and Tool Hang with state- and pixel-based observations.
- 5.1. Comparison to RL Finetuning Algorithms: Starting from 45% success on Tool Hang, DICE-RL surpasses 90% success within roughly 2000 online episodes using only 50 demonstrations.The result is reported for both state and pixel inputs.
- 5.2. What makes a pretrained policy easy to finetune?: Pretrained-policy finetunability depends on action-distribution structure: intermediate checkpoints balance high GOODCOV with reduced BADENT better than later checkpoints.Later checkpoints suppress low-value variability further but trade away too much GOODCOV.
- 5.3. Understanding DICE-RL: RL reshapes the pretrained action distribution toward high-value actions, with larger value gains accompanying larger entropy reductions.The effect is strongest at critical states such as pre-insertion and insertion.
- 5.3. Understanding DICE-RL: RL-policy rollouts show more stable and typically smaller pairwise-distance evolution than pretrained BC and expert rollouts, indicating stronger closed-loop contraction.Under action-noise probabilities from 0.1 to 0.9, the RL policy also degrades more gracefully than the pretrained BC policy.
- 5.4. DICE-RL on Real Robot: On real-robot tasks, DICE-RL reliably finetunes policies around dominant BC failures such as pulley transitions, belt seating, navigation, and insertion.The largest value gains and entropy reductions occur at states corresponding to these failure modes.
- 5.5. Ablation Studies: The BC-loss filter, multi-sample expectation training, and best-of-N action selection generally accelerate convergence and improve peak finetuning performance.DICE-RL also finetunes stably and sample efficiently with both diffusion and flow-matching pretrained backbones.
6. Conclusion
DICE-RL is presented as a stable, sample-efficient framework that contracts pretrained generative policy distributions around high-value behaviors. It achieves strong performance across tasks and observation modalities, including a real-world belt assembly task.
- DICE-RL sharpens and contracts pretrained action distributions around high-value behaviors during RL finetuning.
- The framework is described as stable and sample-efficient for finetuning generative behavior cloning policies.
- DICE-RL achieves strong performance across tasks and observation modalities, including a challenging real-world belt assembly task.
Impact Statement
DICE-RL improves the stability and sample efficiency of RL finetuning for pretrained generative robot policies, with validation in simulation and on a real robot. Deployment in safety-critical settings still requires safeguards because policies may inherit dataset biases or behave unsafely under distribution shift.
- DICE-RL improves RL finetuning stability and sample efficiency for pretrained generative robot policies across simulation and real-robot validation.
- Policies may inherit dataset biases or exhibit unsafe behavior under distribution shift, requiring careful evaluation and safeguards before safety-critical deployment.
B. Detailed Ablations Results
Ablations show that DICE-RL’s components improve sample efficiency and stability, with their effects depending on task horizon, precision, and pretrained-policy architecture. Across hyperparameters, the procedure remains robust while benefiting from task-appropriate regularization and credit assignment.
- BC-loss filtering consistently improves sample efficiency and final performance on most tasks, especially long-horizon Transport and Tool Hang.
- Increasing multi-sample count K consistently improves sample efficiency by optimizing across action-distribution candidates and reducing gradient variance through sample reuse.
- Best-of-N action selection accelerates convergence by executing the highest-valued candidate sampled from the pretrained policy’s latent-conditioned diversity.
- Both flow-matching and diffusion-pretrained policies finetune stably and sample efficiently, reaching comparable peak performance.
- Increasing demonstrations improves downstream finetuning, while higher update-to-data ratios help long-horizon tasks but slightly reduce Can performance.
C. Implementation Details - Simulation
Simulation uses flow-matching behavior cloning with visual or state encoders, followed by off-policy residual actor–critic finetuning over action chunks. The implementation uses frozen pretrained components, critic ensembles, Polyak targets, and fixed update-to-data ratios.
- BC pretraining uses a lightweight MLP for state observations and independent from-scratch ResNet-18 encoders for multi-view pixel observations.
- Flow-matching training interpolates Gaussian noise and data actions along a linear path, then predicts a conditional velocity field from encoded observations.
- The flow-matching implementation uses forward-Euler integration with 10 flow steps for Transport, Square, and Can, and 20 for Tool Hang.
- RL finetuning uses a lightweight residual actor over h-step action chunks while freezing the pretrained generative policy and observation encoder.
- The off-policy critic uses an ensemble, Polyak target updates with τ = 0.01, Adam at learning rate 10^-4, task-specific n-step returns, and batched updates at a fixed update-to-data ratio.
D. Implementation Details - Real Robot
The real-robot implementation uses a CLIP-pretrained ViT diffusion policy with synchronized visual and proprioceptive inputs, then applies asynchronous residual RL updates. Its critic ensemble, BC regularization, and scheduled offline-data ratio are configured for online robot finetuning.
- The real-robot BC policy uses a CLIP-pretrained ViT-Base encoder and conditional U-Net denoiser for RGB-conditioned action prediction.
- Robot states sampled at 500Hz and RGB images sampled at 60Hz are downsampled and temporally aligned for each visual observation.
- Real-robot RL uses residual actor and critic MLPs with hidden layers [1024, 1024, 1024], a five-critic ensemble, and BC loss weight 100.
- The RLPD offline-data ratio decreases linearly from 0.5 to 0.1 over the first 100 online episodes, with asynchronous batched updates every 10 episodes.