PaperlayerView exact revision on arXiv

Source-linked AI summary

From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

Zhanyi Sun, Shuran Song

arXiv:2603.10263v2cs.ROcs.LG

TL;DR

Pretrained generative robot policies can produce broad but imperfect behaviors, while sparse-reward, long-horizon manipulation makes unconstrained RL exploration costly and unstable. DICE-RL uses residual off-policy RL with selective behavior regularization and value-guided action selection to contract the policy toward successful behaviors. It achieves strong, stable, sample-efficient performance across tasks and observation modalities, including real-robot manipulation.

  • Problem

    Sparse-reward, long-horizon manipulation requires improving imperfect generative BC policies without relying on costly, unconstrained online exploration.

    Source: §1 · p. 1

  • Method

    DICE-RL freezes a pretrained generative BC policy and learns residual corrections using selective behavior regularization, value-guided action selection, and off-policy RL.

    Source: §4 · p. 3, §2 · p. 3

  • Results

    DICE-RL achieves the highest final performance with greater stability and sample efficiency across benchmark tasks, reaching ≥90% success on Tool Hang from state or pixel inputs using 50 demonstrations.

    Source: §5.1 · pp. 4, 6

  • Takeaways & Limitations

    RL finetuning can sharpen and contract generative policy distributions around high-value behaviors while retaining controlled deviations for manipulation skill improvement.

    Source: §2 · p. 3, §6 · p. 9

    The resulting policies may inherit dataset biases or exhibit unsafe behavior under distribution shift, requiring safeguards before safety-critical deployment.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: https://zhanyisun.github.io/dice.rl.2026/.

1. Introduction

DICE-RL treats RL as a distribution contractor that refines a pretrained generative behavior prior into a more successful policy. It addresses sparse-reward, long-horizon manipulation through controllable exploration, residual corrections, selective regularization, and value-guided action selection.

  • RL reweights a pretrained generative BC policy toward high-success behaviors while suppressing failure-prone ones.Source: §1 · p. 1
  • Robotics makes this difficult because actions are continuous, verification is costly, rewards are delayed, and horizons are long.Source: §1 · p. 1
  • A diffusion-based BC policy pretrained on offline demonstrations provides diverse, physically plausible action proposals for structured exploration.Source: §1 · pp. 1–2
  • DICE-RL uses a lightweight residual over the frozen BC prior, preserving expressiveness while reducing the effective search space.Source: §1 · p. 2
  • Selective behavior regularization and value-guided action selection support stable, sample-efficient contraction toward successful behaviors.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper contributes a practical framework, empirical evaluation in simulation and on a real robot, and analyses of generative-policy post-training.Source: §1 · p. 2

2. Related Work

Prior work uses offline or online methods to improve pretrained BC policies, while DICE-RL combines regularization, steering, and residual correction for controlled refinement. Its design targets sharpened action distributions without abandoning the pretrained policy’s support.

Offline BC Pretraining

  • RL post-training is motivated by BC covariate shift and compounding errors, while diffusion and flow policies offer expressive action-distribution modeling.Source: §2 · p. 2

RL Finetuning of Generative BC Policies

  • Existing RL finetuning methods trade off stability, efficiency, and the ability to correct pretrained-policy failures.Source: §2 · pp. 2–3
  • DICE-RL combines BC regularization, value-guided action selection, and residual action learning to balance stability, efficiency, and flexibility.Source: §2 · p. 3

3. Preliminaries

The paper formulates sparse-reward manipulation as an MDP with offline demonstrations and online replay, then models actions using diffusion- or flow-based generative BC policies. These policies generate stochastic actions by transforming latent noise conditioned on the state.

  • The setting is a sparse-reward MDP in which success is observed only at episode termination.Source: §3 · p. 3
  • Training uses an offline demonstration dataset and an online replay buffer collected during finetuning.Source: §3 · p. 3
  • Diffusion and flow-matching BC policies transform latent noise into state-conditioned actions, producing expressive generative behavior.Source: §3 · p. 3
  • The pretrained policy is deterministic given state and latent noise, while DICE-RL applies to both flow-matching and diffusion implementations.Source: §3 · p. 3

4. DICE-RL

DICE-RL freezes a pretrained generative policy and learns a latent-conditioned residual that corrects its action chunks. Selective regularization, multi-sample training, best-of-N selection, action chunking, and adaptive data mixing make exploration more controlled and finetuning more stable.

Residual Policy

  • The pretrained flow-matching policy remains fixed while RL learns a lightweight residual correction instead of differentiating through the generative model.Source: §4 · p. 3
  • The residual conditions on the same latent noise as the base policy, making each correction explicitly aware of its proposed action chunk.Source: §4 · p. 3
  • Residual action chunks combine the pretrained action with a correction, preserving stochastic generation while enabling controlled deviations.Source: §4 · p. 3

Selective Behavior Regularization

  • The actor combines critic-value maximization with BC-style regularization that encourages exploration within the pretrained policy’s support.Source: §4 · p. 3

Training Procedure

  • Action chunking reduces decision frequency and improves temporal consistency, while adaptive offline-to-online mixing shifts learning toward online experience after warm-start.Source: §4 · pp. 3–4

Exploration and Training

  • Multi-sample expectation training averages objectives over latent-induced candidates, and best-of-N selection executes the highest-valued sampled action.Source: §4 · p. 4

Selective Behavior Regularization

  • The BC penalty is selectively relaxed when value improvement is reliable, while critic consistency guards against overestimation-driven drift.Source: §4 · p. 4

5. Experiments

DICE-RL is evaluated against prior RL finetuning methods, across simulation and real-robot manipulation settings, with analyses of finetunability, distribution sharpening, contraction, robustness, and design choices. It achieves strong, stable, sample-efficient finetuning while improving performance on challenging tasks.

5.1. Comparison to RL Finetuning Algorithms

  • DICE-RL attains the highest final performance across Robomimic tasks while remaining more stable and sample efficient across difficulty levels.The comparison includes Can, Square, Transport, and Tool Hang with state- and pixel-based observations.Source: §5.1 · pp. 4, 6
  • Starting from 45% success on Tool Hang, DICE-RL surpasses 90% success within roughly 2000 online episodes using only 50 demonstrations.The result is reported for both state and pixel inputs.Source: §5.1 · p. 6

5.2. What makes a pretrained policy easy to finetune?

  • Pretrained-policy finetunability depends on action-distribution structure: intermediate checkpoints balance high GOODCOV with reduced BADENT better than later checkpoints.Later checkpoints suppress low-value variability further but trade away too much GOODCOV.Source: §5.2 · pp. 6–7

5.3. Understanding DICE-RL

  • RL reshapes the pretrained action distribution toward high-value actions, with larger value gains accompanying larger entropy reductions.The effect is strongest at critical states such as pre-insertion and insertion.Source: §5.3 · p. 7
  • RL-policy rollouts show more stable and typically smaller pairwise-distance evolution than pretrained BC and expert rollouts, indicating stronger closed-loop contraction.Under action-noise probabilities from 0.1 to 0.9, the RL policy also degrades more gracefully than the pretrained BC policy.Source: §5.3 · pp. 7–8

5.4. DICE-RL on Real Robot

  • On real-robot tasks, DICE-RL reliably finetunes policies around dominant BC failures such as pulley transitions, belt seating, navigation, and insertion.The largest value gains and entropy reductions occur at states corresponding to these failure modes.Source: §5.4 · pp. 8–9

5.5. Ablation Studies

  • The BC-loss filter, multi-sample expectation training, and best-of-N action selection generally accelerate convergence and improve peak finetuning performance.DICE-RL also finetunes stably and sample efficiently with both diffusion and flow-matching pretrained backbones.Source: §5.5 · p. 9

6. Conclusion

DICE-RL is presented as a stable, sample-efficient framework that contracts pretrained generative policy distributions around high-value behaviors. It achieves strong performance across tasks and observation modalities, including a real-world belt assembly task.

  • DICE-RL sharpens and contracts pretrained action distributions around high-value behaviors during RL finetuning.Source: §6 · p. 9
  • The framework is described as stable and sample-efficient for finetuning generative behavior cloning policies.Source: §6 · p. 9
  • DICE-RL achieves strong performance across tasks and observation modalities, including a challenging real-world belt assembly task.Source: §6 · p. 9

Impact Statement

DICE-RL improves the stability and sample efficiency of RL finetuning for pretrained generative robot policies, with validation in simulation and on a real robot. Deployment in safety-critical settings still requires safeguards because policies may inherit dataset biases or behave unsafely under distribution shift.

  • DICE-RL improves RL finetuning stability and sample efficiency for pretrained generative robot policies across simulation and real-robot validation.Source: Impact Statement · p. 10
  • Policies may inherit dataset biases or exhibit unsafe behavior under distribution shift, requiring careful evaluation and safeguards before safety-critical deployment.Source: Impact Statement · p. 10

B. Detailed Ablations Results

Ablations show that DICE-RL’s components improve sample efficiency and stability, with their effects depending on task horizon, precision, and pretrained-policy architecture. Across hyperparameters, the procedure remains robust while benefiting from task-appropriate regularization and credit assignment.

C. Implementation Details - Simulation

Simulation uses flow-matching behavior cloning with visual or state encoders, followed by off-policy residual actor–critic finetuning over action chunks. The implementation uses frozen pretrained components, critic ensembles, Polyak targets, and fixed update-to-data ratios.

D. Implementation Details - Real Robot

The real-robot implementation uses a CLIP-pretrained ViT diffusion policy with synchronized visual and proprioceptive inputs, then applies asynchronous residual RL updates. Its critic ensemble, BC regularization, and scheduled offline-data ratio are configured for online robot finetuning.

Related papers

Loading 2603.10263v2…