PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev, Animesh Garg

arXiv:2608.21204v1cs.ROcs.LG

TL;DR

BC policies cannot learn from deployment failures, while RL fine-tuning is difficult to scale to modern robot policies. Q-Planning adds a small off-policy Q-function to a frozen BC policy, using Q-guided selection and Q-only self-improvement. Across simulated and real-robot tasks, repeated deployment-rollout updates improve performance without human intervention or BC updates.

  • Problem

    BC policies are bounded by successful demonstration data, while RL fine-tuning is expensive and brittle for multi-billion-parameter policies.

  • Method

    Q-Planning freezes a large BC policy, trains a small off-policy Q-function on successful and failed rollouts, and selects actions through a single-step Q-weighted average over BC draws.

  • Results

    Ten iterations improve every tested simulated benchmark, including LIBERO-10 93% to 99% and RoboTwin 83.8% to 91.4%, while five iterations raise stack-cups 40% to 90% and insert-wallet 25% to 80%.

  • Takeaways & Limitations

    Q-Planning enables self-improvement from deployment failures while scaling updates with the Q-function rather than the frozen BC policy.

  • Takeaways & Limitations

    Q-Planning cannot learn behaviors the BC policy cannot produce with non-negligible probability, so gains depend on BC quality and diversity.

Abstract

from arXiv · show

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

1 Introduction

Large-scale BC policies achieve strong manipulation performance but cannot learn from deployment failures, while direct RL fine-tuning is costly and brittle at VLA scale. Q-Planning addresses this by freezing the BC policy, adding an off-policy Q-function, and using value-guided selection plus Q-only self-improvement.

  • Motivation: BC policies are bounded by their demonstration data and cannot improve from deployment failures without additional human teleoperation.This demo ceiling becomes increasingly limiting as policies scale to billions of parameters and millions of demonstrations.
  • Motivation: Direct RL fine-tuning of multi-billion-parameter policies is expensive and brittle on sparse-reward, on-policy data, while risking degradation of the BC prior.Value-based planning avoids policy updates but had not been demonstrated at VLA scale because of high-dimensional actions and long horizons.
  • Q-Planning: Q-Planning decouples the actor and critic: BC learns from successful demonstrations, while an off-policy Q-function learns from successful or failed trajectories.The expensive BC policy remains frozen, and deployment signals are absorbed by the smaller Q-function.
  • Q-Planning: Q-Planning selects among N frozen-BC action chunks using a single-step Q-weighted average with softmax weights over candidate Q-scores.The Q encoders are amortised once per planning step, leaving only the action-conditioned decoder to scale with N.
  • Contributions: Q-Planning adds an off-policy Q-function trained alongside the BC policy, a real-time value-guided action selector, and Q-only self-improvement from deployment rollouts.The self-improvement loop incorporates both successful and failed rollouts without changing BC weights.

2 Related work

Related work spans large-scale BC, value-based planning, and policy fine-tuning, but these approaches typically inherit limits in failure learning, VLA-scale search, or policy-update cost. Q-Planning positions itself as a frozen-BC, failure-driven, auxiliary-actor-free alternative.

  • Large-scale BC and VLA policies: Large-scale BC policies imitate demonstrations effectively but inherit their data ceiling and degrade on out-of-distribution states.Q-Planning instead treats BC as a frozen action-proposal distribution and supplies it with a value function.
  • Value functions and planning for control: Value-based planners perform well in low-dimensional or simulated settings, but VLA-scale deployment is hindered by expensive online search over high-dimensional action spaces.Prior value-guidance work does not address active self-improvement through online iteration.
  • Self-improvement and RL fine-tuning: Existing online and offline RL methods generally update policy parameters, which is expensive for large VLAs and risks degrading the BC prior.Q-Planning instead updates only the Q-function while leaving the BC policy untouched.
  • Positioning: Q-Planning is the only compared method combining a frozen BC policy, multi-billion-parameter compatibility, failure-rollout self-improvement, and no auxiliary actor.The comparison distinguishes it from offline value guidance, auxiliary-actor methods, fresh RL actors, and direct policy fine-tuning.

3 Method

Q-Planning combines a frozen BC policy with an off-policy Q-function, using value-guided action selection and Q-only updates from deployment rollouts. Its design supports self-improvement from both successful and failed trajectories without updating the BC policy.

  • Overview: Q-Planning uses three components: an off-policy Q-function over action chunks, Q-weighted averaging of BC draws, and a Q-only self-improvement loop.The base BC policy remains frozen throughout the method.
  • Off-policy Q-function over action chunks: Q-chunking treats each length-H action chunk as a super-action, reducing the effective bootstrapping horizon by a factor of H.The method uses chunked transitions and a one-step-shifted target during training.
  • Off-policy Q-function over action chunks: The Q-function is trained on chunked transitions from demonstrations and later online rollouts, including failed trajectories whose targets are pinned to zero.HL-Gauss categorical regression replaces direct squared-error minimization to stabilize learning under sparse, bimodal returns.
  • Q-weighted action selection: At each planning step, N action chunks are sampled from the BC policy, scored by Q, and combined through a softmax-weighted average.Flow-matching draws preserve multiple plausible action modes while remaining on the BC policy’s action manifold.
  • Q-weighted action selection: The planner evaluates all candidates in one batched Q forward pass, with shared vision-language encoding and decoder computation scaling with N.On RoboTwin, N=32 gives a 400 ms planning step, 1.6× faster than a single 10-step BC inference and within the 960 ms replanning budget.
  • Self-improvement loop: Each self-improvement iteration collects planner rollouts, appends them to replay buffer D, and performs Q-only updates while keeping the BC policy frozen.The procedure uses M=100 episodes per task per simulation iteration, 20 on hardware, and S=200 Q-only gradient steps.

4 Experimental results

Across simulation and real-robot evaluations, Q-Planning improves a frozen BC policy through Q-guided selection and Q-only self-improvement, outperforming alternatives under matched online budgets.

  • Q1: which action source for planning?: Q-Planning’s single-step Q-weighted average matches temporally-smoothed MPPI at 93.0% success on LIBERO-10 while running 1.7× faster and using fewer hyper-parameters.Both methods recover a +3pp gain over the frozen BC; Q-Planning is adopted for subsequent experiments.
  • Q2: online self-improvement: The offline Q-Planning column improves over the frozen BC on 4 of 5 benchmarks by +1.3pp on average.Offline evaluation uses Q-weighted selection over BC draws without self-improvement.
  • Q2: online self-improvement: Ten Q-only self-improvement iterations raise mean success from 92.1% to 97.6% and improve every benchmark tested.LIBERO-Spatial rises 91.5 →98.5%, LIBERO-10 93 →99%, and RoboTwin 83.8 →91.4%.
  • Q2: online self-improvement: On LIBERO-Object and LIBERO-Goal, successful episodes shorten from 139 to 120 and 110 to 99 steps because success was already near ceiling.Figure 2(b) shows the per-iteration trajectories of these gains.
  • Q3: comparison with alternatives: Under the same online budget, Q-Planning is the only compared method that improves stably from failures.Best-of-N plateaus at 95%, filtered SFT at 93.5%, IBRL collapses, DSRL swings between 69% and 91%, and DAWR stays below the frozen BC.
  • Q4: real-world transfer: After five iterations on real robots, Q-Planning raises stack-cups from 40% to 90% and insert-wallet from 25% to 80% without human intervention.SFT on successful rollouts alone stalls at 55% and 30%; Q-Planning uses the failure signal that SFT discards.

5 Conclusion

Q-Planning makes a frozen BC policy self-improving by training a smaller off-policy Q-function on successful and failed rollouts, while retaining real-time action selection. The approach improves all tested simulated benchmarks and two contact-rich real-robot tasks without human intervention.

  • Conclusion: Q-Planning decouples the actor and critic: BC learns from successful demonstrations, while the off-policy Q-function learns from successful or failed rollouts.Only the Q-function is updated, leaving the BC weights untouched.
  • Conclusion: Ten iterations raise mean simulated success from 92.1% to 97.6%, while five iterations raise stack-cups from 40% to 90% and insert-wallet from 25% to 80%.Successful episodes also shorten on suites where success is already at ceiling.
  • Conclusion: Because only Q is updated, self-improvement scales with the critic’s size rather than the policy’s as VLA backbones grow.The conclusion identifies this as increasingly attractive for backbones approaching 10B+ parameters.

6 Limitations

Q-Planning has three stated boundaries: it depends on behaviours the BC policy can generate, scales with candidate count, and requires per-episode success labels.

  • Q-Planning cannot learn behaviours that the BC head cannot generate with non-negligible probability.Multi-modal flow-matching draws widen the exploration envelope beyond a single Gaussian around the BC mode, but tasks with no successful BC chunks remain out of reach.
  • Gains scale with the BC policy’s quality and diversity.
  • The Q decoder scales linearly with candidate count N, eventually becoming the bottleneck if N reaches the hundreds.The deployed configurations use N=64 on LIBERO and N=32 on RoboTwin within the frozen BC baseline’s latency budget.
  • The self-improvement loop assumes a per-episode success detector: simulation uses an environment success bit, while real-robot runs use human-provided labels.Open-ended tasks would require language-conditioned or learned success models.

A Q-function architecture

The Q-function uses parameter-disjoint visual and language encoders, a cross-attending decoder conditioned on candidate action chunks, and an HL-Gauss value head.

  • The Q-function has its own DinoV2 visual encoder and T5 language encoder, parameter-disjoint from the BC policy.
  • A transformer decoder cross-attends to visual and language tokens while using the candidate action chunk as query tokens.
  • The HL-Gauss head outputs B bin logits over discounted returns, whose softmax expectation gives the scalar Qϕ.

B Implementation details

Implementation combines chunked, categorical Q-training with fixed planning and update schedules, while freezing the large BC policy and updating only the approximately 1B-parameter Q-function.

  • The planner uses H = 32, N = 64 on LIBERO, N = 32 on RoboTwin, λ = 1, and γ = 0.99.It executes the first 10 chunk steps on LIBERO and first 24 on RoboTwin before replanning.
  • Each self-improvement iteration collects M = 100 episodes per simulated task or 20 per real-robot task, then performs S = 200 Q-only updates.Minibatches mix original demonstrations and accumulated online rollouts equally, while BC remains frozen.
  • The Q-function has approximately 1B parameters, substantially fewer than the multibillion-parameter FastWAM policy.This size difference makes Q-only self-improvement cheaper than a full-policy gradient step.
  • HL-Gauss categorical regression addresses sparse, bimodal returns, while Q-chunking reduces the effective bootstrapping horizon by H.
  • The Bellman loss bootstraps from the next buffer chunk rather than a planner-selected action, avoiding a planner call during every update.As the buffer accumulates planner-collected rollouts, the shifted target increasingly reflects planner-selected actions; the planner-selected variant showed no gain.

D Temporal-smoothed MPPI (baseline in Sec. 4.2)

Temporal-smoothed MPPI correlates perturbations across action-chunk timesteps to keep candidates coherent and near the BC trajectory manifold, recovering the baseline gain.

  • MPPI initializes its action-chunk proposal mean with the BC policy and perturbs it using additive Gaussian noise.
  • A Gaussian kernel with σ > 0 correlates noise across neighbouring timesteps, producing temporally coherent chunks closer to the BC trajectory manifold.The implementation uses σ = 2 over H = 32; σ →0 recovers vanilla MPPI with independent timestep perturbations.
  • +3pp is recovered by temporally smoothed MPPI, because coherent perturbations stay closer to the BC trajectory manifold.
  • MPPI weights candidates using Q-scores and temperature λ to update the proposal mean across T iterations.After T iterations, the converged mean is executed; Q-Planning is the T=1 limit using multi-modal flow-matching draws instead of smoothed noise.

E Planning-step latency profile

Q-Planning’s deployed planner meets the real-time replanning deadlines under strict evaluation settings, with cost concentrated in the action-conditioned Q decoder rather than the shared encoders.

  • RoboTwin: 42% of the 960 ms RoboTwin budget is used by the deployed N=32 planner, while the earlier iterative variant used 133%.The earlier variant would have missed the deadline.
  • LIBERO: 640 ms for deployed N=64 on LIBERO essentially matches the 646 ms cost of one 10-step frozen-BC draw.Reducing to N=16 fits the 333 ms LIBERO budget with comparable success.
  • Scaling: 23–27 ms encoder execution is amortised once per planning step, while the approximately 500M-parameter Q decoder adds roughly 2–3 ms per candidate.This scaling explains why doubling N does not double total planning time.

F Real-robot setup

The real-robot evaluation uses two bimanual 6-DoF arms on contact-rich tasks, randomised starts, and autonomous deployment rollouts without teleoperation. Q-Planning is compared with full policy fine-tuning that uses only successful rollouts.

  • Platform and control: Two table-mounted, forward-facing 6-DoF YAM arms execute joint-space commands from 30-step action chunks with 1-second replanning.RGB observations come from overhead and wrist cameras, and inference runs on one RTX 5090.
  • Data and evaluation protocol: Each task begins with 100 demonstrations, randomises initial conditions within the demonstration distribution, and collects 20 episodes per self-improvement iteration.The reported per-iteration success rates are measured on these collection episodes.
  • Baseline: Q-Planning uses all rollout data for Q-only updates, whereas the SFT baseline fully fine-tunes the policy on successful rollouts and discards failures.The comparison uses the same episode budget.
Loading 2608.21204v1…