PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Action Chunking

Qiyang Li, Zhiyuan Zhou, Sergey Levine

arXiv:2507.07969v4cs.LGcs.AIcs.ROstat.ML

TL;DR

Offline-to-online RL must use prior data to explore efficiently despite behavior mismatch. Q-chunking runs TD-based RL over action sequences, and experiments show strong offline performance and online sample efficiency across challenging long-horizon tasks. The approach is subject to action-chunk-size selection and additional best-of-N sampling costs.

  • Problem

    Offline-to-online RL lacks an obvious way to use prior offline data to acquire an effective exploratory policy for long-horizon, sparse-reward tasks.

  • Method

    Q-chunking runs RL in a chunked action space, using temporally coherent action sequences and critics that support unbiased n-step backups.

  • Results

    Q-chunking achieves strong offline performance and online sample efficiency, outperforming prior offline-to-online methods across six challenging long-horizon, sparse-reward domains.

  • Takeaways & Limitations

    Q-chunking provides a simple recipe for leveraging offline behavior during online exploration while speeding value backup through temporally extended actions.

  • Takeaways & Limitations

    The method requires choosing a fixed action-chunk length, and QC incurs additional computational costs from best-of-N sampling.

Abstract

from arXiv · show

We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a 'chunked' action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased $n$-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.

1 Introduction

Offline-to-online RL seeks to use prior data for efficient exploration, but mismatched offline behavior makes this difficult. Q-chunking addresses this by running TD-based RL over action sequences, combining temporally coherent exploration with unbiased n-step backups.

  • Motivation: Offline-to-online RL uses prior offline data to improve online exploration, but the data distribution may not match the learned policy.Exploration from scratch can be prohibitively expensive in complex environments.
  • Action Chunking: Action chunking predicts short sequences of future actions instead of a single action from each state observation.The technique is widely used in imitation learning and can better handle non-Markovian offline behavior.
  • Q-Chunking: Q-chunking runs reinforcement learning in a chunked action space to obtain temporally extended, non-Markovian skills for exploration.The approach is intended to leverage temporally consistent behavior from offline data.
  • Q-Chunking: The temporally extended critic estimates the value of executing a whole action sequence, enabling unbiased n-step backups for TD learning.The sequence length determines the effective backup horizon.
  • Contributions: QC instantiates Q-chunking as a practical offline-to-online algorithm and QC-FQL applies the recipe to FQL with minimal modification.The methods are evaluated on long-horizon, sparse-reward domains.

2 Related Work

Prior offline-to-online and temporally extended-action methods address exploration, behavior mismatch, or multi-step value learning separately. Q-chunking instead uses an expressive, continuously parameterized action-sequence policy that is trained and fine-tuned with RL.

  • Offline-to-online RL: Offline-to-online methods commonly pretrain on offline data and continue optimizing with a replay buffer, but pessimism can hinder online exploration.Methods that adjust pessimism may be difficult to tune and can still have limited sample efficiency.
  • Action Chunking: Imitation-learning action chunking predicts and executes action sequences open loop, while existing RL approaches often fine-tune imitation-pretrained policies.Action chunking has been associated with robustness and handling non-Markovian offline behavior.
  • Exploration with Temporally Coherent Actions: Exploration methods based on correlated noise, hierarchical policies, or frozen skills face heuristic, stability, or fine-tuning limitations.Q-chunking represents temporally extended actions with a single policy network.
  • Hierarchical Reinforcement Learning: Hierarchical reinforcement learning uses high-level policies to select among low-level policies, but jointly updating both levels can create a moving objective and instability.The options framework additionally learns initiation and termination conditions.
  • Q-Chunking: Unlike prior approaches using discrete or predefined action sequences, Q-chunking uses a flow-matching policy to parameterize a continuous action-sequence space and train it online.This distinguishes the method from heuristic sequence extraction and fixed sequence sets.
  • Multi-step Latent-space Planning: Multi-step latent-space planning methods optimize short-horizon sequences through learned dynamics and value models.They are presented as a model-based RL approach to action-sequence search.

3 Background

Offline-to-online RL pretrains on a prior dataset before online fine-tuning to maximize discounted return. TD learning propagates value locally, while n-step returns accelerate propagation but can be biased off policy.

  • Offline-to-online RL: The offline-to-online setting assumes an infinite-horizon, fully observable MDP with a prior dataset of transition rollouts.The objective is to maximize expected discounted cumulative reward.
  • Offline-to-online RL: Offline-to-online algorithms typically use an offline pretraining phase followed by online fine-tuning with environment interactions.The online dataset grows with new interactions while retaining prior offline data.
  • Temporal Difference Learning: TD-based algorithms learn Qθ(s, a) with a temporal-difference loss that estimates value from sampled off-policy transitions.A delayed target network is used for learning stability.
  • Temporal Difference Learning: The standard 1-step estimate bootstraps from the immediate reward and the next-state action value under the current policy.The target uses a delayed critic and blocks gradient flow through it.
  • Multi-step Returns: As the effective horizon grows, 1-step TD learning slows because each backup propagates value backward by only one timestep.Longer trajectory segments motivate multi-step returns.
  • Multi-step Returns: n-step returns propagate value backward across n steps, but their estimates can be biased when the data-collection policy differs from the current policy.Despite this bias, their implementation simplicity has led to widespread adoption.

4 Q-Chunking

Q-chunking applies TD-based RL in a temporally extended action space, combining chunk-level value learning with behavior constraints that preserve coherent offline behaviors. Its practical variants use learned flow policies and either implicit KL or Wasserstein constraints, while Gaussian-policy chunking can perform poorly.

  • Q-learning on a temporally extended action space: Q-chunking trains the actor and critic on h-step action sequences rather than individual actions.The policy predicts and executes h actions open loop, while the critic estimates the value of the complete sequence.
  • Q-learning on a temporally extended action space: Q-chunking retains h-step value-propagation speed while avoiding the off-policy bias of conventional n-step returns.Its h-step critic evaluates the same actions that generated the intervening rewards, yielding an unbiased backup according to the paper.
  • Behavior constraints for temporally coherent exploration: A behavior constraint limits policy deviation from the offline behavior distribution in the chunked action space.This is intended to preserve temporally coherent sequences, which can represent non-Markovian structure and support structured exploration.
  • Behavior constraints for temporally coherent exploration: Gaussian policies may inadequately capture action-sequence behavior, and naïvely adding action chunking with behavior cloning can perform poorly.Figure 2 compares RLPD, RLPD-AC, and QC-RLPD; the text identifies Gaussian-policy modeling as an implementation challenge.
  • Practical implementations: QC learns a flow-matching behavior policy and uses best-of-N sampling to impose an implicit KL constraint while selecting high-value action chunks.The selected chunks are used both for environment interaction and for TD backups, avoiding separate policy parameterization; QC-FQL instead uses a 2-Wasserstein constraint.
  • Practical implementations: QC incurs additional computational cost from best-of-N sampling, motivating cheaper QC-FQL and QC-IFQL variants.The paper presents QC-FQL because it performs better empirically among the cheaper alternatives considered.

5 Experimental Results

Experiments evaluate Q-chunking across long-horizon, sparse-reward manipulation tasks, comparing it with prior offline-to-online methods and testing mechanisms and design choices. Q-chunking shows strong offline performance, improved online sample efficiency, more coherent exploration, and sensitivity to chunk length and critic ensemble size.

  • Comparisons: Q-chunking methods outperform prior offline-to-online methods at the end of online training on OGBench.Table 1 reports offline performance after 1M training steps and online performance after 1M additional online steps.
  • Experimental setup: Experiments span six sparse-reward robotic manipulation domains from OGBench and robomimic, covering tasks with varying difficulties.OGBench contributes five domains with five tasks each, while robomimic contributes three tasks.
  • Why action chunking helps exploration: QC exhibits more diverse early state coverage and higher end-effector temporal coherency than BFN on cube-triple-task3.The analysis compares trajectories during the first 1000 environment steps and measures coherency using adjacent end-effector-position differences.
  • Comparisons: On the two hardest OGBench domains, cube-triple and cube-quadruple, QC shows especially strong online sample efficiency and outperforms prior methods by a large margin.The advantage is reported as particularly large on cube-quadruple.
  • Comparisons: Across OGBench and robomimic, Q-chunking methods outperform corresponding 1-step TD and n-step return baselines that do not use chunked critics or policies.The same trend appears in robomimic ablations, where both Q-chunking methods consistently outperform their n-step and 1-step counterparts.
  • Sensitivity analysis: Increasing QC-FQL’s chunk length helps up to h = 10, while h = 50 achieves no success; using K = 10 critics improves both QC and BFN.The authors use h = 5 and K = 2 in other experiments because these settings generally perform well or are cheaper to run.

6 Discussions

Q-chunking integrates action chunking into offline-to-online RL and is reported to improve value backup and online exploration on challenging long-horizon tasks. Its main scope boundaries are fixed chunk sizes and limited support for settings requiring high-frequency feedback.

  • Discussion: Q-chunking integrates action chunking into offline-to-online RL through a simple recipe.The broader approach is presented as a practical way to train non-Markovian policies for online exploration from offline data.
  • Limitations: The approach uses a fixed action-chunk size, and choosing that size currently requires task-specific hyperparameter tuning.Automatically determining chunk boundaries is identified as a future research direction.
  • Limitations: Action chunking covers only a limited subclass of non-Markovian policies and may perform poorly when high-frequency control feedback is essential.The paper proposes training more general non-Markovian policies as a future direction for improving online sample efficiency.
  • Theoretical justification: The chunked Q-function makes n-step return backups unbiased for the same policy’s ground-truth Q-value when the terminal value estimate is unbiased.The proposition contrasts this with n-step backup in the original action space, which does not converge to the correct chunked Q-value.

B.1 OGBench environments.

The experiments cover challenging sparse-reward manipulation environments from OGBench and robomimic, with tasks involving sequential manipulation, button-state puzzles, and moving multiple cubes.

  • Five OGBench manipulation domains include sparse scene and puzzle tasks plus cube-double, cube-triple, and cube-quadruple tasks.Scene and puzzle rewards are −1 until completion and 0 upon completion; cube tasks involve placing two, three, or four cubes.
  • Table 2 records dataset size, episode length, and action dimension; OGBench actions have dimension 5, while robomimic square actions have dimension 7.
  • Scene-sparse tasks require sequential operations such as unlocking controls, moving drawer and window components, and placing a cube.
  • Puzzle-3x3-sparse uses a 3 × 3 button grid whose presses flip selected and adjacent button colors toward a target configuration.
  • The robomimic evaluation uses lift, can, and square tasks, requiring cube pickup, can placement, and precise nut placement respectively.The datasets contain 300 successful trajectories collected by six human operators, and the tasks use binary completion rewards.

C Implementation Details

The implementation applies Q-chunking to temporally extended action sequences, combining chunk-valued critics and policies with behavior modeling, regularization, and multi-step TD updates.

  • QC-FQL: Q-chunking extends FQL by representing the behavior policy, one-step noise-conditioned policy, and Q-network over action chunks.The critic takes a state and action sequence, while the policies generate action sequences conditioned on state and noise or flow variables.
  • QC-FQL: The policy samples Gaussian noise and maps it through a one-step noise-conditioned network to produce an h-action sequence.
  • QC-FQL: Behavior regularization controls adherence to offline behavior, with larger α imposing stronger regularization.
  • Baselines: The FQL-n baseline replaces FQL’s one-step TD update with an h-step TD update while retaining its actor and flow-matching losses.
  • BFN: BFN methods sample multiple behavior-policy action chunks and select the chunk with the highest Q-value for policy execution and TD targets.
  • Baselines: RLPD baselines use mixed offline and online replay, while QC-RLPD additionally incorporates behavior cloning in the actor loss.

D.2 Evaluation protocol

Evaluation uses multiple random seeds, confidence intervals, and success-rate estimates, with baseline-specific hyperparameter sweeps and reporting based on selected settings.

  • Experiments use four seeds per OGBench task and five seeds per robomimic task, with plots showing 95% confidence intervals.Success rate is measured over 50 environment episodes.
  • QC, BFN, and BFN-n tune the expected-max Q operator’s sample count N over {2, 4, 8, 16, 32, 64, 128}.The best setting is selected per domain and method on task2, then reported across tasks.
  • QC-FQL, FQL, and FQL-n tune behavior regularization α using fractions and multiples of FQL’s default, with larger values additionally swept on robomimic.
  • RLPD variants sweep clipped double Q-learning and entropy backup choices, using K = 10 critics for consistency with the original RLPD hyperparameter.
  • The remaining baselines use specified fixed or swept settings, including QC-RLPD α = 0.01, SUPE-GT KL coefficient 0.003, and ReBRAC behavior regularization 0.1.
  • IFQL uses N = 32 and τ = 0.9, while IQL uses τ = 0.9 and task-dependent α values.

E Full Results

The section presents end-effector trajectory visualizations for QC and BFN on cube-triple-task3 across early and late online-training periods.

  • Early training: Figure 8 visualizes the first 9000 environment steps as nine consecutive 1000-step end-effector trajectories.
  • Late training: Figure 9 visualizes another 9000-step trajectory window late in training, spanning environment steps 900000 to 909000.
  • The examples compare trajectory rollouts from QC and BFN over online training on cube-triple-task3.

E.2 OGBench results by individual task

The OGBench results are broken down by task, domain, baseline, and ablation, including comparisons involving n-step returns, Gaussian policies, and action-chunking sizes.

  • Main results by task: Table 1 and Figure 10 provide the performance breakdown for all methods across individual OGBench tasks.The results use 4 seeds and report 95% confidence intervals.
  • Selected baselines: Figure 11 compares selected baselines, including BFN, FQL, RLPD, and RLPD-AC, with summaries by domain and plots by task.
  • n-step return ablation: Figure 12 reports the individual-task results for the n-step return ablation.
  • Gaussian policies: Figure 2 results include Q-chunking with Gaussian policies and the QC-RLPD variant, which adds behavior cloning to RLPD-AC.
  • Action chunking size ablations: Increasing action-chunking size generally helps until h = 25, which is the only tested size yielding non-trivial success on cube-triple-play-task5.
  • Action chunking size ablations: On cube-quadruple-play-task4, FQL-n can initially succeed but quickly collapses, whereas QC-FQL solves the task consistently for h ∈{5, 7, 9}.The comparison covers results over 5 seeds.

E.3 Robomimic ablation results

The robomimic ablations report task-level comparisons, runtime measurements, and parameter counts for Q-chunking and baseline methods. QC-FQL has comparable runtime, while QC is more expensive, especially offline.

  • Task results: Figure 16 breaks down QC, QC-FQL, BFN-n, FQL-n, BFN, and FQL performance across three robomimic tasks.Each method-task combination uses 5 seeds.
  • Runtime: QC-FQL has comparable offline and online runtime to baselines such as FQL and RLPD on cube-triple-task1.
  • Runtime: QC is slower offline because each agent-update training example samples 32 actions, whereas BFN samples 4 actions.
  • Runtime: QC is around 50% more expensive online than other methods when using one gradient update per environment step.
  • Parameter count: Table 7 compares parameter counts on cube-triple-* with h = 5 for all Q-chunking methods.RLPD uses K = 10 critic networks, while other methods and baselines use K = 2.
Loading 2507.07969v4…