PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?

Filippo Lazzati, Kyle Stachowicz, William Chen, Alberto Maria Metelli, Andrew Wagenmaker, Sergey Levine

arXiv:2608.02547v1cs.RO

TL;DR

Action chunking is widely used in robotic behavioral cloning, but why it improves performance remains incompletely understood. Through theoretical analysis and simulated and real-world experiments, the paper identifies delayed policies and implicit ensembling as explanations, and finds that explicit ensembles of delayed policies can outperform action chunking in many settings.

  • Problem

    Although action chunking is widely used in robotic control, the mechanisms behind its performance benefit remain poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The paper combines theoretical analysis and controlled experiments with comparisons of action chunking, delayed policies, and explicit ensembles of delayed policies.

    Source: §1 · pp. 2–3

  • Results

    Delayed policies often capture action chunking’s non-Markovian and compounding-error benefits, randomized-delay ensembles match its performance across considered settings, and explicit delayed-policy ensembles exceed it in many settings.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Action chunking’s benefits are not fully explained by temporal consistency, horizon reduction, or representation learning; implicit ensembling provides an additional explanation.

    Source: §1 · p. 2, §8 · p. 14

    A simple delay captures much necessary non-Markovian behavior, but whether full history conditioning captures compounding-error reduction or implicit ensembling remains unclear.

    Source: §8 · p. 14

Abstract

from arXiv · show

Action chunking---predicting and executing multiple actions instead of a single action---has proven to be a critical component for learning effective robotic control policies. However, our precise understanding of why action chunking improves performance has remained limited. In this work we seek to close this gap. Through rigorous experimental evaluations in both simulated and real-world settings, we show that existing hypotheses for the success of action chunking---temporal consistency, horizon reduction, and representation learning---fail to explain the success of action chunking. Instead, we find that action chunking benefits from greater non-Markovian expressivity and reduced compounding error compared to Markovian policies, but, in many settings of interest, these effects can be fully captured by delayed policies, which at each step predict a single action based on the observation $k$ steps in the past. We then show that there exists an additional benefit of action chunking that we refer to as implicit ensembling. In particular, by learning a diversity of temporal relationships (that is, $a_t | o_t, a_t | o_{t-1}, \ldots$), action-chunked policies exhibit behavior matching that of a model ensemble, increasing their robustness and generalization ability over policies that only learn a single temporal relationship. Building on these insights, we show that in simulated and real-world robotic control settings, we can match the performance of action chunking without action chunking---by deploying an action chunking policy as an ensemble of policies with randomized delays. Furthermore, we propose a policy class that amplifies the benefits of action chunking by explicitly instantiating an ensemble, and which we show significantly improves over the performance of action chunking in many domains.

1 Introduction

Action chunking is widely used in robotic behavioral cloning, but the mechanisms behind its success remain poorly understood. The paper argues that delayed policies explain much of its benefit, while implicit ensembling explains additional gains; explicit ensembles of delayed policies can exceed action chunking in many settings.

1 Introduction

  • Action chunking predicts and executes action sequences rather than a single action, and typically outperforms recomputing one action at each timestep.At deployment, chunked policies may execute actions open-loop using a stale observation.Source: §1 · pp. 1–2
  • Temporal consistency, horizon reduction, and representation learning are common hypotheses, but the paper finds they do not explain action chunking’s success.The paper instead identifies non-Markovian delays, reduced compounding error, and ensemble-like effects as the key factors.Source: Figure 1 · p. 2, §1 · p. 2
  • Delayed policies often capture the non-Markovian behavior and compounding-error reductions that action chunking provides, making chunking’s temporal consistency and horizon reduction unnecessary in many settings.The paper evaluates this claim through theoretical analysis and controlled experiments in simulated and real-world robotic manipulation.Source: §1 · p. 2
  • Implicit ensembling is an additional mechanism: action-chunked policies learn diverse temporal relationships and can be deployed as randomized-delay ensembles that match action-chunking performance.The paper finds that delayed-policy explanations alone do not fully account for action chunking’s benefits in some settings.Source: §1 · p. 2
  • Explicit ensembles of delayed policies exceed action-chunking performance in many settings.The paper also reports that deploying an action-chunking policy as a randomized-delay ensemble can match action chunking in simulated and real-world control.Source: §1 · pp. 2–3

2 Related Work

Related work establishes the broad use of behavioral cloning and action chunking in robotics, while distinguishing this paper’s explanatory focus from prior methods. Prior theory studies control-theoretic conditions and compounding error, whereas other approaches modify chunk sampling, length, or test-time use.

2 Related Work

  • Behavioral cloning is widely used in robotics, and nearly every large-scale robot policy effort since ACT predicts and executes chunked actions.This paper focuses on why chunking helps and how to move beyond it, rather than demonstrating another behavioral-cloning application.Source: §2 · p. 3
  • Prior work improves action chunking through temporal-consistency sampling, mixed chunk lengths, adaptive chunk selection, or test-time switching.The passage also distinguishes receding-horizon policies, which execute only the first action, from chunk execution.Source: §2 · p. 3
  • Prior control-theoretic analyses study when action chunking helps, including how stability conditions can let it mitigate compounding error, while this paper finds additional practical effects.The cited works are Simchowitz et al. [50] and Zhang et al..Source: §2 · p. 3

3 Preliminaries

The paper frames behavioral cloning in potentially non-Markovian, partially observed environments and learns policies from successful demonstrations. It defines Markov, chunked, and delayed policies, and uses validation error to assess data fit rather than task success.

3 Preliminaries

  • The setting permits non-Markovian, partially observed environments, and the paper notes that most conclusions also hold in Markovian, fully observed settings.Episodes proceed through observations and actions under a transition kernel over histories, and success is measured by expected episode reward.Source: §3 · pp. 3–4
  • Behavioral cloning trains a policy by supervised learning to mimic demonstration actions; the experiments parameterize the policy as a diffusion model.The demonstrations are successful, though not necessarily optimal, and are split into training and validation sets.Source: §3 · p. 4
  • Action-chunked policies predict the next k demonstrator actions and execute some or all of the sequence open-loop before querying again.The chunk is a sequence of actions produced from an observation.Source: §3 · p. 4
  • The notation distinguishes chunk length, partial chunk execution, and single-step delayed policies conditioned on an observation from d steps earlier.The paper also defines delayed execution induced by selecting an action from a chunk predicted using a past observation.Source: §3 · p. 4
  • Validation error is mean-squared error of mean policy actions on held-out data, used to measure data fit rather than task success.The measure does not capture multimodality, which the authors say is not a major shortcoming in their experiments because their diffusion policies showed little multimodality.Source: §3 · pp. 4–5

4 Do Existing Hypotheses Explain the Performance of Action Chunking?

This section tests whether temporal consistency, horizon reduction, and representation learning explain action chunking’s performance, using Libero-90 experiments and theoretical analysis. It finds that delayed policies often reproduce chunking’s benefits, while the three hypotheses do not fully explain those benefits.

4.1 Hypothesis 1: Temporal Consistency

  • On Libero-90, actions are best predicted from o_t-10, while delayed policies match or exceed chunked-policy success, making temporal consistency unnecessary.Delayed policies capture the demonstrators’ relevant non-Markovian behavior.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Hypothesis 2: Horizon Reduction

  • Across tasks, non-Markovianity poorly predicts chunking’s success gains; some tasks benefit despite similar validation losses for Markovian and non-Markovian policies.This indicates that non-Markovian expressivity only partially explains chunking’s success.Source: Figure 4 · p. 6, §4.2 · p. 8
  • At drawer decision boundaries, demonstrators pause before changing direction; delayed or chunked policies can model the wait when delay or chunk length covers it.The Markovian policy has higher prediction error around timestep 80 and can deviate from the demonstrated behavior at the boundary.Source: §4.2 · p. 7, Figure 6 · p. 7
  • Under deterministic, smooth dynamics, delayed policies achieve action chunking’s same compounding-error reduction, so horizon reduction is not needed to explain it.The analysis attributes the reduction primarily to conditioning on past observations.Source: §4.2 · pp. 8–9

4.3 Hypothesis 3: Representation Learning

  • Action chunking provides representational benefits over single-step training, but no meaningful benefit over delayed-policy training; its representation benefit is limited to short prediction horizons.The delayed policy can again reproduce the benefit for actions predicted with more substantial delays.Source: §4.3 · p. 9, Figure 7 · p. 10

5 Action-Chunked Policies as Implicit Ensembles

Action-chunked policies learn multiple temporal relationships, whose implicit ensemble improves action prediction beyond any single delayed policy. Randomized-delay ensembles nearly match or outperform action chunking across the evaluated settings.

5 Action-Chunked Policies as Implicit Ensembles

  • On Libero, delayed policies capture the benefits of action chunking, but on Robomimic they do not generally match its success rate.The results identify settings where delayed policies alone do not explain action chunking’s performance.Source: §5 · p. 10
  • The temporal ensemble averages predictions induced by the action-chunked policy, while the true ensemble averages independently trained policy predictions.The induced temporal ensemble and a true ensemble have similar predictive properties, with the latter approaching lower validation loss.Source: §5 · pp. 10–11
  • The implicit temporal ensemble achieves lower validation loss than a single delayed policy and nearly matches a true ensemble.For the correct delay, a delayed policy has lower validation loss than the action-chunked policy.Source: §5 · p. 11
  • Randomized-delay deployment samples a delay at each step, leveraging multiple learned temporal relationships without relying on standard chunk execution.The proposed approach is intended to retain past-observation prediction while using the policy’s learned temporal relationships.Source: §5 · p. 11
  • Randomized-delay ensembles nearly match or outperform action chunking across settings, while action chunking significantly beats delayed policies in several benchmarks.The randomized-delay ensemble also exceeds the temporal ensemble formed by averaging predictions.Source: §5 · p. 11, Table 1 · p. 11
  • Action chunking implicitly ensembles delayed policies by learning action predictions conditioned on observations from multiple past timesteps.This ensemble-like effect improves action prediction ability and success rate.Source: §5 · p. 11

6 Randomized Delay Ensembles Match the Performance of Action Chunking in Real-World Robotic Control

In three real-world manipulation tasks, Markovian policies perform poorly and delayed policies improve on them but generally do not match action chunking. Randomized-delay ensembles match or exceed action-chunking performance.

6 Randomized Delay Ensembles Match the Performance of Action Chunking in Real-World Robotic Control

  • The evaluation used a Franka Emika arm on carrot-in-bowl, bread-toaster, and sushi-in-cup tasks.The robot used delta joint-position control at 15 Hz.Source: §6 · p. 12
  • For each task, the authors collected 50 human demonstrations and trained a diffusion policy with action chunk length 20.They evaluated action chunks, Markovian and delayed policies, and a randomized-delay ensemble.Source: §6 · p. 12
  • Randomized-delay ensembles match or exceed action-chunking performance across the real-world tasks.Markovian policies perform poorly, and delayed policies improve on them but do not generally match action chunking.Source: §6 · p. 12
  • The results support matching action-chunking performance without playing action chunks when policies predict from past observations and replicate implicit ensembling.The paper presents this conclusion for certain real-world robotic-control settings.Source: §6 · p. 13

7 Going Beyond Action Chunking with Ensembled BC Policies

Motivated by delayed-observation conditioning and implicit ensembling, the authors test explicit ensembles of independently trained behavioral-cloning policies. These ensembles improve over action chunking across the evaluated settings.

7 Going Beyond Action Chunking with Ensembled BC Policies

  • The approach is motivated by the finding that conditioning on delayed observations and implicit ensembling are key benefits of action chunking.The authors frame this evaluation as a first attempt to improve behavioral-cloning performance using those insights.Source: §7 · p. 13
  • The tested ensembles use independently trained policy members and either average their predicted actions or sample one member per action.An additional variant samples both an ensemble member and a delay at each step.Source: §7 · p. 13
  • Explicit ensembles improve over action-chunked policies across settings, with an approximately +30% boost on Robomimic Transport.The paper reports improvements for both full action-chunked ensembles and ensembles of induced delay policies.Source: §7 · p. 13

8 Conclusion

The conclusion attributes action chunking’s benefits to non-Markovian expressivity, reduced compounding error, and implicit ensembling, while identifying limits and open questions. The appendices describe additional experiments and explain the choice of validation error over diffusion loss.

8 Conclusion

  • Non-Markovian expressivity, reduced compounding error, and implicit ensembling together almost fully explain action chunking’s success, and explicit ensembles of delayed policies can improve further.These conclusions hold on standard simulated benchmarks and real-world robotic manipulation tasks.Source: §8 · p. 14
  • The authors propose testing whether adaptively selecting delays with the lowest validation error can improve on using multiple delays.Figure 2 showed lowest validation error for delays around 5–15.Source: §8 · p. 14
  • Simple delayed policies capture much of the needed non-Markovian behavior, but history conditioning may not also capture chunking’s error-reduction and implicit-ensemble effects.The authors identify combining full history conditioning with these other benefits as an open question.Source: §8 · p. 14
  • At control frequencies of 50–60 Hz, delayed policies did not replicate chunking’s success, but treating five-step sub-chunks as actions significantly improved performance.The authors report that temporal consistency matters at high frequencies and relate this to the demonstrator’s behavior frequency.Source: §8 · p. 14
  • In real-world deployment, inference pauses can shift the robot state relative to demonstrations and potentially increase compounding error; how much pause reduction helps remains an open question.Action chunking naturally reduces inference pauses.Source: §8 · p. 14

A Additional Experimental Details

  • The appendix outlines additional simulation details and notes that simulations used one Nvidia H100 GPU and ten Nvidia A100 GPUs.It covers environments and datasets, network architectures, and simulation-specific details.Source: A Additional Experimental Details · p. 20

A.1 Validation Error vs Validation Loss

  • The authors use validation error on predicted actions because diffusion loss cannot isolate individual chunk timesteps or straightforwardly ensemble predictions across timesteps.They note that validation error may not capture multimodality effectively, but their trained diffusion policies were unimodal in practice.Source: §A.1 · p. 20

A.2 Description of Environments and Datasets

The experiments use Robomimic and Libero behavioral-cloning benchmarks, with distinct task sets, demonstrations, and observation modalities.

A.2 Description of Environments and Datasets

  • Robomimic experiments cover square, can, transport, and tool hang using the PH dataset of 200 successful trajectories per task from a single experienced teleoperator.Demonstrations were collected at 20 Hz; observations use low-dimensional proprioception and ground-truth object states, with no images.Source: §A.2 · pp. 20–21
  • Libero experiments use all 90 tasks, each with 50 successful human demonstrations, and combine proprioception with front-view and wrist-camera images.Libero actions are seven-dimensional delta translation/rotation and gripper commands; observations do not include ground-truth object states.Source: §A.2 · pp. 20–21

A.3 Network Architecture

The appendix specifies diffusion-policy architectures, training and normalization choices, evaluation protocols, and additional simulation procedures. It describes validation and success-rate calculations across Libero and Robomimic experiments, but the supplied passages do not report their result values.

A.3 Network Architecture

  • The experiments train conditional DDPM diffusion policies, using MLP observation encoders for Robomimic and a ViT encoder for Libero, with task-specific versus multitask policies respectively.The denoising MLP has five layers; Libero uses a task one-hot input and two image views.Source: §A.3 · p. 21
  • Both benchmark suites use 20 denoising steps and shared DDPM hyperparameters, while Libero’s epoch counts rescale gradient iterations to be more comparable with Robomimic.Training uses 3000 epochs, AdamW, a cosine learning-rate schedule, and EMA weights; states and actions are min-max normalized, with Libero images scaled separately.Source: §A.3 · pp. 21–22

A.4.1 Evaluation Protocol

  • Evaluation uses up to 250 rollouts per Robomimic task and 40 per Libero task, with task-specific maximum episode lengths and action-sample counts for validation.The appendix also notes that delayed policies may be initialized by playing a chunk, though the supplied passage does not fully specify the procedure.Source: §A.4.1 · p. 22, §A.4 · p. 22

A.4.2 Libero Validation Loss (Figure 2)

  • For Libero validation, three policies trained on 25 demonstrations per task are evaluated across delays 0–19 using sampled mean actions and squared action error, then averaged across tasks, trajectories, and seeds.Validation skips the first 20 trajectory timesteps; the AC(n) curve uses cumulative means of delay values.Source: §A.4.2 · p. 23

A.4.3 Libero Success Rate (Figure 3)

  • Libero evaluation trained and rolled out three differently seeded policies and reported mean/SEM; the comparison uses validation averaged across three seeds, not tasks, and success averaged across 15 policy seeds (40 rollouts/task).The line is the least-squares solution over all plotted points.Source: §A.4.3 · p. 23, §A.4.4 · p. 23

A.4.5 Libero With and Without Action Chunking (Figure 7)

  • The Libero chunking comparison evaluates trained chunked, single-action, and delayed policies over 40 rollouts per task, averaging success across tasks.For one delayed policy, the first five timesteps use an action predicted from the initial observation.Source: §A.4.5 · p. 23

A.4.6 Robomimic Success Rate (Figure 8)

  • Robomimic success-rate experiments train ten policies per task and report means and standard errors across 40 policy-condition rates; Table 5 covers square PH.The evaluated conditions vary chunk and delay parameters.Source: §A.4.6 · pp. 23–24, Table 5 · p. 24

A.4.7 Robomimic Validation Loss (Figure 9)

  • Robomimic validation compares action-chunking and delay curves using ten policy seeds, while general-result tables aggregate Libero and Robomimic policy evaluations across specified seed groups and parameter values.For the ensemble validation curve, predictions average mean actions from groups of five seeds; the appendix points to Tables 5–8 for complete Robomimic results.Source: §A.4.7 · p. 24, §A.4.8 · p. 24

A.5 Additional Details for Real-World Experiments

The real-world experiments use diffusion policies and evaluate chunked, delayed, and randomized-delay approaches. Additional simulations test action chunking without temporal consistency and describe further simulation directions.

A.5 Additional Details for Real-World Experiments

  • Real-world policies use delta-joint control and 15 denoising steps to speed inference, and evaluation runs 50 trajectories per method with a one-step delay.The one-step delay allows chunk inference to run asynchronously from a past state; chunk sizes and delays were fixed in advance without tuning.Source: §A.5 · pp. 24–26
  • In real-world manipulation, randomized-delay action chunking matched or slightly outperformed action chunking, while Delay(7) outperformed Delay(2) but remained below it.The passage identifies the randomized-delay method as AC(10)-Delay(2)-RDE and the chunked comparator as AC(10)-Delay(2).Source: §A.5 · p. 26

B Additional Simulations

  • The additional-simulations appendix covers alternatives to temporal consistency, sample efficiency, dynamics smoothness, other datasets and suites, and OpenPi policies.The supplied passage outlines these topics without reporting their specific results.Source: B Additional Simulations · p. 26

B.1 Action Chunking without Temporal Consistency

  • AC(10)-Ordered, which plays marginal action predictions in sequence rather than matching the joint action distribution, matched AC(10) across all considered simulation tasks.The comparison is intended to test whether temporal consistency is required for chunking’s performance.Source: §B.1 · p. 26

B.2 Sample Efficiency in Libero

The appendix examines sample efficiency, smoothness of Robomimic dynamics, and results across additional suites, then illustrates why stationary Markov policies may fail to match non-Markovian demonstrators.

B.2 Sample Efficiency in Libero

  • As Libero demonstration data grows from 10 to 50 trajectories, the success-rate gap between AC(10) and AC(1) shrinks from 0.39 to 0.20.The gap is 0.31 at 25 trajectories; Delay(6) shows a similar pattern.Source: Figure 12 · p. 27, §B.2 · p. 27

B.3 Robomimic Dynamics are Smooth

  • In the Robomimic square trajectory, s_(h-3) + a_(h-3) usually predicts the next end-effector position, with exceptions mostly during object interactions.The appendix uses action components for translation along all three axes to illustrate the dynamics.Source: §B.3 · p. 27, Figure 14 · p. 28, Figure 13 · p. 28

B.5 Results with π0.5

  • The fine-tuned π0.5 policy’s Libero results are described as compatible with the authors’ DDPM results; the authors conjecture preprocessing that removes pauses helps explain stronger Markovian performance.The policy has an overall chunk size of 10.Source: §B.5 · p. 28, Table 15 · p. 29

C Stationary Markov Policies are Not Expressive Enough for Non-Markov Experts

  • In a finite-horizon line-world example, a demonstrator that moves consistently left or right succeeds within n steps, while behavior-cloned Markov policies produce a random walk and succeed with probability below 1.The appendix notes the difference from standard literature stems from assuming a finite horizon for success; action chunking is described as inducing longer random-walk steps that reach endpoints faster.Source: C Stationary Markov Policies are Not… · pp. 29–30, Figure 16 · p. 30

D Predicting Actions Based on Delayed States Provably Mitigates Compounding Error

Under smoothness assumptions, the section analyzes how delayed policies and action-chunked policies bound compounding error, and constructs settings where Markovian and chunked learners can fall short.

D.1 Upper Bound for Delayed Policy

  • Under deterministic dynamics and Lipschitz assumptions on dynamics, reward, and policies, the section derives upper bounds on deviation for delayed and chunked policies.The appendix cautions that the delayed-policy bound is not tight and its k-factor may be doubled.Source: D Predicting Actions Based on Delayed… · p. 30, §D.1 · pp. 31–32

D.2 Lower Bound for Markovian Policies

  • A constructed smooth R2 environment shows that a Markov learner’s bounded generalization error can still accompany growing deviation from the expert trajectory.The construction uses an expert that advances along one coordinate and a learner whose actions include a state-dependent offset.Source: §D.2 · pp. 34–35

D.3 Lower Bound for Action-Chunked Policies

  • Theorem 3 gives a constructed setting where an action-chunking learner with bounded chunkwise Wasserstein error has lower return than the Markov demonstrator by an Ω-sized gap.The theorem assumes the environment, demonstrator, and learner satisfy the stated conditions.Source: §D.3 · p. 35

E Individualized Result Plots

This section presents per-task plots of validation loss and success rates rather than averages across tasks.

E Individualized Result Plots

Loading 2608.02547v1…