Understanding Behavior Cloning with Action Quantization

Haoqun Cao; Tengyang Xie

Source-linked AI summary

Understanding Behavior Cloning with Action Quantization

Haoqun Cao, Tengyang Xie

arXiv:2603.20538v1Mar 20, 2026cs.LGstat.ML

TL;DR

The paper addresses the limited theoretical understanding of quantized actions in behavior cloning for continuous control. It analyzes quantization and statistical errors under stable dynamics and policy smoothness, and introduces a model-based augmentation. The results establish matched fundamental limits, characterize quantizer behavior, and improve horizon dependence without smoothness through augmentation.

Problem
Action quantization is widely used with autoregressive models for continuous control, but its interaction with horizon-wise error propagation and statistical complexity is poorly understood.
Method
The paper analyzes log-loss behavior cloning with quantized actions using stability and probabilistic smoothness, compares quantizers, and proposes a model-based augmentation.
Results
The analysis gives a quadratic-in-horizon regret upper bound capturing finite-sample and quantization error, while the upper bound generally matches information-theoretic limits.
Takeaways & Limitations
Binning-based quantizers are preferable to general learning-based quantizers for log-loss behavior cloning, especially when cloning deterministic policies.
Takeaways & Limitations
The main guarantees assume stable dynamics and probabilistically smooth quantized policies, although Lipschitz continuity alone may still permit exponential compounding error.

Abstract

from arXiv · show

Behavior cloning is a fundamental paradigm in machine learning, enabling policy learning from expert demonstrations across robotics, autonomous driving, and generative models. Autoregressive models like transformer have proven remarkably effective, from large language models (LLMs) to vision-language-action systems (VLAs). However, applying autoregressive models to continuous control requires discretizing actions through quantization, a practice widely adopted yet poorly understood theoretically. This paper provides theoretical foundations for this practice. We analyze how quantization error propagates along the horizon and interacts with statistical sample complexity. We show that behavior cloning with quantized actions and log-loss achieves optimal sample complexity, matching existing lower bounds, and incurs only polynomial horizon dependence on quantization error, provided the dynamics are stable and the policy satisfies a probabilistic smoothness condition. We further characterize when different quantization schemes satisfy or violate these requirements, and propose a model-based augmentation that provably improves the error bound without requiring policy smoothness. Finally, we establish fundamental limits that jointly capture the effects of quantization error and statistical complexity.

1 Introduction

The paper studies when action quantization preserves learnability in behavior cloning and how quantization interacts with statistical error over finite horizons. It establishes bounds under stability and smoothness assumptions, compares quantizers, and develops a model-based augmentation and matching lower bounds.

Motivation: Action quantization discretizes continuous actions for autoregressive behavior-cloning models, but its interaction with distribution shift and statistical error remains poorly understood.Quantization introduces deployment mismatch that can compound over long horizons even with perfect fitting.
Main results: The paper establishes regret upper bounds as functions of sample size and quantization error under stable dynamics and probabilistically smooth quantized policies.The analysis targets log-loss behavior cloning with quantized actions.
Quantizer comparison: General quantizers may have small in-distribution error yet violate smoothness and incur large regret, whereas binning-based quantizers are better behaved.The practical implication is that low training-distribution error need not predict deployment error for nonsmooth quantizers.
Model-based augmentation: A model-based augmentation bypasses smoothness requirements and improves the horizon dependence of the quantization term.The augmentation effectively copies quantized actions along an in-distribution trajectory.
Fundamental limits: The paper proves information-theoretic lower bounds depending jointly on sample size and quantization error, with upper bounds that generally match them.These results connect quantization effects with existing imitation-learning sample-complexity limits.

2 Preliminaries

The preliminaries formalize finite-horizon continuous-action MDPs, quantizers, induced policies, and log-loss behavior cloning. They show that training controls a quantized-model distribution, while deployment occurs in the original environment and therefore requires separate quantization-error analysis.

Markov Decision Process: The environment is modeled as a finite-horizon MDP with continuous state and action spaces, transition kernels, rewards, and Markov policies.The trajectory distribution and cumulative reward are defined for deploying a policy under the dynamics.
Quantization: A quantizer maps continuous actions to a finite set of representative actions, while a dequantization kernel reconstructs raw-action distributions consistent with each representative.The expert-induced kernel is supported on the corresponding quantizer preimage.
Types of quantizers: Binning-based and learning-based quantizers are analyzed through a quantization-error parameter εq and continuity assumptions under an action metric.Binning-based quantization is a special case of the learning-based formulation.
Statistical baseline: Standard maximum-likelihood guarantees yield regret rates O(H log |Π|/n) for deterministic and stochastic experts, matching prior results.These rates arise from Hellinger-distance control combined with trajectory-level guarantees.
Log-loss behavior cloning: Log-loss behavior cloning fits a quantized policy by maximum likelihood on trajectories generated under the quantized expert and composed dynamics.The learned distribution is P_ˆπ,(T ◦ρ), not directly the deployment distribution P_ˆπ,T.
Log-loss behavior cloning: Optimizing the quantized training objective controls statistical discrepancy but does not eliminate the deployment mismatch caused by executing representative actions in the original environment.This approximation effect persists even as the sample size increases.

3 Regret Analysis Under Stable Dynamic and Smooth Policy

The analysis bounds regret from quantized behavior cloning by combining statistical and quantization errors under incremental stability and policy continuity assumptions. The resulting guarantees avoid exponential error compounding and give explicit stochastic- and deterministic-expert bounds.

Stability and smoothness: Quantization creates action mismatch that propagates through dynamics, while stability mitigates amplification and policy smoothness limits suboptimal quantized actions.The analysis uses probabilistic incremental-input-to-state stability and relaxed total variation continuity to formalize these requirements.
Assumption boundary: The expert quantized policy must satisfy TVC or RTVC, and Wasserstein continuity alone is insufficient because Lipschitz policies can still incur exponentially compounding error.The paper uses thresholded 0–1 transport cost to ignore action perturbations below ε′.
General reduction: The general regret bound reduces performance loss to total variation between extended trajectory laws plus in-distribution terms evaluated under the expert.The statistical term is controlled through log-loss, while the remaining terms depend on the expert and quantizer.
Stochastic quantized experts: Under globally P-EIISS dynamics and linear-modulus TVC policies, the upper bound combines sample-size and quantization terms without exponential horizon compounding.The theorem assumes a stochastic quantized expert for the first concrete guarantee.
Deterministic quantized experts: For deterministic quantized experts, binning quantizers and kεq-RTVC policies yield a bound involving κ(γ((k + 1)εq)) and γ((k + 1)εq) + (k + 1)εq.This result assumes γ-global P-IISS and states that the 1/n rate matches Foster et al. (2024).

4 When Policies Violate Smoothness: Characterization and Model-Based Remedy

The paper characterizes when quantized policies satisfy the smoothness needed for regret guarantees and shows that non-smooth quantization can cause large deployment regret. It also introduces model-based augmentation to bypass policy smoothness and improve horizon dependence.

When Policies Violate Smoothness: Deterministic policies generally cannot satisfy TVC unless locally constant, because arbitrarily close states can induce different actions.The resulting modulus is trivial, making the associated upper bound uninformative.
When Policies Violate Smoothness: Binning quantizers preserve the uniform structure required for deterministic quantized policies to satisfy RTVC, unlike general learning-based quantizers.This yields an explicit nontrivial modulus under the proposition’s conditions.
The Price of Non-Smooth Policies: Small expert-distribution error can produce drastically different deployment state distributions and large regret, even with stable dynamics and benign unquantized policies.The failure is attributed to non-smoothness of the quantized policy.
A Model-Based Augmentation: Model-based augmentation bypasses the RTVC requirement by executing learned actions along an auxiliary trajectory generated from training data and a learned transition model.The resulting actions are then executed in the real environment from the same initial state.
A Model-Based Augmentation: The model-augmented bound has milder horizon dependence for quantization error because it avoids the additional cost of correcting OOD states through RTVC.The method requires realizability for the transition-model class and may introduce additional complexity through log |M|.

5 Lower Bound

The lower bounds show that offline imitation under action quantization faces additive statistical and intrinsic quantization errors, with horizon-dependent limits for deterministic and stochastic experts.

Two additive sources—finite-sample estimation and intrinsic quantization—remain even when either source is eliminated.
The lower bounds provide separate deterministic- and stochastic-expert constructions, while the quantization term has the same rate in both cases.
H · (errorstat + errorquantization) is generally unimprovable because the learner must recover information across all H stages.
Stochastic experts incur a statistical lower-bound term scaling as H √n, matching Foster et al. (2024).
Theorem 2, Theorem 3, and Theorem 7 match the sample-size-dependent terms, while Theorem 7 also matches the quantization-dependent term.

6 Conclusion

For fixed quantizers, the paper establishes a quadratic-in-horizon log-loss behavior-cloning upper bound combining finite-sample and quantization errors under stable dynamics and probabilistically smooth quantized policies.

A quadratic-in-horizon upper bound captures both finite-sample error and quantization error for log-loss behavior cloning on quantized actions.
The guarantee assumes a stability notion for the dynamics and probabilistic smoothness of the quantized policy class.
The paper discusses two widely used quantizers and characterizes when they satisfy the required smoothness conditions.

A Supervised Learning Guarantee

This section develops maximum-likelihood guarantees for quantized behavior cloning, connects trajectory distances under projected representations, and establishes noise-based transition representations.

The maximum likelihood estimator receives finite-sample guarantees for finite or potentially infinite model classes under log-loss covering numbers.
For deterministic experts, the paper connects log-loss analysis with trajectory-wise discrepancies between learned and expert actions.
Quantized policies have support on finite action points, allowing positive mass on expert actions and making statistical learning possible.
Total variation and Hellinger distances are preserved between extended trajectory measures and their relevant projections.
Transition kernels that are deterministic or have strictly positive densities admit invertible noise representations, with uniform noise recovering the transition law.

B.2 Stability of the Dynamic

The paper develops a probabilistic, localized stability framework for stochastic dynamics and policies, showing how shared-noise coupling controls trajectory deviations under contractive conditions.

Incremental input-to-state stability bounds state differences using a KL function and an input-gain function γ.
The proposed probabilistic stability notion extends prior open-loop and closed-loop frameworks to stochastic, localized settings without requiring global IISS.
Contractive dynamics and Lipschitz mean policies provide the core assumptions for the stability construction.
The resulting expert trajectory law is (γ, δ)-d-P-IISS, with failure probability of order O(Hεq) under stated contractivity and variance conditions.
Shared-noise coupling transfers bounded policy and system perturbations into trajectory-level stability events.

B.3 Smoothness of the Policy

The section develops policy smoothness conditions for controlling distributional changes and regret. It shows that Gaussian smoothing and quantization preserve a useful TVC property, while Wasserstein continuity alone is insufficient.

Gaussian policies are TVC with a linear modulus when their means are Lipschitz.The section states that Gaussian smoothing yields this property for arbitrary policies.
Wasserstein continuity does not ensure that in-distribution regression error controls regret.Even deterministic Lipschitz learners can incur exponentially compounding error.
Gaussian smoothing makes any policy TVC with a linear modulus γ_TVC(u) = u 2σ.The displayed expression gives the modulus associated with smoothing variance parameter σ.
Quantizing a TVC policy with any measurable quantizer preserves TVC.This closure property connects smooth raw policies to their quantized counterparts.
Under invertible dynamics, a maximal trajectory coupling can also be constructed as a shared-noise coupling.The coupling result is used to connect trajectory distributions while retaining common environmental noise.

C.2 Proof of Theorem 1

The proof establishes Theorem 1 by constructing shared-noise couplings across policy pairs and applying stability, smoothness, and reward regularity to bound regret.

The proof first bounds a bad event and then derives Theorem 1 using the law of total expectation.The argument combines trajectory couplings with event-wise control of action mismatches.
The regret bound uses local P-IISS stability, RTVC smoothness, and a Lipschitz reward function.These conditions control state and action deviations along the coupled trajectories.
A maximal coupling between trajectory laws can be chosen as a shared-noise coupling under invertible dynamics.This is the coupling ingredient used to transfer stability arguments across trajectories.
The proof glues successive couplings to obtain a joint law for the extended trajectories and the final policy.The glueing construction preserves the required pairwise marginals and shared-noise structure.
Theorem 1 is specialized to quantized behavior cloning by substituting the quantization error into the smoothness modulus.The resulting upper bound follows after applying the statistical guarantee.

D.1 Proof of Proposition 5

The proof shows that binning quantizers satisfy the required RTVC smoothness condition when applied to Lipschitz policies.

For sufficiently close states, a binning quantizer maps the policy outputs identically, yielding zero quantized Wasserstein cost.The proof uses the quantizer’s bounded distortion and the policy’s Lipschitz property.
For states farther than δ0 apart, the RTVC modulus equals one and trivially bounds the quantized policy distance.Combining the near and far cases gives κ(r) = 1{r > δ0}.

D.2 Proof of Theorem 6

The proof constructs a stable scalar example showing that a carefully designed quantizer can repeatedly route states through regions with large action error, including under stochastic dynamics.

The deterministic example uses stable linear dynamics, an arctangent expert policy, and a reward penalizing action deviation.The closed-loop contraction parameter is λ = A + B < 1.
On IT1 ∪ IT2, the constructed quantization error is at least A.These regions are the target of the recurrent bad-action construction.
The quantizer creates a loop IP → IT1 → IT2 → IP under noiseless dynamics while maintaining small error elsewhere.The construction uses three non-overlapping intervals and a binning quantizer outside them.
Under noisy dynamics, the quantized chain has a stable distribution and converges to it geometrically.The proof establishes this using a Foster–Lyapunov drift condition and Gaussian transition density.
The stochastic construction requires noise scale σ_ω = o(ε_q) to keep the relevant probability bound below one.This condition ensures the selected lower-bound parameter remains positive.
The stationary distribution assigns nontrivial mass to the transition regions, and finite-time mass approaches this stationary behavior geometrically.The displayed bounds involve Tmix and a decay term Mρ^t.

E.1 Proof of Theorem 8

The proof constructs paired behavior-cloning instances whose quantized demonstrations are indistinguishable, then applies two-point testing arguments to derive lower bounds involving horizon and quantization effects.

Lower-bound construction: The proof constructs two stable dynamics-policy-reward instances and compares their trajectory distributions under quantized demonstrations.The instances use shared initial distributions but different policies, rewards, and dynamics, with stability asserted for the constructed pair.
Testing argument: A metric based on the difference between expected first actions separates the policies, enabling a Le Cam two-point lower-bound argument.The proof defines ρ using expected first actions and invokes the standard two-point argument after constructing the hypothesis test.
Testing argument: The n-trajectory testing error is controlled by total variation, with the proof using bounds involving 1 − (1 − 1/(2D²) H²(Pa, Pb))^n and a large-n constant bound.The displayed fragments state total-variation bounds for product trajectory distributions, including a bound of at most 0.8 for sufficiently large n under a selected Δ.
Quantization scheme: Uniform midpoint binning partitions [0, 1] into K equal intervals and maps each input to its interval midpoint.The quantizer is defined by consecutive intervals of length 1/K, with midpoint representatives.
Indistinguishability: Quantization can make the observed data distribution identical across policies, so any learner using only quantized data produces the same estimate in both instances.The proof explicitly attributes this indistinguishability to identical quantized-data distributions under the two policies.