PaperlayerView exact revision on arXiv

Source-linked AI summary

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

Irving Giovani Bronzatti Petrazzini, Eric Aislan Antonelo

arXiv:2609.38407v1cs.LG

TL;DR

Behavior cloning can compound errors when learned policies visit states outside the demonstration distribution, and DRIL’s effectiveness for image-based control with continuous bounded actions is uncertain. This study evaluates DRIL across policy supports, expert sources, demonstration counts, and retained training stages; score-selected DRIL leads in most stochastic regimes, but Beta behavior cloning leads with 20 bounded-action trajectories.

  • Problem

    It is unclear whether ensemble disagreement remains useful for visual control with continuous bounded actions, where policy support may affect both behavior cloning and the disagreement signal.

    Source: §1 · p. 2

  • Method

    The study adapts DRIL to stacked visual observations and bounded continuous actions, comparing Gaussian and Beta learners across expert sources, demonstration counts, and behavior-cloning, score-selected, and final checkpoints.

    Source: §1 · p. 3, §3.3 · p. 9

  • Results

    Score-selected DRIL has the highest stochastic mean in three of four demonstration regimes, while Beta behavior cloning is strongest with 20 bounded-action trajectories.

    Source: §7 · p. 28

  • Takeaways & Limitations

    The usefulness of disagreement regularization depends on policy representation, reward informativeness, and checkpoint choice; bounded Beta policies can be a competitive alternative when more demonstrations are available.

    Source: §1 · p. 3, §7 · p. 28

    With Beta learners, bounded rollouts can produce low scores without large action-direction variance in the fixed Gaussian ensemble, so disagreement may not identify states outside the demonstrations.

    Source: §6.2 · p. 27

Abstract

from arXiv · show

Purpose: Behavior cloning can accumulate errors when a learned controller visits states outside the demonstrated distribution. This study evaluates whether Disagreement-Regularized Imitation Learning (DRIL), which converts disagreement among cloned policies into a reinforcement-learning reward, improves image-based continuous control. Methods: A controlled CarRacing study combines Gaussian and Beta learner policies, demonstrations from either a clipped Gaussian expert or an intrinsically bounded Beta expert, one or 20 trajectories, deterministic and stochastic evaluation, and three retained stages: behavior cloning, the highest 10-episode training-score checkpoint, and the final DRIL checkpoint. The disagreement ensemble contains five Gaussian policies in every variant. Each retained policy is evaluated over 100 procedurally generated episodes. Results: Score-selected DRIL produced its largest gains in the few-demonstration setting, improving over the strongest behavior-cloning mean by 61% with clipped-action demonstrations and by 112% with bounded-action demonstrations. With 20 trajectories, the advantage of DRIL narrowed; in the bounded-action regime, Beta behavior cloning remained about 7% above the best DRIL checkpoint. The experiments also show that the informativeness of the disagreement reward changes with the learner representation and training stage. Conclusion: DRIL can substantially improve few-demonstration visual continuous control, while bounded Beta policies provide strong behavior-cloning performance when more demonstrations are available. The results highlight the joint importance of learner support,ensemble response, and checkpoint selection.

1 Introduction

Behavior cloning is vulnerable to distribution shift, and it remains unclear whether disagreement-based regularization works for visual control with continuous bounded actions. This study tests how policy support, demonstration source and count, evaluation mode, and checkpoint choice affect DRIL.

1 Introduction

  • DRIL uses disagreement among bootstrapped behavior-cloned policies as a thresholded reward for PPO, interleaving behavior-cloning and policy-gradient updates.This substitutes an imitation-uncertainty-derived reward for explicit task-reward design.Source: §1 · p. 2
  • Gaussian policies require clipping for bounded actuators, whereas Beta policies generate bounded actions directly; changing policy support can also alter rollout states and ensemble-variance scale.Thus, bounded policies may improve behavior cloning while weakening the disagreement reward on which DRIL relies.Source: §1 · p. 2
  • The study evaluates the previously untested combination of high-dimensional visual observations and continuous bounded actions, varying learner policy, expert-action source, demonstration count, and checkpoint selection.Its comparisons include deterministic and stochastic evaluation and behavior-cloning, score-selected DRIL, and final DRIL checkpoints.Source: §1 · pp. 2–3
  • Score-selected DRIL achieves the highest stochastic mean in three of four demonstration regimes, while Beta behavior cloning leads with 20 bounded-action trajectories.The study concludes that disagreement regularization’s value depends on policy representation, reward informativeness, and which checkpoint is retained.Source: §1 · p. 3

2 Related work

Related work addresses imitation-learning distribution shift through interactive supervision, reward learning, uncertainty signals, and alternative policy representations. The paper situates DRIL among these approaches and examines how its disagreement reward responds to policy support and rollout states.

2.1 Imitation learning and covariate shift

  • Prior approaches address covariate shift through online aggregation, feedback, hybrid RL–IL, or occupancy matching, while DRIL uses disagreement without new expert labels or recovery of a latent task reward.DRIL’s binary disagreement reward also acts as a constraint encouraging proximity to the demonstrated distribution, although it trains with environment interaction.Source: §2.1 · pp. 3–4

2.2 Visual imitation learning for autonomous control

  • Visual driving work spans camera-to-control, conditional and privileged policies, BEV representations, and hybrid controllers; CarRacing offers procedural tracks, image observations, and continuous actions for closed-loop experiments.Closed-loop evaluation matters because low supervised error does not ensure route completion or recovery from deviations.Source: §2.2 · pp. 4–5

2.3 Uncertainty and ensemble disagreement

  • DRIL thresholds ensemble prediction variance to penalize uncertain states, but for Beta learners the fixed Gaussian ensemble often stays below its expert-data threshold, leaving an almost uniformly positive reward.The paper relates this representation-dependent response to the distinction between epistemic and aleatoric uncertainty.Source: §2.3 · p. 5

2.4 Continuous policy representations

  • Gaussian policies have unbounded support and clipping may bias gradients, whereas Beta policies have finite support; more expressive energy-based, transformer, and diffusion policies represent richer behaviors.Prior Gaussian-versus-Beta PPO results motivate the policy choices here, rather than constituting a contribution of this paper.Source: §2.4 · p. 5

3 Method

The method clones demonstrations, derives an action-aligned disagreement reward from a bootstrapped Gaussian ensemble, and alternates imitation and PPO updates across Gaussian and Beta learners.

3.1 Problem formulation and behavior cloning

  • BC fits continuous expert actions by minimizing squared error between demonstrated actions and policy means, using an 80/20 random training/validation split.Source: §3.1 · p. 6

3.2 Disagreement ensemble

  • Five bootstrapped Gaussian BC policies define disagreement by projecting their covariance along the learner action, thresholding the score at the expert-data 98th percentile into +1/−1 rewards.Source: §3.2 · pp. 6–7

3.3 DRIL optimization

  • DRIL alternates BC and PPO updates using disagreement rewards, while retaining the BC initialization, highest 10-episode-score checkpoint, and final budget-end policy.Source: §3.3 · pp. 7, 9

3.4 Gaussian and Beta learner policies

  • Gaussian learners clip sampled actions outside the valid range, whereas Beta learners generate bounded actions; Beta mean updates also change both shape parameters and variance.Source: §3.4 · p. 10

4 Experimental design

The experiment moves from staged implementation checks to a controlled CarRacing comparison of image preprocessing, expert-data sources, and learner evaluation. It varies demonstration count, evaluation mode, and retained training checkpoint.

4.1 Staged implementation checks

  • Breakout and LunarLanderContinuous implementation checks precede CarRacing, verifying qualitative behavior in DRIL’s image/discrete-action and low-dimensional/continuous-action regimes.These checks are described as verification, not new algorithmic contributions.Source: §4.1 · p. 10

4.2 CarRacing task and preprocessing

  • CarRacing frames become 4 × 84 × 84 grayscale stacks, and retained policies are evaluated on 100 procedurally generated tracks distinct from demonstrations.The stack combines the current frame with the previous three to supply short-term motion information without an explicit speed sensor.Source: §4.2 · p. 10

4.3 Clipped-action and bounded-action experts

  • The study contrasts clipped Gaussian-expert actions with intrinsically bounded Beta-expert actions, using either one successful trajectory or all 20.About 73% of sampled Gaussian expert actions fell outside the valid interval in the analyzed trajectory; stored actions were clipped to [−1, 1]².Source: §4.3 · pp. 11–12

4.4 Factorial comparison

  • The factorial comparison evaluates BC, DRIL-peak, and DRIL-final in deterministic and stochastic modes; each configuration has one main training run.DRIL-peak is selected by the highest 10-episode training-score mean, while DRIL-final is the last checkpoint; evaluation variability is conditional on a checkpoint, not independent retraining seeds.Source: §4.4 · p. 12, Table 2 · p. 13
  • The controlled comparison uses BC as the common supervised baseline to isolate when disagreement regularization improves on it.The authors avoid direct numerical comparisons with methods using substantially different architectures, training protocols, or supervision assumptions.Source: §4.4 · p. 13

5 Results

Across CarRacing regimes, score-selected DRIL often outperformed BC, with gains concentrated in few-demonstration settings; Beta BC led when 20 bounded-action trajectories were available. Performance also depended on checkpoint selection and the informativeness of the disagreement reward.

5.1 Gaussian learner policies

  • Stochastic Gaussian DRIL-peak exceeds BC under both demonstration sources, but DRIL-final scores fall sharply.Training scores rise early and later decline; the orange curve marks the 10-episode moving average used for DRIL-peak selection. Stochastic Gaussian DRIL-peak is generally stronger than its deterministic counterpart.Source: §5.1 · pp. 14, 18, §5 · p. 14

5.2 Beta learner policies

  • With 20 trajectories, stochastic Beta BC scores 758±237 (clipped) and 794±227 (bounded); DRIL-peak improves BC in the one-trajectory regimes but remains below it with 20.All four DRIL-final stochastic means are between 99 and 267.Source: §5.2 · p. 18

5.3 Cross-factor comparison

  • DRIL-peak leads three regimes (61.1%, 5.8%, and 111.8% gains over strongest BC means); with 20 bounded trajectories, BC leads by 7.1%.The comparison uses the strongest stochastic BC and DRIL-peak policy in each demonstration regime.Source: Table 6 · p. 23, §5.3 · p. 23

5.4 Reward response with the Beta learner

  • For Beta learners, ensemble variance often stays below threshold even during poor or off-track episodes, leaving the binary reward positive for most steps.A nearly constant reward gives PPO limited information for distinguishing actions, while updates continue to move the actor away from BC initialization.Source: §5.4 · pp. 23, 27

6 Discussion

The discussion finds that DRIL’s apparent benefit depends on checkpoint selection, learner support, demonstration source, and dataset size. In particular, the best intermediate DRIL checkpoint can outperform later checkpoints, while Beta behavior cloning is strong with more bounded-action demonstrations.

6.1 Checkpoint selection and training dynamics

  • With 20 trajectories, stochastic Gaussian means fall from 802 to 218 for clipped demonstrations and from 720 to 240 for bounded demonstrations between DRIL-peak and DRIL-final.The evolution may reflect retention dynamics between BC and PPO objectives; tested reward and ensemble modifications did not eliminate the degradation, and score selection identifies the strongest intermediate policy in simulation.Source: §6.1 · p. 27

6.2 Policy support and ensemble response

  • Beta policies keep sampled actions within actuator bounds and optimize both mean and concentration, supporting strong stochastic BC with 20 demonstrations.However, bounded rollouts can score poorly without producing large action-direction variance in the fixed Gaussian ensemble, so actuator validity alone does not ensure an informative disagreement signal.Source: §6.2 · p. 27

6.3 Expert demonstrations and dataset size

  • 802±197 is the largest score, from Gaussian DRIL-peak trained on 20 clipped trajectories; the strongest pure BC score is 794±227 from Beta BC trained on 20 bounded trajectories.Increasing the dataset from one to 20 trajectories generally improves BC, while DRIL’s clearest relative benefit occurs with one trajectory and its advantage narrows with 20 bounded demonstrations.Source: §6.3 · p. 28

6.4 Relation to current imitation-learning methods

  • Modern generative policies can represent multimodal or temporally extended actions, but valid behavioral ambiguity may make action-level disagreement harder to interpret as out-of-distribution evidence.The paper’s explicit Gaussian and Beta actors do not cover those richer policy representations.Source: §6.4 · p. 28

7 Conclusion

The study evaluates DRIL for stacked visual observations and continuous bounded actions across policy distributions, demonstration sources, dataset sizes, evaluation modes, and checkpoints. Score-selected DRIL leads in most demonstration regimes, while stochastic Beta BC is strongest with 20 bounded-action demonstrations.

7 Conclusion

  • Score-selected DRIL has the highest stochastic mean in three of four demonstration regimes, while stochastic Beta BC is strongest with 20 bounded-action demonstrations.The largest relative DRIL gains occur in one-trajectory settings.Source: §7 · p. 28
  • Gaussian and Beta learners induce different rollout distributions, and the fixed Gaussian ensemble responds differently; Beta learners often retain positive reward over long rollout portions.The strongest intermediate checkpoints occur early in training, making BC, score-selected, and final checkpoints complementary views of learning.Source: §7 · p. 28
  • Future work will extend factorial evaluation across independent training and evaluation-track seeds, use matched experts, and test transfer in scenario-rich benchmarks including CARLA and Bench2Drive.The study concludes that DRIL can be especially effective in few-demonstration visual continuous control when the ensemble reward remains informative throughout training.Source: §7 · pp. 28–29

A Implementation-validation experiments

The implementation checks qualitatively test DRIL in the original study’s discrete-visual and continuous-low-dimensional regimes before combining visual observations with continuous actions. Across both checks, the reported DRIL score changes vary with demonstration count.

A Implementation-validation experiments

  • The checks precede the main study’s combination of visual observations and continuous actions and assess qualitative behavior in the two original DRIL regimes.The implementation-validation figure presents those original discrete-visual and continuous-low-dimensional regimes.Source: A Implementation-validation experiments · p. 30, Figure 14 · p. 31
  • DRIL reaches Breakout means of 355 and 339, compared with BC means of 5 and 7, for one- and three-trajectory settings, respectively.Each point represents one of 100 consecutive episodes; the figure also marks the DRIL mean, BC mean, and expert-level threshold of 300.Source: A Implementation-validation experiments · p. 31
  • In the continuous-low-dimensional check, DRIL changes the one-trajectory mean from 194 to 257 and the 20-trajectory mean from 232 to 228.Each setting is evaluated over consecutive episodes, with horizontal lines showing DRIL and BC means and the expert-level threshold of 200.Source: A Implementation-validation experiments · pp. 30–31

Statements and Declarations

The declarations report funding sources, no competing interests, no applicable ethics approval, available code repositories, and the authors’ contributions and approval.

Statements and Declarations

  • The work was partially funded by CNPq, Brazil, and in part by CAPES, Brazil.The passage identifies CNPq Grant No. 420148/2025-6 and CAPES Finance Code 001.Source: Statements and Declarations · p. 32
  • The authors declare no competing interests, and ethics approval is not applicable because the study uses simulated environments without human participants or animals.Source: Statements and Declarations · p. 32
  • Code for the DRIL adaptation and the previously published PPO-Beta expert is available in separate GitHub repositories.Source: Statements and Declarations · p. 32
  • The authors divide contributions across methodology, software, investigation, validation, analysis, and writing, and both approved the manuscript.The passage also attributes conceptualization, supervision, and project administration to Eric Aislan Antonelo.Source: Statements and Declarations · p. 32
Loading 2609.38407v1…