Source-linked AI summary
Implicit Behavioral Cloning
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson
TL;DR
The paper asks whether direct explicit policy representations limit behavioral cloning on complex robotic tasks. It reformulates imitation as implicit energy-based inference and evaluates the resulting policies across synthetic, benchmark, and real-world settings. Implicit behavioral cloning often outperforms explicit baselines and can be competitive with state-of-the-art offline reinforcement learning methods.
Problem
Behavioral cloning commonly maps observations directly to actions, motivating investigation of whether the policy representation itself is a fundamental design decision for complex robotic imitation.
Method
The paper reformulates behavioral cloning as conditional energy-based modeling, using inference-time implicit regression to select actions that minimize a learned energy.
Results
Implicit behavioral cloning often outperforms explicit policy baselines across visual, high-dimensional, contact-rich, and real-world tasks, while providing competitive results with state-of-the-art offline reinforcement learning methods.
Takeaways & Limitations
Implicit policies can learn long-horizon, closed-loop visuomotor behaviors and represent discontinuous and multi-valued functions within behavioral cloning.
Takeaways & Limitations
Compared with MSE behavioral cloning, implicit models require more training and inference computation and have greater implementation complexity.
Abstract
from arXiv · showhide
We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models. We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions. On robotic policy learning tasks we show that implicit behavioral cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) behavioral cloning policies, including on tasks with high-dimensional action spaces and visual image inputs. We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information. In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision.
1 Introduction
The paper revisits behavioral cloning’s policy representation, replacing direct observation-to-action prediction with implicit energy-based inference. It studies this change across challenging robotic tasks and motivates it through the ability to represent complex functions.
- Behavioral cloning commonly uses explicit continuous models that map observations directly to actions, but the paper questions whether this representation is appropriate.
- The proposed policy composes argmin with a continuous energy function, representing actions as minimizers conditioned on observations.
- At inference time, the energy-based policy finds an optimal action through sampling or gradient descent, framing imitation as conditional energy-based modeling.
- Experiments report improvements across contact-rich tasks including scooping, 1mm-tolerance block pushing, and color-based block sorting.
- The paper investigates why implicit models may succeed by examining multimodal distributions and discontinuous functions.
2 Background: Implicit Model Training and Inference
Implicit models represent outputs through energy minimization rather than direct prediction. Training uses negative counter-examples and an InfoNCE-style objective, while inference can use several optimization procedures.
- An implicit model is the composition argmin_y ◦ Eθ(x,y), where a function approximator defines an energy and inference selects minimizing outputs.
- Training generates negative counter-examples for each observed sample and uses an InfoNCE-style loss to train the energy model.
- The conditional model assigns likelihood through an energy-based form proportional to exp(−Eθ(x,y)).
- Inference solves the energy minimization problem using derivative-free sampling, autoregressive coordinate descent, or gradient-based Langevin sampling.
3 Intriguing Properties of Implicit vs. Explicit Models
The experiments compare implicit and explicit models on discontinuities, extrapolation, multimodal functions, and visual coordinate regression. Implicit models more sharply preserve discontinuities and often generalize better beyond sparse training data.
- The comparison tests discontinuity handling, multi-valued function fitting, and extrapolation using nearly identical ReLU-MLP architectures.
- Discontinuities: Implicit models approximate discontinuities sharply without intermediate artifacts, whereas explicit models interpolate through discontinuities.
- Discontinuities: For random uncorrelated data without regularization, implicit models exhibit nearest-neighbor-like behavior with non-zero-slope segments around samples.
- Extrapolation: Outside the training convex hull, implicit models often perform piecewise linear extrapolation from the model region nearest the domain boundary.
- Multi-valued functions: Multi-valued functions can be represented through argmin sets, interpreted either as likely-value samples or as sets of optimization minimizers.
- Visual generalization: 1 to 2 orders of magnitude lower test-set MSE loss is achieved by ConvMLP-EBM in the low-data visual coordinate-regression regime.
4 Policy Learning Results
The experiments compare implicit and explicit behavioral-cloning policies across simulated, benchmark, and real-world robotic tasks. Implicit policies generally perform better, including on discontinuous, high-dimensional, visual, contact-rich, and combinatorially complex tasks.
- Cross-Domain Comparison: Figure 5 compares implicit and explicit policies across six simulated and real domains, including author-reported D4RL baselines.The figure averages the best policy performance within each policy type and domain across that domain’s tasks.
- D4RL Human-Expert Tasks: On D4RL human-expert tasks, implicit and explicit policies significantly outperformed reported BC baselines and were competitive with state-of-the-art offline RL results.Prioritizing the top 50% of demonstrations by return generally improved implicit policies and sometimes produced new state-of-the-art performance.
- N-D Particle Integrator: 95% successful implicit policies reached 16 dimensions in the N-D Particle Integrator, while explicit MSE policies reached only 8 dimensions with the same success rate.The task isolates discontinuities while holding the number of demonstrations constant; nearest-neighbor policies performed well only in 1D.
- Simulated Bi-Manual Sweeping: EBM policies outperformed MSE policies by 14% on simulated bi-manual sweeping with 12-DoF actions and many mode changes.The task uses two coordinated arms, visual inputs, and 700-step episodes.
- Visual Policy Learning: Image-based EBM policies achieved 78.2 ±2.7 Success %, compared with 63.9 ±7.7 for image-based MSE policies using the listed encoders.This result comes from the visual particle-sweeping comparison.
- Real Robot Manipulation: Across four real-world manipulation tasks, implicit policies achieved significantly higher performance than the explicit baseline.The tasks included multimodal pushing, 1mm-tolerance oriented insertion, and sorting blocks by color; results were averaged over three models per task and policy type.
5 Theoretical Insight: Universal Approximation with Implicit Models
The theoretical analysis shows that continuous energy functions combined with argmin can represent broad classes of set-valued and discontinuous functions. Approximation can remain accurate without requiring the energy approximator to inherit steep gradients from the represented function.
- Theorem 2: Continuous function approximators can implicitly represent multi-valued and discontinuous functions to arbitrary accuracy.The paper frames this as a broader representational class than earlier results based on finitely many polynomial inequalities.
- Theorem 1: For any closed-graph set-valued function F, a continuous g exists such that argmin_y g(x,y)=F(x) for every x.This establishes exact implicit representation for the stated class of functions.
- Approximation Guarantee: Theorem 2 guarantees that the inferred point’s distance to the graph of F is controlled when the underlying energy is approximated with arbitrarily small bounded error.The guarantee is visualized through level sets and slices of a continuous energy function constructed from distance to the function graph.
- Lipschitz Behavior: Implicit functions with bounded Lipschitz constants can approximate explicit functions with arbitrarily small or large Lipschitz constants, including steep or discontinuous functions.The paper contrasts this with explicit continuous approximators, which must match the steepness of the represented function.
6 Related Work
Prior work explores energy-based models in generative modeling, reinforcement learning, and imitation learning, while imitation-learning methods often require additional data, rewards, or environment interaction. This paper situates Implicit BC among these approaches and addresses approximation of discontinuous functions.
- Energy-Based Models: Energy-based models have been studied for implicit generation, reinforcement learning policy representation, planning, and imitation learning.Prior applications include Langevin sampling, model-based planning, and on-policy imitation learning.
- Imitation Learning: Imitation-learning alternatives to behavioral cloning may require on-policy data, expert action labels, rewards, or millions of environment interactions.Distribution-matching and offline reinforcement-learning methods differ in their information and interaction requirements.
- Imitation Learning: ValueDice uses sample-efficient off-policy distribution matching, but had not been proven on image observations or high-degree-of-freedom action spaces.
- Discontinuous Functions: Function-approximation research has developed approaches for discontinuous functions, typically outside neural-network modeling.
7 Conclusion
The paper reformulates supervised imitation learning as conditional energy-based modeling with inference-time implicit regression. It reports that this approach often outperforms explicit policy baselines across high-dimensional, visual, and real-world tasks, while requiring more compute than explicit models.
- Conclusion: Implicit BC often greatly outperforms traditional explicit policy baselines across high-dimensional action spaces, visual observations, and real-world tasks.The reported action-space scope reaches 30 dimensions in D4RL human-expert tasks.
- Conclusion: The paper develops intuitive and theoretical analyses of implicit models’ ability to represent discontinuities and provides a distinct universal-approximation notion.
- Limitations: Implicit policies require more training and inference compute than explicit models, although the paper reports real-time vision-based control and modest training time relative to offline RL.
A Contributions Statement
The contributions introduce Implicit BC, evaluate it in simulation and real-world robotics, analyze implicit-model behavior, and develop theoretical results on representation and approximation.
- Method: Implicit BC casts imitation learning as conditional energy-based modeling with sampling-based or gradient-based optimization at inference.
- Real-World Experiments: Real-world robots learn image-driven, contact-rich pushing behaviors from human demonstrations, including precision insertion and multi-item sorting.Implicit BC performs significantly better than the explicit baseline across all real-world tasks, with an order-of-magnitude increase on precision insertion.
- Simulation and Benchmarks: Implicit and explicit BC are compared in simulation and on D4RL human-expert tasks, where both achieve competitive or state-of-the-art performance without reward information.
- Model Analysis: The paper analyzes implicit models in 1D examples, focusing on behavior at discontinuities and during extrapolation.
- Theory: Theoretical results establish implicit models’ representational and approximation abilities for discontinuous and set-valued functions.
B Energy-Based Model Training and Implicit Inference Details
The paper trains conditional energy models and performs implicit inference using derivative-free, autoregressive derivative-free, or Langevin methods. These variants trade simplicity, memory, gradient stabilization, and scalability as action dimensionality increases.
- Training: Training uses energy-based modeling with counterexamples sampled from a uniform action range and an InfoNCE loss.The implementation typically uses batches of 512 samples and 256 counterexamples per sample.
- Inference: Implicit inference solves ˆy=argmin_y Eθ(x,y) using derivative-free sampling, autoregressive coordinate descent, or gradient-based Langevin sampling.
- Variant Comparison: The joint derivative-free method fails beyond N=5 dimensions, whereas autoregressive and Langevin variants solve reliably up to 16 dimensions and achieve nonzero success at 32 dimensions.These results use 2,000 demonstrations in the N-D particle environment.
- Variant Comparison: The autoregressive variant uses N separate models and no new gradient stabilization, while Langevin uses one model but requires gradient stabilization.
- Evaluation Tasks: The evaluation environments span low-data regimes and high observation, state, or action dimensionalities.The task summary reports demonstrations and observation/state/action dimensionalities for each environment.
C.2 Training and Inference Times, Implicit vs. Explicit Comparison
The comparison covers training and inference protocols for implicit and explicit policies across D4RL, visual real-world tasks, and several simulation analyses. Implicit models incur greater D4RL runtime but remain practical, while visual training times are comparable and inference adds only a modest delay.
- D4RL timing: D4RL experiments compare training steps per second and full train+eval time through 100k training steps, with 100 episodes evaluated every 10k steps.The protocol reports both throughput and total experiment duration.
- D4RL timing: 5.6x: the best-performing 100-iteration Langevin implicit models take this much longer to train and evaluate than the best explicit models.The implicit model’s 3.4-hour full train+eval time remains below the 16.3 hours reported for CQL on a comparable D4RL task.
- Real-world timing: Visual real-world models require 5.0 hours for implicit training and 5.8 hours for explicit training.Visual processing dominates training because the models use identical visual processing through late fusion.
- Real-world timing: Implicit real-world models show a modest inference-time increase, reaching up to 7.22 ms, while the learned policy operates at 5 Hz.Inference runs on a single GPU after training on an eight-GPU server.
- Experimental scope: The experiments span state and image observations, action dimensions from 1 to 32, and optimization methods including derivative-free optimization and Langevin dynamics.The reported protocols also include 100-episode averages over three seeds for Simulated Pushing.
E Model Architectures
The architectures use MLPs for state observations and ConvMLP-style visual models, with EBMs receiving actions as inputs and returning scalar energies. Visual models combine CNN image encodings with actions before the MLP.
- Architecture notation: The architecture notation defines T as sequence length, m as observation dimensionality, n as action dimensionality, and W, H, C as image width, height, and channels.These symbols describe the state-observation and visuomotor diagrams.
- State-observation models: State-observation EBMs use MLPs that take actions as inputs and output a scalar energy, whereas MSE models output actions directly.The implementations use ReLU activations and optionally configure dropout, residual connections, or spectral normalization.
- Visuomotor models: Visuomotor EBMs concatenate actions with CNN image encodings before the MLP portion of a ConvMLP-style architecture.The visual architecture is illustrated separately from the state-observation architecture.
- Model comparison: The theoretical analysis compares explicit y=fθ(x) models with implicit argmin_y Eθ(x,y) models using nearly identical ReLU fully connected MLPs.The implicit model differs by receiving y as an additional input.
F.2 Proofs
The proofs construct implicit functions from distances to function graphs, showing that closed-graph set-valued functions can be represented by continuous energies whose minimizers recover the function values. Approximation error yields a one-sided graph-distance guarantee, but not symmetric recovery.
- Distance-function lemmas: A distance function to any non-empty set is well-defined and 1-Lipschitz, with the infimum attained for closed sets.The proofs establish continuity using the triangle inequality and compactness of a bounded intersection.
- Single-valued functions: For a continuous single-valued function, distance to its graph is continuous and has the function value as the unique argmin over y.At graph points the distance is zero, while off-graph points have positive distance.
- Set-valued representation: Theorem 1: every nonempty set-valued function with a closed graph has a 1-Lipschitz implicit function whose argmin exactly equals the function’s value set.The construction uses distance to the function graph and allows discontinuous or multi-valued functions.
- Approximation guarantee: Theorem 2: if the continuous energy is approximated with bounded error ϵ, every point in the learned argmin graph lies within ϵ of the original function graph.The proof scales the distance function by two and bounds the resulting graph distance.
- Limitations: The guarantee is not symmetric: small energy error can remove values from the learned set, and discontinuities can cause arbitrarily large output errors.The theorems establish existence and approximation bounds, not that learning or inference will recover the required model.
- Policy implications: Implicit functions can represent steep, discontinuous, and multi-valued policies without requiring large gradients in the function approximator.At imperfectly estimated discontinuity boundaries, inferred values remain on one side rather than averaging between sides.
H Limitations
Implicit behavioral cloning has higher computational and implementation costs than explicit policies, though the reported real-world inference increase is modest and supports real-time control.
- Implicit behavioral cloning requires more training and inference computation than a simple MSE behavioral cloning policy.
- The increased inference time for real-world models is modest, and the authors validated them for real-time vision-based control.
- Implicit models are more complex to implement than explicit models.
- The presented models have modest training times compared with reported offline reinforcement learning methods.