Smooth Imitation Learning for Online Sequence Prediction

• Imitation learning is useful when complex planning and control tasks make manually specifying a good policy difficult.Source: §1 · p. 1
• Smooth imitation learning targets online decisions from environmental inputs while requiring smooth predictions in continuous action spaces.Source: §1 · p. 1
• Automated camera planning motivates the setting because camera actions must respond to noisy detections while maintaining smooth motion for aesthetic video.Source: §1 · p. 1
• Conventional supervised imitation learning violates i.i.d. assumptions because predicted actions alter future state distributions, motivating iterative learning reductions.Source: §1 · p. 1
• SIMILE combines smooth policy classes with policy aggregation to lift supervised-learning guarantees and achieve faster convergence than previous work.Source: §1 · pp. 1–2
• The method formalizes smooth policy classes, provides agnostic performance guarantees, uses adaptive learning rates, and empirically demonstrates performance gains.Source: §1 · p. 2

2. Problem Formulation

• Contexts are exogenous sequences, actions lie in a bounded continuous space, and policies map context-action states to action sequences.Source: §2 · p. 2
• Policies may use recent contexts and actions, with the state represented by the current context and previous action.Source: §2 · p. 2
• Exogenous contexts model sequential control settings such as camera planning, autonomous driving, helicopter aerobatics, and smart-grid management; smoothness is enforced on internal state.Source: §2 · p. 2
• Imitation learning minimizes convex loss under the state distribution induced by executing the learned policy, including an agnostic setting where zero loss may be impossible.Source: §2 · p. 2

2.1. Smooth Imitation Learning & Smooth Policy Class

• Smooth imitation learning minimizes imitation loss over policies whose action responses are smooth, motivated by stable behavior under nearly constant context streams.Source: §2.1 · pp. 2–3
• The proposed policy class regularizes complex models such as neural networks or trees toward smooth functions, turning optimization over complex models into constrained optimization.Source: §2.1 · pp. 2–3
• Concrete regularizers include previous-action smoothing, whose λ controls closeness to the previous action, and linear autoregressors that model actions dynamically.Source: §2.1 · p. 3
• An H-state-smooth policy bounds action sensitivity, equivalently bounding each component’s Hessian for twice-differentiable policies.Source: §2.1 · p. 3

• Earlier imitation-learning work largely targeted discrete actions or restrictive dynamics models, whereas this paper addresses complex smooth functions over continuous states and actions.Source: §3 · p. 3
• Naive supervised learning fails to account for trajectory distribution mismatch, motivating iterative reductions that collect data under the current policy.Source: §3 · p. 3
• Iterative rounds execute the current policy, collect expert labels on visited states, train a new policy, and update the current policy.Source: §3 · p. 3
• SEARN mixes policies stochastically, while DAgger aggregates datasets; DAgger’s training time can grow super-linearly with the number of iterations.Source: §3 · pp. 3–4
• Previous variation-distance analyses require learning rates below 1/T, which can make convergence slow for long horizons.Source: §3 · p. 4
• SIMILE removes horizon dependence from its policy-improvement bound, supports adaptive learning rates and virtual expert feedback, returns deterministic policies, and provides fast-convergence guarantees for smooth finite-horizon classes.Source: §3 · p. 4

• SIMILE iteratively builds state distributions, trains a policy under updated smooth regularization, adapts the interpolation rate, and outputs the final policy.Source: §4 · p. 4
• The policy update deterministically interpolates the previous and newly learned policies, converging at least as fast as stochastic interpolation for smooth policy classes.Source: §4 · p. 4
• Interpolation provides greedy refinement and limits state-distribution drift, preserving an approximate i.i.d. relationship for supervised regression and supporting convergence.Source: §4 · p. 4
• SIMILE adaptively increases the interpolation rate according to the learned policy’s quality, reducing the tension between distribution stability and aggressive exploration.Source: §4 · pp. 4–5
• Virtual feedback simulates expert demonstrations, reducing the need for expert queries while ensuring each learned policy remains feasible and smooth.Source: §4 · p. 5
• When the learner is far from the expert trajectory, smoother corrective feedback makes training more stable and is designed to converge toward the expert action.Source: §4 · p. 5
• The implementation gradually reduces the feedback-smoothing parameter σ toward zero as iterations increase.Source: §4 · p. 5

5.1. Stability Conditions

• Smooth policy classes satisfy stability conditions that control action changes under fixed context, with the stronger Lipschitz condition enabling stronger policy-improvement guarantees.Both conditions follow from sufficiently small smoothness regularization, while Condition 2 removes dependence on the time horizon in the strongest guarantee.Source: §5.1 · p. 5, §5.3 · p. 6

5.2. Deterministic versus Stochastic

• Deterministic interpolation matches stochastic interpolation in expected trajectory behavior under stability, and convex loss makes deterministic policies no worse in expectation.This avoids the polynomial number of stochastic roll-outs needed to approximate deterministic trajectories.Source: §5.2 · pp. 5–6
• Unstable policies can diverge exponentially from expert behavior even when the learned policy is correct, so iterative interpolation alone cannot ensure convergence without stability.The constructed policy π(s)=ka with k>1 remains unstable after every convex interpolation update.Source: §5.2 · p. 6

5.3. Policy Improvement

• Earlier reduction methods require conservative β<1/T bounds, which slow convergence and do not generally guarantee monotonic improvement in continuous settings.The limitation follows from variation-distance analyses used by SEARN and related methods.Source: §5.3 · p. 6
• Condition 2 removes the time-horizon dependence in policy-improvement bounds, enabling monotonic improvement for an appropriate adaptive interpolation parameter β.The guarantee is stronger than previous approaches because it exploits smoothness conditions rather than requiring β to scale inversely with T.Source: §5.3 · p. 6

5.4. Smooth Feedback Analysis

• Smooth virtual feedback still supports progress because interpolated targets point the learning update in a negative-gradient direction.The target is formed by combining the current action and expert action using σ, with smooth feedback corresponding to σ approaching 1.Source: §5.4 · pp. 6–7
• When the current policy performs poorly, smooth virtual feedback can be necessary to keep the next learned policy feasible and smooth.The method uses smoother targets in early iterations, when the average gap from expert behavior is larger.Source: §5.4 · p. 7

Automated Camera Planning

• SIMILE learns trajectories that are both smooth and accurate for automated camera planning, where fluid movement is preferred to jittery perfect tracking.The policy uses regression-tree ensembles regularized by linear autoregressors to model smooth actions in a continuous setting.Source: §6 · pp. 7–8

Adaptive vs. Fixed β

• Adaptive β converges substantially faster than conservative fixed values, while very large fixed β can overshoot and worsen the combined policy.The adaptive rule uses the relative empirical losses of the current and newly trained policies.Source: §6 · pp. 7–8

Smooth Feedback Generation

• Larger σ produces smoother but less accurate targets, so σ is kept near 1 early and approaches 0 as training proceeds.This gradual schedule improves stability when the initial policy performs poorly, whereas low-σ targets can be difficult and less stable.Source: §6 · p. 8
• When the initial policy performs poorly, setting σ closer to 1 in early iterations and letting σ approach 0 as iterations increase produces more stable policies; smooth targets allow learning a smooth policy in the subsequent round.Figure 6 visualizes intermediate policies learned after different iteration counts, showing that each intermediate policy is smooth.Source: §6 · p. 8

Deterministic vs. Stochastic Interpolation

• Deterministic policy interpolation has higher empirical error than stochastic policy sampling under β=0.5 and 50 stochastic roll-outs.Source: §6 · p. 8, Figure 7 · p. 8

• SIMILE uses an adaptive learning rate that provably enables faster convergence than previous learning-reduction approaches.Source: §7 · p. 8
• SIMILE improves sample complexity through fully deterministic learning and virtual simulation of training labels.Source: §7 · p. 8
• The approach was validated for efficiency and practicality in online camera planning.Source: §7 · p. 8

A.1. Proof of lemma 5.1

• The proof applies the mean value theorem and gradient smoothness to derive the required self-bounding inequalities.Source: §A.1 · p. 10
• A.1 proves Lemma 5.1 by reducing the multidimensional case to a one-dimensional function along the line between two actions.The proof uses self-bounding and one-dimensional smoothness properties of non-negative H-smooth functions.Source: §A.1 · p. 10
• Lemma 5.1 establishes that action differences in the policy representation are bounded by smoothness and the squared distance between actions.The derived bound is expressed as 6H(ϕ(a)+ϕ(a1))||a-a1||^2.Source: §A.1 · p. 10

A.2. Proof of lemma 5.2

• A.2 proves that, under policy stability, the expected action sequence from stochastic roll-outs matches the deterministic policy's action sequence at every time step.The result is established by induction from a common starting state.Source: §A.2 · pp. 10–11

A.3. Proof of theorem 5.6 and corollary 5.7 - Main policy improvement results

• Theorem 5.6 bounds the updated policy’s loss difference under convexity, L-Lipschitz continuity, and Condition 2, for β∈(0,1), using ε as maximum policy disagreement and a reduction-loss term.The reduction term is the expected loss of trained policy ˆπ on the state distribution induced by policy π.Source: §A.3 · pp. 11–12
• The policy loss is defined as expected trajectory loss over starting states induced by the policy.Source: §A.3 · p. 11
• The update deterministically interpolates the current policy π with the trained policy ˆπ using learning rate β.The learning rate may be chosen adaptively at each iteration.Source: §A.3 · p. 11
• The proof compares trajectories from a common starting state and the same input stream, then uses policy-class smoothness to control state divergence.Source: §A.3 · p. 11
• As β approaches zero, the new policy's loss cannot degrade much; when the trained policy improves the reduction loss, adaptive β can enforce monotonic policy improvement.Source: §A.3 · pp. 11–12

A.4. Proof of theorem 5.5 - T-dependent improvement

• Theorem 5.5 assumes convex L-Lipschitz loss and Condition 1 to bound the loss difference between the updated and current policies.Source: §A.4 · p. 12
• The proof obtains the result by comparing two roll-outs from the same starting state and recursively bounding their state differences across time.Source: §A.4 · p. 12
• For β in (0,1/T), the policy-improvement bound includes the policy disagreement term ϵL and the reduction-loss difference.Source: §A.4 · p. 12

A.5. Proof of proposition 5.8 - smooth expert proposition

• The policy class regularizes supervised predictors toward a smooth function, with λ trading off smoothness against imitation accuracy.The regularized minimization balances closeness to smooth predictions and human actions.Source: §A.5 · pp. 12–13
• When the old trajectory is sufficiently far from the human trajectory, training directly on ideal human actions causes the learned trajectory to be non-smooth.The proposition formalizes this failure for policies regularized toward smooth behavior.Source: §A.5 · p. 13

B.1. Variant of SIMILE Using Smooth Regression Forest Policy Class

• The smooth regression-forest variant combines random forests with linear autoregressors so complex tree-based predictors can generate smooth sequential actions.The autoregressor uses previous actions, while the tree component preserves expressive state-dependent prediction.Source: §B.1 · p. 14

B.1.1. SMOOTH REGULARIZATION WITH LINEAR AUTOREGRESSORS

• Each iteration rolls out the current policy, forms smooth one-step feedback targets, trains an updated autoregressor and forest policy, and interpolates policies using weighted empirical errors.The feedback interpolation and policy mixing are intended to address changing state distributions during iterative training.Source: §B.1.1 · pp. 14–15, §B.1 · p. 14
• Across iterations, the autoregressor and λ enforce trajectory smoothness while the recurrent forest improves imitation, with satisfactory policies generally reached after 5–10 iterations.This convergence claim is reported for the sport broadcasting datasets.Source: §B.1.1 · p. 15

B.2. Smooth Regression Tree Training

• The recurrent tree branches on states containing previous predictions, and its autoregressor captures temporal policy dynamics while ignoring exogenous inputs.The autoregressor predicts current behavior from a fixed history of prior actions.Source: §B.2 · p. 15
• The leaf prediction minimizes a loss combining squared imitation error with smoothness regularization, controlled by λ.λ determines the relative emphasis on smoothness versus absolute prediction accuracy.Source: §B.2 · p. 15
• With fixed tree structure and autoregressor, a regularized leaf value averages transformed training targets and autoregressive predictions; λ = 0 recovers the conventional leaf mean.The conventional regression-tree solution is the mean target within each leaf.Source: §B.2 · p. 15
• Node splits are selected by maximizing impurity reduction, with child impurities computed using the regularized terminal-node values.This retains the standard regression-tree splitting principle while changing the leaf-value objective.Source: §B.2 · p. 16