Maximum Causal Tsallis Entropy Imitation Learning

1 Introduction

• Stochastic expert policies matter because multiple action sequences can perform equally well, and learning them can improve robustness, exploration, and stability.The paper gives overtaking or turning choices as examples of differing driver actions in the same situation, and links multi-sequence learning to recovery from unexpected events.Source: §1 · p. 1
• MCTE models expert stochasticity with a policy whose supporting set can adjust from uni-modal to multi-modal behavior.Its optimal discrete-action policy follows sparsemax, which can assign zero probability to unnecessary actions, unlike softmax.Source: §1 · pp. 1–2
• MCTEIL uses a sparse mixture density network with sparsemax mixture weights to model state-dependent stochastic behavior in continuous action spaces.The paper states that the Tsallis entropy of a sparse MDN encourages exploration and mixture utilization, while causal entropy is less effective at preventing collapsed mixture means.Source: §1 · p. 2
• MCTEIL outperforms existing methods in average cumulative return across four MuJoCo control problems, with its strongest reported result on reacher using fewer demonstrations.GAIL often fails to learn reacher; in the first simulation, sparse MDN learns multi-modal behavior without performance loss, unlike a single Gaussian and a softmax-based MDN.Source: §1 · p. 2

Markov Decision Processes

• An MDP frames sequential decision-making through states, actions, transition probabilities, rewards, and stochastic policies.The stated objective is to find a policy maximizing expected discounted rewards.Source: §2 · p. 2

Maximum Causal Entropy Inverse Reinforcement Learning

• Maximum causal entropy IRL matches the optimized policy’s feature expectations to the expert’s while maximizing policy causal entropy.Its optimal solution is a softmax distribution.Source: §2 · p. 2

Generative Adversarial Imitation Learning

• GAIL avoids directly solving the soft MDP inner minimization, which can be intractable for large state and action spaces and require unavailable transition probabilities.The framework represents imitation learning with a discriminator whose output can be interpreted as a cost.Source: §2 · p. 3

Sparse Markov Decision Processes

• Sparse MDPs add causal sparse Tsallis entropy to expected discounted rewards, yielding sparse, multi-modal optimal policies.Sparsemax can assign zero probability to actions whose state-action value is below a threshold.Source: §2 · p. 3

3 Principle of Maximum Causal Tsallis Entropy

• MCTE switches the optimization variable from a policy distribution to state-action visitation measures before deriving optimality conditions.The framework’s optimal solution is characterized using concavity and KKT conditions.Source: §3 · p. 4

3.1 Optimality Condition of Maximum Causal Tsallis Entropy

• The optimal policy is sparsemax: actions below a threshold receive zero probability, allowing the supporting set to vary with the expert’s behavior.The policy can have a larger support for multi-modal behavior and a smaller support otherwise.Source: §3.1 · p. 5
• The MCTE optimality condition is equivalent to the sparse Bellman optimality equation, so the policy and Lagrange multiplier can be computed by solving a sparse MDP.The reward is r(s, a) = θ*⊺φ(s, a), with θ* the optimal dual variable.Source: §3.1 · p. 5

3.2 Interpretation as Robust Bayes

• MCTE is equivalent to a minimax game in which a sequential decision maker minimizes worst-case prediction Brier score.The resulting estimator is interpreted as the best decision maker against the worst nature can offer.Source: §3.2 · p. 5

4 Maximum Causal Tsallis Entropy Imitation Learning

• The MCTE objective can be recast as minimizing a distance between expert and training-policy feature expectations in a generative-adversarial formulation.The reformulation uses concavity in visitation measures and convexity in the reward parameter to interchange minimization and maximization.Source: §4 · p. 6
• MCTEIL extends MCTE to a model-free generative-adversarial setting, alternating discriminator updates with policy optimization using a Tsallis entropy bonus.The discriminator distinguishes policy-generated trajectories from expert trajectories.Source: §4 · pp. 5–6
• A sparse MDN models continuous, potentially multi-modal expert behavior with Gaussian mixtures whose weights use sparsemax.The number of active mixture components can vary by state, with unnecessary components assigned zero weight.Source: §4 · p. 6
• The causal Tsallis entropy of an MDN has an analytic form, unlike the intractable Gibbs-Shannon entropy described here.The entropy is proportional to distances between mixture means, encouraging exploration of diverse directions and discouraging use of several mixtures to model one mode.Source: §4 · pp. 6–7

5 Experiments

• The comparisons include behavior cloning, single-Gaussian GAIL, and softmax-weighted MDN GAIL, with multi-task imitation methods excluded because they target a different kind of multi-modality.The excluded methods focus on demonstrations spanning multiple tasks, whereas this study targets multiple optimal actions within one task.Source: §5 · p. 7

5.1 Multi-Goal Environment

• At equal mixture counts, MCTEIL achieves greater reachability than soft GAIL and learns all demonstrated modes by the end of training.Average returns rise similarly because a uni-modal policy can earn high rewards in the multi-modal reward environment.Source: §5.1 · p. 7, Figure 1 · p. 8

5.2 Continuous Control Environment

• The MuJoCo evaluation uses Halfcheetah, Walker2d, Reacher, and Ant, with 4, 11, 18, or 25 demonstrations and three random-seed repetitions.Each trained policy is evaluated over 50 episodes; MDN methods use a mixture count selected by brute-force search.Source: §5.2 · p. 8
• MCTEIL outperforms other methods on three of four MuJoCo tasks as demonstrations increase, while matching soft GAIL on Walker2d.On Reacher, MCTEIL performs best at all tested demonstration counts.Source: §5.2 · p. 8, Figure 2 · p. 8

6 Conclusion

• The MCTE framework has a sparsemax optimal solution, and its analysis includes concavity, optimality conditions, and a robust Bayes interpretation.The robust Bayes connection is expressed in terms of the Brier score.Source: §6 · p. 8
• MCTEIL uses Tsallis entropy over Gaussian mixtures to encourage exploration and efficient mixture use in continuous-action problems.The conclusion attributes its multi-modal learning ability to searching in diverse directions with a sparse MDN.Source: §6 · p. 9
• MCTEIL outperforms behavior cloning, GAIL, and soft-MDN GAIL on the standard MuJoCo imitation-learning problems reported.The paper concludes that MCTEIL is an efficient and principled way to learn multi-modal expert behavior.Source: §6 · p. 9

A Analysis

• The proof of Theorem 2 is described as checking two equalities.The passage does not state the equalities themselves.Source: A Analysis · p. 9

A.1 Concavity of Maximum causal Tsallis Entropy

• The inequality argument uses the concavity of F(x) = x(1 − x) to derive the needed bound.The proof applies the property of a concave function to the expression under consideration.Source: §A.1 · pp. 9–10
• The proof concludes that the maximum causal Tsallis entropy objective is concave with respect to the occupancy measure.The derivation establishes an inequality for every state-action pair and sums it over states and actions.Source: §A.1 · p. 10

A.2 Optimality Condition from Karush–Kuhn–Tucker (KKT) conditions

• The KKT derivation obtains stationary conditions by differentiating the objective and the Bellman-flow constraints.For positive-occupancy actions, complementary slackness sets the corresponding multiplier to zero.Source: §A.2 · p. 10
• For each state, the right-hand side is constant across actions, and the normalization argument sums over actions with positive occupancy.The proof identifies K as the number of actions with positive occupancy.Source: §A.2 · p. 11
• The optimal policy condition uses the same threshold as sparsemax, giving a sparse distribution over actions.The optimality-condition discussion characterizes the policy as sparse and multi-modal.Source: §A.2 · pp. 10–11

A.3 Interpretation as Robust Bayes

• MCTE estimation is equivalent to a minimax game with the Brier score, and the reformulated problem is equivalent to maximum causal Tsallis entropy.Source: §A.3 · pp. 11–12

A.4 Generative Adversarial Setting with Maximum Causal Tsallis Entropy

• Concavity in ρ and convexity in θ, on the stated domains, allow minimax duality to make the maximization and minimization interchangeable.Source: §A.4 · pp. 12–13

A.5 Tsallis Entropy of a Mixture of Gaussians

• The section derives the causal Tsallis entropy of a mixture of Gaussian distributions.Source: §A.5 · p. 13

• The proposed W(π) approximation uses sampled trajectories, and its integral component is computed analytically without additional computational cost.Source: B Causal Entropy Approximation · p. 13
• The alternative approximation tested for W(π) performs poorly compared with approximation (17).Source: B Causal Entropy Approximation · p. 13
• For soft GAIL, H(π) is approximated using a sum of discounted likelihoods.Source: B Causal Entropy Approximation · p. 13
• The W(π) approximation is unavailable for H(π) because the Gibbs-entropy integral is intractable for a Gaussian-mixture policy.Source: B Causal Entropy Approximation · p. 13

• Figure 3 plots average return and reachability for MCTEIL and soft GAIL across mixture counts k and entropy regularization coefficients α.Source: Figure 3 · p. 14, C Additional Experimental Results · p. 14