Source-linked AI summary
Notes on Kullback-Leibler Divergence and Likelihood
Jonathon Shlens
TL;DR
The paper examines how KL divergence arises from likelihood theory, specifically as the negative logarithm of the average multinomial log-likelihood. The paper derives the relationship using multinomial likelihood, independent observations, algebra, and Stirling’s approximation. KL divergence is the negative logarithm of the average multinomial log-likelihood.
Problem
The paper examines how KL divergence arises from likelihood theory, specifically as the negative logarithm of the average multinomial log-likelihood.
Method
The paper derives the relationship using multinomial likelihood, independent observations, algebra, and Stirling’s approximation.
Results
KL divergence is the negative logarithm of the average multinomial log-likelihood.
Takeaways & Limitations
The derivation provides an interpretation of KL divergence through its relationship with multinomial likelihood.
Abstract
from arXiv · showhide
The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions. Although difficult to understand by examining the equation, an intuition and understanding of the KL divergence arises from its intimate relationship with likelihood theory. We discuss how KL divergence arises from likelihood theory in an attempt to provide some intuition and reserve a rigorous (but rather simple) derivation for the appendix. Finally, we comment on recent applications of KL divergence in the neural coding literature and highlight its natural application.
Appendix A: Derivation
The appendix derives the relationship in Equation 2 using Stirling’s approximation, algebraic rearrangement, and recognition of an implicit probability distribution. It shows that KL divergence is the negative logarithm of the average multinomial log-likelihood and connects entropy to combinatorial normalization.
- Appendix A: Derivation: The derivation applies Stirling’s approximation, algebraic manipulation, and recognition of an implicit probability distribution to establish Equation 2.Stirling’s approximation is log n! ≃ n log n − n and becomes quite good for n > O(100).
- Appendix A: Derivation: The multinomial likelihood expresses the probability of observing histogram c given model q, while independent observations combine into their joint probability.The appendix defines average likelihood through the geometric mean of the multinomial likelihood across histogram counts.
- Appendix A: Derivation: KL divergence is the negative logarithm of the average multinomial log-likelihood.The derivation recognizes the normalized histogram as a probability distribution in the limit n →∞.
- Appendix A: Derivation: The normalization constant in the multinomial likelihood yields Σ_i p_i log p_i, identifying the distribution’s entropy through combinatorial reasoning.The appendix attributes this entropy derivation from combinatorial notions to Jaynes (2003).