PaperlayerView exact revision on arXiv

Source-linked AI summary

The Epic Story of Maximum Likelihood

Stephen M. Stigler

arXiv:0804.2996v1stat.ME

TL;DR

Maximum likelihood offers a simple route to broadly applicable estimates and accuracy assessments, but examples and unresolved assumptions complicate its claims. This article reviews its history and Fisher’s proofs, then concludes that the method remains useful while its safe scope is not fully guaranteed.

  • Problem

    Maximum likelihood’s broad claims of optimal estimation and accuracy face counterexamples and require assumptions whose scope is difficult to guarantee, especially in complex applications.

    Source: §1 · p. 2, §14 · pp. 17–18

  • Method

    The article traces maximum likelihood’s history and examines Fisher’s proofs, including their assumptions and mathematical basis.

    Source: §1 · p. 2, §11 · p. 12, §10 · p. 12

  • Results

    Maximum likelihood remains useful, including where no general theorem supports its use, but complex applications can still enter regions of inapplicability.

    Source: §14 · p. 18

  • Takeaways & Limitations

    The history clarifies that maximum likelihood has solid mathematical support in a wide class of problems, while its useful scope can exceed what available proofs establish.

    Source: §14 · pp. 17–18

    Fisher’s proofs rely on implicit regularity conditions and assumptions, and current understanding cannot guarantee safe use in complex situations.

    Source: §11 · p. 12, §14 · pp. 17–18

Abstract

from arXiv · show

At a superficial level, the idea of maximum likelihood must be prehistoric: early hunters and gatherers may not have used the words ``method of maximum likelihood'' to describe their choice of where and how to hunt and gather, but it is hard to believe they would have been surprised if their method had been described in those terms. It seems a simple, even unassailable idea: Who would rise to argue in favor of a method of minimum likelihood, or even mediocre likelihood? And yet the mathematical history of the topic shows this ``simple idea'' is really anything but simple. Joseph Louis Lagrange, Daniel Bernoulli, Leonard Euler, Pierre Simon Laplace and Carl Friedrich Gauss are only some of those who explored the topic, not always in ways we would sanction today. In this article, that history is reviewed from back well before Fisher to the time of Lucien Le Cam's dissertation. In the process Fisher's unpublished 1930 characterization of conditions for the consistency and efficiency of maximum likelihood estimates is presented, and the mathematical basis of his three proofs discussed. In particular, Fisher's derivation of the information inequality is seen to be derived from his work on the analysis of variance, and his later approach via estimating functions was derived from Euler's Relation for homogeneous functions. The reaction to Fisher's work is reviewed, and some lessons drawn.

1. INTRODUCTION

The introduction frames maximum likelihood as a powerful but mathematically complicated method, then traces early attempts to formulate and justify it. The history shows how similar-looking approaches could lead to different estimates and rationales.

1. INTRODUCTION

  • Maximum likelihood promises a broadly applicable optimum estimate and accuracy assessment, but superefficient and inconsistent estimates challenge that picture.The article reviews the theory’s history and Fisher’s claims rather than treating these counterexamples as the whole story.Source: §1 · p. 2
  • The article traces maximum likelihood’s history from early work through Fisher’s bold claims and the arguments, including unpublished ones, offered in their support.Source: §1 · p. 2

2. THE EARLY HISTORY OF MAXIMUM LIKELIHOOD

  • In 1769, Lagrange obtained multinomial probability estimates equal to sample relative frequencies, then imposed a symmetric error curve and arrived at method-of-moments estimates (Lagrange, 1776; Stigler, 1999; Hald, 1998).Source: §2 · pp. 2–3
  • Bernoulli first used a hypothesized curve to iteratively reweight observations, then treated it as a density and maximized the product of observation densities (Bernoulli, 1769, 1778; Laplace, 1774).Source: §2 · p. 3
  • Early treatments used maximum-likelihood-like reasoning but offered no reasoned defense of the method or its performance.Their stated rationale was often simply that the chosen value made the observed data’s probability as large as possible.Source: §2 · p. 3
  • Gauss used a uniform prior and chose the posterior mode, yielding least squares under normally distributed errors; later textbooks often omitted the prior invoked in his justification (Gauss, 1809).Source: §2 · p. 3

3. KARL PEARSON AND L. N. G. FILON

The article reviews Pearson and Filon’s suggestive but flawed treatment of estimation, then follows Fisher’s development of a rationale for maximum likelihood. Fisher’s first proof linked sufficiency to optimality under conditions that constrained its reach.

3. KARL PEARSON AND L. N. G. FILON

  • Pearson and Filon’s 1898 expansion of a likelihood ratio led them to claim an error distribution, but their limiting procedure did not depend on the estimation method and was at most valid for maximum likelihood (Pearson and Filon, 1898).Their later steps implicitly used a Bayesian approach and could yield a multivariate normal posterior when higher-order terms were neglected.Source: §3 · p. 3
  • Their formula was applied indiscriminately, including to method-of-moments estimates, and the result was a mess that failed even for the examples presented.The approach nonetheless produced some correct bivariate-normal correlation results and may have suggested ideas to Fisher.Source: §3 · p. 4

4. R. A. FISHER

  • After Pearson challenged the rationale for maximizing a frequency function, Fisher discovered sufficiency in 1918 and began developing his 1922 theory of statistics.Fisher’s earlier Gaussian approach tacitly followed Gauss’s Bayesian reasoning, while Pearson’s rejection focused on the lack of a clear rationale.Source: §4 · p. 4

5. FISHER’S FIRST PROOF

  • Fisher’s first proof argued that a sufficient estimate cannot have greater mean squared error than another approximately normal estimate, yielding optimality with consistency and asymptotic normality.The argument treats approximately normal estimates as exactly normal; Fisher then incorrectly generalized that maximum-likelihood estimates were always sufficient.Source: §5 · pp. 4–5
  • Fisher acknowledged he lacked a rigorous proof that maximum likelihood always produced a sufficient statistic, and his 1922 paper added likelihood-expansion arguments for asymptotic normality and accuracy that did not apply to moment estimates.He also contrasted his maximum-likelihood results for Pearson’s Type-III distributions with Pearson and Filon’s earlier erroneous results for that family.Source: §5 · p. 5

6. THREE YEARS LATER

In 1925 Fisher shifted maximum likelihood theory from sufficiency toward efficiency, then used an analysis-of-variance argument to derive the information inequality and assess information loss. The proof depended on asymptotic assumptions, and Fisher later replaced part of its argument.

6. THREE YEARS LATER

  • Fisher’s 1925 account began with efficiency rather than sufficiency, after he recognized that sufficient statistics of the parameter’s dimension do not always exist.The Cauchy location problem featured in the 1925 paper, though the account only casually noted the possibility of no sufficient statistic.Source: §6 · pp. 5–6
  • Fisher claimed that when an efficient statistic exists, the maximum-likelihood estimate is efficient under the stated large-sample normality and variance conditions.He also noted that sufficient and consistent estimates are maximum likelihood, but sufficiency is not necessary for efficiency; any two efficient estimates become increasingly correlated as n grows.Source: §6 · p. 6

7. THE 1925 “ANOVA” PROOF

  • Fisher’s ANOVA-like proof decomposed score variation into variation conditional on an estimate and variation in its conditional mean, yielding the information inequality.For consistent, asymptotically normal estimates in regular parametric problems, the expected conditional variance is nonnegative, so 1/V ≤ I(θ); equality holds for efficient estimates.Source: §7 · pp. 6–7
  • Fisher treated the expected conditional score variation as residual information loss and introduced ancillary statistics as covariates to reduce it.For efficient but non-maximum-likelihood estimates, he found the information difference tended to a finite limit, later termed second-order efficiency by Rao.Source: §7 · p. 7
  • By 1935 Fisher replaced the 1925 argument for showing that the maximum-likelihood estimate achieves the lower bound 1/I(θ).The 1925 discussion did not specify conditions under which its linear approximation was adequate.Source: §7 · pp. 6–7

8. AFTER 1925: CORRESPONDENCE WITH HOTELLING

Fisher and Hotelling planned a mathematical text on statistical theory, but the project was not completed. Hotelling’s 1930 paper addressed consistency and asymptotic normality of maximum-likelihood estimates, while its proof had a consequential limitation.

8. AFTER 1925: CORRESPONDENCE WITH HOTELLING

  • Fisher invited Hotelling to help develop a mathematical text, but the project was never completed as their interests and circumstances shifted.Hotelling’s move to Columbia and Fisher’s focus on genetics were described as likely reasons for the loss of interest.Source: §8 · p. 8
  • Hotelling’s 1930 paper treated consistency and asymptotic normality of maximum-likelihood estimates, and Fisher said their definitions of consistency differed.The paper may have reflected Fisher’s views after Hotelling’s visit, but the account says Fisher apparently had no direct role in writing it.Source: §8 · p. 8
  • Hotelling’s proof did not establish the needed uniformity beyond discrete distributions with bounded parameter sets.The combined parameter transformation and grouping of observations did not ensure that uniformity; Hotelling later listed validation of the double limiting process as an open problem.Source: §8 · p. 9

9. THE GEOMETRIC SHADOW OF A NASTY LITTLE FACT

Hotelling’s geometric view prompted Fisher to clarify when maximum likelihood is optimal and what class of estimators permits a fair comparison. Hotelling’s proposed improvements showed why consistency alone might not settle the question.

9. THE GEOMETRIC SHADOW OF A NASTY LITTLE FACT

  • Consistency alone does not define a safe comparison class: arbitrary functions of the observations can have arbitrarily small variance, so Hotelling asks how to restrict estimators without reference to the true parameter.He also asks whether accuracy should mean variance or mean squared deviation from the true value.Source: §9 · p. 9
  • In Hotelling’s multinomial geometry, observed relative frequencies x lie in the simplex, while f(p) traces possible probability vectors; the MLE is approximately the curve point nearest x.The figure marks the true parameter p0, the MLE, and a likelihood contour.Source: §9 · p. 9, Figure 2 · p. 10
  • An alternative projection can have smaller variance near high-density regions but larger variance near low-density regions, so its average performance may be worse.Hotelling notes that averaging across parameter values would presume equal a priori probabilities.Source: §9 · p. 10
  • Hotelling’s challenge was that the geometry offered no obvious guarantee against improving on maximum likelihood, even when estimates were consistent.His symmetric-beta example was undermined by a differentiation error, though he continued to speculate about superefficient estimates.Source: §9 · pp. 10–11

10. FISHER’S REPLY: A THIRD PROOF OF THE EFFICIENCY OF MAXIMUM LIKELIHOOD

Fisher’s 1930 correspondence introduced a third proof of maximum likelihood efficiency, using a geometric argument for multinomial models. Its restrictions and use of Euler’s Relation gave a distinct route to the result.

10. FISHER’S REPLY: A THIRD PROOF OF THE EFFICIENCY OF MAXIMUM LIKELIHOOD

  • Fisher’s enclosure also supplied multinomial variances and covariances, and corrected Hotelling’s differentiation error in the beta-density calculation.Fisher later published the proof in disguised estimating-function form, without its geometric origin (Fisher, 1935, pages 45–46).Source: §10 · pp. 11–12
  • Fisher’s third proof assumed multinomial distributions and smoothly differentiable degree-zero homogeneous estimates, a narrower class that made the geometric argument tractable.The estimates depend on relative frequencies rather than sample size.Source: §10 · pp. 11–12
  • Under consistency, Fisher’s geometric characterization says efficient statistics have surfaces touching the expectation line, and maximum likelihood has a plane surface of this type.This summarizes the conclusion of the proof in his enclosure.Source: §10 · p. 11
  • Euler’s Relation for differentiable homogeneous functions let Fisher derive asymptotic variances for the restricted class, then minimize them by Lagrangian methods to recover the maximum-likelihood equations.This route avoided regularity assumptions used in the earlier proof’s normal approximations and approximately constant variances.Source: §10 · p. 12

11. THE SITUATION TO 1950

By 1950, Fisher’s three proofs had differing assumptions and degrees of rigor, while later mathematicians formalized conditions for optimality. Debate also continued over the originality and scope of his contributions.

11. THE SITUATION TO 1950

  • Fisher’s three proofs traded generality for support: the 1922 and 1925 arguments relied on implicit regularity assumptions, while the 1930 proof restricted both distributions and estimators.The third proof was more satisfactory within its multinomial setting, but less general.Source: §11 · p. 12
  • Doob and Dugué encountered new difficulties, while Wald corrected Doob and Mourier later identified Dugué’s error.The effort to supply rigorous support for broad claims continued beyond Fisher’s proofs.Source: §11 · p. 13
  • Wald and Cramér gave the most satisfactory rigorous treatments, but their conditions were complex and limiting; Wald’s 1943 argument excluded normal-standard-deviation estimation.Cramér made conditions explicit while closely following Fisher’s structure.Source: §11 · p. 13
  • Neyman’s 1951 review challenged both Fisher’s priority and the validity of efficiency proofs, claiming the assertion was false in full generality.The review revived an earlier comparison with Edgeworth amid a longstanding dispute (p40; p39).Source: §11 · pp. 13–14
  • Edgeworth had earlier treated likelihood maximization within a restricted class of location estimates, but his proof differed from Fisher’s and did not establish the priority accusation.His result was a partial anticipation, not an instance of Fisher’s proof (p28); the author rejects the 1951 charge (p42).Source: §11 · p. 14
  • Fisher denied intellectual debt to Edgeworth, while the author judges Edgeworth’s work to have merit but not to warrant Neyman’s priority accusation.Fisher granted Edgeworth an “inkling” but no more (p32).Source: §11 · pp. 14–15

12. DOUBTS ABOUT MAXIMUM LIKELIHOOD

Examples from Wald, Neyman–Scott, and Hodges made concrete how maximum likelihood can fail or be improved in particular settings. Later work refined the theory, while the examples remained relevant as cautions or computational concerns.

12. DOUBTS ABOUT MAXIMUM LIKELIHOOD

  • Wald’s 1938 line-fitting problem showed that maximum likelihood could yield estimates whose stochastic limits differ from the parameter values.The example concerned observed points on a line with independent errors in both coordinates; Wald treated normally distributed errors, possibly with unequal variances (p49).Source: §12 · pp. 15–16
  • In the Neyman–Scott example, the maximum-likelihood estimate of σ^2 consistently estimates half its correct value when Xij are independent N(µj,σ^2), for i=1,2 and j=1,...,n.Neyman and Scott published a simplified version of Wald’s example.Source: §12 · p. 16
  • Le Cam’s dissertation included Hodges’s example and proved that the parameter points permitting superefficiency have Lebesgue measure zero.Hodges’s estimate improves locally on maximum likelihood by shrinking toward zero, but performs worse away from zero in finite samples (p56).Source: §12 · p. 16
  • Hodges’s example became a minor technical result, whereas Wald–Neyman–Scott remained a warning for highly parameterized problems where data information may be too thin for consistency.The normal-mixture example also matters computationally because likelihood may require local maxima or parameter constraints.Source: §12 · p. 16
  • Rigorous descriptions of conditions under which maximum likelihood would not mislead became increasingly refined, but proof conditions could be unwieldy and exclude basic examples.Later coverage often restricted the range of cases, and Bahadur’s theorem was limited to one-dimensional parameters and asymptotically normal estimates with continuous variances.Source: §12 · p. 17

13. OF ERRORS IN THEORY

The article catalogs consequential errors in early estimation theory and argues that these missteps also helped drive later advances.

13. OF ERRORS IN THEORY

  • Fisher incorrectly assumed in 1921 that sufficient statistics always exist, while Hotelling’s 1930 proof claimed consistency and asymptotic normality too broadly.The article also notes errors by Lagrange, Gauss, and Pearson and Filon.Source: §13 · p. 17
  • Lambert’s n = 2 example claimed the sample mean was always most probable, but that claim fails for the Cauchy density.The article calls his presentation sketchy and says the example arguably gives the wrong answer.Source: §13 · p. 17
  • Despite these errors, the pioneers’ bold and flawed explorations helped launch twentieth-century theoretical statistics.The article argues that missteps and boldness could contribute to major advances.Source: §13 · p. 17

14. CONCLUSION

Maximum likelihood remains widely useful, and Fisher’s arguments can be defended under implicit regularity conditions. Its use nevertheless carries unresolved risks, especially in complex settings.

14. CONCLUSION

  • Fisher’s proofs can be defended under implicit regularity assumptions, including smooth differentiability and consistency, with stated restrictions in 1922 and 1925.The article says Fisher’s 1922 argument was limited to sufficient estimates and his 1925 argument to score functions linearly approximable by maximum likelihood estimates.Source: §14 · p. 17
  • Increasing parameter counts, unbounded likelihoods, and local improvements over maximum likelihood are potential problems identified by later mathematical investigations.Fisher might have viewed consistency problems with increasing dimension as expected when information is spread over a space whose dimension grows with sample size.Source: §14 · pp. 17–18
  • Fisher–Neyman hostilities helped bring theoretical problems and limitations into public discussion, though they also produced uncivil discourse.The article says the problems were not on public view during the earlier period of relative peace.Source: §14 · p. 18
  • Maximum likelihood has often proved useful without a general theorem defending its use, but complex applications may still enter regions of inapplicability.The article says its limitations are better understood than in Fisher’s time, but not well enough to guarantee safety in complex situations.Source: §14 · p. 18

APPENDIX 1: FISHER’S DECEMBER 1928 DRAFT TABLE OF CONTENTS [HOTELLING PAPERS, BOX 3]

Fisher’s 1928 draft contents outline ranges from distribution theory and estimation to experimental design and statistical mechanics.

APPENDIX 1: FISHER’S DECEMBER 1928 DRAFT TABLE OF CONTENTS [HOTELLING PAPERS, BOX 3]

  • The draft lists topics on distribution forms and variables, including characteristic functions, cumulative moment functions, and distributions derived from the normal.It also names uniqueness of the normal distribution, multinomial and multiple Poisson distributions, and correlation coefficients.Source: APPENDIX 1: FISHER’S DECEMBER 1928… · p. 18
  • The outline includes moment estimates, distributions of estimates, combinatorial methods, sufficient statistics, maximum likelihood, and Bayes’ theorem.It also proposes illustrating inefficiency of moments with Pearsonian curves.Source: APPENDIX 1: FISHER’S DECEMBER 1928… · p. 18
  • Further planned topics include experimental design, amount of information, statistical mechanics, biological problems, and the method of steepest descent.The draft remarks that the steepest-descent method seems indirect and limits statistical argument.Source: APPENDIX 1: FISHER’S DECEMBER 1928… · pp. 18–19

[HOTELLING PAPERS, BOX 45]

Fisher’s draft characterizes consistency and efficiency geometrically: consistency fixes a statistic on the expectation line, while efficiency determines how its surface meets that line.

[HOTELLING PAPERS, BOX 45]

  • For large samples, the draft qualifies its argument by requiring no bias of order as high as n^-1/2 and refers to the multinomial case.The differentials are taken at the expectation point.Source: [HOTELLING PAPERS, BOX 45] · p. 19
  • The draft proposes minimizing the variance subject to the consistency condition and gives equations for admissible derivatives when φ is homogeneous of degree zero in x.The displayed setup identifies the expectation line as x = f(θ) and the statistic as T = φ(x1,...,xs).Source: [HOTELLING PAPERS, BOX 45] · p. 19
  • Consistency fixes the value of T along the expectation line, and efficiency fixes the direction in which its equistatistical surface cuts that line.Statistics satisfying both criteria have surfaces touching the line; the maximum-likelihood surface is the plane surface of this type.Source: [HOTELLING PAPERS, BOX 45] · p. 19

APPENDIX 3: HOTELLING ON PARAMETER SPACES

This appendix reproduces Hotelling’s summary of his work on statistical parameter spaces. It introduces a metric defined through efficient-estimate variances and covariances, then describes the resulting curved geometry.

APPENDIX 3: HOTELLING ON PARAMETER SPACES

  • Hotelling’s paper was read by Oystein Ore in his absence, and only its abstract was published; the surviving summary is the manuscript Ore read.The summary is from the Hotelling Papers at Columbia University.Source: APPENDIX 3: HOTELLING ON PARAMETER… · p. 19

SPACES OF STATISTICAL PARAMETERS

  • The space represents distribution parameters and uses variances and covariances of efficient estimates to define a statistically significant metric.Hotelling’s abstract says ordinary distribution types yield curved spaces.Source: SPACES OF STATISTICAL PARAMETERS · p. 19
  • For the normal law, the two-parameter manifold includes a negatively curved surface of revolution; Pearson Type III distributions share that surface at fixed shape.The unrestricted Type III curve has three parameters, whose space is investigated along with general metrical properties.Source: SPACES OF STATISTICAL PARAMETERS · p. 19

SUMMARY OF “SPACES OF STATISTICAL PARAMETERS”

Hotelling treats parameter estimates and populations as points in a curved space whose metric comes from the likelihood-based estimation framework. He uses its geometry to interpret discrepancies, biological examples, and short-step migration problems.

SUMMARY OF “SPACES OF STATISTICAL PARAMETERS”

  • For samples of N, maximizing the log likelihood gives optimum parameter estimates whose errors approach a normal distribution as N grows.The passage defines the likelihood from observations and the parameters of their frequency distribution.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • The tensor property of the second-derivative quantity motivates using it as a distance element, yielding curved parameter spaces whose curvature depends on the initial population distribution.The passage notes that the second derivative itself is not a tensor under parameter transformations.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • Short-step random migration in parameter space is equivalent to problems of heat conduction and geodesics in the curved space.The passage mentions biological evolution by small mutations and the dilution method of counting soil bacteria as examples.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • For a fixed-shape curve, location and scale are the two parameters; for a normal curve, scale varies along a pseudosphere’s generator and mean by rotation about its axis.Greater variance corresponds to closer proximity to the axis.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • For two related species with similar variance but different means, the geodesic argument implies their most likely common ancestors had greater variance than either species.The passage bases this conclusion on a geodesic passing closer to the pseudosphere’s axis than the meridian.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • For Pearson Type III curves, position and scale vary along loxodromes rather than geodesics.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • The excess area of a geodesic circle on the curved mean–variance surface supplies a correction to χ2 with n = 2 for the probability of a greater discrepancy.The passage applies this to jointly testing differences in means and variances.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
  • On a pseudosphere, a geodesic-circle circumference represents equally likely sample statistics around a population, or equally likely populations given a sample.Source: SUMMARY OF “SPACES OF STATISTICAL… · p. 20
Loading 0804.2996v1…