PaperlayerView exact revision on arXiv

Source-linked AI summary

Data analysis recipes: Fitting a model to data

David W. Hogg, Jo Bovy, Dustin Lang

arXiv:1008.4686v1astro-ph.IMphysics.data-an

TL;DR

The paper examines why standard linear fitting is often inadequate when uncertainties, outliers, intrinsic scatter, or nuisance parameters complicate the data. It advocates generative models to compute likelihoods or posterior distributions, while noting limits of orthogonal-only modeling and frequentist treatment of nuisance parameters.

  • Problem

    Standard weighted least-squares assumptions are rarely met because data can have heterogeneous, covariant, unknown, or intrinsically scattered uncertainties and substantial outliers.

  • Method

    The paper develops generative-model-based likelihood and posterior approaches for fitting straight lines with arbitrary two-dimensional uncertainties, outliers, unknown uncertainties, intrinsic scatter, and nuisance parameters.

  • Results

    Generative models make fitting non-arbitrary by enabling direct computation of parameter likelihoods or posterior probability distributions.

  • Takeaways & Limitations

    Posterior construction is indispensable when powerful nuisance parameters must be marginalized, whereas maximum likelihood alone may reveal little about the parameters of interest.

  • Takeaways & Limitations

    The orthogonal-distribution procedure models only the density perpendicular to the relationship, while fully general methods require modeling and deconvolving the intrinsic two-dimensional density.

Abstract

from arXiv · show

We go through the many considerations involved in fitting a model to data, using as an example the fit of a straight line to a set of points in a two-dimensional plane. Standard weighted least-squares fitting is only appropriate when there is a dimension along which the data points have negligible uncertainties, and another along which all the uncertainties can be described by Gaussians of known variance; these conditions are rarely met in practice. We consider cases of general, heterogeneous, and arbitrarily covariant two-dimensional uncertainties, and situations in which there are bad data (large outliers), unknown uncertainties, and unknown but expected intrinsic scatter in the linear relationship being fit. Above all we emphasize the importance of having a "generative model" for the data, even an approximate one. Once there is a generative model, the subsequent fitting is non-arbitrary because the model permits direct computation of the likelihood of the parameters or the posterior probability distribution. Construction of a posterior probability distribution is indispensible if there are "nuisance parameters" to marginalize away.

1 Standard practice

Standard weighted least squares fits a straight line by weighting y-direction residuals with known Gaussian uncertainties while treating x as exact. The same matrix framework can be extended to covariances and quadratic models, but its interpretation depends on the assumptions.

  • The standard setup assumes N > 2 points with known Gaussian uncertainties σy_i in y and no uncertainty in x.
  • The best-fit slope m and intercept b are obtained from a covariance-weighted matrix solution.The covariance matrix can be generalized beyond diagonal form to include covariances among points.
  • The method is correct when its uncertainty assumptions hold, because it gives greater relative weight to data with higher statistical significance.
  • The matrix solution minimizes χ2, the total squared error scaled by the uncertainties.Under the stated Gaussian assumptions, χ2 provides the objective minimized by the fitted parameters.
  • The exercises apply the method to subsets and larger data sets, then generalize it from a straight line to a quadratic relationship.

2 The objective function

The paper argues that fitting should begin with a scientifically justified objective function, preferably derived from a generative model for the data. For Gaussian line-fitting errors, likelihood maximization reproduces χ2 minimization, while Bayesian posteriors incorporate prior knowledge and nuisance-parameter marginalization.

  • A generative model is a parameterized quantitative description of a statistical procedure that could reasonably have generated the data.
  • The likelihood supplies a justified scalar objective by measuring the probability of the observed data under candidate line parameters.For independent points, the likelihood is formed from the product of conditional probabilities.
  • Likelihood maximization is identical to χ2 minimization for the Gaussian straight-line model, thereby justifying the standard procedure.
  • Bayesian inference combines the likelihood with prior knowledge to produce a posterior over the line parameters and other quantities.The posterior denominator acts as a normalization obtained by marginalizing the numerator over parameters.
  • With an uninformative prior, the line maximizing likelihood comes very close to the line maximizing the posterior probability.
  • Gaussian distributions make the justified objective quadratic in the data, so its optimum is a linear function of the data and is trivial to obtain.

3 Pruning outliers

Standard linear fitting is sensitive to outliers, so the paper replaces ad hoc rejection with probabilistic models that represent good and bad data. A mixture model permits posterior inference for the line while marginalizing uncertainty about which points are outliers and about the outlier distribution.

  • Standard linear fitting is highly sensitive to points far from the expected relation because of unmodeled uncertainty, rare noise, or restricted model scope.
  • The outlier model introduces binary indicators q_i, a bad-point probability P_b, and parameters describing the bad-point mean and variance.The indicators mark each point as good or bad, while the additional parameters define the outlier distribution.
  • The model permits objective ranking of individual points by posterior badness, although it assumes equal prior probabilities that points are bad.
  • After marginalizing each indicator, every observation is modeled as a mixture of the straight-line and outlier distributions.The mixture amplitudes are 1−P_b for foreground points and P_b for bad points.
  • Marginalizing the mixture posterior over P_b, Y_b, and V_b yields a posterior for the line parameters without exploring an exponential number of alternatives.The resulting integration involves only three continuous parameters rather than explicitly evaluating all 2^N assignments.
  • Marginalization is necessary to account for covariance between the line parameters and nuisance parameters when estimating uncertainties or confidence intervals.

4 Uncertainties in the best-fit parameters

Uncertainty estimates from standard linear fitting are reliable only under strict assumptions that are rarely met; posterior distributions and empirical resampling provide alternatives when those assumptions fail.

  • Underestimated observational uncertainties can make posterior behavior pathological, including degraded outlier-probability inference and multi-modal distributions.The exercises illustrate worsening of the marginalized Pb distribution and possible multi-modality when uncertainties are reduced.
  • The standard parameter-uncertainty formula is strictly valid only with correctly estimated variances, exact χ2 treatment, and an accurate negligible-scatter Gaussian generative model.These conditions commonly fail because uncertainties may be rough or missing, data may be rejected, and relationships may have intrinsic scatter or curvature.
  • Posterior distributions for (m, b) can provide uncertainty estimates that respond to how well the model fits the data.Their second moments change as the data become inconsistent with a narrow straight-line model, unlike standard covariance elements that do not depend on fit quality.
  • Off-diagonal covariance terms matter because linear combinations of slope and intercept can be known much better than either parameter individually.Propagation of best-fit results generally requires the full 2 × 2 uncertainty tensor or a two-dimensional sampling of the marginalized posterior.
  • Bootstrap resamples N observations with replacement across M trials, while jackknife refits the model N times, leaving out one observation each time.Bootstrap estimates parameter variability across resampled datasets; jackknife measures sensitivity to individual data points.
  • Bootstrap and jackknife are useful when uncertainties are unknown or distrusted, but require data to represent a reasonable stationary sampling and the model to fit reasonably well.The jackknife correction factor [N −1]/N is justified only in the limit of Gaussian, identically noisy points.

5 Non-Gaussian uncertainties

Non-Gaussian uncertainties require methods beyond standard Gaussian fitting. The paper favors modeling the uncertainty distribution itself, while noting that this may make optimization substantially harder.

  • Standard linear fitting is justified only when measurement uncertainties are Gaussian with known variances, a condition often violated by real-world noise processes.Non-Gaussian uncertainty may arise from limits, constrained true values, or probabilistic inference underlying a measurement.
  • Alternative strategies include estimating total variance and assuming a conservative Gaussian, rejecting data, or heuristically softening the likelihood.The paper generally recommends data rejection over softening that makes the objective function difficult to justify.
  • The paper identifies fully understanding and modeling non-Gaussianity as the only genuinely justified general approach.Mixtures of Gaussians can represent distributions with different variances, offsets, and amplitudes, and can approximate any reasonable frequency distribution in principle.
  • Arbitrarily complicated non-Gaussian uncertainties and joint distributions can be incorporated, although fitting then becomes more challenging from an optimization standpoint.The resulting objective function can remain completely general when the uncertainty structure is modeled explicitly.
  • For upper limits, lower limits, and forbidden regions, the recommended procedure is to model the uncertainty distribution and construct the corresponding justified scalar objective function.Optimization may be challenging, but the paper treats that difficulty as an engineering issue rather than a reason to sacrifice scientific correctness.

6 Goodness of fit and unknown uncertainties

Goodness-of-fit assessment and uncertainty estimation are coupled: testing assumptions requires trusted uncertainties, while testing uncertainties requires trusted assumptions. The section therefore examines χ2 diagnostics and Bayesian inference with marginalization when uncertainties are unknown.

  • Coupled assessment: χ2 model assessment depends jointly on the fit assumptions and the individual uncertainty estimates.The standard frequentist paradigm cannot test one confidently without confidence in the other.
  • χ2 interpretation: A χ2 value near the expected range makes a good model conceivable but does not establish that the model is correct.The section presents χ2 as a diagnostic rather than a decisive validation criterion.
  • Correlated uncertainties: Nonzero covariances require replacing the independent-point sum with the matrix form involving the inverse covariance elements wij and cross terms.The covariance matrix’s off-diagonal elements encode correlations among measurements.
  • Unknown uncertainties: Adjusting variances or covariances until χ2 looks reasonable is rarely reliable because the model may not be independently known to be good.Directly understanding the uncertainty structure is preferred to tuning it through χ2.
  • Unknown uncertainties: Bayesian inference can place unknown individual variances in the parameter set, infer their joint posterior, and marginalize them to obtain a posterior for (m, b).The procedure requires priors on the unknown variances and can be computationally expensive.
  • Unknown uncertainties: With a relatively flat variance prior and a reasonable model, inferred uncertainties are pushed away from both implausibly large and implausibly small values.Large uncertainties make the model insufficiently specific, while small uncertainties expose model inaccuracies.

7 Arbitrary two-dimensional uncertainties

For data with uncertainties in both coordinates, the paper constructs a likelihood from two-dimensional Gaussian measurement models and fits the line through projected uncertainties. This avoids procedures such as forward–reverse fitting and can be extended to outliers.

  • Uncertainty model: Two-dimensional measurements require accounting for uncertainty in both x and y, including covariance between them.The covariance tensor collects the directional variances and covariance for each point.
  • Uncertainty model: Gaussian two-dimensional uncertainties define the probability of each observed point given a true point and its covariance tensor.This provides the measurement model used to construct the fitting likelihood.
  • Likelihood construction: The likelihood specifies a candidate line, evaluates each observation given possible true locations on that line, and marginalizes over those unobserved locations.The resulting construction treats the true position along the line as a missing datum for each point.
  • Likelihood construction: Projecting each point’s two-dimensional uncertainty onto the direction orthogonal to the line yields the displacement and variance used in the likelihood.The line can be parameterized by slope and intercept or by angle and perpendicular offset.
  • Parameterization: Maximum-likelihood fitting or posterior inference is performed directly from the constructed likelihood rather than by separately fitting forward and reverse relations.The paper recommends coordinates (θ, b⊥), with priors naturally taken as flat in angle and perpendicular offset.
  • Scope and extensions: The method assumes a narrow linear relation with Gaussian measurement offsets, and extensions are needed for intrinsic width or outliers.The paper applies a mixture-based outlier model in an exercise for the full data set.

8 Intrinsic scatter

When the underlying relationship is near-linear but intrinsically broad, the paper introduces an orthogonal intrinsic Gaussian variance and incorporates it into the likelihood. Bayesian marginalization then yields posterior information about that variance, while the approach remains limited by its one-dimensional treatment orthogonal to the relation.

  • Motivation: Real relationships are often not exact even with noise-free observations, motivating a model with intrinsic scatter around the line.The paper attributes such scatter generically to additional unmeasured or unmeasurable quantities.
  • Model: An intrinsic Gaussian variance V is introduced orthogonal to the line, expanding the relationship parameters to (θ, b⊥, V).This models a linear relationship that is not narrow.
  • Likelihood: The likelihood convolves each projected measurement Gaussian with the intrinsic-scatter Gaussian, adding their variances.Because Gaussian convolution is analytic, the resulting likelihood includes a penalty for broad variances.
  • Limitation: The procedure models only the distribution orthogonal to the relationship rather than the full intrinsic two-dimensional density.Fully general treatment would involve density estimation and deconvolution after convolving the intrinsic density with measurement uncertainty.
  • Inference: Maximum-likelihood fitting produces an intrinsic relation represented by the fitted line together with ±√V boundaries.The corresponding exercise asks for the maximum-likelihood intrinsic relation after excluding only data point 3.
  • Inference: Bayesian fitting marginalizes over (θ, b⊥) to obtain a posterior distribution for the intrinsic variance V.The paper’s exercise requests 95 and 99 percent upper limits on V.

Notes

The notes qualify the paper’s scope, explain why fitting procedures should follow justified objectives and generative assumptions, and identify limits of standard and ad hoc methods.

  • Scope: For data-driven prediction, standard line fitting can remain the best linear predictor under some assumptions even when it is not an appropriate model.The applicability conditions depend on model selection and exactly what is being predicted.
  • Methodological standards: Testing ad hoc procedures on simulated data evaluates recovery of the artificial model’s parameters, not necessarily performance on real data.The critique is that procedures may not optimize any objective a scientist actually cares about.
  • Standard fitting: The standard least-squares problem is uniquely and globally solvable because its objective is quadratic and convex, but realistic modifications break linearity and convexity.The resulting optimization can require substantially more thought, especially as assumptions about uncertainties and data generation are relaxed.
  • Model assumptions: The paper treats x_i as experimental-design inputs rather than data, creating a total asymmetry between x_i and y_i in the standard approach.Under the stated generative assumptions, marginalization can also reconstruct the true position of an individual missing data point.
  • Inference limits: With nuisance parameters, marginalization can make regions with specific data explanations dominate the posterior, whereas frequentists cannot summarize the full acceptable parameter range in the same way.The paper also notes that PCA is strongly affected by individual noise covariances and becomes applicable only when data uncertainties are negligible.
Loading 1008.4686v1…