Everything you wanted to know about Data Analysis and Fitting but were afraid to ask

• The notes cover averaging numerical data and fitting it to functional forms, including data obtained from simulations.Source: §I · pp. 2–3
• They aim to derive the results they use in a style for physicists, addressing gaps the author sees in existing books.The author says physicist-oriented books often omit needed discussion or derivations, while mathematical treatments can feel intimidating to physicists.Source: §I · p. 3
• Python, Perl, and gnuplot scripts are included for selected data-analysis and fitting tasks.Source: §I · p. 3

• Error-bar estimates require uncorrelated data; in Monte Carlo work, binning measurements over intervals longer than the relaxation time can make adjacent bins almost uncorrelated.Independent, re-equilibrated runs provide an alternative, using each run’s average as a data point.Source: A. Basic Analysis · p. 4
• The sample mean estimates the distribution mean, and the sample variance provides an unbiased estimate of the distribution variance.The notes derive these results by considering repeated samples and using statistical independence.Source: A. Basic Analysis · pp. 5–7
• For large N, the error bar in the mean decreases as 1/√N, so reducing it tenfold requires 100 times as much data.Source: A. Basic Analysis · p. 8
• For large N and distributions with finite first two moments, the sample mean is approximately Gaussian, but natural distributions may have heavier tails and more outliers.The Gaussian probabilities are described as a reasonable guide even when N is not very large, though deviations may occur.Source: A. Basic Analysis · pp. 8–9
• For multiple measured variables, analysis must account for cross-correlations, represented by their covariance.The notes distinguish sample averages from exact averages over the unknown distribution.Source: A. Basic Analysis · pp. 5, 9

• Nonlinear functions of averages include fluctuations such as ⟨x2⟩−⟨x⟩2 and the kurtosis ⟨x4⟩/⟨x2⟩2, used in finite-size scaling analyses.The notes state that subtracting 3 from this kurtosis makes it zero for a Gaussian.Source: B. Advanced Analysis · p. 10
• Error propagation, jackknife, and bootstrap are presented as approaches to estimating error bars for nonlinear functions of averages.Jackknife and bootstrap automate error propagation at the price of additional computation.Source: B. Advanced Analysis · p. 10
• For nonlinear functions, the estimate can have a bias of order 1/N, generally smaller than the statistical error of order 1/√N; jackknife and bootstrap can eliminate its leading contribution.Source: B. Advanced Analysis · p. 11

• A second-order expansion separates the leading error, from first-order deviations, from the leading bias, which comes from second-order terms.Source: §1 · p. 11
• The leading bias is of order 1/N, vanishes for a linear function, and can be removed by correcting the estimate using the leading bias term.Source: §1 · p. 12
• The uncertainty estimate must include both the variances of the component quantities and their covariance.The resulting error estimate is of order 1/√N.Source: §1 · p. 12
• For a linear function of one average, the method gives zero bias and an error-bar expression matching the simple-mean result.Source: §1 · p. 12
• Unlike the traditional calculation, jackknife and bootstrap resampling account for error propagation without manually entering partial derivatives or tracking variances and covariances.Source: §1 · p. 13

2. Jackknife

• Each jackknife data set omits one point, and the overall estimate averages the function evaluated on the resulting jackknife averages.A jackknife estimate can be used for nonlinear combinations such as the Binder ratio ⟨x4⟩/⟨x2⟩2.Source: §2 · p. 13
• For linear functions of averages, jackknife and standard averages are identical; for nonlinear functions, resampling can automate bias and error-bar estimation.The method avoids explicitly tracking partial derivatives and variances and covariances when estimating error bars.Source: §2 · pp. 13–14
• The leading bias is usually negligible for large N because it is O(1/N), while the statistical error is O(1/√N).Eliminating the leading bias is therefore usually unnecessary when N is sufficiently large.Source: §2 · p. 14
• The jackknife error bar is obtained from the variance of the jackknife estimates, without entering partial derivatives by hand.For very large N, grouping observations into bins can reduce roundoff errors from subtracting nearly equal numbers.Source: §2 · p. 15

3. Bootstrap

• Each bootstrap data set contains N points sampled with replacement from the original N observations, and Nboot such data sets are generated.A point may appear zero, one, or multiple times in a resampled set.Source: §3 · pp. 15–16
• The bootstrap average of the sample mean is unbiased for the exact mean, and its uncertainty is estimated from variation across bootstrap data sets.For sufficiently large N, the square-root factor in the stated uncertainty relation can be replaced by unity.Source: §3 · p. 17
• For nonlinear functions of averages, bootstrap estimates provide an error bar, while their bias is O(1/N) and leading-bias correction is usually unnecessary for large N.The leading bias can be reduced to O(1/N^2) by a correction, but the text says this is not normally needed when bias is much smaller than statistical error.Source: §3 · p. 18
• Bootstrap cannot give correct error bars for correlated data, and the author has not systematically established how many resamples are needed in practice.Nboot values in the range 100 to 500 are described as typical and seemingly adequate, not systematically validated.Source: §3 · pp. 18–19

4. Jackknife or Bootstrap?

• Use bootstrap when data covering the full range of the sample distribution are needed, and jackknife for quick error bars on combinations of moments.Jackknife requires less calculation and gives identical results on repeated runs with the same data; bootstrap estimates vary across runs.Source: §4 · p. 19

III. FITTING DATA TO A MODEL

• A fitting procedure should report parameter values, parameter-error estimates, and a measure of fit quality; a very poor fit can make the parameter results meaningless.The data are points with corresponding error bars, and the model has M adjustable parameters.Source: §III · pp. 19–20
• Least squares minimizes a weighted sum of squared residuals, weighting points according to their error bars; its usual form assumes uncorrelated data.For linear models, the equations determining the fit parameters are themselves linear; a generalization for correlated data is given later.Source: §III · p. 20
• Even for moderately large samples, natural error distributions may have heavier tails than a Gaussian, making outliers more probable than a Gaussian model predicts.The notes also state that the minimized χ2 distribution result is exact for linear models and is often used as an approximation for nonlinear ones.Source: §III · p. 20
• Linear models are linear in the fitting parameters and generally lead to straightforward linear equations, whereas nonlinear models can yield equations with many or no solutions.A model may be nonlinear in x while remaining linear in its fitting parameters.Source: §III · p. 21
• A log transformation can make some nonlinear parameter dependence linear, but it does not exactly preserve Gaussian errors unless relative errors are small.For the stated log transformation, the condition is σi/yi ≪ 1.Source: §III · p. 21

A. Fitting to a straight line

• Minimizing χ² with respect to a0 and a1 yields two linear equations whose solution determines the straight-line fit.The equations can be solved by eliminating one variable or by using the inverse-matrix formulation.Source: A. Fitting to a straight line · p. 22
• The matrix notation is introduced here for convenience in extending the method to general polynomial fits.For this straight-line case, U is symmetric, with U10 = U01.Source: A. Fitting to a straight line · pp. 22–23
• The slope a1 and intercept a0 are determined from the N data points (xi, yi) and their error bars σi.Source: A. Fitting to a straight line · p. 23

B. Fitting to a polynomial

• A polynomial fit of order M − 1 minimizes χ² with respect to its M parameters aα.Source: B. Fitting to a polynomial · p. 23
• Setting the derivatives of χ² to zero gives M linear equations, one for each α.Source: B. Fitting to a polynomial · p. 23
• The equations are solved using U^-1, which is called the covariance matrix.Uαβ and vα are defined in Eqs. (80) and (81).Source: B. Fitting to a polynomial · p. 23

C. Error Bars

• For a linear model, the simulated-data parameter spread about the observed fit equals the real-data spread about the true parameters; nonlinear models require an effective linear approximation over the needed range.The real-data spread cannot be measured directly from a single actual dataset.Source: C. Error Bars · pp. 24–25
• Diagonal elements of U^-1 give squared parameter error bars, but the simulated-spread equality does not hold precisely for nonlinear fits.For nonlinear models, the covariance matrix is the inverse curvature matrix at the χ² minimum, and treating it as informative about error bars assumes the difference in spreads is small.Source: C. Error Bars · p. 26
• Q is the probability, assuming Gaussian data noise, of obtaining χ² at least as large as the observed value; very small Q values make fit parameters probably meaningless.The notes caution that non-Gaussian statistics increase outliers, so Q around 0.01 may still be acceptable.Source: C. Error Bars · p. 26
• For the Fig. 3 data, χ²/NDOF = 7.44/9 = 0.866 and Q = 0.592; the fitted line is consistent with y = 1 + 2x within the error bars.The data were generated from y = 1 + 2x with zero-mean noise.Source: Figure 3 · p. 27, C. Error Bars · p. 27
• Off-diagonal elements of U^-1 encode fitted-parameter correlations, and the correlation coefficient measures their dimensionless correlation.A good fitting program should also report those coefficients alongside parameter values, error bars, χ²/NDOF, and Q.Source: C. Error Bars · pp. 27–28
• For linear models, the curvature matrix equals U and is constant, whereas U is not constant for nonlinear models.The curvature matrix is defined with a conventional factor of 1/2.Source: C. Error Bars · pp. 28–29

D. Interpolating

• At a chosen x, the best estimate of y is the fitted function evaluated using the optimized parameters, with a separately derived squared error bar.The discussion assumes a general linear model.Source: D. Interpolating · p. 29
• Substituting the optimized parameters expresses the prediction using a kernel that is independent of the observed yi.Source: D. Interpolating · p. 29
• The notes state that Eq. (110) holds generally when one basis function is constant, the usual situation.They also say it is expected to be approximately true without a constant basis function.Source: D. Interpolating · p. 30

E. Fitting to a non-linear model

• Non-linear fitting minimizes χ2, but the resulting equations may have multiple or no solutions, and solving them requires initial parameter values.The notes identify Levenberg–Marquardt as a common method and mention scipy and gnuplot routines as alternatives.Source: E. Fitting to a non-linear model · p. 30

Appendix H.

• For non-linear fits, the curvature matrix varies with the parameters, so its inverse at the χ2 minimum estimates parameter errors only when the uncertainty range remains effectively linear.The covariance matrix’s off-diagonal elements describe correlations between fit parameters.Source: Appendix H · pp. 30–31

F. Confidence limits

• A confidence limit specifies parameter values for which ∆χ2 stays below a chosen threshold; the notes develop the one-parameter-at-a-time case.For that case, the other parameters are varied to minimize χ2 while the selected parameter is held fixed.Source: F. Confidence limits · p. 32
• For Gaussian noise and a linear model—or a non-linear model adequately described by an effective linear model—one-standard-deviation confidence spans 68%, and ∆χ2 = 1 when one parameter varies while the others are re-optimized.The corresponding two-standard-deviation probability is 95.5%.Source: F. Confidence limits · p. 33
• If uncertainty extends beyond an effective linear region, or linear-model data have non-Gaussian noise, bootstrap resampling is needed for confidence limits.The notes suggest using the range ∆χ2 ≤ 1 as an error bar when resampling is unavailable, but say it does not provide a precise confidence limit.Source: F. Confidence limits · pp. 34–35
• When non-linear χ2 is asymmetric, the ∆χ2 = 1 error range can differ on each side from the symmetric local-curvature error bar.For Fig. 4’s data, the parameter is more tightly constrained on the positive side than on the negative side.Source: Figure 4 · p. 34, F. Confidence limits · p. 35

G. Confidence limits by resampling the data

• Bootstrap resampling of raw data yields confidence limits for linear fits with non-Gaussian noise and error bars and limits for nonlinear fits with uncertainty beyond the effective-linear range.Fitting bootstrap datasets gives a parameter distribution; the interval containing 68% of bootstrap values can be interpreted as a 68% confidence limit.Source: G. Confidence limits by resampling the… · p. 35
• When raw data are unavailable but noise is believed close to Gaussian, simulated datasets with standard deviations σi can be fitted to estimate nonlinear parameter distributions without assuming effective linearity.This approach corresponds to the analytical method in Appendix E without requiring the model to remain effectively linear over the parameter range.Source: G. Confidence limits by resampling the… · p. 36

H. A tale of two probabilities. When can one rule out a fit?

• Q measures the probability of the data given a fit, whereas relative fit probabilities change substantially when χ2 changes by unity, so the two probabilities answer different questions.Q changes substantially only for χ2 changes of order NDOF.Source: H. A tale of two probabilities. When… · p. 36, confidence limits · p. 36

confidence limits,

• The cited probability is approximately 1.78 × 10−6, and although a reasonable Q alone does not establish the model, careful analysis concludes the straight-line fit is unlikely to be correct.Source: confidence limits · p. 38

I. Model selection (i.e. how to avoid over-fitting): Maximum Likelihood versus Bayes

• For polynomial fits, χ2 decreases with parameter count, but Q peaks at M = 3 for the parabolic data and then generally declines, supporting the generating model over higher-order fits.For M > 3, Q varies non-monotonically in the example.Source: §I · p. 39, Figure 6 · p. 39

1. Maximum likelihood

• Least squares is equivalent to maximum likelihood under Gaussian noise, and the example’s Q peak at M = 3 correctly selects a parabola for the ten-point dataset.Although χ2 reaches zero at M = N = 10, the fit overfits; its Q peak at M = 3 identifies the parabolic model.Source: §1 · pp. 42–43

2. A Bayesian approach

• Bayes’ theorem relates conditional probabilities as P(X|Y) = P(Y|X)P(X)/P(Y), with the denominator normalizing the conditional probabilities.The section introduces conditional probabilities as the probability of one variable’s value given another’s.Source: §2 · p. 44
• A Gaussian prior adds a regularizer to χ2 that penalizes large fit parameters and suppresses overfitting.The cost function also includes terms that penalize model complexity; the regularizer helps prevent its minimum from occurring at γ = 0, which would recover maximum likelihood.Source: §2 · pp. 47–49
• The procedure iterates fit-parameter and γ updates to convergence for each M, then selects the M minimizing the cost function.It begins with an M and initial γ, updates the fitting parameters, uses them to update γ, and repeats across M values.Source: §2 · p. 50
• Unlike maximum likelihood, the Bayesian cost function is not invariant to shifts or scaling; coupling γ to the constant term means even adding a constant can change the fit.The notes therefore recommend transforming x and y to lie between −1 and 1 before Bayesian analysis.Source: §2 · pp. 50–51
• M = 3 was recovered for both datasets after scaling x and y to −1 to 1 (Fig. 11), whereas unscaled selection failed for Fig. 7 data (Fig. 10).Both datasets were generated from parabolas plus noise, so the expected model had three parameters.Source: §2 · pp. 51–52, Figure 10 · p. 51, Figure 11 · p. 52

3. Conclusions for Model Selection

• Maximum likelihood uses Q to penalize increasing M unless χ2 decreases substantially, while Bayesian model selection is more complicated and must be applied with care.The Bayesian method can be generalized with separate hyperparameters for each fit variable, at substantial additional complexity.Source: §3 · pp. 52–53

Appendix A: Central Limit Theorem

• For large N, sums of independent, identically distributed variables with finite mean μ and variance σ² approach a Gaussian with mean Nμ and variance Nσ².The characteristic function of the sum is Q_N(k) = Q(k)^N; higher-order terms become negligible for large N. The finite-moment assumption excludes distributions such as the Lorentzian.Source: Appendix A · pp. 54–56

central limit theorem

• For finite-mean, finite-variance variables, sums approach a Gaussian as N grows, but convergence is non-uniform and slow in the tails, where deviations can exceed the Gaussian’s 32% probability for more than σ.The notes illustrate slow tail convergence even for a distribution with mean 0 and standard deviation 1 whose higher moments do not exist.Source: central limit theorem · pp. 56–58

Appendix B: The number of degrees of freedom

• With Gaussian data noise, a linear fit using M parameters leaves NDOF = N − M degrees of freedom; the chi-squared statistic is a sum of squares of that many standard Gaussian variables.The notes say this result is often used as a reasonable approximation for nonlinear fitting functions as well.Source: Appendix B · p. 58

Appendix C: The chi-squared distribution and the goodness of fit parameter Q

• The chi-squared goodness-of-fit parameter Q is the probability that the observed χ² or a larger value occurs by chance; for large m, the χ² distribution approaches a Gaussian and χ² per degree of freedom should be around unity.For m > 2, the distribution peaks at X = m − 2.Source: Appendix C · pp. 60–61

Appendix D: Asymptotic standard error and how to get correct error bars from gnuplot

• When data-point errors are unknown but assumed equal, setting χ² per degree of freedom to one estimates their common error and yields an asymptotic standard error for fit parameters.For a constant fit, the fitted parameter is the data average, and the resulting squared error matches the expression for the error in the mean.Source: Appendix D · pp. 61–62, y. The number of degrees of freedom · p. 62

, (gnuplot) .

• Gnuplot’s asymptotic standard errors are incorrect when data points have error bars; divide them by the square root of χ² per degree of freedom to correct them.The correction applies to linear and nonlinear models and to the curve fitting routine; the notes report that Python’s leastsq gives correct error bars.Source: , (gnuplot) · pp. 62–63

Appendix E: The distribution of fitted parameters determined from simulated datasets

• For an arbitrary linear model, the appendix derives the distribution of fitted parameters across simulated datasets, assuming Gaussian noise.The setup starts with one dataset and its fitted parameters, then considers an ensemble of simulated datasets.Source: Appendix E · p. 63

Appendix F: The distribution of fitted parameters from repeated sets of measurements

• For a linear model with Gaussian noise, fitted parameters from simulated datasets relative to one actual dataset have the same distribution as parameters from repeated actual datasets relative to the true value.The repeated-dataset distribution is Gaussian; the same result is stated to apply to nonlinear models when the needed parameter range is small enough for an effective linear model, and is often used as an approximation beyond that condition.Source: Appendix F · p. 65

Appendix G: Fitting correlated data

• Correlated-data fitting changes the expressions for U and v but retains the parameter covariance of Eq. (98), although correlations are rarely known well enough to use the method.The derivation assumes Gaussian noise and a positive-definite matrix; it uses a linear model for notation, with nonlinear generalization described as straightforward.Source: Appendix G · pp. 66–67

Appendix H: Scripts for some data analysis and fitting tasks

• The scripts appendix provides Perl, Python, and Gnuplot examples for basic data-analysis and fitting tasks, with output on datasets available on the web.Source: Appendix H · p. 67

1. Scripts for a jackknife analysis

• 1.8215 ± 0.0368 is the reported jackknife average for ⟨x4⟩/⟨x2⟩2; the Python script gives the same output as Perl.The scripts compute the ratio and its jackknife error bar from input values of x.Source: §1 · p. 67, a. Perl · p. 68, b. Python · p. 69

a. Perl, writing out the formulae by hand

• Perl and Python scripts implement a straight-line fit by hand, and the Python results are identical to the Perl results.The scripts calculate slope, intercept, their uncertainties, and chi-squared per degree of freedom from x_i, y_i, and σ_i.Source: a. Perl, writing out the formulae by… · p. 70, b. Python, writing out the formulae by… · p. 71

c. Python, using a built-in routine from scipy

• The SciPy curve_fit script performs a straight-line fit and corrects the error bars; its results match those of the preceding scripts.It computes chi-squared per degree of freedom and rescales parameter errors using that value.Source: c. Python, using a built-in routine… · p. 72

d. Gnuplot

• The Gnuplot script fits a straight line and displays the data, fit, fit parameters, error bars, chi-squared per degree of freedom, and goodness-of-fit parameter.The script corrects the displayed parameter error bars by dividing by FIT_STDFIT.Source: d. Gnuplot · pp. 72–73

a. Python

• The nonlinear-fit Python script fits y = Tc + A / x^w to determine Tc, A, and w.It uses SciPy curve_fit and computes chi-squared and the quality-of-fit parameter Q.Source: §3 · p. 73, a. Python · p. 74
• Python reports Tc = -0.2570 +/- 1.4775, A = 2.7878 +/- 0.8250, w = 0.2060 +/- 0.3508, chi squared / NDF = 0.2541, and Q = 0.9073.Source: a. Python · p. 75

b. Gnuplot

• Gnuplot plots a fit to y = Tc + A / x^w, and its results are reported to agree with the Python results.Source: b. Gnuplot · pp. 75–76, a. Python · p. 75, Figure 15 · p. 76
• The quoted symmetric Tc error is clearly unreasonable because chi-squared varies asymmetrically near its minimum; profiling Tc or bootstrapping is preferable, but the raw data needed for bootstrapping are unavailable.Source: b. Gnuplot · p. 77