A Unified Approach to the Classical Statistical Analysis of Small Signals

I. INTRODUCTION

• The method uses a likelihood-ratio ordering within Neyman’s construction to unify upper limits and two-sided confidence intervals.This ordering yields intervals that are never unphysical or empty and removes a motivation for the PDG’s Bayesian procedures.Source: §I · p. 2
• Data-dependent switching between upper-limit and central-interval tables can produce intervals that undercover the true value.The paper identifies this flip-flopping problem as a failure of the classical confidence-interval criterion.Source: §I · p. 2
• The unified construction also decouples interval calculation from goodness-of-fit testing, unlike traditional classical upper-limit calculations.Source: §I · p. 2
• The construction is applied to Poisson processes with background and Gaussian errors with a bounded physical region, then generalized to neutrino-oscillation searches.Source: §I · p. 2

A. Bayesian Intervals

• Bayesian intervals use a posterior based on a prior for the unknown parameter, whereas classical intervals are constructed from P(x|µ) and do not assign a probability distribution to µ_t.The paper emphasizes that confidence intervals must not be interpreted as Bayesian statements about P(µ_t|x_0).Source: A. Bayesian Intervals · p. 3

B. Classical Confidence Intervals

• For each parameter value, Neyman’s confidence-belt construction selects an acceptance interval in the observable with probability α.The acceptance region must be specified by auxiliary criteria or an ordering principle that is not influenced by the observed data.Source: B. Classical Confidence Intervals · pp. 4–5
• One-sided acceptance choices produce upper confidence limits, while symmetric tail choices produce central confidence intervals.Source: B. Classical Confidence Intervals · p. 5
• After observing x_0, the confidence interval is the set of parameter values whose acceptance regions contain that observation.Graphically, it is obtained by intersecting the horizontal acceptance regions with a vertical line through x_0.Source: B. Classical Confidence Intervals · p. 5
• Because the acceptance condition holds for every parameter value, the resulting intervals have the stated frequentist coverage for the fixed unknown true value.Source: B. Classical Confidence Intervals · p. 5

A. Gaussian with Boundary at Origin

• In the Gaussian problem, the physical parameter is restricted to µ ≥ 0, so the confidence belt excludes negative parameter values.Source: A. Gaussian with Boundary at Origin · p. 5
• Standard upper-limit and central-interval belts are valid when the reporting choice is made before the experiment, but not when it is selected from the observed result.Source: A. Gaussian with Boundary at Origin · pp. 5–6
• A flip-flopping policy that reports an upper limit for x < 3σ and a central interval otherwise produces undercoverage over a substantial range of µ.For µ = 2.0, the implied acceptance interval contains only 85% of P(x|µ), so the quoted intervals are neither confidence intervals nor conservative confidence intervals.Source: A. Gaussian with Boundary at Origin · p. 6
• Standard belts can yield an empty confidence interval when x = −1.8, despite having correct ensemble coverage when used without flip-flopping.The ordering principle introduced later is stated to solve both the flip-flopping and empty-set problems.Source: A. Gaussian with Boundary at Origin · p. 6

B. Poisson with Background

• The construction uses Neyman acceptance regions for signal mean µ with known background b, comparing it with standard upper-limit and central-interval orderings.Representative intervals are tabulated at 68.27%, 90%, 95%, and 99% confidence levels; italicized entries require caution when the observed count is very unlikely even for µ = 0.Source: B. Poisson with Background · pp. 6–7, A. Poisson with Background · pp. 8–9
• Because n is discrete, the acceptance region can exceed the stated confidence level, making some conservatism unavoidable in the Poisson problem.The construction adds only slight further conservatism to compensate for the two pathologies, while discreteness remains the dominant source.Source: B. Poisson with Background · pp. 6–7, A. Poisson with Background · p. 8
• Standard Poisson belts can undercover when data determine whether upper-limit or central-interval tables are used, and can become empty when b = 3 and no events are observed.The likelihood-ratio ordering is introduced to solve both problems.Source: B. Poisson with Background · p. 7

A. Poisson with Background

• Values of n are ranked by R = P(n|µ)/P(n|µbest), and added until the acceptance probability reaches the desired confidence level.For µ = 0.5 with b = 3, the resulting 90% acceptance region is n = [0, 6].Source: A. Poisson with Background · pp. 7–8
• For fixed observed n, mild discreteness-induced irregularities are corrected by forcing the upper endpoint µ2 to be non-increasing with background b.The correction was investigated over b in [0, 25] using a fine grid.Source: A. Poisson with Background · pp. 8–9
• The likelihood-ratio construction produces two-sided intervals at large n and automatic upper limits with µ1 = 0 for n ≤4, eliminating data-dependent flip-flopping and empty intervals.At large n, background is effectively subtracted without constraint; at small n, the construction provides one coherent confidence-belt family.Source: A. Poisson with Background · pp. 7–8

B. Gaussian with Boundary at Origin

• The Gaussian construction orders outcomes using the likelihood ratio relative to the best physically allowed mean, µbest = max(0, x).For each µ, acceptance-region endpoints are solved numerically on a grid and inverted to obtain [µ1, µ2].Source: B. Gaussian with Boundary at Origin · p. 9
• At large x, the bounded and unconstrained confidence intervals coincide because the physical boundary no longer constrains the result.Figure 10 is compared with Figure 3 to show this large-x agreement.Source: B. Gaussian with Boundary at Origin · p. 9
• Below x = 1.28, the lower endpoint is zero, creating an automatic transition from two-sided intervals to the upper limit µ2.For 90% confidence, the transition follows from the acceptance interval at µ = 0, where x2 = 1.28.Source: B. Gaussian with Boundary at Origin · p. 10
• At x = 0, the new 90% C.L. upper limit is µ2 = 1.64, equal to the standard 95% limit rather than the standard 90% value 1.28.The loosening reflects compensation for undercoverage caused by choosing between upper and central intervals using the data.Source: B. Gaussian with Boundary at Origin · p. 10
• For sufficiently negative x, the upper limits decrease asymptotically as 1/|x|, but such limits require caution when all physical µ values make the measurement unlikely.This caveat concerns interpretation of measurements that are unlikely under every physically allowed parameter value.Source: B. Gaussian with Boundary at Origin · p. 10

• The approach mirrors ordinary χ2 fitting, where goodness-of-fit and parameter-interval confidence levels can otherwise be chosen independently.Source: C. Decoupling of Goodness-of-Fit C.L.… · p. 10
• Standard intervals can be empty or unphysical, effectively coupling the goodness-of-fit level to the interval confidence level.At 90% confidence, the empty set can occur 10% of the time in the constrained Gaussian case and up to 10% in the Poisson case, depending on background and discreteness.Source: C. Decoupling of Goodness-of-Fit C.L.… · pp. 10–11
• The construction always supplies a confidence interval at the desired level while permitting an independent goodness-of-fit assessment.For discrete problems, some conservatism remains.Source: C. Decoupling of Goodness-of-Fit C.L.… · p. 11
• In the constrained Gaussian example, x0 = −2.0 yields the 90% C.L. interval [0, 0.4], with goodness-of-fit assessed by P(x ≤ x0) under µ = 0.Source: C. Decoupling of Goodness-of-Fit C.L.… · p. 11
• In the Poisson example, n0 = 1 with b = 3 yields the 90% C.L. interval [0, 1.88], with goodness-of-fit assessed by P(n ≤ n0) under µ = 0.Source: C. Decoupling of Goodness-of-Fit C.L.… · p. 11

A. The Experimental Problem

• Neutrino oscillation experiments search for transformation between muon- and electron-type neutrinos, modeled as mixing between two mass eigenstates.Source: A. The Experimental Problem · pp. 11–12
• The transformation probability depends on baseline L, neutrino energy E, and the mass-squared difference ∆m2 = |m2^2 − m1^2|.Source: A. The Experimental Problem · p. 12
• Results are represented as two-dimensional confidence regions over sin2(2θ) and positive ∆m2.Source: A. The Experimental Problem · p. 12
• In the example described, no oscillation evidence is observed, so the confidence region is the area to the left of the plotted curve.Source: A. The Experimental Problem · p. 12

• The method addresses bounded sin2(2θ), where small signals amid background can produce unconstrained best fits with negative, physically meaningless values.Source: B. The Proposed Technique for… · p. 12
• The measured data comprise event counts across energy or other-variable bins, with known background B and oscillation contribution T parameterized by sin2(2θ) and ∆m2.Source: B. The Proposed Technique for… · p. 12
• For each parameter point, acceptance regions are built from simulated experiments using the likelihood-ratio ordering adopted from the one-dimensional construction.Observed ∆χ2 is compared with a critical value calibrated from simulations.Source: B. The Proposed Technique for… · pp. 12–13
• In the Gaussian regime, the likelihood-ratio construction is equivalent to using the difference between χ2 at a tested point and at the best fit.Source: B. The Proposed Technique for… · p. 13
• For the toy model, the critical ∆χ2 varies from about 2.4 to 6.6 rather than remaining at 4.61, because of boundary proximity, sinusoidal oscillation structure, and effectively one-dimensional regions.The one-dimensional behavior occurs at very low ∆m2, where the lowest-energy bin dominates.Source: B. The Proposed Technique for… · pp. 13–14

• The proposed technique is evaluated against three classical alternatives: Raster Scan, Flip-Flop Raster Scan, and Global Scan.These methods use Gaussian approximations instead of a full Neyman construction.Source: C. Comparison to Alternative Classical… · pp. 14–15
• All three alternatives can either exclude the entire physical plane or produce limits unrepresentative of the experiment’s sensitivity.Source: C. Comparison to Alternative Classical… · p. 14
• The Raster Scan gives exact coverage but lacks power because it cannot distinguish likely from unlikely ∆m2 values at fixed ∆m2.Source: C. Comparison to Alternative Classical… · p. 14
• For a positive-signal example, both the proposed technique and Raster Scan provide exact coverage, but only the proposed technique isolates the signal, with one ghost region.Source: C. Comparison to Alternative Classical… · pp. 14–15
• The Flip-Flop Raster Scan undercovers, with coverage dropping as low as 85% because the data determine whether an upper limit or two-sided interval is used.Source: C. Comparison to Alternative Classical… · pp. 14–15
• The Global Scan is powerful but has improper coverage, varying from about 76% to 94% across the parameter plane.Source: C. Comparison to Alternative Classical… · p. 15

• The method still yields lower upper limits for fewer observed events than expected background, a behavior the authors identify as troublesome, especially with zero observations.Source: §VI · p. 15
• The proposed sensitivity is the average upper limit from experiments with the expected background and no true signal.Source: §VI · p. 15
• For two observed events with expected background 3.5, the method gives a 90% C.L. upper limit of 2.7 events and sensitivity of 4.6 events.Source: §VI · p. 16
• Sensitivity curves should accompany upper-limit curves when a substantial portion of the limit lies below the experiment’s sensitivity.Source: §VI · p. 16

• The ordering produces intervals that transition automatically from upper limits to two-sided intervals, eliminating undercoverage from data-dependent flip-flopping.Source: §VII · p. 16
• The construction solves nonphysical or empty-set intervals and avoids coupling the goodness-of-fit confidence level to the interval confidence level in the stated applications.Source: §VII · p. 16
• The method generalizes to neutrino oscillation searches with physical constraints and multiple local minima in the probability density.Source: §VII · p. 16
• The authors recommend publishing measured values, uncertainties, observed events, expected background, and conversion factors so results can be recalculated or combined.Source: §VII · p. 16
• The ordering is described as naturally implied by likelihood-ratio test theory, although the authors were unaware of its prior application to scientific confidence-interval presentation.Source: §VII · pp. 16–17

• Figure 8 plots the upper endpoint µ2 against known background b for observed counts n0 from 0 through 10, with annotations for nonzero µ1 and low goodness-of-fit probability.Source: Figure 8 · p. 27
• Figure 9 gives the corresponding lower endpoint µ1 values in regions where the proposed intervals have nonzero lower bounds.Source: Figure 9 · p. 28
• Figure 10 presents the proposed 90% confidence intervals for the mean of a Gaussian constrained to be non-negative.Source: Figure 10 · p. 29
• Figures 11–14 illustrate neutrino-oscillation confidence regions and under- or overcoverage for the proposed and alternative scans, including the Raster Scan, Flip-Flop Raster Scan, and Global Scan.Source: Figure 11 · p. 30, Figure 12 · p. 31, Figure 13 · p. 32
• Tables II–IX provide Poisson intervals across confidence levels, observed counts, and background ranges, while Table X gives constrained-Gaussian intervals.Source: Table II · p. 36, Table III · p. 36, Table IV · p. 37
• Table XI summarizes properties of the proposed neutrino-oscillation technique and three alternative classical techniques, and Table XII reports experimental sensitivity versus expected background.Source: Table XI · p. 41, Table XII · p. 41