PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding New Physics Searches with Unsupervised Learning

Andrea De Simone, Thomas Jacques

arXiv:1807.06038v3hep-phhep-exphysics.data-an

TL;DR

Model-specific searches and practical constraints leave a need for model-independent ways to compare observations with expectations for signs of new phenomena. The paper proposes an unsupervised, Nearest-Neighbors compatibility test; it reports sensitivity to small numbers of new-physics events and usefulness for locating discrepancies even when background simulation is insufficient for discovery claims.

  • Problem

    The paper seeks a statistically sound way to determine whether observed data are compatible with expectations or contain signals of new phenomena.

    Source: §1 · p. 2

  • Method

    An unsupervised statistical test compares two samples by estimating their local density ratio with a Nearest-Neighbors approach.

    Source: §1 · p. 3

  • Results

    The method is reported to be highly sensitive even to a small number of New Physics events in LHC data.

    Source: §5 · p. 22

  • Takeaways & Limitations

    The method can identify regions of discrepancy to guide further analyses, including when background accuracy is insufficient to claim discovery.

    Source: §3.3 · p. 19, §5 · p. 22

    Accurate Standard Model background estimation remains a significant challenge, and matching simulations to data in control regions introduces model-dependent assumptions.

    Source: §3.3 · p. 19

Abstract

from arXiv · show

We propose a new scientific application of unsupervised learning techniques to boost our ability to search for new phenomena in data, by detecting discrepancies between two datasets. These could be, for example, a simulated standard-model background, and an observed dataset containing a potential hidden signal of New Physics. We build a statistical test upon a test statistic which measures deviations between two samples, using a Nearest Neighbors approach to estimate the local ratio of the density of points. The test is model-independent and non-parametric, requiring no knowledge of the shape of the underlying distributions, and it does not bin the data, thus retaining full information from the multidimensional feature space. As a proof-of-concept, we apply our method to synthetic Gaussian data, and to a simulated dark matter signal at the Large Hadron Collider. Even in the case where the background can not be simulated accurately enough to claim discovery, the technique is a powerful tool to identify regions of interest for further study.

1 Introduction

The paper addresses the need for model-independent searches for new phenomena, since typical LHC searches assume a specific signal model and use limited features. It proposes an unsupervised compatibility test that compares samples without distributional assumptions or fixed bins, retaining high-dimensional information.

1 Introduction

  • The proposed unsupervised test measures compatibility between samples using a Nearest-Neighbors estimate of their local density ratio.It is framed for comparing observations with expectations to detect discrepancies.Source: §1 · pp. 2–3
  • Typical LHC searches assume a New Physics model, face impractical likelihood evaluation, and select a few features, limiting model-independent searches.The paper notes that relatively less effort has gone into model-independent strategies.Source: §1 · p. 3
  • The method is model-independent and non-parametric, requiring no assumed signal model or knowledge of the underlying distributions.It assesses traces of new phenomena regardless of the specific physical model that may have generated the data.Source: §1 · p. 3
  • Its un-binned partition avoids fixed rectangular bins and retains information across the full high-dimensional feature space.This is motivated by cases where a few variables or high-dimensional histograms cannot preserve useful information.Source: §1 · pp. 3–4
  • Although focused on LHC searches, the method can also detect incompatibilities between samples in other settings.The authors identify near-indistinguishable data and reference distributions as a particularly useful situation.Source: §1 · p. 4

2 Statistical test of dataset compatibility

The section formulates compatibility as a two-sample test of whether unknown probability densities match, then estimates a density-based statistic and calibrates its p-value by resampling. It also extends the test to uncertain feature positions, for which weaker rejection evidence is expected.

2.1 Definition of the problem

  • The test asks whether benchmark and trial samples come from the same unknown PDF, using a distribution-free whole-distribution comparison that retains multidimensional information without fixed bins.Source: §2.1 · pp. 4–5

2.2 Test statistic

  • The test statistic is the empirical estimated KL divergence between trial and benchmark PDFs; its logarithm makes it linearly sensitive to small distribution differences.Source: §2 · p. 4, §2.2 · p. 6

2.3 Probability density ratio estimator

  • For each trial point, K-neighbor distances in both samples estimate local PDFs; varying-radius hyperspheres avoid empty fixed bins, though K has no a priori optimum.Source: §2.3 · pp. 7–8

2.4 Distribution of the test statistic and p-value

  • A permutation null is built by pooling and relabeling samples; repeated permutations estimate the TS distribution, from which the two-sided p-value and rejection decision follow.Source: §2.2 · p. 6, §2.4 · p. 8

2.6 Extending the test to include uncertainties

  • Feature uncertainty is modeled by shifting benchmark and/or trial points with random vectors, recomputing TS to obtain noise U, and convolving f(U) with the permutation null.Source: §2.6 · pp. 9–11

2.5 Summary of the algorithm

  • Table 1 gives pseudocode for the nearest-neighbor density-ratio two-sample test, implemented in Python with an open-source package available on GitHub.Source: §2.5 · p. 9, Table 1 · p. 10

2.6 Extending the test to include uncertainties

  • Accounting for uncertainty widens the TS null distribution, raising p-values and lowering equivalent Z, so rejection conclusions weaken.Source: §2.6 · pp. 9, 11

3 Applications to simulated data

The Gaussian studies test whether the nearest-neighbor statistic tracks KL divergence and detects differences across dimensions, sample sizes, and uncertainty settings. In D=2, the test detects a 15% mean shift at 4.9σ, and its power is comparable to parametric competitors.

3.1 Case study: Gaussian samples

  • In the large-sample limit, the test statistic converges to the true KL divergence when the parent distributions are known.Figure 2 compares the statistic with the exact KL divergence for two 2-dimensional Gaussian distributions.Source: §3.1 · p. 12, Figure 2 · p. 12
  • At D=2, TG0 and TG1 are not distinguished from the benchmark above 99% CL, while TG2 and TG3 are distinguished.The comparison uses K = 5 and Nperm = 1000.Source: §3.1 · p. 12, Table 4 · p. 13
  • For two random samples B and TG3 from 2-dimensional distributions differing only by a 15% mean shift along each dimension, the test rejects the null hypothesis at 4.9σ significance.Source: §3.1 · p. 13
  • For D=2, samples of 2,000 points yield Z = 1.4σ, 1.9σ, 1.9σ, and 2.3σ for TG0–TG3, while NB ≤ 10^4 does not exceed 99% CL for TG3.The paper reports stronger discrimination at higher dimensionality, with more than 5σ for TG1, TG2, and TG3.Source: §3.1 · p. 13
  • For normal location alternatives with ∆ from 0.05 to 1.0, power at 5% significance was comparable to Student’s t-test (or Hotelling’s T²-test for D > 1), sometimes lower by at most a factor of 3.Source: §3.1 · p. 14
  • Adding 10% relative error to B lowers the D=2 significance for TG0 versus TG3 from 4.9σ to about 4.1σ.The uncertainties are uncorrelated and added to B only.Source: §3.1 · p. 14, Figure 3 · p. 14

3.2 Case study: Monojet searches at LHC

The monojet case study frames the method as a way to search for unknown new-physics signals by comparing observed data with simulated Standard Model expectations. Its proof-of-principle tests sensitivity to a simulated dark matter signal injected into background events.

3.2 Case study: Monojet searches at LHC

  • Model-independent searches for BSM physics, including dark matter, remain challenging, and a signal may have been overlooked despite extensive efforts.Source: §3.2 · p. 14
  • The compatibility test compares simulated Standard Model expectations with data without requiring knowledge of the nature of new physics.Source: §3.2 · p. 14
  • The proof-of-principle evaluates sensitivity to a monojet dark matter signature by adding simulated signal events to simulated Standard Model background in T.Source: §3.2 · p. 15

Model and simulations

The simulations use a simplified dark matter model and an eight-dimensional event representation, with the study designed as a proof of principle rather than a full detector-level analysis.

Model and simulations

  • The signal uses a fermion dark matter particle χ and an s-channel vector Z′ mediator, with gχ = 1, gq = 0.1, gℓ = 0.01, and dark matter mass 100 GeV.Mediator masses are 1200, 2000, and 3000 GeV.Source: Model and simulations · p. 15
  • Signal and background events are simulated at √s = 13 TeV with a minimal Emiss_T > 90 GeV cut, followed by hadronization and detector simulation.The monojet signal includes missing energy and at least one high-pT jet; 40,000 leading-background events are simulated.Source: Model and simulations · p. 15
  • Each event is represented by eight scaled features, while the proof-of-principle omits sub-leading Standard Model backgrounds and full detector effects.The feature choice balances event information with running time; a full feature-choice study is left to future work.Source: Model and simulations · p. 15

Test Statistic distribution under null hypothesis

The null distribution is estimated empirically by permutation, allowing the test to obtain p-values from the observed statistic. The study reports fast convergence of that distribution, alongside computational costs that may require optimization for real LHC data.

Test Statistic distribution under null hypothesis

  • The null distribution f(TS|H0) is constructed empirically to compute the p-value, using K = 5 and 3,000 permutations.Source: Test Statistic distribution under null… · pp. 15–16
  • The pooled sample contains 40,000 background events plus signal events proportional to the signal cross-section, with B and T assigned equal background counts.The permutation procedure requires the sample sizes NB and NT, not the division of background events between B and T.Source: Test Statistic distribution under null… · p. 16
  • The simulations take about an hour per 1,000 permutations on a standard laptop, and the null-distribution statistics converge quickly.Computation grows as a power law with event count, so optimization and high-performance computing will be necessary for real LHC datasets with many thousands of events.Source: Test Statistic distribution under null… · p. 16

Observed Test Statistic

The authors compare observed test statistics with a null-hypothesis distribution to assess whether background-only data are compatible with the trial samples. In the simulated study, the background-only hypothesis is strongly excluded for T1 and T2, while the authors caution that real-world results may be less clean.

Observed Test Statistic

  • For a practical application, B would be background simulations and T the real-world observation, so only one TSobs measurement would be made.In the study, multiple TSobs measurements are simulated by permuting the 40,000 background events between B and T, with 20,000 in each sample.Source: Observed Test Statistic · p. 16
  • The p-value and Z for each TSobs use the empirical null distribution where possible, with a Gaussian approximation for values beyond its measured range.The Gaussian approximation uses the null distribution's estimated mean and variance.Source: Observed Test Statistic · p. 16
  • The background-only hypothesis is strongly excluded for T1 and T2, although those points were not yet excluded by traditional LHC searches.Table 5 summarizes the comparison of background-only samples with samples containing background plus dark-matter signal; the authors caution that real-world results may be less clean.Source: Observed Test Statistic · p. 16, Table 5 · p. 17

Inclusion of uncertainties

The authors test how background uncertainty and sample size affect the method's significance. In their simulated study, significance decreases with 10% background noise, while signal and sample counts show thresholds and power-law growth patterns.

Inclusion of uncertainties

  • With 10% noise, Z falls from (40, 13, 2.7) to (26, 12, 2.5) for T1, T2, and T3, respectively.The authors warn that limited knowledge of the expected background degrades results and that simulation-only results should be treated cautiously.Source: Inclusion of uncertainties · p. 17
  • The uncertainty study adds uncorrelated Gaussian noise to B; experimental applications could instead seed B or T with their respective simulation or systematic uncertainties.With uncertainties, B and T must be fixed at the outset, leaving one TSobs measurement for a random draw of background events.Source: Inclusion of uncertainties · p. 17

Discussion

  • For Mmed = 2 TeV, significance is negligible below a signal-event threshold and grows as a power law above it; T2 crosses the threshold, but T3 does not.The authors identify this threshold pattern as explaining the rapid drop in significance for T3.Source: Discussion · p. 18
  • For Mmed = 2 TeV, significance grows approximately as a power law with the number of samples.The study uses Nperm = 1000 and the minimum Z over 100 measurements of Tobs.Source: Discussion · p. 18

3.3 Future application to real data

For real-data use, the method compares background simulations with observations under minimal cuts and can use either low- or high-level features. Its practical reach depends on accurate background estimation, while discrepancies can also be assessed in control regions.

3.3 Future application to real data

  • In practice, B would be SM-background simulations and T the observation, with minimal trigger-driven cuts and either low-level or high-level features.The method is model-independent, though physical knowledge can guide cuts and feature choice to manage data and simulation loads.Source: §3.3 · p. 18
  • The monojet case study is a proof of concept with a generic signal and naive background estimation; the method's strength is sensitivity to unspecified signals.The authors state that the method is limited only by background-simulation accuracy.Source: §3.3 · p. 19
  • Accurate SM-background estimation is a significant challenge; matching simulations to control-region data remains possible but introduces model-dependent assumptions.The test could alternatively validate Monte-Carlo compatibility with data in control regions.Source: §3.3 · p. 19
  • Across multiple control regions, the test can assess global simulation-data compatibility and pinpoint feature-space regions with the largest mismatch.This is proposed in the context of nuisance-parameter tuning to match Monte-Carlo background simulations to control-region data.Source: §3.3 · p. 19
  • When background simulations are insufficiently accurate, the test alone may not support discovery claims, but can identify excess regions for follow-up analyses.The authors describe this as a model-independent way to guide hand-crafted analyses of potential signal regions.Source: §3.3 · p. 19

4 Directions for extensions

The authors outline two extensions: selecting the nearest-neighbor count adaptively and using a score field to locate discrepant regions. They leave fuller analysis of these ideas to future work.

4 Directions for extensions

  • The extensions section sketches two directions for improving the method and leaves their fuller analysis to future work.The authors limit this section to outlining ideas.Source: §4 · p. 20

4.1 Adaptive choice of the number of nearest neighbors

  • One adaptive option selects K by minimizing an estimated loss between the true and estimated density ratios over the benchmark density.The true ratio is unknown, but the loss is rewritten to remove its constant term and estimate it using empirical averages.Source: §4.1 · p. 20
  • This model-selection approach can be computationally intensive because it requires running the full algorithm for each candidate K.The selected K is the value minimizing the estimated loss.Source: §4.1 · p. 20
  • Point-Adaptive k-NN (PAk) automatically balances high variance at small K against high bias at large K when point density varies.The authors cite Refs. for PAk.Source: §4.1 · p. 20

4.2 Identifying the discrepant regions

  • A z-score field over trial points can identify clusters contributing most to a large test statistic; in the example, z > 3 selects regions where most discrepancy occurs.The circular benchmark and cross-like trial samples show higher z-field density at the cross's corners.Source: §4.2 · pp. 21–22, Figure 7 · p. 21
  • Once isolated, high-discrepancy regions can guide investigation of the deviations or inform data-selection cuts.The authors also suggest clustering these regions with standard clustering algorithms.Source: §4.2 · p. 22

5 Conclusions

The paper presents an unsupervised, model-independent test for detecting discrepancies between samples, including unknown signals in physics data. It reports sensitivity to small numbers of New Physics events and describes broader uses for the test.

5 Conclusions

  • Because the searched-for phenomenon is unknown, the paper argues for methods that can detect unknown signals rather than testing only specific models and parameters.Source: §5 · p. 22
  • The test is model-independent and non-parametric, requiring no signal parameters or spectrum, and uses the full multidimensional feature space without binning.The paper frames new phenomena as deviations from the expected distribution.Source: §5 · p. 22
  • Nearest-neighbor estimates of relative local densities underpin a test statistic whose extreme-tail values indicate that the samples come from different underlying densities.Source: §5 · p. 22
  • The method can identify regions of discrepancy to guide further analysis even when the background sample is too inaccurate to support a discovery claim.Source: §5 · p. 22
  • The statistical test also applies to joint dataset analysis, outlier detection, distribution changes over time, and anomalous time-series events.Source: §5 · p. 22
  • The method is reported to be highly sensitive to even a small number of New Physics events when LHC observations are compared with Standard Model simulations.The paper presents this as evidence of the technique's strong potential.Source: §5 · p. 22

A Kullback-Leibler divergence

The section introduces Kullback-Leibler divergence as a measure between probability density functions and describes empirical estimation when the distributions are unknown but samples are available.

A Kullback-Leibler divergence

Related papers

Loading 1807.06038v3…