The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics

Motivation

• The LHC’s discovery of the Higgs boson completed the Standard Model, while unresolved dark matter, neutrino, cosmological, naturalness, and quantum-gravity problems motivate searches for BSM physics.The LHC has the potential to address these fundamental challenges.Source: §1 · p. 5

Existing search paradigm

• Current LHC BSM searches select model-specific signals and backgrounds, so even nominally model-independent limits retain model-dependent event selection and background estimation.These analyses generate simulated signal and background data, define signal-like selections, and calibrate background rates before applying strategies to data.Source: §1 · p. 5

Need for complementary searches

• Model dependence may create blind spots: despite thousands of BSM searches, substantial phase space and many unconventional signals remain unexplored.Generic searches such as bump hunts have discovered particles including the Higgs boson, but their limited use of event properties reduces sensitivity.Source: §1 · p. 5

Limitations of existing alternatives

• More differential model-independent searches incur sensitivity loss from the look-elsewhere effect and depend heavily on simulation accuracy and difficult systematic-uncertainty characterization.These approaches compare data with simulation across many exclusive final-state bins, while the bump hunt assumes a localized enhancement over a smoothly falling background.Source: §1 · pp. 5–6

Machine-learning opportunity

• Machine learning can extend model-independent searches through semi-, weak-, or unsupervised training, but HEP anomaly detection targets ensemble over-densities rather than isolated off-manifold events.Because HEP data differ from common machine-learning inputs, public datasets are needed to develop and benchmark dedicated methods.Source: §1 · p. 6

LHC Olympics 2020

• The LHC Olympics 2020 challenge tests whether mostly Standard Model black boxes contain new physics and whether its properties can be identified, using hadronic final states, sidebands, particle lists, and R&D datasets.The challenge provides anomaly-free and benchmark-signal simulations for development, alongside black boxes; over one hundred researchers participated and more than a dozen teams submitted results.Source: §1 · p. 6

2 Dataset and Challenge

• The challenge made datasets available through its website and Zenodo, while contestants submitted a black-box number, method abstract, null-hypothesis p-value, new-physics description, and preselection signal yield.Source: §2 · p. 8

2.1 R&D Dataset

• The R&D dataset contained 1M QCD dijet background events and 100,000 Z′ →XY signal events, generated with Pythia 8.219 and Delphes 3.4.1 using default settings without pileup or multiparton interactions.The signals had mZ′ = 3.5 TeV, mX = 500 GeV, and mY = 100 GeV, with selection by a single R = 1 anti-kT jet trigger above 1.2 TeV.Source: §2.1 · p. 8

2.2 Black Box 1

• Black Box 1 preserved the R&D signal topology but changed the masses to mZ′ = 3.823 TeV, mX = 732 GeV, and mY = 378 GeV, including 834 signal events among 1M total events.Its backgrounds used altered Pythia and Delphes settings, including tracking-efficiency, momentum-smearing, and 10% calorimeter-resolution variations motivated by detector measurements.Source: §2.2 · pp. 9–10

2.3 Black Box 2

• Black Box 2 contained 1M background-only events generated with Herwig++ [53] and a modified Delphes detector card differing from Black Box 1 while retaining similar modifications relative to the R&D card.Source: §2.3 · p. 10

2.4 Black Box 3

• Black Box 3 combined 1200 KK-graviton dijet events and 2000 trijet events with QCD backgrounds, using modified Pythia and Delphes settings and a 4.2 TeV resonance with an mR = 2.217 TeV radion decay mode.A power-shower setting sculpted a bump-like multijet-background feature that was flagged as anomalous and labeled “Human NN” in Figure 5.Source: §2.4 · p. 10

Method categories

• Methods are grouped as unsupervised, weakly supervised, or (semi)-supervised according to the label information available during training, though category boundaries are not rigid.Unsupervised methods learn from background-dominated data, weakly supervised methods use noisy labels, and (semi)-supervised methods use signal simulations or labels for some events.Source: Individual Approaches · p. 12

Result types

• Results are classified as blinded black-box contributions, unblinded black-box results or updates, or results using only the R&D dataset, with each type serving a different purpose.Blinded contributions emulate a pure challenge analogous to a real data analysis, while unblinded and R&D-only results extend the datasets’ utility beyond the initial challenge.Source: Individual Approaches · p. 12

Overview

• Table 2 summarizes the methods and result types presented across the unsupervised, weakly supervised, and (semi)-supervised sections.For some results, the table gives precedence to blinded workshop presentations over subsets reported in subsequent sections.Source: Individual Approaches · p. 12, Table 2 · p. 13

3.1.1 Method

• The VRNN models variable-length jet-constituent sequences with recurrent variational inference, avoiding fixed-length zero-padding effects and assigning event scores from the more anomalous leading or sub-leading jet.Its loss combines constituent reconstruction with pT-fraction-weighted KL divergence, whose optimized coefficient is 0.1.Source: §3.1.1 · pp. 14–15

3.1.2 Results on LHC Olympics

• A 0.75 Event Score selection with central-jet requirements reveals an enhancement just below 4000 GeV in Black Box 1, consistent with a 3800 GeV Z′ signal.In the R&D validation, the selection increased local significance from 0.18σ to 2.2σ without significantly sculpting the background shape.Source: §3.1.2 · p. 16, Figure 5 · p. 17
• Black Box 2 shows no significant invariant-mass excess and selection effects similar to background, consistent with its QCD-only composition and independently trained network weights.The Event Score distributions were produced from unique weights for each dataset.Source: §3.1.2 · pp. 16–17, Figure 6 · p. 18
• The method is insensitive to Black Box 3 because it targets substructure within large-R jets, whereas that signal has varied final states beyond the described two-prong configuration.Black Box 3 contains a new 4200 GeV particle.Source: §3.1.2 · pp. 17–18, Figure 7 · p. 18

3.1.3 Lessons Learned

• The study identifies preprocessing, especially k_t-ordered constituent sequencing, as highly influential and proposes future mass-decorrelation, high-level-feature, and supervised-training extensions.The authors expect constituent-based learning to have less jet-mass correlation than standard variables such as n-subjettiness, but note that this requires further characterization.Source: §3.1.1 · p. 14, §3.1.3 · pp. 18–19

3.2.1 Method

• ANODE estimates pdata(x|m) in the signal region and pbackground(x|m) from sidebands, interpolating the latter into the signal region and applying R(x|m) > Rcut to select anomalies.The method scans a localized feature m, defines signal and sideband regions, and uses additional features x to improve a bump hunt.Source: §3.2.1 · pp. 19–20

3.2.2 Results on LHC Olympics

• In the test-set signal region, background events concentrated near R(x|m) = 1, while signal formed a high-R over-density in relatively background-rare phase space.The R distribution was narrower near 1 at high background density and more dispersed where the background density was low.Source: §3.2.2 · pp. 20–21, Figure 8 · p. 21
• AUC 0.82 was achieved by ANODE, whose performance was comparable to CWoLa hunting, although CWoLa performed better at higher and especially lower signal efficiencies.The comparison used ROC and SIC curves, with CWoLa trained on 200 GeV sidebands on either side of the signal region.Source: Figure 9 · p. 21, §3.2.2 · p. 22
• Signal significance was enhanced by about a factor of 7, enabling a local significance above 5σ under the stated starting S/B condition.The passage attributes the difference from CWoLa partly to CWoLa’s supervised likelihood-ratio approach versus ANODE’s unsupervised density estimation.Source: §3.2.2 · p. 22

3.2.3 Lessons Learned

• ANODE is robust to correlations but struggles to estimate background densities precisely at very small S/B and to extend density estimation to higher dimensions.The demonstrated implementation used MAF, while the general method can employ other density estimators such as Neural Autoregressive Flows and Neural Spline Flows.Source: §3.2.1 · p. 20, §3.2.3 · p. 22

3.3.1 Method

• BuHuLaSpa represents each event with jet masses, n-subjettiness ratios, and a soft-drop-like observable, then uses a VAE to infer latent variables and reconstruct the event data.The method modifies the decoder so one sampled mass variable, with σ(mi) = 0.1mi, explicitly conditions reconstruction while the encoder can remain mass-agnostic.Source: §3.3.1 · pp. 23–24
• Explicit mass conditioning makes optimization local in invariant mass and enables decoder scans over latent variables and mass to visualize learned correlations without relying on data.Events with similar latent representations but different masses are therefore treated differently by the decoder.Source: §3.3.1 · p. 24

3.3.3 Lessons Learned

• Adagrad and Adadelta outperformed momentum-based optimizers, while stopping when per-observable reconstruction-loss variance reached a minimum preserved peak unsupervised classification performance.The classifier’s performance peaked and then declined as the VAE began ignoring anomalous events or over-training on only one or two observables; the latent-vector norm performed best as the signal classifier.Source: §3.3.1 · pp. 24–25, §3.3.3 · pp. 25–26

3.3.2 Results on LHC Olympics

• A bump at ∼3.8 TeV emerged in Black Box 1, with selected events showing jet masses near 750 and 400 GeV and low-end τ2/τ1 values, while Black Boxes 2 and 3 showed no clear signals.The result followed training on the full event samples without invariant-mass cuts or binning.Source: §3.3.2 · p. 25, Figure 11 · p. 26, Figure 12 · p. 26

3.4.1 Method

• The full workflow applies GAN-AE to reduce background, derives a shaping function from black-box data, and supplies the resulting reference to BumpHunter for global significance.BumpHunter scans variable-width windows and uses pseudo-experiments to account for the look-elsewhere effect; a reference background is required but unavailable directly for black boxes.Source: §3.4.1 · pp. 27–29
• GAN-AE alternately trains an autoencoder and MLP using reconstruction, binary-crossentropy, and distance-correlation terms, then uses Euclidean reconstruction error as the discriminant.The MLP exposes autoencoder weaknesses, while the autoencoder is trained to reconstruct inputs and mislead the MLP; DisCo reduces correlation with invariant mass.Source: §3.4.1 · pp. 27–28

3.4.2 Results on LHC Olympics

• The method obtained good discrimination on the R&D dataset, although the Euclidean distance remained correlated with dijet mass, potentially impairing bump hunting.The AE was trained only on background events; Figure 13 reports the Euclidean-distance distributions and ROC curves.Source: R&D dataset · p. 30, Figure 13 · p. 30
• The black-box distance distributions were shifted and broader than those from the R&D background, indicating sensitivity to background modeling; imperfect shaping-function fits could create fake bumps or deficits.The resulting reference backgrounds did not fit post-cut data, so meaningful BumpHunter p-values could not be evaluated for potential signals.Source: R&D dataset · pp. 30–31, Figure 14 · p. 30, Figure 16 · p. 31

3.4.3 Lessons Learned

• The challenge demonstrated GAN-AE’s potential on R&D data but exposed limits from background-modeling sensitivity and residual distance–dijet-mass correlation, while unavailable matching simulations hindered BumpHunter.These limitations persisted despite the DisCo term.Source: R&D dataset · p. 30, §3.4.3 · p. 31

3.5.1 Method

• GIS iteratively Gaussianizes one-dimensional projections, models conditional dependence through binned mappings, and scores local excess density as α = psignal/pbackground.The score is expected to be near one for smooth background and greater than one for anomalous events.Source: §3.5 · p. 32, §3.5.1 · p. 33

3.5 Gaussianizing Iterative Slicing (GIS): Unsupervised In-distribution Anomaly Detection through Conditional Density Estimation10

• The analysis reduced events to jet summary statistics, using MJJ, leading-jet masses, and leading-jet n-subjettiness ratios as the primary features.Jet clustering used R = 1.0 and |η| < 2.5; the final black-box run retained 744,217 events after cuts.Source: §3.5 · p. 32, §3.5.2 · p. 34

3.5.2 Results on LHC Olympics

• The anomaly score strongly peaks around MJJ ≈ 3750 GeV, while selected events cluster in jet masses, mass differences, and small τ21 without comparable neighboring-mass counterparts.The analysis used 3600 GeV < MJJ < 3900 GeV and examined events surviving α thresholds.Source: §3.5.2 · pp. 34–35, Figure 18 · p. 35, Figure 19 · p. 35
• 3772.9 ± 8.3 GeV particle decays into two particles with masses 727.8 ± 3.8 and 374.8 ± 3.5 GeV, each forming a two-pronged jet.Using α > 2.0, the authors estimated 1000 ± 200 such events among one million black-box events.Source: §3.5.2 · p. 36, Figure 20 · p. 36

3.5.3 Lessons Learned

• GIS’s low noise and limited hyperparameter tuning enabled the authors to focus on the blind search rather than method-parameter failures.Gaussian convolution with σ = 250 GeV improved the conditional-density estimates, while autoencoder features added noise and were excluded from the final submission.Source: §3.5.2 · p. 34, §3.5.3 · p. 36

3.6 Latent Dirichlet Allocation11

• LDA represents each event as measurements generated from mixtures of multinomial latent themes, with Dirichlet-distributed mixing proportions allowing different measurements to arise from different themes.For two themes, event classification can use a likelihood ratio of the learned theme distributions; parameters are inferred from unlabelled data using stochastic variational inference.Source: §3.6 · p. 37

3.6.1 Method

• The method uses coarse, discriminative binnings that preserve within-event co-occurrences, considering mass-basis and Lund-basis representations for the two highest-pT jets.It optimizes hyperparameters separately in overlapping invariant-mass bins, selects themes and thresholds, then performs a bump hunt with sideband-normalized background and systematic-error estimates.Source: §3.6.1 · pp. 38–39

3.6.2 Results on LHC Olympics

• Black Box 1 produced significances of 1.8σ for simulated background and 3.8σ for the black-box sample, while no compelling candidates appeared in Black Boxes 2 or 3.The inferred dijet resonance mass was compatible with the unveiled value, but the decay-product masses were substantially overestimated, possibly because of binning and sculpting effects.Source: §3.6.2 · pp. 39–40, Figure 22 · p. 40

3.6.3 Lessons Learned

• The challenge showed that LDA searches should test multiple representations and binnings, include more jets, and incorporate global jet or event variables beyond dijet substructure.Restricting the analysis to two leading-jet substructure missed rare signals lacking rich substructure there.Source: §3.6.3 · p. 40

3.7.1 Method

• PGAEs represent each jet as a fully connected particle graph and use edge convolutions to message-pass, compress particle four-momenta into a two-dimensional latent space, and reconstruct them.The architecture preserves graph structure, supports variable-size graphs without zero-padding, and is insensitive to particle ordering.Source: §3.7.1 · pp. 41–42, Figure 23 · p. 41

3.7.2 Results on LHC Olympics

• The model was tested with MSE and permutation-invariant Chamfer losses, but MSE provided better discrimination for an unseen W′ →XY signal in the R&D dataset.Chamfer loss avoids dependence on particle ordering, whereas MSE requires particles to be reconstructed in their input order.Source: §3.7.2 · p. 42, §3.7.1 · p. 42
• The LHC Olympics search flagged both leading jets as outliers using a 90% loss quantile, then estimated the outlier-region background with a fourth-order-polynomial transfer factor and applied a bump hunter.The procedure searched the dijet invariant mass in BB 1, containing a resonant signal, and BB 2, containing no signal.Source: §3.7.2 · pp. 42–43
• A 2.1 σ excess at 3.9 TeV in BB 1 and a 0.8 σ bump at 3.3 TeV in BB 2 were found with MSE, while Chamfer yielded only 1.5 σ and −1.4 σ excesses.The MSE result was near the injected dijet resonance, whereas neither Chamfer result was significant.Source: §3.7.2 · p. 44, Figure 27 · p. 45

3.7.3 Lessons Learned

• The authors conclude that particle graph autoencoders are promising, but call for better-performing permutation-invariant losses, more generic multidimensional resonance searches, and larger R&D datasets with additional black boxes.They also suggest hosting future competitions on platforms such as Kaggle to increase engagement.Source: §3.7.3 · pp. 44–45

3.8.1 Method

• M-flow combines autoencoder-style reconstruction error with normalizing-flow density estimation to learn a lower-dimensional data manifold and its density.Its latent space separates manifold coordinates u from off-manifold variables v; training sequentially minimizes reconstruction error and negative log likelihood.Source: §3.8.1 · p. 46
• The combined M-flow anomaly score outperformed its reconstruction-error and manifold-density components and a basic normalizing flow on the R&D dataset, as shown by ROC curves.The score combines density on the learned manifold with reconstruction error from projection onto that manifold.Source: §3.8.1 · p. 47, Figure 29 · p. 48

3.8.2 Results on LHC Olympics

• The resulting R_mjj score reduced, but did not eliminate, a bias toward high-m_jj events: raising the threshold selected slightly higher-m_jj events without revealing a sharp resonance peak.R_mjj incorporated the KDE-modeled marginal likelihood of m_jj to decouple the score from dijet mass.Source: §3.8.1 · p. 47, §3.8.2 · p. 48, Figure 30 · p. 49
• The method could not reliably identify the hidden signal in the black boxes, regardless of the R_mjj threshold choice.Differences in black-box modeling introduced significant biases, making transfer from R&D performance difficult.Source: §3.8.1 · p. 47, §3.8.2 · p. 48

3.8.3 Lessons Learned

• The challenge showed that neural networks alone cannot achieve strong anomaly detection without a good background model, while flawed modeling can create fake signals and training on data introduces further issues.The broader lesson is that anomaly detection depends on analysis details beyond the neural network architecture itself.Source: §3.8.3 · pp. 48–49

3.9.1 Method

• UCluster combines a neural event embedding with jet-mass classification and a differentiable clustering loss, encouraging similar events to be close and dissimilar events to separate.Jet labels use 20 mass intervals from 10 to 1000 GeV for particles in the two heaviest jets, with zero-padding or truncation to 100 particles.Source: §3.9.1 · pp. 49–50

3.9.2 Results on LHC Olympics

• UCluster grouped many BSM events into one cluster, increasing S/B from 1% to 2.5% with two clusters, while more clusters further enhanced S/B.The embedding-space visualization compared true labels with UCluster cluster labels for 1000 evaluation events.Source: §3.9.2 · p. 52, Figure 32 · p. 52
• S/B increased to around 28% with more clusters, while significance improved by factors of 3-4 for initial significances of 2-6; stability decreased with many clusters.These results used cluster size 30 for the significance study, with uncertainties from five retrainings.Source: Figure 33 · p. 53, §3.9.2 · p. 53

3.9.3 Lessons Learned

• The method achieved conceptual clustering on the R&D dataset but produced no conclusive black-box results because identifying important clusters was not fully investigated.Future applications require an automatic procedure to evaluate cluster importance.Source: §3.9.3 · p. 53
• Jet-substructure classification was useful for the studied dijet anomaly but may be suboptimal for different decay topologies; auto-encoders were proposed as a more general embedding approach.The limitation arose because only one BB3 decay mode had jet-substructure properties differing from the main QCD background.Source: §3.9.3 · pp. 53–54

4.1.1 Method

• CWoLa Hunting trains a classifier to distinguish a resonant signal region from adjacent sidebands using features orthogonal to the resonance variable.With background distributions matched between regions, no signal should yield poor test performance, whereas signal events localized in the signal region can drive discrimination.Source: §4.1.1 · p. 54
• The CWoLa analysis uses mJJ from the two highest-pT R = 1 anti-kt jets, rescales jet masses by mJJ, and applies nested cross-validation to preserve statistical independence.The rescaling reduces background correlations between jet masses and mJJ; the strategy scans overlapping signal regions and uses a sideband fit for the bump hunt.Source: §4.1.1 · pp. 54–56

4.1.2 Results on LHC Olympics

• CWoLa finds a large 5σ excess in BB1 at mJJ = 3500 GeV, while finding no significant excess in BB2.The result is obtained after selection cuts at efficiencies of 10%, 1%, and 0.2%.Source: §4.1.2 · p. 56, Figure 34 · p. 57

4.1.3 Lessons Learned

• Rescaling mJ by mJJ eliminated correlations more effectively than the original high-mass cut, which would have missed the BB1 signal.The earlier cut severely distorted the QCD background shape by rejecting many low-mJJ events.Source: §4.1.3 · p. 57

4.2.1 Machine Learning Setup

• The CWoLa–autoencoder comparison uses a resonant dijet signal, with CWoLa expected to favor larger signal fractions and autoencoders lower signal statistics.CWoLa uses sideband-versus-signal-region classification, while autoencoders learn reconstruction errors from mixed signal and background samples.Source: §4.2.1 · pp. 58–61

4.2.2 Results on LHC Olympics

• CWoLa and autoencoders were compared across eight injected-signal benchmarks spanning S/B ∈[1.2 · 10−3, 6 · 10−3] and a no-signal benchmark.Results were averaged over five independent runs for each signal-to-background benchmark.Source: §4.2.2 · p. 61
• CWoLa enhances signal significance by 3σ−8σ above S/B ∼3 · 10−3, while autoencoders improve it by at least 2σ−3σ below that range, demonstrating complementary performance.The AUC scores cross near S/B ∼3 · 10−3; CWoLa reaches AUC values above 0.90 at large S/B, whereas the autoencoder remains solid across the full range.Source: §4.2.2 · pp. 61–62, Figure 37 · p. 62

4.3.1 Method

• Tag N’ Train uses anomalous Object-1 examples to tag signal- and background-rich samples, then trains Object-2 with CWoLa and repeats the procedure symmetrically.It starts from unlabeled events and two initial classifiers, often autoencoders, and iterative retraining can improve performance until a plateau.Source: §4.3.1 · p. 63
• The search partitioned events for autoencoder training, TNT training, and anomaly selection, using jet images, CNNs, two iterations, and scanned 500 GeV dijet-mass windows from 3000 to 5000 GeV.Blackboxes used 200k events for autoencoders, 400k for TNT, and 400k for searching, without full cross-validation because of computational limits.Source: §4.3.1 · pp. 63–65

4.3.2 Results on LHC Olympics

• On the R&D dataset, Tag N’ Train was generally competitive, but insufficient signal at 0.3% and 0.1% made TNT significantly worse than autoencoders.Adding a dijet-mass cut improved performance at 1% signal and exceeded CWoLa hunting at 0.1%, without improving on autoencoders.Source: §4.3.2 · p. 65
• A 3800 ± 50 GeV resonance with local significance 4σ was found in Blackbox 1, while initial scans of Blackboxes 2 and 3 showed no significant evidence.The corresponding peak in one subset had a local p-value around 3σ.Source: §4.3.2 · p. 65, Figure 39 · p. 65

4.3.3 Lessons Learned

• The method could not find Blackbox 3’s signal because it targeted dijet resonances with anomalous substructure, whereas Blackbox 3 involved 3-jet decays and gluon-jet dijet decays.Although the correct Blackbox 1 resonance was found, CNN jet images made the learned signal-like features difficult to interpret, and low selection efficiency hindered cross-section estimates.Source: §4.3.3 · p. 66

4.4 Simulation Assisted Likelihood-free Anomaly Detection21

• SALAD uses Dctr to learn a conditional weight w(x|m) from sideband data-versus-simulation classification, then applies those weights in the signal region.The classifier is parameterized by the resonant feature so the learned reweighting can be interpolated into signal-region masses not present during training.Source: §4.4 · pp. 66–67, §4.4.1 · p. 67, §4.4.2 · p. 68

4.4.1 Method

• SALAD trains a data-versus-simulation classifier in sidebands, interpolates its reweighting function into the signal region, then classifies reweighted simulation against data to identify anomalous events.For the LHC Olympics, m is the two-jet invariant mass and x contains the two jets’ substructure information.Source: §4.4.1 · pp. 67–68

4.4.2 Results on LHC Olympics

• SALAD extends useful signal discrimination from S/B ∼1% to about 0.5% and predicts background within a few percent down to about 1% data efficiency, outperforming Herwig-only reweighting.The interpolated reweighting improves Herwig-based sensitivity, while unweighted background predictions are too low by a factor of two or more below 10% data efficiency.Source: §4.4.2 · p. 69, Figure 41 · p. 70

4.4.3 Lessons Learned

• SALAD reduces simulation mismodeling by learning sideband weights, but direct background estimation requires residual-bias and high-dimensional systematic-uncertainty control.Validation regions, simulation variations, auxiliary measurements, low-dimensional reweighting, or proper high-dimensional uncertainty estimates are proposed safeguards.Source: §4.4.3 · pp. 69–70

4.5 Simulation-Assisted Decorrelation for Resonant Anomaly Detection23

• The two approaches use simulations differently: SALAD reweights simulated backgrounds, whereas SA-CWoLa penalizes classifiers for distinguishing signal and sidebands in signal-free simulation to reduce resonant-feature dependence.CWoLa avoids simulation but is sensitive to dependencies, motivating SA-CWoLa as a robustness modification.Source: §4.5 · p. 71, §4.5.1 · p. 72

4.5.1 Method

• SA-CWoLa adds a λ > 0 simulation penalty to the CWoLa loss, with λ →0 recovering ordinary CWoLa and sideband weights balanced against the signal region.The penalty discourages learning signal-region versus sideband differences present in background-only simulation.Source: §4.5.1 · p. 72

4.5.2 Results on LHC Olympics

• With λ = 0.5, SA-CWoLa nearly recovers CWoLa’s ideal performance when dependencies between mJJ and classification features make ordinary CWoLa anti-tagging.The study strengthens the mJ–mJJ dependence by setting mJ 7→mJ + 0.1mJJ and injects 1500 signal events corresponding to about 2σ.Source: §4.5.2 · pp. 72–73, Figure 42 · p. 73

4.5.3 Lessons Learned

• Background-only simulations mitigate classifier sensitivity to resonant-feature dependencies in both SALAD and SA-CWoLa, although multiple methods remain necessary for broad BSM sensitivity.The study presents these methods as contributions to making automated anomaly detection more robust and potentially extending higher-dimensional searches.Source: §4.5.3 · p. 74

5.1.1 Method

• The semi-supervised hybrid achieved 92% CNN accuracy and 95.38% BDT accuracy by combining ResNet-34 image scores with kinematic observables.The BDT used evolutionary hyperparameter optimization, and its test-set scores and ROC curves are shown in Figures 43 and 44.Source: §5.1.1 · p. 76, Figure 43 · p. 77, Figure 44 · p. 77

5.1.2 Results on LHC Olympics

• Applied to a 200K-event black-box subsample, the trained classifier identified a likely 3.5 GeV heavy resonance decaying to 100 GeV and 500 GeV particles.The analysis did not examine the entire black box because of limited time.Source: §5.1.2 · p. 76

5.1.3 Lessons Learned

• The classifier was not robust enough to predict the heavy-resonance mass range because it was specific to the new-physics type used during training.This lack of generalization motivated development of the semi-supervised Anomaly Awareness algorithm [133].Source: §5.1.3 · p. 76

5.2 Factorized Topic Modeling25

• Factorized topic modeling translates leading-order dijet factorization into a non-parametric generative model that learns jet-component fractions and observable distributions from mixed samples.The model assumes sample independence and conditional independence of the two jets given their joint types, and uses alternating minimization despite non-convexity.Source: §5.2 · p. 80, §5.2.1 · pp. 80–82

5.2.3 Lessons Learned

• For factorized topic modeling, performance degrades below 0.1% signal, likely because of the chosen histogram bins, while larger signal fractions yield robust solutions.The method is proposed as a possible drop-in replacement for density-estimation steps and as a proof of concept for encoding scattering constraints statistically.Source: §5.2.2 · p. 83, §5.2.3 · p. 84

5.2.2 Results on the LHC Olympics

• AUCs of 0.88 and 0.81 were obtained using likelihood-ratio discrimination from factorized topic modeling on the LHC Olympics R&D dataset.The model recovered exotic jet masses of 100 GeV and 500 GeV at 10% and 1% signal fractions, although lower fractions increased noise.Source: §5.2.2 · pp. 83–84, Figure 48 · p. 83

5.3 QUAK: Quasi-Anomalous Knowledge for Anomaly Detection26

• QUAK embeds possibly inaccurate signal priors into model-independent anomaly searches, enhancing sensitivity to signals resembling those priors without degrading the pre-existing search.The approach extends comparisons between classification without labels and physics-informed likelihood or autoencoder methods, whose sensitivity depends on available signal regions and complete background knowledge.Source: §5.3 · pp. 84–85

5.3.1 Method

• QUAK constructs an N-dimensional loss space from separate probabilistic models for background and approximate signal priors, then searches regions associated with broadly similar anomalies.Background events have low background loss and high signal loss, whereas signal-like or otherwise anomalous events occupy different loss regions.Source: §5.3.1 · p. 86

5.3.2 Results on LHC Olympics

• QUAK remains relatively stable as test signals deviate from the training prior, whereas supervised performance worsens and becomes abysmal for the 3-prong signal.This supports using priors to scan broader BSM possibilities when signal modeling is uncertain.Source: §5.3.2 · p. 87
• Adding approximate signal priors improves QUAK sensitivity, approaches or sometimes exceeds supervised discrimination, and preserves model-independent anomaly-search capability.The 3-prong signal gains substantially from adding a 2-prong prior, while inaccurate priors can still provide powerful discrimination.Source: §5.3.2 · p. 87, Figure 50 · p. 88

5.3.3 Lessons Learned

• QUAK adds generic signal-like and background-like information without degrading prior-free sensitivity, extending anomaly searches to settings with large signal-modeling uncertainties.The approach is presented as broadly applicable beyond the demonstrated LHC search.Source: §5.3.3 · p. 88

5.4.2 Results on LHC Olympics

• A supervised LSTM-based jet model identifies 95% of R&D test signals but performs poorly on black boxes, especially when their signal type differs from training data.Black Box 1 produces low-confidence, potentially spurious selections, while Black Boxes 2 and 3 resemble pure-background outputs.Source: §5.4.1 · p. 89, §5.4.2 · p. 89

5.4.3 Lessons Learned

• The supervised-model study concludes that classification trained on specific signals is unsuitable for varying-background anomaly detection and proposes feature-space occupancy methods instead.The proposed direction uses independent features and sparsely populated multidimensional regions, but requires background coverage that does not mislabel signal regions.Source: §5.4.3 · pp. 89–90

6.1 Overall Results

• Four approaches identified Black Box 1’s resonance mass, but only Density Estimation accurately predicted the other observables.The submissions included nine blind approaches, with results summarized using predicted resonance mass, signal events, and daughter-particle masses.Source: §6.1 · p. 91, Figure 51 · p. 92
• Black Box 3’s true 4.2 TeV resonance with two decay modes was not detected by any approach, while Black Box 2 results exposed vulnerability to tail-distribution effects.Black Box 2 produced several resonance predictions despite containing no injected signal.Source: §6.1 · pp. 91–92
• After unblinding, additional methods found resonances near 3.5–3.9 TeV in Black Box 1, but no further signals were reported for Black Boxes 2 or 3.These later contributions included VRNN, BuHuLaSpa, deep ensembles, Latent Dirichlet Allocation, Particle Graph Autoencoders, and CWoLa hunting.Source: §6.1 · p. 92

6.2 Overall Lessons Learned

• The first-box resonance was detected repeatedly before unblinding, with the closest mass estimates mainly coming from likelihood-based methods that benefited from the shared development topology.Tag N Train and Density Estimation used signal-to-background likelihood ratios, while LSTM used a signal likelihood.Source: §6.2 · pp. 92–93
• Black Box 3 highlighted the need for greater sensitivity and improved statistical interpretation because its unfamiliar topology, two decay modes, and limited substructure handle challenged existing methods.No confident signal absence could be established for Black Box 2 because of high-mass false positives.Source: §6.2 · p. 93
• The challenge demonstrated the value of targeted community studies and motivates anomaly-detection development with richer detector effects, track and vertex information, and eventual experimental data.Toy studies remain important for method development, while credible detection or rejection of Black Box 3-like anomalies is still needed.Source: §6 · p. 91, §6.2 · p. 93

7.1 Prospects for the (HL)-LHC

• Model-agnostic searches require complementary methodologies and may need a dedicated analysis group or subgroup connected to statistics and machine-learning forums.The existing ATLAS and CMS structures are organized by physics models, while technical development and physics review depend on the analysis-group home.Source: §7.1 · pp. 94–95, Figure 52 · p. 94
• Scaling anomaly detection to high-dimensional features, non-resonant signals, and online triggers requires new methodological, computational, and background-estimation strategies.Successful LHC Olympics methods used model-inspired dimensionality reduction; non-resonant searches lack a general background estimator, while online methods face one-pass trigger decisions and latency constraints.Source: §7.1 · pp. 95–96
• Model-agnostic results pose major interpretation challenges because data-dependent selections require retraining for each signal hypothesis, complicating limits, reanalysis, and reinterpretation.The ATLAS CWoLa hunting exclusion plots in Ref required training tens of thousands of neural networks, and reproducible preservation must include the optimization procedure.Source: §7.1 · pp. 96–97, Figure 53 · p. 97

7.2 Prospects for Future Colliders and Beyond

• Future colliders enable co-design of accelerator, detector, and analysis procedures, while collider anomaly-detection methods may also transfer to other fundamental-physics experiments.An anomaly-detection method developed with the LHC Olympics is being studied for astrophysical data.Source: §7.2 · p. 97

7.3 The Role of Theory and Theorists

• Theory remains essential: physics-informed machine learning can encode symmetries, improve interpretability and robustness, and use broad theoretically motivated priors to enhance sensitivity.The authors also call for low barriers among theory, experiment, computation, and statistics, with public challenges supporting broader engagement.Source: §7.3 · pp. 97–98

7.4 The Future of the LHC Olympics

• Future LHC Olympics iterations should improve method comparison and accessibility while expanding black boxes and signal topologies, including leptons, heavy-flavor quarks, and long-lived particles.The 2020 challenge lacked a single metric because false-positive-rate estimation is non-trivial; broader exposure, participant support, and more robust black-box checks were also identified as improvements.Source: §7.4 · pp. 98–99

• Machine-learning anomaly detection complements dedicated searches amid the lack of convincing evidence for new particles or forces and represents a snapshot of a rapidly developing field.The paper frames these methods as less model-dependent approaches for exploring collider data.Source: §8 · p. 100
• The LHC Olympics tested at least 18 anomaly-detection methods across three black boxes containing different Standard Model backgrounds and potential anomalies.The datasets emulated unlabeled collider data paired with labeled research-and-development data.Source: §8 · p. 100
• Teams applied their strategies to unknown black-box signals, refined methods after workshop feedback, and gained experience despite varying proximity to the correct answers.Results were presented before teams knew the signals’ nature, and several strategies continued to be refined for future collider-data applications.Source: §8 · p. 100
• Unsupervised, semisupervised, and supervised methods each have advantages and disadvantages, while scaling to higher dimensions remains a shared challenge.The paper suggests ultimate performance will likely combine approaches and require new method development.Source: §8 · p. 100
• The LHC Olympics marks a starting point for exploring natural high-dimensional LHC collisions and pursuing discovery in current and future datasets.The authors describe machine learning as catalyzing a data-driven revolution with substantial discovery potential.Source: §8 · p. 100