PaperlayerView exact revision on arXiv

Source-linked AI summary

Symmetries, Safety, and Self-Supervision

Barry M. Dillon, Gregor Kasieczka, Hans Olischlager, Tilman Plehn, Peter Sorrenson, Lorenz Vogel

arXiv:2108.04253v1hep-ph

TL;DR

Collider searches need representations that expose physical symmetries, retain discriminating information, and do not depend on a new-physics hypothesis. JetCLR uses self-supervised contrastive learning with a permutation-invariant transformer encoder to construct such representations for jets. In linear-classifier tests, it outperforms simple jet images and is competitive with energy flow polynomials.

  • Problem

    Collider searches seek high-dimensional representations that manifest physical symmetries, retain discriminating features, and remain agnostic to new-physics hypotheses.

    Source: §1 · p. 2

  • Method

    JetCLR uses self-supervised contrastive learning with a permutation-invariant transformer encoder and symmetry and theory-driven augmentations to learn jet representations without truth labels.

    Source: §1 · p. 2, §3 · p. 6, §5 · p. 12

  • Results

    JetCLR outperforms simple jet images and is competitive with energy flow polynomials in linear classifier tests distinguishing top jets from QCD jets.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The paper concludes that symmetry principles and physics knowledge can be incorporated into self-supervised tools and latent representations, with JetCLR particularly suited to enhancing and controlling anomaly searches.

    Source: §5 · p. 12

    JetCLR enforces only approximate infrared and collinear safety through augmentations, rather than explicitly fixing angular correlations or pT scalings.

    Source: §2 · p. 3, Symmetries and augmentations · p. 5

Abstract

from arXiv · show

Collider searches face the challenge of defining a representation of high-dimensional data such that physical symmetries are manifest, the discriminating features are retained, and the choice of representation is new-physics agnostic. We introduce JetCLR to solve the mapping from low-level data to optimized observables though self-supervised contrastive learning. As an example, we construct a data representation for top and QCD jets using a permutation-invariant transformer-encoder network and visualize its symmetry properties. We compare the JetCLR representation with alternative representations using linear classifier tests and find it to work quite well.

1 Introduction

The paper frames symmetry-aware representations as a way to move beyond a limited set of observables and full process control in unsupervised collider analyses. It introduces self-supervised contrastive learning for jet representations and evaluates them with a linear classifier test.

1 Introduction

  • Supervised classifiers can yield task-optimized observables from low-level data or simulations, but those observables lack theoretical calculability.The passage notes that calibration and understood systematic errors can make this lack of calculability acceptable for supervised classification.Source: §1 · p. 2
  • Unsupervised analyses motivate replacing a limited set of high-level observables and full process control with high-dimensional representations structured by symmetries and fundamental theory.The cited examples include anomaly searches, generalized side-band analyses, and generalized model hypotheses.Source: §1 · p. 2
  • Jets are a common LHC machine-learning application, with representations including jet images, graphs, trees, the Lund plane, Lorentz-inspired networks, and energy flow polynomials.Jet images use rapidity versus azimuthal angle and typically exploit rotation symmetry in preprocessing.Source: §1 · p. 2
  • JetCLR uses self-supervised contrastive learning to map jet constituents into a representation designed to retain discrimination while becoming invariant to specified symmetries and augmentations.The method uses a transformer encoder, and jet truth labels are not used in optimization.Source: §1 · p. 2
  • A linear classifier test measures how well processes separate by a linear cut in the learned representation space.The paper uses this test to benchmark JetCLR.Source: §1 · p. 2

2 Contrastive learning

Contrastive learning maps jets and symmetry-augmented copies into a representation space, using positive and negative pairs to balance invariance with discrimination. The paper develops the loss and its symmetry and theory-inspired augmentations, including approximate infrared and collinear safety.

2 Contrastive learning

  • Positive pairs bring original jets and augmented copies together, while negative pairs separate different jets; training never uses top-versus-QCD truth labels.The jet phase space has dimension 3nC, and the representation is normalized to lie on a unit hypersphere.Source: §2 · p. 3, Loss function · p. 3

Loss function

  • The contrastive loss combines positive-pair similarity and negative-pair contributions, with temperature τ controlling their relative influence.The passage also defines an angular distance from the angle between representation vectors.Source: Loss function · p. 4

Uniformity vs alignment

  • Alignment favors mapping each jet and its augmented counterpart to the same point, while uniformity spreads jets across the hypersphere and supplies discriminative power.The combined contrastive objective does not produce representations that are perfectly aligned or perfectly uniform.Source: Uniformity vs alignment · pp. 4–5

Symmetries and augmentations

  • Jets are centered by their pT-weighted centroid, then augmented with rotations and translations in the η−φ plane and soft-constituent smearing.Rotations are sampled over 0 to 2π; translations in each direction range from −1 to 1, and soft-coordinate smearing uses pT-suppressed variance relative to Λsoft = 100 MeV.Source: Symmetries and augmentations · p. 5
  • Collinear augmentations split constituents while preserving total pT locally, making the resulting representation approximately infrared and collinear safe.The method lets contrastive optimization determine the representation rather than explicitly enforcing a fixed set of angular correlations or pT scalings.Source: Symmetries and augmentations · pp. 5–6

3 JetCLR

JetCLR combines contrastive learning with a permutation-invariant transformer to map jet constituents into a representation space. Its attention and masking choices support permutation symmetry and infrared-safe handling of variable-length inputs.

3 JetCLR

  • JetCLR combines contrastive loss with a permutation-invariant network architecture to define jet representations.Source: §3 · p. 6

Attention

  • Self-attention weights constituent values using query-key scores, and the operation is permutation equivariant across set elements.Source: Attention · pp. 6–7

Transformer encoder

  • The encoder embeds constituents, repeats attention and residual feed-forward blocks, sums their outputs, and applies a final feed-forward head.Source: Transformer encoder · p. 7

Variable-length inputs

  • Masked attention and constituent summation exclude zero-pT padding; continuous pT weighting makes the transformer infrared-safe by construction.Source: Variable-length inputs · pp. 7–8

4 Pretty good results

The section tests JetCLR’s augmentation choices, symmetry encoding, signal-fraction sensitivity, and linear-classifier performance against other representations. It reports that combined augmentations perform best, rotation training yields rotational invariance, and JetCLR slightly outperforms EFPs in the stated linear-classifier comparison.

4 Pretty good results

  • The study uses 1000-dimensional representations and trains a linear classifier on 50,000 top and 50,000 QCD jets for 5000 epochs.The temperature range found to work best is τ = 0.1 ... 0.2; performance gains from larger transformer dimensions plateau around 1000 ... 1500 dimensions.Source: §4 · p. 8

JetCLR

  • Combining translations, rotations, and soft-plus-collinear augmentations gives the best representations, and LCT gains track optimization of the contrastive loss.Soft-plus-collinear augmentation works best individually; IR-safe masking gives a slight boost when all augmentations are combined.Source: JetCLR · p. 9, Table 2 · p. 9
  • JetCLR’s LCT performance is hardly sensitive to the training sample’s top-signal fraction, indicating that it can encode fundamental structures from QCD jets and the chosen symmetries and augmentations alone.The authors note that each S/B < 1 signal model was trained only once, so results may be noisy.Source: JetCLR · p. 9

Encoded symmetries

  • With rotational transformations in training, JetCLR representations are rotationally invariant; without them, similarity varies from 0.5 to 1.0 across rotation angles.The comparison includes 100 jets rotated at evenly spaced angles and toy two- and three-prong jets.Source: Encoded symmetries · p. 10, Figure 5 · p. 11

JetCLR performance

  • At the same latent dimension, JetCLR slightly outperforms EFPs with a linear network and binary cross-entropy, while EFPs improve under a linear discriminant analysis assumption unavailable to JetCLR.The authors caution that an LCT is only a proxy for representation quality; performance in an actual analysis is the real test.Source: JetCLR performance · p. 11, Figure 6 · p. 12

5 Conclusions

JetCLR uses a transformer encoder and contrastive learning to build jet representations that encode physical symmetries and constituent augmentations. In a linear-classifier test, it outperforms simple jet images and is competitive with energy flow polynomials.

5 Conclusions

  • The transformer-encoder network encodes rotation, translation, and permutation symmetries, along with invariance to soft and collinear constituent augmentations.The approach uses contrastive learning to design observables that respect symmetries and augmentations while retaining discriminative power.Source: §5 · p. 12
  • JetCLR outperforms simple jet images and is competitive with energy flow polynomials in a linear-classifier test.The test evaluates a simple supervised classifier trained on the learned representations to distinguish top jets from QCD jets.Source: §5 · p. 12
  • The authors argue that incorporating symmetry principles and physics knowledge into self-supervised representations is particularly suited to enhancing and controlling anomaly searches.They connect this suitability to JetCLR’s use of symmetries from a single augmented dataset.Source: §5 · p. 12

A Linear classifier tests

Comparing representations without a specific downstream task is difficult. Although linear classifier tests restrict classifier expressivity, their outcomes can still depend on the loss and optimization choices.

A Linear classifier tests

  • Without a direct application to a specific task, comparing data representations is difficult.Downstream tasks may include anomaly detection, classification, or regression.Source: A Linear classifier tests · p. 13
  • A linear classifier test is intended to measure representation expressivity by removing much of the classifier’s expressive power.The passage cautions that this test can nevertheless be ambiguous.Source: A Linear classifier tests · p. 13
  • Linear classifier test results depend more on inductive biases from the chosen loss and optimization than the test’s motivating intuition suggests.The comparison in Table 3 covers JetCLR and energy flow polynomials using different tests; reported performance averages 10-fold cross-validation results after hyperparameter selection.Source: A Linear classifier tests · p. 13

Binary cross entropy loss

The binary cross-entropy classifier assumes class probabilities follow a sigmoid of a linear function of the data. Its parameters are fit against labeled data, with regularization selected by cross-validation.

Binary cross entropy loss

  • The binary cross-entropy classifier assumes p(y = 1|x) = sigmoid(w^T x + c), where x denotes data and y ∈ {0, 1} the two classes.The passage identifies this classifier as logistic regression.Source: Binary cross entropy loss · p. 13
  • The classifier fits w and c by minimizing the KL divergence between its model and labeled data, which in practice means minimizing binary cross entropy.Source: Binary cross entropy loss · p. 13
  • The regularization parameter λ is selected from {10^-6, 10^-4, 10^-2} by 10-fold cross-validation; the optimization problem is convex.Regularization is not strictly necessary but can improve performance, and setting λ = 0 turns it off.Source: Binary cross entropy loss · p. 13

Support vector machine

The support vector machine seeks a wide separating margin and handles nonseparable classes by penalizing misclassified points. Its results depend on a tuned hyperparameter, and the paper compares classifier tests for JetCLR and energy flow polynomials.

Support vector machine

  • For non-linearly separable classes, an SVM penalizes how far misclassified points lie beyond the decision boundary but ignores safely classified points.It seeks the widest possible margin when separating two classes.Source: Support vector machine · p. 13
  • Table 3 compares JetCLR and energy flow polynomials using different linear classifier tests.Source: Table 3 · p. 14
  • Unlike logistic regression, the SVM does not push already-correct points farther across the boundary, and it makes no assumptions about the data distribution.The SVM is expressible as a convex problem.Source: Support vector machine · p. 14
  • SVM results can vary with the hyperparameter controlling the strictness of misclassification penalties, which the authors select using another metric such as classification accuracy.The squared hinge variant penalizes points just on the correct side less strongly and points on the incorrect side more strongly; λ is selected by 10-fold cross-validation.Source: Support vector machine · p. 14

Linear discriminant analysis

Linear discriminant analysis models two equally likely Gaussian classes with a shared covariance, yielding a linear decision score when the quadratic covariance terms cancel. Its Gaussian-data assumption is not met by JetCLR representations, which are optimized for uniformity on a unit hypersphere.

Linear discriminant analysis

  • LDA assumes two equally likely Gaussian classes with a shared covariance matrix, whose means and covariance are typically estimated from labeled data.Source: Linear discriminant analysis · p. 14
  • When the shared covariance terms cancel in the Bayes log-probability ratio, the classifier becomes linear in x, with score wT x+c.Source: Linear discriminant analysis · p. 14
  • JetCLR representations are optimized for uniformity on a unit hypersphere, violating LDA’s Gaussian-data assumption and preventing it from capturing their structure.Source: Linear discriminant analysis · p. 14
  • Reported results are averages over 10-fold cross validation, although no cross validation is needed to select LDA hyperparameters.Source: Linear discriminant analysis · p. 14

Related papers

Loading 2108.04253v1…