Source-linked AI summary
Neural Scaling Laws for Boosted Jet Tagging
Matthias Vigl, Nicole Hartman, Michael Kagan, Lukas Heinrich
TL;DR
This paper addresses how neural-network compute scaling can characterize performance limits for boosted jet classification in HEP. Using JetClass, it jointly varies model capacity and dataset size, then studies repetition, input features, and particle multiplicity. It finds an asymptotic performance limit that scaling approaches, while richer lower-level features raise the ceiling and improve fixed-size-data results.
Problem
HEP has comparatively limited study of how model capacity, dataset size, and compute govern boosted jet-tagging performance and its fundamental limits.
Method
The study jointly varies model capacity and training dataset size on JetClass, deriving compute-optimal scaling relations and testing data repetition, input features, and particle multiplicity.
Results
An irreducible loss defines an asymptotic performance limit; data-scaling exponents remain approximately constant across input configurations, while asymptotic loss varies significantly.
Takeaways & Limitations
More expressive, lower-level features can raise the performance ceiling and improve results at fixed dataset size without substantially changing the data-scaling rate.
Takeaways & Limitations
The effective dataset-size interpretation for repeated data is viable only if the scaling form holds as model size approaches infinity.
Abstract
from arXiv · showhide
The success of Large Language Models (LLMs) has established that scaling compute, through joint increases in model capacity and dataset size, is the primary driver of performance in modern machine learning. While machine learning has long been an integral component of High Energy Physics (HEP) data analysis workflows, the compute used to train state-of-the-art HEP models remains orders of magnitude below that of industry foundation models. With scaling laws only beginning to be studied in the field, we investigate neural scaling laws for boosted jet classification using the public JetClass dataset. We derive compute optimal scaling laws and identify an effective performance limit that can be consistently approached through increased compute. We study how data repetition, common in HEP where simulation is expensive, modifies the scaling yielding a quantifiable effective dataset size gain. We then study how the scaling coefficients and asymptotic performance limits vary with the choice of input features and particle multiplicity, demonstrating that increased compute reliably drives performance toward an asymptotic limit, and that more expressive, lower-level features can raise the performance limit and improve results at fixed dataset size.
1 INTRODUCTION
HEP jet tagging increasingly relies on expressive neural networks, but scaling behavior and performance limits remain important open questions. This work uses neural scaling laws to study how model size, data, and compute shape boosted-jet classification.
- Motivation: Jet tagging is a central HEP task because boosted-jet substructure encodes the identity of highly energetic parent particles.Boosted jets arise when decay products of heavy particles merge into one large-radius jet.
- Motivation: State-of-the-art HEP jet-tagging models use far less training compute than industry foundation models.The supplied passage introduces this gap as motivation for studying scaling in HEP.
- Motivation: Scaling models across data, parameters, and compute is presented as essential for realizing larger architectures' potential in HEP.The motivation concerns architectures trained on diverse, multi-modal inputs.
- Related work: Prior studies report power-law scaling in jet-tagging loss, with classifier rankings and scaling exponents changing as datasets grow.Related work also includes transformer and foundation-model scaling studies in particle and nuclear physics.
- Contribution: This work varies model and dataset size on JetClass to derive compute-optimal relations, identify an asymptotic loss, and test feature and multiplicity effects.It also studies how data repetition modifies scaling and frames scaling laws as a resource-allocation tool.
2 DATASET AND TRAINING SETUP
The study uses simulated JetClass jets and set Transformers to examine scaling under controlled changes in model capacity, dataset size, and particle multiplicity. Training compute is estimated from parameter count, samples, and average constituent multiplicity.
- Dataset: JetClass provides 100M training, 5M validation, and 20M testing jets across 10 classes, with up to 128 particles per jet.The dataset simulates top, W, Z, and Higgs processes, with QCD as the background class.
- Model: The models use a set Transformer encoder that is invariant to constituent ordering and accepts variable-length particle sequences.Particles are sorted by decreasing pT only to define deterministic truncation when varying particle count.
- Compute: C = 6npND FLOPs per epoch estimates training compute for a model with N parameters on D samples.Here np is the particle multiplicity, averaging approximately 40 particles per jet.
- Training: Model capacity is scaled by varying embedding dimension while training uses batch size 128, AdamW, and cross-entropy loss.The supplied setup specifies learning rate 10^-4 and weight decay 10^-2.
3 SCALING LAWS
The paper derives compute-optimal scaling laws for boosted jet classification, then examines how fixed datasets, repeated training, overfitting thresholds, and input representations alter scaling and asymptotic performance.
- Compute-optimal scaling: The loss is modeled as an irreducible term plus finite-model and finite-dataset contributions, L(N,D)=L∞+A/N^α+B/D^β.The exponents α and β determine how rapidly model-capacity and dataset-size errors diminish.
- Compute-optimal scaling: Compute-optimal training minimizes loss at fixed compute by jointly allocating model size and dataset size, yielding N ∝ C^a, D ∝ C^(1−a), and L ∝ C^−γ.The compute constraint follows from the per-sample Transformer forward-and-backward cost.
- Scaling under data repetition: Beyond the regime where the full dataset is seen once, repeated epochs become necessary in typical HEP datasets but make updates less compute-optimal, causing loss saturation or overfitting.The paper notes that practical jet datasets contain roughly 10^8 to 10^10 jets, while simulation remains expensive.
- Scaling under data repetition: Data repetition preserves approximately the same scaling exponent βrep≈β but changes the normalization, requiring roughly 10 times more compute to reach the same loss as compute-optimal training without repetition.Models above the overfitting threshold minimize validation loss at each fixed dataset size.
- Scaling under data repetition: The overfitting threshold scales as N ∝ D^0.47, while increasing model size beyond it no longer reduces converged loss in the repeated-data regime.The threshold separates underfitting models, whose validation loss plateaus, from overfitting models, whose loss later increases.
4 PHYSICS PERFORMANCE
The paper translates scaling-law behavior into QCD background rejection and compares tagging performance across input representations and particle multiplicities. Larger, richer representations achieve higher rejection, while individual signal processes can scale differently.
- Physics metrics: QCD background rejection is evaluated at fixed signal efficiency, with 50% used for most signal classes and higher efficiencies for H →lνqq′ and t →blν.The scaling laws map cross-entropy loss to physics-relevant rejection metrics.
- Physics metrics: The ParT benchmark trained on 100M samples is crossed at the corresponding 100M-jet scale, supporting consistency between scaling predictions and established performance.The benchmark is shown as dashed reference lines in Figure 5.
- Input representations: Richer input representations and higher particle multiplicities yield substantially higher QCD rejection, consistent with lower asymptotic losses.Figure 6 compares four feature configurations and asymptotic performance obtained from fitted L∞ values.
- Input representations: Individual signal processes benefit differently from retained physics information, producing different scaling exponents when particle multiplicity or features are restricted.The 10-particle, 128-particle, and kinematic-only configurations exhibit distinct behavior in Figure 6b.
5 CONCLUSION
The study establishes neural scaling laws for boosted jet classification and identifies asymptotic performance limits shaped by input representation and particle multiplicity. Data repetition improves efficiency by reducing the scaling prefactor, but broader studies are needed to test the generality of the feature-dependence result.
- Scaling laws: Cross-entropy loss follows a scaling form in model capacity and dataset size, with irreducible loss L∞ representing the infinite-model, infinite-data performance limit.The study jointly varies model capacity N and training dataset size D using JetClass.
- Input dependence: The asymptotic limit depends on input representation and particle multiplicity, while the data scaling exponent β remains approximately constant across the studied configurations.Richer, lower-level features can raise the performance ceiling without substantially changing the rate at which data reduces loss.
- Data repetition: Training beyond the overfitting threshold effectively reduces the scaling-law prefactor B, providing a data-efficiency gain at the cost of additional compute.The benefit of repeated data eventually saturates and can degrade validation performance through overfitting.
- Scope: A broader study across more input modalities is needed to confirm whether the observed feature-dependence pattern is general.This is the paper’s explicit scope limitation for the input-representation conclusion.