PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Collider Event Generation with Residual-Quantized Tokens

Dan Godi, Dmitrii Kobylianskii, Eilam Gross

arXiv:2610.00569v1hep-excs.LGhep-phphysics.data-an

TL;DR

Full detector simulation and reconstruction are projected to become a computational bottleneck at the HL-LHC. The paper applies residual-quantized autoregressive modeling to conditional full-event generation and studies how model and data scaling relate to physical fidelity. It reports that token-level loss predicts generated physical-observable fidelity, with the relationship described by a power law.

  • Problem

    Full detector simulation and reconstruction are projected to exceed available compute resources as collider synthetic-dataset requirements grow.

    Source: §I · p. 1

  • Method

    A frozen residual-quantized tokenizer represents full events as discrete tokens, and an autoregressive transformer generates particle-level detector responses conditioned on truth particles.

    Source: C. Autoregressive Generative Model · p. 3, §IV · p. 9

  • Results

    Achieved next-token loss is a strong predictor of global distributional distances and individual physical metrics, with their relationship described by a power law.

    Source: §IV · p. 9

  • Takeaways & Limitations

    Token-level scaling laws may help estimate downstream physical performance and guide compute allocation for collider-physics surrogate models.

    Source: §IV · p. 9

Abstract

from arXiv · show

Full detector simulation and reconstruction of collider events are projected to become major bottlenecks at the High-Luminosity Large Hadron Collider, motivating the development of fast, ML-based surrogates. At the same time, LLMs have driven fast progress in generative discrete modeling: autoregressive transformers trained on tokenized data now represent the state of the art across a range of generative tasks. We extend the discrete modeling paradigm by introducing a particle-level generative model trained on residual-quantized full-event data. We demonstrate the ability of this model family to perform conditional generation from detector-stable particles; we study its scaling behavior across a range of dataset and model sizes, characterize the effects of repeated data exposure and demonstrate that token-level loss systematically predicts downstream physical fidelity. These results provide an empirical framework for scalable collider full-event generation based on residual-quantized representations.

I. INTRODUCTION

Full-event detector simulation and reconstruction are projected to exceed available compute at the HL-LHC, motivating fast surrogates. This work extends discrete residual-quantized modeling to particle-level full events, with a dataset and method designed for conditional generation.

I. INTRODUCTION

  • Conditional generation models detector response from truth-level particles, a setting preferred for generalizing to rare or novel processes and potentially serving multiple signals and backgrounds.The generated Particle Flow objects preserve event structure and compatibility with downstream pipelines.Source: §I · pp. 1–2
  • The study extends discrete residual-quantized representations to full-event point-cloud generation and examines model and dataset scaling, repeated exposure, and links between token loss and physical fidelity.The authors report that improvements in cross-entropy translate predictably into higher fidelity across particle-, jet-, and global event observables.Source: §I · p. 2, §II · p. 2

II. METHODS

  • The method separates a learned discrete representation from autoregressive sequence modeling, which samples tokens that are decoded back into continuous particle-level quantities.The tokenizer is separately trained and fixed while the transformer models token sequences.Source: §II · p. 2

A. Dataset

  • The dataset uses generator-level truth particles and reconstructed Particle Flow particles from simulated and reconstructed CMS Open Data events; training mixes t¯t and QCD, with held-out and unseen-process tests.The particle collections include kinematics, class, and production-vertex information, and particles are ordered by transverse momentum.Source: A. Dataset · p. 2, Table I · p. 2

B. RQ-VAE

The RQ-VAE maps inputs to ordered discrete codes, using residual quantization to expand effective representation capacity while avoiding reliance on very large codebooks.

B. RQ-VAE

  • A conventional VQ-VAE represents an encoded latent with one nearest-neighbor codebook index, but fidelity is constrained by codebook size and large codebooks can collapse.Codebook collapse means only a small fraction of entries are selected, limiting effective capacity.Source: B. RQ-VAE · p. 3
  • Residual quantization represents a latent with an ordered sequence of M codes, quantizing and subtracting the best code for the running residual at each depth.This raises effective codebook size to KM without requiring K to grow as much.Source: B. RQ-VAE · p. 3
  • The input becomes a sequence of code indices, whose code vectors are summed to form the quantized latent and then decoded into a reconstructed input.The tokenizer combines a transformer encoder, residual-quantization module, and transformer decoder, trained end-to-end for reconstruction.Source: B. RQ-VAE · p. 3
  • With the tokenizer frozen, each particle maps to M residual codes and an event of N particles becomes a sequence of NM tokens.Source: B. RQ-VAE · p. 3

C. Autoregressive Generative Model

A decoder-only transformer autoregressively models PFlow tokens conditioned on tokenized truth events, while a cardinality head predicts the number of reconstructed particles. The evaluation tests in- and out-of-distribution generation and compares detector-response residuals rather than requiring exact particle-feature matches.

C. Autoregressive Generative Model

  • The transformer models PFlow token sequences conditioned on a tokenized truth-event prefix, with shared frozen RQ-VAE tokenization for truth and PFlow events.Token embeddings combine code-index, residual-depth, and codebook-vector components.Source: C. Autoregressive Generative Model · p. 3
  • A feed-forward cardinality head predicts reconstructed multiplicity, which is supplied through a countdown embedding, and training combines next-token and cardinality cross-entropy objectives.Source: C. Autoregressive Generative Model · p. 3
  • Generation samples the reconstructed multiplicity, then exactly NPF · M tokens autoregressively, and decodes the sequence into particle-level quantities.Source: C. Autoregressive Generative Model · p. 3

III. RESULTS

  • Evaluation uses held-out QCD and t¯t events plus unseen H →4ℓ and QCD 1000 samples, examining feature distributions and matched residuals for particles and jets.The residual-based evaluation targets stochastic detector response rather than deterministic reproduction of exact PFlow features.Source: §III · pp. 3–4

A. Tokenizer fidelity and codebook utilization

Tokenizer fidelity is assessed through reconstructed-feature distributions, particle-class confusion, and codebook usage and perplexity across residual depths. Generation performance is examined through residual spectra at particle, event, and jet levels across test datasets.

A. Tokenizer fidelity and codebook utilization

  • The tokenizer uses four residual stages with codebooks of 8192 16-dimensional vectors to assess reconstruction fidelity.Figure 1 presents original-versus-reconstructed feature distributions and a particle-class confusion matrix; vertex residuals are computed only for charged particles.Source: A. Tokenizer fidelity and codebook… · p. 4, Figure 1 · p. 4
  • Codebook usage and perplexity are reported for each residual depth across the three test datasets.Usage is the percentage of codes used at least once; the supplied passages do not provide the table values.Source: A. Tokenizer fidelity and codebook… · p. 4, Table II · p. 4

B. Generation Performance

  • The generation study reports the largest model configuration trained for four epochs on the full dataset, with ablation comparisons in Appendix D..Source: B. Generation Performance · p. 5
  • Particle residual spectra are shown as normalized histograms, with rows for datasets or processes and columns for observables.The figure covers all three testing datasets.Source: Figure 2 · p. 5
  • Event-level residuals cover particle count, missing transverse energy in x and y, and scalar summed particle pT; jet-level residuals cover kinematics and substructure against matched truth jets.The event-level variables probe global correlations, while jet substructure observables test intra-jet particle correlations and detector effects.Source: B. Generation Performance · pp. 5–6, Figure 3 · p. 5, Figure 4 · p. 6

C. Scaling study

The study varies model size, training-set size, and data repetition, then examines how held-out token loss relates to physical observables. Lower loss predicts higher fidelity on global and individual physical metrics.

C. Scaling study

  • The scaling study varies model size, training-set size, and repeated exposure, evaluating held-out token cross-entropy and generated physical observables.Each model configuration is trained across eight data and repetition settings.Source: C. Scaling study · p. 6, Table III · p. 6

1. Loss scaling

  • Repetition-aware fits apply a prescriptive penalty or combine that penalty with the coupled Skaling form; all fitted laws predict similar compute-optimal allocation parameters.The repetition-aware laws use all 48 runs, while Chinchilla and Skaling use the 18 single-epoch runs.Source: §1 · p. 7, Table V · p. 7
  • N ∝ C^0.53 is the isoFLOP-fitted compute-optimal model-size scaling; runs remain data-constrained, and larger models overfit sooner on smaller datasets.Larger models achieve lower loss when sufficient unique data are available.Source: §1 · p. 7, Figure 5 · p. 8

2. Scaling of physical metrics

  • The held-out loss strongly predicts Sliced Wasserstein Distance on global particle-, event-, and jet-level feature manifolds.The relationship is fitted as a power law.Source: §2 · p. 8, Figure 6 · p. 8
  • Each individual residual feature distribution exhibits scaling behavior, and lower achieved loss predicts higher fidelity across those features.Improvements in loss transfer to individual features, including when marginal differences can approach two-sample noise.Source: §2 · p. 8, Figure 7 · p. 9

IV. CONCLUSION

The paper applies residual-quantized autoregressive modeling to conditional full-event simulation and studies its scaling across model and dataset sizes. Token-level loss predicts fidelity across physical observables, supporting empirical scaling laws as proxies for downstream performance and tools for compute allocation.

IV. CONCLUSION

  • Residual-quantized point-cloud representations enable discrete autoregressive generation of particle-level detector responses for conditional full-event simulation.The model family is evaluated across particle-, event-, and jet-level observables, including processes not encountered during training.Source: §IV · p. 9
  • The model family exhibits familiar next-token-loss scaling, and a repetition-aware scaling law captures effects of repeated data exposure.The study spans model and dataset sizes.Source: §IV · p. 9
  • Token-level loss predicts global distributional distances and individual physical metrics, with their relationship described by a simple power law.The paper presents token-level scaling laws as useful proxies for downstream physical performance and empirical guides for compute allocation.Source: §IV · p. 9

Appendix A: Methods details

The methods represent particles with standardized features and train a residual-quantized set-to-set autoencoder to encode and reconstruct particle sets. The tokenizer uses a transformer architecture and is frozen after training.

1. Input representation

  • Each particle is represented by 12 features, with standardized kinematics and vertex coordinates; neutral-particle vertices are set to zero.Particles are pT-sorted, padded to 400, and masked; evaluation applies |η| < 2.5 acceptance.Source: §1 · p. 14, Table VI · p. 14

2. RQ-VAE tokenizer

  • The set-to-set tokenizer uses bidirectional self-attention, with per-particle input and output MLPs around a five-layer, width-128 transformer.The output MLP produces a 16-dimensional latent representation.Source: §2 · p. 14
  • Residual codebooks are initialized by 100-iteration k-means and updated using exponential moving averages with decay 0.8.Encoder gradients pass through the quantizer via the rotation trick.Source: §2 · p. 14
  • The tokenizer is trained on 16.5 million events, using Truth and PFlow sets for 33 million point clouds per epoch, then frozen.Training uses reconstruction loss plus a small per-codebook commitment loss.Source: §2 · p. 14

Appendix B: Autoregressive model

The autoregressive model consumes tokenized Truth and PFlow events in a fixed two-block sequence layout and predicts PFlow tokens conditioned on Truth. A cardinality head and countdown embeddings support variable-length generation.

Appendix B: Autoregressive model

  • Each event uses separate padded Truth and PFlow blocks of 1,600 code positions, yielding a 3,202-position sequence with padding excluded from computation.The vocabulary has 8,195 entries, including padding, code, and two start tokens.Source: Appendix B · pp. 14–15
  • The backbone is a decoder-only causal transformer with rotary position embeddings, QKV normalization, and SwiGLU feed-forward blocks.The PFlow block always begins at position 1601, independent of Truth multiplicity.Source: Appendix B · p. 15
  • The cardinality head masked-means final hidden states over the truth block, applies Linear(dmodel, dmodel) → GELU → Linear(dmodel, 401), and returns logits for NPF values 0–400.Its gradient flows into the shared backbone.Source: Appendix B · p. 15
  • Training applies next-token loss only to the PFlow block and uses a composite cross-entropy objective with cardinality prediction.The reported parameter count includes embedding tables but excludes the frozen tokenizer.Source: Appendix B · p. 15
  • At inference, the model samples cardinality and then generates four codes per particle autoregressively before decoding with the frozen tokenizer.The throughput table measures full generation, including quantization, prefill, autoregressive decoding, and RQ-VAE decoding, on A100 and H200 GPUs.Source: Appendix B · p. 15, Table VIII · p. 16
  • The 22M model runs slower than its parameter count suggests because its 48-wide attention head lacks an optimized FlashAttention inference kernel.The inference system had no optimized kernel for that head dimension.Source: Appendix B · p. 16

Appendix C: Scaling study methods

The scaling study fits loss laws against model size, unique data, and—in repetition-aware fits—data exposure. Shared fitting choices and run conditions define how the fitted laws are compared.

Appendix C: Scaling study methods

  • The fits use parameter count N and unique events U, with amplitudes A, B, and P treated as free parameters.Events serve as the data indicator, and the architecture family and training recipe are held constant across runs.Source: §1 · p. 16
  • All four loss laws use a shared pair of terms: one scaling with parameter count and another with unique events adjusted for repetition.The shared form is F(N, U, R) = A N^-α + B [U(1 + R)]^-β.Source: §1 · p. 16
  • Chinchilla and Skaling omit repetition and fit 18 single-epoch runs, while Prescriptive and Custom include a penalty and fit all 48 runs.The penalty is zero at R = 0; parameters are fitted by unweighted least squares to held-out in-distribution losses.Source: §1 · p. 16

2. IsoFLOP analysis

The analysis constructs compute-matched loss slices from training checkpoints and relates physical-observable distances to model loss. The physical comparisons use truth-relative residuals and explicit convergence and overfitting criteria.

2. IsoFLOP analysis

  • Checkpoints saved up to every 1/16 epoch are evaluated on held-out data, with loss at a target compute estimated by local quadratic interpolation in log step count.Each run contributes 8 to 16 logs.Source: §2 · pp. 16–17
  • For each compute budget, the analysis fits available checkpoint losses to estimate the minimum, requiring at least four model-size values; Figure 8 shows the selected slices and loss points.The slices are built from single-epoch training curves, and intermediate checkpoints may have learning rates that have not fully annealed.Source: §2 · pp. 16–17, Figure 8 · p. 17

3. Scaling in the physical metrics

  • Physical scaling compares generated and reference events after common acceptance and pT thresholds, using Truth-relative event, particle, and jet residuals.Particle and jet objects are matched to Truth; the projected one-dimensional W1 distances are averaged over 256 random projections, with a statistically identical-sample noise floor.Source: §3 · p. 18
  • The loss–metric relation is fitted in ln SWD only for runs meeting compute, token-per-parameter, convergence, and non-overfitting criteria.Intermediate checkpoints are included to probe the relation across training dynamics.Source: §3 · p. 18

Appendix D: Additional material

Appendix D reports codebook usage and perplexity across the three test datasets, and supplements generation results with normalized marginal spectra and per-feature W1 comparisons.

1. Quantization results

  • Codebook usage and perplexity are evaluated across all three testing datasets.Figure 9 orders codes by usage rank, from most to least used.Source: §1 · p. 18, Figure 9 · p. 19

2. Generation results

  • Normalized spectra show particle-, jet-, and event-level features for the three testing datasets, with rows for datasets or processes and columns for observables.The particle-, jet-, and event-level histograms are each normalized to unity.Source: §2 · p. 18, Figure 10 · p. 19, Figure 11 · p. 20
  • Table X reports per-feature W1 distances for FS/PF residuals after four-epoch full-dataset training, alongside RQ-VAE reconstruction and independent-PF-sample noise references on 50,000 held-out events.Distances are given in the units of each feature.Source: Table X · p. 21
Loading 2610.00569v1…