PaperlayerView exact revision on arXiv

Source-linked AI summary

RINO: Renormalization Group Distillation with No Labels

Zichun Hao, Raghav Kansal, Abhijith Gandrakota, Chang Sun, Haoyang Li, Zihan Zhao, Ngadiuba Jennifer, Javier Duarte, Maria Spiropulu

arXiv:2610.15919v1hep-excs.AI

TL;DR

Simulation-trained HEP classifiers face a simulation-to-data gap, while existing SSL approaches often use views that do not reflect jets’ multi-scale structure. RINO uses kT clustering to build multi-scale jet views for DINO-style self-distillation. It improves OOD top tagging and transfers to unseen signal classes under the reported evaluation setup.

  • Problem

    Simulation-trained HEP classifiers face a simulation-to-data gap, and existing SSL methods largely use single-scale views despite jets’ multi-scale structure.

    Source: §1 · pp. 1–2

  • Method

    RINO uses different kT clustering scales of the same jet as views and aligns their representations through teacher-student self-distillation.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    RINO improves OOD top tagging by 5.9% over the strongest baseline, matches supervised training with 100× fewer labels, and transfers to unseen H →4q jets.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    DINO-style self-distillation retains scale-dependent structure in the backbone, a difference associated with an 8.6% OOD accuracy gap versus JetCLR-scale.

    Source: §1 · p. 2

    The JETNET-to-JETCLASS proxy may not faithfully reflect simulation-to-data shift, and direct real-data validation is beyond this work’s scope.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Supervised machine learning in high-energy physics (HEP) requires training on labeled simulations of particle collisions, which are known to be unfaithful to the underlying physics. This leads to significant inaccuracies and biases in the transfer of trained models to real data, limiting the power of deep learning methods. Self-supervised learning (SSL) is a promising alternative for pretraining directly on real, unlabeled data. However, existing techniques largely operate with inductive biases inspired by domains such as images and natural language. We propose RINO (Renormalization Group DIstillation with NO Labels), which, for the first time, exploits the fundamentally multi-scale renormalization group structure of high-energy jet formation as a physically grounded source of SSL views. We define teacher-student representations across energy scales using different levels of hierarchical clustering, aligned via DINO-style self-distillation. To mirror the background-dominated nature of real collider data, RINO pretrains exclusively on jets from quantum chromodynamics (QCD) processes and is evaluated on out-of-distribution (OOD) transfer to rare signal classes. In our controlled data-efficiency study, RINO matches fully supervised training on JetNet after finetuning on only 1% of the labeled data. With the full labeled dataset, RINO improves OOD classification accuracy on top-quark jets by 5.9% and, without retraining, improves AUC on the unseen Higgs boson decay channel $H \to 4q$ by 4.1% over the strongest SSL baseline. Scale-structure diagnostics reveal that DINO-style self-distillation permits our backbone to retain scale-dependent structure. Ablation studies show that this accounts for an 8.6% OOD accuracy gap with respect to alternative SSL paradigms such as SimCLR. Code is available at https://github.com/zichunhao/RINO.

1 INTRODUCTION

RINO addresses the simulation-to-data gap in HEP by using physically grounded, multi-scale jet views for self-supervised learning. Its QCD-only pretraining improves out-of-distribution top tagging, while ablations associate scale-aware representations and multi-scale views with performance gains.

Motivation

  • The mismatch between simulated training data and real measurements is a fundamental ML bottleneck; in HEP, simulation-based classifiers can lose 20% to 40% of performance on real data.Existing physics-motivated SSL approaches encode useful symmetries but operate at a single physical scale.Source: §1 · pp. 1–2

Contributions

  • RINO is presented as the first SSL framework to align jet representations across multiple kT clustering scales.The multi-scale views use clustering to represent different resolutions of the same jet.Source: §1 · p. 2, Figure 1 · p. 3
  • 5.9% higher OOD top-tagging performance than the strongest baseline, with lower seed-to-seed variance, was achieved under QCD-only pretraining and the JETNET-to-JETCLASS evaluation protocol.RINO also showed ∼3× smaller W1 and 6–7× smaller MMD2 between finetuning and evaluation domains.Source: §1 · p. 2
  • An 8.6% OOD accuracy gap is associated with DINO retaining scale-dependent structure in the backbone, unlike the nearly scale-invariant JetCLR-scale backbone.Both methods achieve comparable cross-scale invariance in the full system, but locate it differently between backbone and projection head.Source: §1 · p. 2
  • Multi-scale view construction contributes +11.4% to +14.0% over single-scale alternatives.The paper motivates these views with the multi-scale structure of QCD radiation and its coarse-grained descriptions of jets.Source: §1 · pp. 2–3

2 RELATED WORK

Prior jet-tagging methods are largely supervised, while SSL methods adapt view alignment, masking, generative, and joint-embedding approaches to jets. RINO instead derives its multi-scale views from the physically ordered kT clustering sequence.

Jet Tagging

  • Most jet-tagging advances are supervised and depend on labeled Monte Carlo simulations, whose imperfect modeling creates a persistent gap when models are deployed on real collision data.RINO focuses instead on pretraining transferable representations from unlabeled data, with jet tagging as its primary downstream benchmark.Source: §2 · p. 3

Learning from Augmented Views

  • Vision SSL commonly aligns augmented views such as global and local crops, but those augmentations do not transfer directly to variable-length jet point clouds.For jets, view generation shapes the learned representation’s inductive bias.Source: §2 · p. 3

SSL in HEP

  • Jet SSL spans contrastive, masked, generative, and joint-embedding methods, but the approaches described construct views at a single physical resolution.RINO uses kT clustering to construct views from the unclustered particle cloud to a few subjets and evaluates transfer to a distinct target.Source: §2 · p. 4

Multi-Scale and Hierarchical SSL

  • Unlike vision methods that define scale spatially or architecturally, RINO uses the QCD-motivated kT sequence, where subjet count indexes physically ordered resolution scales.Stopping clustering at N subjets yields a coarse-grained jet description, and varying N traces the multi-scale axis used for representation alignment.Source: §2 · p. 4

3 METHOD: THE RINO FRAMEWORK

RINO pretrains on QCD jets and constructs views of each jet at multiple kT clustering scales. Its DINO-based objective aligns those views, alongside masked-particle prediction and embedding-diversity regularization.

3.1 PRETRAINING AND FINETUNING DATA

  • RINO pretrains exclusively on JETCLASS QCD jets, excluding simulated signal classes, then finetunes on smaller JETNET and evaluates OOD on held-out JETCLASS signals.JETNET serves as a proxy for limited labeled simulation data; the datasets differ in pT range, constituent count, and detector simulation.Source: §3.1 · p. 4

3.2 JET CLUSTERING AS MULTI-SCALE VIEW CONSTRUCTION

  • Stopping exclusive kT clustering at different subjet counts creates physically ordered views of the same jet, from the unclustered particle cloud to coarse-grained descriptions.The ladder uses N ∈{2, 3, 4, 6, 8, 16}; the teacher sees intermediate scales, while the student also sees the coarse-grained and unclustered extremes.Source: §3.2 · p. 5

3.3 SETUP: TRAINING OBJECTIVE AND ARCHITECTURE

  • The DINO loss aligns student representations across views with teacher representations at intermediate scales, while Sinkhorn–Knopp centering is used to prevent collapse.The teacher parameters track the student through an exponential moving average.Source: §3.2 · p. 5, §3.3 · p. 5
  • The objective also includes iBOT masked-particle prediction and KoLeo embedding-diversity regularization, and jets are encoded by a shared student-teacher transformer.The input point cloud is preprocessed into seven kinematic features, with polar-binned positional encoding for masked tokens.Source: §3.3 · p. 5

4 EXPERIMENTS

Under a controlled JETNET-to-JETCLASS protocol, RINO leads OOD top tagging, matches full-label supervised training with 1% of labels, and transfers to unseen signal topologies. Ablations associate its gains with multi-scale kT views, cross-scale DINO distillation, and the energy-ordered kT clustering sequence.

4.1 SETUP AND CONTROLLED COMPARISON PROTOCOL

  • The controlled protocol shares pretraining data, downstream splits, finetuning hyperparameters, and seed counts across methods, with a common transformer backbone except for OmniJet-α.JETNET supplies finetuning labels and JETCLASS is used for OOD evaluation; frozen-embedding probes and finetuned heads assess different representation outcomes.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 OOD TOP TAGGING

  • RINO improves OOD top-tagging accuracy by 5.9% over the strongest baseline, MPMv1, with an MLP head.The comparisons use JETNET finetuning and JETCLASS evaluation; RINO is also the only method whose MLP head significantly exceeds its linear head.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 DATA EFFICIENCY

  • RINO with 1% of JETNET labels reaches 0.797 ± 0.015 JETCLASS accuracy, matching supervised training on all JETNET labels at 0.786 ± 0.023 within uncertainty.RINO leads at all tested label fractions and improves monotonically, while JetCLR-scale outperforms JetCLR across fractions.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

4.4 CROSS-TOPOLOGY GENERALIZATION

  • RINO achieves the highest AUC on unseen H →4q (0.894±0.007) and H →b¯b (0.788±0.016) after finetuning only on top tagging.No model sees Higgs samples during training; the H →b¯b methods cluster within ∼2% AUC, and kinematic-only inputs lack the track-level flavor signatures useful for that task.Source: §4.4 · p. 7

4.5 DESIGN ABLATIONS

  • Adding cross-scale DINO distillation yields +14.0%, while teacher-schedule refinement adds +1.0%.The ablation variants share the same polar-binned positional encoding, backbone, and finetuning protocol.Source: §4.5 · p. 7, Table 3 · p. 8
  • JetCLR-scale reaches 0.780 versus JetCLR’s 0.700 (+11.4%), while RINO’s remaining gain over JetCLR-scale is +7.6% before teacher tuning and +8.6% for the final model.JetCLR-scale replaces JetCLR augmentations with kT views while retaining the contrastive objective; the compared methods use the same multi-scale views and differ in training objective.Source: §4.2 · p. 6, §4.5 · pp. 7–8
  • With the same training objective, kT yields 0.839 ± 0.005 accuracy, above Cambridge/Aachen at 0.796 ± 0.012 and anti-kT at 0.693 ± 0.026.The algorithms define resolution differently: kT coarsens by energy scale, C/A by angular proximity, and anti-kT by hardness.Source: §4.5 · p. 8

5 DISCUSSION

Controlled comparisons attribute RINO’s OOD gains to both kT-scale views and self-distillation, while its backbone retains scale-aware geometry. RINO also has smaller measured representation shifts between JETNET and JETCLASS, though the study’s simulation-to-simulation proxy and task-specific kT bias bound the conclusions.

5.1 WHY CROSS-SCALE TRAINING HELPS

  • Replacing JetCLR augmentations with kT views raises accuracy from 0.700 to 0.780 (+11.4%); adding cross-scale distillation gives +14.0% (0.736 →0.839).Switching from contrastive alignment to self-distillation adds +7.6% for JetCLR-scale →RINO before teacher tuning.Source: §5.1 · p. 8, Figure 3 · p. 9
  • RINO’s backbone is scale-aware while its head realizes invariance; the 8.6% OOD accuracy gap over JetCLR-scale suggests scale-aware backbones outperform scale-invariant ones.The methods share views and architecture; the text notes top jets resolve into three prongs near N ∼3 and develop characteristic soft-radiation patterns at finer scales, unlike QCD jets.Source: §5.1 · pp. 8–9, Figure 3 · p. 9

5.2 WHY RINO GENERALIZES OOD

  • RINO has the smallest reported JETNET–JETCLASS representation shifts: QCD W1=0.043 and MMD2=0.007, versus baseline minima of 0.132 and 0.045.The reported metrics quantify source-to-target representation distance; RINO’s shift is already small before finetuning and does not grow during finetuning.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 LIMITATIONS

  • The JETNET→JETCLASS simulation proxy may not reflect the real simulation-to-data shift; pretraining also overlaps with the test domain through JETCLASS QCD, and kT’s advantage is largest for multi-prong tasks.The advantage narrows for signatures such as H →b¯b; direct real-data validation requires physics analyses beyond this work’s scope.Source: §5.3 · p. 9

6 CONCLUSION

RINO uses multi-scale kT views and DINO-style self-distillation, pretraining on JETCLASS QCD before finetuning on smaller JETNET data. The paper reports OOD top-tagging gains, label efficiency, and transfer to unseen H →4q jets; it also documents how AI-assisted work was reviewed and provides code and public dataset references.

6 CONCLUSION

  • RINO combines multi-scale kT views with DINO-style self-distillation, pretraining on JETCLASS QCD and finetuning on smaller labeled JETNET data before OOD evaluation.The conclusion reports improved top-tagging, matching supervised training with 100× fewer labels, and transfer without retraining to unseen H →4q jets.Source: §6 · p. 10

AI USE STATEMENT

  • AI tools assisted with writing, reference discovery, experiments, limited code work, result interpretation, and figure creation, with author review and approval described for these uses.The authors state that the core implementation and preprocessing pipeline were largely complete before AI use, and that derived numbers were recomputed and checked against experiment outputs.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

  • Code is available, and the paper lists public datasets and appendix locations for key implementation and experimental details.The listed appendices cover pretraining hyperparameters, losses, architecture, datasets and preprocessing, finetuning, baselines, and compute resources.Source: REPRODUCIBILITY STATEMENT · p. 11

A APPENDIX

The appendix passage points readers to the paper’s code repository.

A APPENDIX

A.1 PRETRAINING HYPERPARAMETERS

RINO’s production pretraining configuration uses cosine schedules for teacher EMA momentum and learning rate across 100 epochs. The recipe is held fixed across ablations apart from view-construction parameters and positional encoding, while the headline model uses a tuned teacher schedule.

A.1 PRETRAINING HYPERPARAMETERS

  • The headline RINO model scores 0.847 ± 0.003 using a tuned teacher schedule, fixed teacher temperature, and lower final EMA momentum.The schedule was chosen using OOD top-vs-QCD accuracy on JETCLASS.Source: §A.1 · p. 20
  • Teacher EMA momentum and the learning-rate scheduler follow cosine schedules over all 100 training epochs, with warmup.The listed hyperparameters are for the production configuration.Source: §A.1 · p. 20, Table 5 · p. 20
  • The pretraining recipe stays fixed across ablations except for view-construction parameters and input positional encoding; untuned RINO already exceeds every baseline.Tuning contributes the final +1.0% row of Table 3; before tuning, RINO scores 0.839 ± 0.005.Source: §A.1 · p. 20

A.2 PRETRAINING LOSS TERMS

RINO combines cross-entropy self-distillation across clustered views with masked-particle prediction and a representation-spreading term. The teacher’s output is normalized with Sinkhorn–Knopp and receives no gradient, and loss terms are averaged over mini-batch jets.

A.2 PRETRAINING LOSS TERMS

  • The teacher distribution over K prototypes is Sinkhorn–Knopp normalized and receives no gradient, and each loss term is averaged over mini-batch jets.Student and teacher distributions are obtained with temperatures τs and τt.Source: §A.2 · p. 20
  • The DINO term aligns each student view with every teacher view at a different scale, excluding pairs where the view scales match.Teacher views use N = {6, 8, 16}; student views also include coarse and unclustered views.Source: §A.2 · p. 21
  • The iBOT term predicts teacher outputs for masked particle tokens while the student input masks 25% of particles and the teacher sees the unmasked jet.The iBOT head operates on output tokens at masked particle positions.Source: §A.2 · p. 21
  • The KoLeo term spreads different jets within a view and never compares two views of the same jet.It acts on ℓ2-normalized student [CLS] representations and is averaged over teacher-scale views.Source: §A.2 · p. 21

A.3 ARCHITECTURE DETAILS

RINO uses a shared transformer backbone with fixed, robustly standardized particle features and a polar-binned positional encoding. The encoding is selected to reduce masked-token bin collisions while balancing collision reduction against the cost of a larger grid.

A.3 ARCHITECTURE DETAILS

  • The eight-layer transformer has 8.4M trainable parameters, uses a learnable [CLS] token as the global jet representation, and accepts sequences up to 90 particles.It has hidden dimension 256, 16 attention heads, and four register tokens; a further 2.6M parameters reside in the positional-encoding lookup table.Source: §A.3 · p. 21
  • Particle features use fixed affine standardization whose centering and scale come from the QCD pretraining set’s median and interquartile range.The same values are used during training, inference, and downstream datasets.Source: §A.3 · p. 21
  • The positional encoding maps constituent positions onto a 100×100 polar grid, with logarithmic radial bins and uniform azimuthal bins.Its radial scale is the pretraining-set median radius and its maximum radius is the 99th percentile, clamping extreme-tail constituents.Source: §A.3 · pp. 21–22
  • At Polar 100×100, only 0.76% of masked particles share a bin, versus 3.03% of particles overall; Polar also ranks best in bin efficiency across grid sizes.The mask–mask collision rate is the operationally relevant measure for iBOT; Log-Cartesian reduces the 100×100 per-particle rate from 16.82% to 4.19%, with Polar refining it to 3.03%.Source: §A.3 · pp. 22–23
  • Polar 100×100 is the selected middle ground: 120×120 requires 44% more patches for smaller collision reductions, while 80×80 saves 36% of patches but raises the per-particle collision rate.At 100×100, the per-particle rate is 3.03%; it would be 2.16% at 120×120 or 4.52% at 80×80.Source: §A.3 · p. 23
  • The independent DINO and iBOT projection heads are three-layer MLPs with 2048-dimensional outputs, operating on [CLS] representations and masked-particle outputs, respectively.Both use hidden width 1024 and bottleneck dimension 256.Source: §A.3 · p. 23

A.4 DATASET DETAILS

RINO pretrains on QCD jets from JETCLASS and evaluates on downstream signal-versus-QCD tasks using disjoint samples. The evaluation datasets differ in kinematic range, constituent counts, or detector simulation, and the coarse views are genuine coarse-grainings for most pretraining jets.

A.4 DATASET DETAILS

  • JETCLASS supplies 10M QCD training jets for pretraining, while its signal classes are reserved for downstream evaluation.JETCLASS has 10 classes spanning QCD, Higgs, electroweak, and top topologies.Source: §A.4 · p. 23
  • Only 11% of QCD jets have 16 or fewer constituents, and just 0.13% have six or fewer, so views with N ∈ {2, 3, 4} are genuine coarse-grainings for nearly all jets.The median QCD jet contains 32 constituents.Source: §A.4 · p. 23, Figure 4 · p. 24
  • Downstream evaluation uses JETNET and TOP QUARK TAGGING, with OOD evaluation on JETCLASS; all downstream samples are disjoint from pretraining and their signal classes are unseen during pretraining.JETNET differs from JETCLASS in pT range, constituent-count distribution, and detector-level simulation.Source: §A.4 · p. 23

A.5 PHYSICAL SCALES OF THE kT VIEWS

The subjet count N imposes a hierarchy of kT views whose merge scales and softest-subjet fractions vary across the ladder, supporting N as a physically meaningful scale coordinate.

A.5 PHYSICAL SCALES OF THE kT VIEWS

  • The hierarchy is imposed by the number of subjets N; each N+1→N merge defines a characteristic kT scale q_N.q_N is the kT distance at the clustering step from N+1 to N subjets.Source: §A.5 · p. 24
  • q_N's median falls from 9.91 to 0.46 GeV, while the softest-subjet momentum fraction falls from 0.23 to 0.0033.The reported medians and IQRs are over JETCLASS QCD jets with more than 16 constituents.Source: §A.5 · p. 24, Table 8 · p. 24
  • The intended teacher {6, 8, 16}/student-only {2, 3, 4, uncl.} assignment scores 0.839 ± 0.005, versus 0.823 ± 0.006 with swapped roles and 0.730 ± 0.009 with mixed scales.Source: §A.5 · p. 24

A.6 FINETUNING PROTOCOL

The appendix details shared finetuning, baseline, and implementation protocols, and reports JetCLR-scale’s gain over JetCLR alongside compute costs. It also covers evaluations of RINO’s robustness, ablations, scale structure, and transfer behavior.

A.6 FINETUNING PROTOCOL

  • The locked recipe uses LP-FT for SSL models—five frozen-backbone head-training epochs followed by joint training—while the supervised baseline trains end-to-end from random initialization.The proposed experimental workflow is unlabeled-data pretraining, labeled-simulation finetuning, then calibration in data using control-region efficiencies and data-to-simulation scale factors.Source: §A.6 · pp. 24–25, §A.13 · pp. 30–31

A.7 ROBUSTNESS TO FINETUNING REGULARIZATION

  • RINO’s worst regularization setting reaches 0.843 OOD accuracy, above every other method’s best; its default beats the strongest target-selected baseline, MPMv2, by 0.037.Across settings, RINO’s accuracy spans 0.009, compared with 0.050 for MPMv2; the best setting is selected using JETCLASS labels and is therefore retrospective.Source: Table 10 · p. 25, §A.7 · p. 25

A.8 EXTENDED ABLATION RESULTS

  • Hierarchical views outperform mixed-scale views by 13%, and DINO+iBOT reaches 0.839 versus 0.677 for DINO-only and 0.736 for iBOT-only.Polar-binned positional encoding hurts mixed views by 5.9% but improves hierarchical views by 1.3%; iBOT masked prediction is applied only to the unclustered view.Source: Table 12 · p. 26, §A.8 · pp. 26–27

A.9 SCALE STRUCTURE

  • RINO’s backbone has role-dependent, non-monotonic scale geometry: N=2 is more similar to the unclustered view (0.54) and N=16 (0.53) than to teacher scales N=6 (0.36) and N=8 (0.37).Swapping teacher-student roles changes N=4’s nearest neighbor from N=3 (0.85) to N=6 (0.87), while the projection head yields cross-scale cosines of at least 0.88.Source: Figure 5 · p. 27, §A.9 · p. 28, Figure 6 · p. 28

A.10 CROSS-TOPOLOGY TRANSFER WITH MLP HEAD

  • On cross-topology transfer, RINO reaches H→4q AUC 0.896 ± 0.002, within one standard deviation of MPMv2’s 0.900 ± 0.005; its scale-factor deformation is 2.5%, less than half the next-lowest 5.4%.For the MLP-head evaluation, RINO and JetCLR-scale are statistically tied on H→4q, at 0.828 ± 0.006 and 0.824 ± 0.020, respectively.Source: §A.10 · pp. 28–30, Table 13 · p. 29, Table 15 · p. 30

A.14 FULL DATA-EFFICIENCY CURVES

  • Several baselines have non-monotonic OOD data-efficiency curves: MPMv2 peaks at 10% labels and declines at 50–100%, and its 50% accuracy changes from 0.789 to 0.761 ± 0.027 with batch size reduced to 256.This check indicates that the non-monotonicity is not simply a batch-size effect.Source: Figure 8 · p. 31, §A.14 · p. 31
  • RINO improves monotonically and leads at every JETNET label fraction, whereas supervised accuracy stays near 0.78 beyond 10% and JetCLR reaches 0.700 ± 0.031 at 100%.The paper attributes RINO’s transfer behavior to cross-scale agreement discouraging reliance on soft, low-scale details that are modeled approximately in simulations.Source: §A.14 · p. 32

A.15 MLP VS. LINEAR HEAD DIAGNOSTIC

  • RINO is the only method with significantly positive MLP-versus-linear OOD performance (t=+3.0, p≈0.01); its pretrained W1 (QCD) domain shift is 0.055 and falls to 0.043 after finetuning.The controlled baseline setup shares the encoder backbone, QCD-only pretraining data, and finetuning protocol to isolate pretraining objectives and view construction as far as possible.Source: Table 17 · p. 32, §A.15 · p. 32, §A.16 · p. 32

A.17 BASELINE IMPLEMENTATION DETAILS

  • 0.700 vs. 0.780 in the JetCLR-to-JetCLR-scale comparison supports the contribution of kT clustering views.JetCLR-scale uses kT clustering views with an NT-Xent objective, pairing scales from different tiers; a fine-scale MAE reconstruction auxiliary stabilizes low-data training.Source: §A.17 · p. 34

A.18 COMPUTE

  • RINO pretraining takes about 80 hours per run, while finetuning typically takes about 50 minutes per JETNET seed or 6 hours per TOP QUARK TAGGING seed.Pretraining uses the QCD-only JETCLASS subset; finetuning is early-stopped, typically after about 25 epochs. The per-method pretraining comparison uses shared JTE encoders (matched JTD for OmniJet-α), with MPMv1 and OmniJet-α also requiring a shared VQ-VAE tokenizer trained once for about 8 hours.Source: §A.18 · p. 34, Table 20 · p. 34

Related papers

Loading 2610.15919v1…