Scaling Laws, Tabular Data and Actuarial Ratemaking Models

1 Introduction

• Scaling laws are framed as irreducible loss plus power-law improvements in parameters, data, and compute, extending questions previously studied for large language models (Kaplan et al., 2020; Hoffmann et al., 2022).The paper asks whether analogous regularities hold despite actuarial data being smaller, noisy, and heterogeneous.Source: §1 · p. 2
• The paper systematically evaluates how GLMs, regularized variants, feed-forward networks, and Transformers scale with training data using out-of-sample Poisson deviance on motor-insurance claim frequency.Data size is measured by policy-period rows, with exposure-years as the actuarial analogue of example count; progressive representative subsets address the finite dataset.Source: §1 · p. 3
• The study seeks model-selection guidance across data regimes, saturation points, power-law exponents, and problem characteristics, while diagnosing Transformer scaling failures and developing new ratemaking models.Its stated contributions include the first systematic empirical investigation of actuarial scaling laws, explicit functional forms, and practical guidance for actuaries.Source: §1 · pp. 2–3

1.1 Credibility as an actuarial scaling law

• Credibility theory provides an actuarial precedent: beyond large exposure, residual uncertainty decays as n^-1, while fitted scaling exponents summarize model-specific rates of approach to a performance floor.The paper defines reducible Poisson deviance relative to an empirical asymptote and interprets larger α as faster improvement with data.Source: §1.1 · p. 4
• The proposed credibility-style scaling curve supports minimum credible data thresholds and connects modern tabular-model exponents to established actuarial scaling relationships.It relates tolerance-based data requirements to classical credibility and notes familiar -1/2 rates for Poisson and GLM estimation errors and Monte Carlo error.Source: §1.1 · p. 5

1.2 Organization of the Manuscript

• Straightforward supervised tabular Transformers show weak and sometimes non-monotone parameter returns, motivating architectural modernization and swap-style self-supervised learning.The manuscript progresses through pooling, stabilization, value-side capacity mechanisms, mixture-of-experts routing, and additional self-supervised signal.Source: §1.2 · p. 5
• The manuscript proceeds from rationale and literature through data, methodology, model families, empirical scaling results, and conclusions, with appendices and supplementary material supporting notation, formulas, allocation theory, and reproducibility.The supplementary workflow uses public French MTPL frequency data to make the empirical pipeline easier to inspect.Source: §1.2 · p. 5

2.1 Why scaling laws matter for actuarial pricing

• Scaling-law analysis asks how quickly held-out loss improves with portfolio growth and whether additional model capacity improves ratemaking predictions.The motivation draws on predictable diminishing returns in held-out loss and actuarial concerns about credibility, stability, data volume, and model capacity.Source: §2.1 · p. 6

2.2 The frequency task in actuarial terms

• Frequency modeling predicts claim rates from rating factors and converts them into expected claim counts using earned exposure as a Poisson offset.The target is claim frequency rather than severity or pure premium because frequency is observed for every policy-period record, whereas severity is observed only for claims.Source: §2.2 · p. 6

2.3 Why Poisson deviance is the loss

• Poisson deviance is a likelihood-based held-out loss in which lower values indicate better fit to observed claim counts.A mean deviance improvement of ∆ corresponds to an average held-out log-likelihood lift of ∆/2 nats per policy-period row, but does not by itself establish premium adequacy or profitability.Source: §2.3 · p. 7

2.4 The three resources: data, parameters, and compute

• The scaling resources are labelled data N, trainable parameters P, and training compute C, with exponents α, β, and γ measuring improvement and diminishing returns.The fitted floor L∞ is an empirical floor over the experimental range, not a universal Bayes-risk claim; doubling N multiplies remaining reducible loss by 2^-α.Source: §2.4 · p. 7

2.5 A short example and how to read the scaling claims

• Doubling training rows multiplies the reducible loss by 2^-0.409 ≈0.75, removing about 25% of the remaining gap to the fitted floor within the studied range and TabM/TabM-mini family.The preview uses policy-period rows and the best-achieved TabM/TabM-mini data-scaling envelope; its conclusions are conditional on the dataset, target, features, tuning protocol, and IID split.Source: §2.5 · pp. 7–8, Figure 1 · p. 8

2.6 How to read the model families

• The model-family guide contrasts an additive GLM baseline with FFNs, Transformers, TabM/TabM-mini ensemble-style models, and a Transformer+TabM-mini hybrid.These families differ in how much rating structure is specified by the modeller versus learned from data, and added complexity may help more in larger-data regimes than in small-data regimes.Source: §2.6 · pp. 9–10

3.1.1 Empirical Regularities and Power-Law Improvement

• Scaling-law research finds approximate power-law improvements in loss with model parameters, data, and compute, making scale a predictive design variable (Kaplan et al., 2020; Hestness et al., 2017).Kaplan et al. (2020) report L ∝P −0.076, L ∝D−0.095, and L ∝C−0.050 for cross-entropy test loss.Source: §3.1.1 · p. 10

3.1.2 Compute-Optimal Training and Data–Parameter Balance

• Compute-optimal studies argue that performance depends on balancing model parameters and training data, rather than increasing model size alone (Hoffmann et al., 2022).For a fixed compute budget, resource allocation follows an efficient frontier between model size and data volume, affecting training recipes and deployment economics.Source: §3.1.2 · p. 11

3.1.3 Estimating and Interpreting Scaling Laws

• Scaling-law estimates should account for intermediate checkpoints, model-size similarity, and random-seed variability; this study therefore trains every model five times and averages predictions (Choshen et al., 2024).Intermediate-checkpoint fits improve accuracy over fully trained models alone, while multiple small models can sometimes outperform one large model because of seed variance.Source: §3.1.3 · p. 11

3.1.4 Beyond Simple Power Laws: Regime Changes and Emergent Capabilities

• Scaling relationships may be non-monotone, piecewise, or metric-dependent, so extrapolation must consider finite-sample regimes, evaluation design, and apparent emergent capabilities (Caballero et al., 2022; Schaeffer et al., 2023).These qualifications are especially relevant for noisy actuarial tasks, where aggregate regularities may conceal regime changes or metric artifacts.Source: §3.1.4 · pp. 11–12

3.2.1 Tabular Learning and the Persistence of Tree-Based Methods

• Actuarial tabular learning remains anchored by GLMs and strong tree-based baselines because insurance data often contains skewness, heavy tails, sparse categories, and low signal-to-noise ratios (Richman, 2021a, 2021b; Grinsztajn et al., 2022; McElfresh et al., 2023).Entity embeddings offer a neural strategy for sharing information across categorical variables (Guo & Berkhahn, 2016; Richman, 2021a, 2021b; Kuo, 2019).Source: §3.2.1 · pp. 12–13

3.2.2 Tabular Transformers: Adapting Attention to Structured Features

• Tabular Transformers such as TabTransformer, FT-Transformer, and SAINT can compete with GBDTs, but no deep architecture is universally dominant and weak priors may require additional inductive bias or auxiliary training signal (Huang et al., 2020; Gorishniy et al., 2021; Somepalli et al., 2021).Their challenges include sparse categorical structure, weak spatial or sequential priors, and low signal-to-noise ratios.Source: §3.2.2 · p. 13

3.2.3 Tabular Foundation Models and In-Context Learning

• Tabular foundation models use synthetic-task pretraining or in-context learning to amortize inference, making them especially relevant to small-data portfolios and fragmented actuarial datasets (M¨uller et al., 2021; Hollmann et al., 2022; Hollmann et al., 2025).The cited literature primarily emphasizes classification and synthetic-task pretraining rather than actuarial regression.Source: §3.2.3 · p. 13

3.2.4 When Tabular ML Meets Millions of Rows

• This study addresses a literature gap by examining scaling in a roughly 4.5 × 10^6-row ratemaking dataset across six training fractions, asking whether losses, model rankings, and capacity requirements change in the multi-million-row regime.The study fits data-scaling power laws in held-out deviance, with α measuring improvement slope and AN measuring finite-sample distance from the asymptote.Source: §3.2.4 · p. 14, Table 1 · p. 14

4 Data

• The portfolio contains approximately 4.5 million policy-period records, about 60–70 raw rating-factor columns, and roughly 1 million exposure-years with 240,000 claims.Claim frequency is 0.240 claims per exposure-year, while approximately 5% of policy-periods have a claim, highlighting substantial sparsity.Source: §4 · pp. 16–17
• Claim counts y_i are modeled with a Poisson likelihood using earned exposure E_i as an offset, with predicted rate r_i giving mean λ_i = E_i r_i.Rating factors span policyholder, driver, vehicle, usage, geography, insurance, and claims-history dimensions.Source: §4 · p. 16
• The data show actuarially plausible heterogeneity: claim frequency declines with driver age after ages 25–34, newer vehicles have higher frequency, and territorial effects are more moderate.Figure 3 reports anonymized one-way claim-frequency summaries across selected rating factors.Source: §4 · p. 17, Figure 3 · p. 18

4.1 Data Preparation

• Preprocessing removes non-positive-exposure records, defines a 90/10 IID split, and creates nested training subsets at fractions from 0.05 to 1.00 with a fixed test set.Split and sampling columns are excluded from covariates, and unknown codes remain distinct categories.Source: §4.1 · p. 17
• Continuous predictors are quantile-binned on training data and, like categorical predictors, represented with learned entity embeddings, while categories above 100 levels are excluded.This standardizes tokenization and limits sparse embedding tables that could confound capacity comparisons.Source: §4 · p. 16, §4.1 · p. 17
• Because record-level timestamps are insufficiently granular, evaluation uses the provided IID split rather than temporal validation, which is treated as a limitation.The limitation is discussed in Section 8.2.Source: §4.1 · p. 19

5.1 Data Processing and Sampling Strategy

• Nested training subsets at fractions 0.05, 0.10, 0.25, 0.50, 0.75, and 1.00 use one fixed held-out set, isolating added-data effects from resampling noise.Data size is operationalized as the number of policy-period rows N in each subset, with exposure-years expected to scale linearly with N.Source: §5.1 · p. 19

5.2 Training Methodology

• Models are trained with Poisson deviance and fixed expert-selected hyperparameter recipes reused across fractions and seeds, while AdamW, regularization, and early stopping support robust fitting.Five independent runs per configuration quantify initialization variance; tuning compute is excluded from the training-compute proxy.Source: §5.2 · pp. 20–21

5.3 Evaluation Framework

• Held-out mean Poisson deviance is the primary metric, with lower values indicating better fit and decreases of Δ corresponding to Δ/2 nats of average held-out log-likelihood lift per policy.The study evaluates frequency objectives across data amount, model configuration, and random seed; severity and pure-premium scaling are left for future work.Source: §5.3 · p. 21

5.4 Ensemble Methodology

• Five-seed predictions are averaged at inference using the seed-averaged ensemble procedure, reducing initialization variance while distinguishing this averaging from TabM’s jointly trained within-model submodels.Results report ensemble and mean single-seed scores to separate averaging effects from architectural improvements.Source: §5.4 · p. 22

5.5 Scaling Law Analysis

• Compute scaling fits an irreducible-loss-plus-power-law frontier using the Pareto set and Cens = 5C1, treating per-epoch FLOPs as the controlled compute proxy despite early stopping.Replacing Cens with C1 changes the fitted intercept but not the compute-scaling exponent γ.Source: §5.5 · p. 22
• Data scaling uses best-of-size envelopes over nested training fractions, while full-data parameter scaling uses Pareto runs and decomposes parameters into embedding and non-embedding components.All exponents are estimated by linear regression in log space after subtracting the shared fitted floor L∞, with (α, β, γ) reported in Section 7.5.Source: §5.5 · p. 23

6.1 Shared Notation and Exposure Scaling

• All models tokenize categorical and quantile-binned continuous covariates into learned embeddings and train an exposure-scaled Poisson objective that predicts a log-rate.At inference, the predicted log-rate is multiplied by earned exposure.Source: §6 · p. 23, §6.1 · p. 24, §6.2 · p. 26

6.3 GLM and FFN Baselines

• The GLM provides an interpretable additive reference, whereas FFN learns nonlinearities and cross-feature interactions through dense mixing of concatenated embeddings.TabM-mini and TabM add parameter-efficient implicit ensembles using shared MLP kernels and rank-1 adapters, with TabM allowing members to diverge across layers.Source: §6 · pp. 23–24

6.5 Transformer Enhancements

• The enhanced Transformer family combines feature weighting, normalization, stochastic depth, LayerScale, attention temperature and bias, value gating, and head-fix mechanisms to improve stability and capacity.These modifications are motivated by limited effective sample sizes and brittle optimization in heterogeneous actuarial data.Source: §6 · p. 24, §6.5 · p. 31, §6.5.2 · p. 32
• TokenMoE, TokenMoE+SSL, and Transformer+TabM-mini extend capacity through expert routing, swap-style self-supervision, and parameter-efficient ensembling, respectively.TokenMoE uses experts as a scaling dimension, self-supervision supplies an auxiliary representation-learning signal, and the hybrid combines attention-based interactions with a TabM-mini head.Source: §6 · p. 24, §6.5 · p. 32

6.4 Transformer Architecture for Tabular Data

• The tabular Transformer retains feature embeddings as a token sequence, using attention to model covariate interactions rather than requiring manual interaction construction.Learned positional embeddings distinguish column identities, while multi-head attention represents different interaction patterns in parallel.Source: §6.4 · p. 27, §6.4.1 · p. 28, §6.4.2 · p. 28

6.4.6 Multi-CLS Pooling and Readout

• Multi-CLS Transformers address the single-CLS pooling bottleneck by learning multiple attention-based summaries, with concatenated or ensemble readouts and ncls as an explicit capacity knob.This design increases representation capacity linearly in ncls without changing the number of feature tokens.Source: §6.4.6 · p. 30, §6.4.8 · p. 31

6.5.3 Layer-wise Value Mixing

• Layer-wise value mixing expands value-side content while keeping query/key dimensions fixed, using learned layer-specific embeddings and cached value memory.Enhanced variants split covariate embeddings into base and layer-wise value streams; learned softmax mixtures combine current, cached, and projected values, with v embeddings set to zero for CLS tokens.Source: §6.5.3 · pp. 32–33

6.5.4 Stability Enhancements

• Enhanced Transformer variants combine normalization, attention, gating, residual-scaling, sink-token, and stochastic-depth mechanisms to support deeper and wider models.The listed stabilizers include RMS-normalized Q/K, learned per-head temperatures and biases, per-head value gates, optional sink tokens, LayerScale, an FFN residual gate, and drop-path.Source: §6.5.4 · p. 33

6.6 Token-wise Mixture-of-Experts (TokenMoE)

• TokenMoE routes both base and value embedding streams through lightweight expert increments, increasing conditional capacity without uniformly increasing computation.A load-balancing penalty discourages expert collapse, while explicit residual coefficients control MoE contribution strength; the resulting v-MoE design leaves attention patterns relatively constrained.Source: §6.6 · pp. 33–34

6.6.1 Swap-style Self-supervision

• Swap-style self-supervision detects minibatch donor-row swaps, adding a weighted binary cross-entropy objective that trains representations to capture cross-feature consistency.Swapped embeddings preserve approximate column marginals while disrupting within-row coherence; corruption is disabled at inference, where predictions use only the supervised Poisson head.Source: §6.6.1 · p. 34

6.7 TabM and TabM-Mini Ensembles

• TabM packs K implicit MLP submodels into shared kernels with lightweight rank-1 adapters, enabling prediction averaging with substantially fewer added parameters than duplicated networks.TabM-mini retains only the first multiplicative adapter, whereas TabM restores per-layer adapters with better initialization; Transformer-plus-TabM-mini hybrids ensemble pooled Transformer representations.Source: §6.7 · pp. 34–35

7.2 Results roadmap: from baselines to scaling fixes

• TabM and TabM-mini achieve the strongest full-data performance overall, while TokenMoE+SSL is the best Transformer-family configuration and Transformer+TabM leads at 5%–25% of training data.At 50%, the ensemble gap between TabM and Transformer+TabM is only 0.00004 and is not stable, whereas TabM has stronger seed-level support at 75% and 100%.Source: §7.2 · p. 36, §7.3.1 · p. 36, §7.3.2 · p. 38

7.6 Compute-optimality and the “big model” intuition

• Larger models help conditionally: Transformer+TabM improves with size even at 5%–10% data, whereas FFNs and TabM/TabM-mini can worsen with size at 5%, and TabM favors larger sizes as N grows.In the large-N regime, TabM’s stronger parameter scaling supports the shift toward larger compute-optimal configurations; in data-scarce settings, size is efficient mainly when paired with sample-efficiency or variance-reduction mechanisms.Source: §7.2 · p. 36, §7.6 · p. 44

7.6.1 Compute scaling (Kaplan-style)

• The compute-efficient frontier follows L∞≈0.28749 and γ ≈0.0575, transitioning from GLMs at low compute to FFNs and then TabM/TabM-mini at the highest compute levels.The fit has R2 ≈0.94 on (L −L∞), but the small γ indicates slow compute returns.Source: §7.6.1 · p. 44, §7.6 · p. 44

7.6.2 Data scaling by family

• TabM improves faster with data than supervised Transformers, with supervised Transformer α ≈0.18–0.19 versus α ≈0.409 when the largest full-data TabM-mini extension is included.The higher TabM exponent has lower fit quality, about R2 ≈0.80, consistent with capacity crossovers and piecewise rather than universal scaling.Source: §7.6.2 · p. 45, Table 8 · p. 45

7.6.3 Parameter scaling at fixed (full) data

• TabM has markedly stronger parameter scaling than vanilla Transformers and FFNs, with β ≈0.148 versus β ≈0.025 and β ≈0.028, respectively.TokenMoE without SSL is nearly flat at β ≈0.008, whereas TokenMoE+SSL reaches βemb ≈0.053 and βnon-emb ≈0.058; TabM is backbone-dominated, with only about 0.8%–8.3% of parameters in embeddings.Source: §7.6.3 · pp. 45–46, §7.6.4 · p. 46

7.6.4 TabM exhibits a different scaling law than independent MLP ensembles

• TabM’s advantage is not explained by ordinary ensemble averaging alone: its main-sweep α ≈0.309 and β ≈0.148 exceed the corresponding FFN values, while seed-averaging gains are smaller for TabM-style models.Average seed-averaging gain is 0.000360 for TabM-style models versus 0.001003 for vanilla Transformers; the proposed mechanisms include joint training, shared weights with lightweight adapters, and improved width utilization.Source: §7.6.4 · pp. 46–47

7.6.5 Chinchilla-style compute-optimal implication

• The TabM/TabM-mini bivariate fit gives α ≈0.430 and β ≈0.170, implying β/α ≈0.396, P⋆∝C0.716, and N⋆∝C0.284 under C ∝NP.Because training uses early stopping, this is a first-order per-epoch compute heuristic rather than an exact total-training-FLOPs optimum.Source: §7.6.5 · pp. 47–48, Figure 8 · p. 49

7.6.6 A pooled multi-family scaling fit (global exponents, family terms)

• The pooled multi-family fit estimates α ≈0.425 and β ≈0.087 with R2 ≈0.916, implying a parameter-heavy split P⋆∝C0.830 and N⋆∝C0.170.This descriptive fit uses heterogeneous architectures and does not replace family-wise exponents or capture small-data traps and objective-induced shifts such as SSL.Source: §7.6.6 · pp. 48, 50

7.8 Why transformers improve when we add TabM ideas or SSL

• Purely supervised Transformers show meaningful data scaling but weak parameter scaling, with α ≈0.18–0.20 and β ≈0.025–0.037.TokenMoE without SSL has β ≈0.008, indicating that added routing capacity alone produces essentially flat parameter scaling.Source: §7.8 · p. 50

7.7 Regime diagnostics: when do compute and capacity matter?

• Compute scaling is nearly flat at 5% data (γ5% ≈ 0.0066) but steepens at full data (γ100% ≈ 0.0580), as capacity scaling changes across regimes.TabM and FFNs shift from negative to positive parameter exponents with more data, whereas supervised Transformers remain near zero.Source: §7.7 · p. 50, Figure 10 · p. 52

7.8.1 TabM-style adaptation inside the transformer

• TabM-style adaptive transformations yield the best models in the small-N regime by injecting structured feature adaptability that improves sample efficiency.The adaptation operates over token embeddings under input perturbations.Source: §7.8.1 · p. 53

7.8.2 Self-supervised learning (SSL) via feature corruption prediction

• TokenMoE+SSL reaches full-data Transformer-family performance of 0.29027 and improves more consistently with model size than supervised Transformers.The row-swap corruption objective supplies a column-level signal about what changed, raising β from ≈0.008 without SSL to ≈0.055 with SSL.Source: §7.8 · p. 50, §7.8.2 · p. 53

7.9 Variance and the benefit of seed-averaged ensembles

• TabM achieves the best test deviance, while supervised Transformers gain more from seed averaging and show smaller train–test gaps than TabM.At full data, TabM’s train–test gap is 0.00984 versus 0.00293 for vanilla Transformers, while vanilla Transformer ensemble gain averages 0.001003 versus 0.000018 for GLMs.Source: §7.9 · p. 53, Figure 11 · p. 54, Table 11 · p. 55

8.1 Discussion

• TabM-style models scale most strongly with data, while the best architecture changes by regime and TabM-mini achieves the lowest full-data deviance.TabM models have α ≈0.31–0.41; Transformer+TabM leads at 5%–25%, while TabM becomes better supported at 75%–100% after accounting for seed uncertainty.Source: §8.1 · p. 55
• Seed averaging consistently improves inference, especially for attention models, while the compute-optimal strategy shifts from regularized baselines toward higher-capacity tabular deep models as data grow.The average ensemble gain is 0.001003 for supervised Transformers, 0.000018 for the GLM, and 0.000360 for TabM; TabM’s compute-optimal model size grows faster than data volume under C ∝NP.Source: §8.1 · p. 56
• Deviance improvements translate directly into held-out log-likelihood lift but should not be interpreted as direct business-value multipliers without pricing, calibration, severity, and governance context.At full data, TabM (xlarge) improves over the GLM by 0.00216 deviance, while the best TabM-mini extension improves by 0.00315; Figure 12 visualizes the corresponding per-policy lift.Source: §8.1 · pp. 56–57, Figure 12 · p. 57

8.2 Limitations

• The empirical conclusions are conditional on one motor dataset, an IID split, studied model families, limited tuning, and scaling fits within the observed data and capacity range.The paper calls for temporal evaluation, broader baselines including GBDTs, matched parameter-allocation ablations, paired per-policy bootstrap uncertainty, and experiments beyond approximately 10^7 rows.Source: §8.2 · pp. 57–58

8.3 What did we learn?

• For practitioners, GLMs remain stable and difficult to beat with limited data, whereas TabM benefits more from additional rows and parameters in the multi-million-row regime.Additional parameters alone do not reliably improve supervised Transformers; stronger results require tabular inductive bias or auxiliary self-supervised signal.Source: §8.3 · p. 58
• Small Poisson-deviance differences represent systematic likelihood gains and can correspond to materially different actuarial interaction surfaces, but not directly to loss ratio, profit, or premium adequacy.The FMTPL supplement illustrates changed driver-age and age-by-bonusmalus patterns, while business interpretation still requires severity modeling, calibration, expenses, margins, retention, and governance.Source: §8.1 · p. 57, §8.3 · p. 59

8.4 Conclusions

• The study establishes scaling laws in actuarial tabular data and shows that Transformer scaling depends on depth, width, inductive bias, and training signal rather than size alone.It recommends higher-capacity models and small ensembles beyond approximately 10^5 rows, while warning that compute increases without more data or better inductive bias face diminishing returns.Source: §8.4 · p. 59

A Notation Reference

• The notation appendix centralizes symbols used throughout the paper, while explanations of frequency modeling, Poisson deviance, and scaling-law quantities remain in Section 2.Dense component-level model formulae are retained in Appendix B.2.Source: A Notation Reference · p. 64

A.1 Quick reference: symbols and dimensions

• The appendix defines core notation for the data, loss, and scaling-law quantities.Source: Table 12 · p. 64
• It separately documents notation used in neural architecture descriptions.Source: Table 13 · p. 65

B Technical Appendix: Model Configurations

• The technical appendix records component-level mathematical definitions used in the implementations.Source: B Technical Appendix: Model… · p. 66
• It specifies the main architectural hyperparameters for all model variants.Source: B Technical Appendix: Model… · p. 66
• Unless noted, positional-embedding dimension equals embedding dimension across the model configurations.Source: B Technical Appendix: Model… · p. 66
• Baseline and multi-CLS variants concatenate positional embeddings, whereas head-fix and TokenMoE variants add them, optionally with a projection.Source: B Technical Appendix: Model… · p. 66
• All models use the exposure-scaled Poisson objective.Source: B Technical Appendix: Model… · p. 66

B.1 Experiment Model Identifiers · B.2 Component Formulae

• The appendix groups the experiment models by family and establishes shared notation for feature tokens, CLS tokens, attention heads, token dimensions, and row- or batch-wise operations.These identifiers and conventions organize the subsequent component formulae.Source: §B.1 · p. 66, Table 14 · p. 66, §B.2 · p. 66

B.2.1 Exposure-scaled Poisson mean and loss · B.2.2 GLM baseline

• Exposure-scaled Poisson prediction uses earned exposure Ei, predicted claim rate ˆri, predicted log-rate ηi, and a numerically stabilized per-observation Poisson negative log-likelihood.The GLM baseline applies a linear predictor to encoded features and maps it to ˆλi = Ei exp(ηi).Source: §B.2.1 · p. 66, §B.2.2 · p. 67

B.2.3 Feed-forward network (FFN) · B.2.4 Quantile binning and entity embeddings

• Feed-forward baselines flatten learned policy embeddings and apply layered nonlinear transformations, while quantile binning and entity embeddings convert continuous and categorical covariates into token embeddings.The FFN uses ReLU and may apply dropout between layers.Source: §B.2.3 · p. 67, §B.2.4 · p. 67

B.2.5 Extended embeddings for v streams and MoE banks

• Enhanced Transformer variants extend embeddings into layer-wise value streams and TokenMoE expert banks, with CLS-token value embeddings zeroed to anchor the value pathway in observed covariates.TokenMoE stores expert slices alongside base and value components for later routing.Source: §B.2.5 · pp. 67–68

B.2.6 Positional embeddings (concatenation vs addition) · B.2.7 CLS tokens and multi-CLS pooling · B.2.8 Column weighting

• Transformers incorporate positional embeddings through concatenation or addition, append learned CLS tokens, and produce log-rates through concatenated or per-CLS ensemble readouts.The two readouts coincide when ncls = 1; column weighting separately learns clipped per-token weights.Source: §B.2.6 · p. 68, §B.2.7 · p. 68

B.2.9 Self-attention with learned bias and temperature · B.2.10 Gated FFN (SwiGLU)

• Self-attention uses learned additive biases and per-head temperatures, while gated FFNs return projected outputs to dimension d within residual Transformer blocks.The attention blocks apply layer normalization and dropout around residual branches, and the gated FFN uses a SwiGLU-style projection.Source: §B.2.9 · p. 69, §B.2.10 · p. 69

B.2.11 Head-fix value mixing, sink tokens, and stabilizers

• Head-fix variants mix current, cached, and embedded values, and add RMS-normalized queries and keys, optional sink tokens, value gates, LayerScale, FFN gates, drop-path, and near-zero output initialization.These mechanisms make residual branches start close to the identity map to improve early optimization stability in deep stacks.Source: §B.2.11 · pp. 69–70

B.2.12 TokenMoE routing and load balancing · B.2.13 Swap-style self-supervision · B.2.14 TabM-mini and TabM adapters

• TokenMoE performs token-wise top-1 expert routing with straight-through gates and load balancing, while swap-style self-supervision adds swap-token binary cross-entropy; TabM uses multiplicative or rank-1 BatchEnsemble adapters whose exposure-scaled member rates are averaged.SSL optimizes the Poisson loss with the swap loss and, when enabled, the TokenMoE load-balancing term.Source: §B.2.12 · pp. 70–71, §B.2.13 · p. 71

• The mathematical definitions of the matrix components are provided in Appendix B.2.Source: §B.3 · p. 72
• Table 15 compactly compares the major Transformer-derived experiment families.The matrix uses checkmarks to indicate components central to each variant.Source: Table 15 · p. 72, §B.3 · p. 72

• Configuration tables define the GLM and FFN model settings used in the experiments.Source: Table 16 · p. 72
• TabM and TabM-Mini configurations are specified, including TabM-Mini xlarge variants with expanded embeddings.Source: Table 17 · p. 72, Table 18 · p. 73
• Vanilla, Multi-CLS, Transformer-plus-TabM-Mini, and enhanced Transformer configurations are provided across the corresponding model families.Enhanced variants share hyperparameters with the corresponding Transformer Multi-CLS sizes but use the head-fix attention block; one large variant widens the FFN and disables dropout and weight decay.Source: §B.4 · p. 72, Table 19 · p. 73, Table 20 · p. 73
• TokenMoE baseline and TokenMoE+SSL configurations are documented, with all TokenMoE runs using moe hard select = False and moe router hidden = 0.TokenMoE+SSL uses swap-style self-supervision and was named transformer tokenmoe * selfsup in experiment logs.Source: Table 22 · p. 74, Table 23 · p. 74, §B.4 · p. 74
• The paper uses transformer tokenmoe ssl xlarge as the naming convention for the internal alias transformer tokenmoessl xlarge.Source: §B.4 · p. 74

• The baseline size-by-fraction results cover GLM, FFN, and vanilla Transformer ladders using seed-averaged ensemble test Poisson deviance.Source: Table 24 · p. 74
• Progressive Transformer fixes are evaluated across model sizes and training fractions, with the TokenMoE+SSL xlarge variant reported only on the full-data split.Source: Table 25 · p. 75
• TabM and TabM-mini receive their own size-by-fraction comparison, including two full-data-only extensions.Source: Table 26 · p. 75

C Chinchilla-Style Scaling: Technical Explainer

• The appendix derives the Chinchilla-style allocation statement used in Section 7.6.Source: C Chinchilla-Style Scaling: Technical… · p. 76

C.1 Setup and notation

• The setup denotes labelled training rows by N, trainable parameters by P, and their scaling exponents by α and β.Source: §C.1 · p. 76
• The additive law separates reducible loss into finite-data and finite-parameter contributions.Source: §C.1 · p. 76
• At fixed P, doubling N multiplies the data term by 2−α, while at fixed N, doubling P multiplies the parameter term by 2−β.Source: §C.1 · p. 76
• These isolated doubling comparisons do not by themselves determine compute-optimal scaling.Source: §C.1 · p. 76
• Under fixed compute C, increasing P forces N down and increasing N forces P down, making the problem a constrained allocation rather than a direct comparison of α and β.Source: §C.1 · p. 76

C.2 Derivation

• The derivation minimizes the compute proxy while ignoring constants.Source: §C.2 · p. 76
• First-order conditions are used to characterize the allocation.Source: §C.2 · p. 76
• Multiplying the first equation by N and the second by P yields a marginal-balance condition, which gives the allocation rule reported in the main text.Source: §C.2 · p. 76

• The apparent paradox arises because the resource with the larger isolated exponent is easier to improve, so optimization shifts growth toward the slower-shrinking bottleneck.The optimizer balances marginal returns rather than assigning the larger share to the resource with the larger isolated exponent.Source: §C.3 · p. 77
• β/α < 1 implies parameter-heavy co-scaling: along the fitted frontier, P grows faster than N, while TabM/TabM-mini show more balanced joint scaling than weaker Transformer ratios.Ratios near zero indicate weak parameter scaling; ratios closer to one indicate that both additional labelled rows and parameters measurably reduce held-out deviance.Source: §C.3 · p. 77
• For TabM/TabM-mini, α ≈0.409 and β ≈0.148 give β/α ≈0.36, implying parameter growth remains faster while data growth remains meaningful.Using the main-sweep α ≈0.309 instead gives β/α ≈0.48 and approximately 1.39× N⋆ growth per doubling of P⋆, with the same qualitative allocation rule.Source: §C.3 · p. 77
• Both resources increase with compute, but this conclusion is only a per-epoch heuristic because early stopping makes realized compute proportional to NP E_stop(P, N).An exact total-compute optimum therefore requires modelling stopping time within the scaling law.Source: §C.3 · p. 78