PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws, Tabular Data and Actuarial Ratemaking Models

Ronald Richman

arXiv:2609.03106v1cs.LGq-fin.RM

TL;DR

The paper asks whether neural scaling laws extend to actuarial tabular ratemaking, where GLMs remain strong baselines. It evaluates model families across training-data fractions and random seeds using Poisson deviance, finding architecture-dependent scaling and limited gains from simply enlarging supervised Transformers.

  • Problem

    The paper investigates whether scaling regularities observed in modern deep learning also arise for heterogeneous, noisy actuarial tabular data.

  • Method

    Using an anonymized motor insurance dataset, the study compares GLMs, FFNs, Transformers, and TabM variants across nested training fractions and multiple random seeds with out-of-sample Poisson deviance.

  • Results

    TabM-style models show the strongest data scaling, with α ≈0.31–0.41 and the lowest full-data deviance, while supervised Transformers show weak parameter scaling.

  • Takeaways & Limitations

    Model selection for actuarial tabular tasks should account for data regime and architecture rather than relying on simple increases in Transformer size.

  • Takeaways & Limitations

    The scaling claims are conditional on one motor insurance dataset, a fixed feature set, and the study’s IID/random train/test split, with no comparable full GBDT baseline sweep.

Abstract

from arXiv · show

Scaling laws in modern deep learning describe how held-out loss improves as model capacity, training data, and compute increase, often following power-law trends. We investigate whether analogous scaling regularities arise in actuarial ratemaking, where data are tabular, heterogeneous, and noisy, and where classical models such as GLMs remain strong baselines. Using a real-world motor insurance portfolio, we train models from different families across increasing fractions of the training data and multiple random seeds, evaluating out-of-sample Poisson deviance, a likelihood-based loss for Poisson count predictions in which lower values indicate better held-out fit. We find that all model families improve with additional data, but scaling exponents differ substantially: TabM exhibits markedly stronger data scaling than purely supervised tabular Transformers and standard MLP baselines. Transformer variants show weak parameter scaling unless augmented with additional inductive biases (TabM-style adaptation or self-supervision). These results provide quantitative guidance on model selection by data regime and suggest that effective scaling on actuarial tabular tasks depends on architecture and loss function objective design, with simple increases in Transformer size providing limited gains.

1 Introduction

The paper asks whether scaling-law regularities extend to actuarial ratemaking and connects empirical loss scaling with classical credibility ideas. It evaluates how data and model choices affect out-of-sample Poisson deviance across actuarial model families.

  • 1 Introduction: The paper quantifies how ratemaking model performance changes with training data and modelling choices using a scaling-law framework.Scaling laws are presented as a way to predict how held-out loss evolves with key resources.
  • 1 Introduction: Poisson deviance measures out-of-sample performance because the response is a claim count and predictions are exposure-adjusted Poisson means.Lower mean deviance corresponds to higher held-out Poisson likelihood.
  • 1 Introduction: The study notes that gradient boosting machines are an intermediate model family left for future investigation.The reported comparisons therefore do not cover every major tabular modelling family.
  • 1.1 Credibility as an actuarial scaling law: Classical credibility provides an actuarial analogue: when exposure greatly exceeds the credibility constant, residual uncertainty decays approximately as n^-1.Modern fitted exponents instead summarize empirical rates of improvement that can vary by model, task, metric, and data regime.
  • 1.1 Credibility as an actuarial scaling law: The scaling analysis reframes credibility-style improvement through a fitted distance-to-asymptote curve and a model-specific exponent.The exponent functions as an empirical summary of how quickly reducible loss falls as data increase.
  • 1 Introduction: Straightforward supervised tabular Transformers show weak and sometimes non-monotone parameter returns, motivating investigation of architectural and training modifications.This behavior contrasts with the monotonic parameter scaling often reported for large language model Transformers.

2 Rationale, Metric, and Reading Guide

This section explains the paper’s data, Poisson-frequency setup, scaling-law notation, and model-family distinctions. It frames fitted exponents as empirical measures of diminishing returns across data, parameters, and compute.

  • 2.1 Frequency modelling: The experiment uses policy-period rows, with observed claim counts and earned exposure converted into expected claim counts through predicted frequency rates.The target is claim frequency rather than severity or pure premium.
  • 2.3 Why Poisson deviance is the loss: Poisson deviance is a likelihood-based loss in which lower held-out values indicate better fit to observed claim counts.A mean deviance improvement of Δ corresponds to an average held-out log-likelihood lift of Δ/2 nats per policy-period row.
  • 2.4 The three resources: data, parameters, and compute: The paper studies three scaling resources: labelled training data N, trainable parameters P, and training compute C.For neural networks, parameters include embeddings, dense weights, attention weights, and prediction heads.
  • 2.4 The three resources: data, parameters, and compute: The data exponent α measures how quickly held-out deviance improves with additional rows, while β and γ measure parameter and compute scaling effectiveness.A small exponent indicates improvement with strong diminishing returns; doubling N multiplies reducible loss by 2^-α.
  • 2.5 A short example and how to read the scaling claims: For the previewed TabM/TabM-mini envelope, doubling training rows multiplies reducible loss by 2^-0.409 ≈0.75.Within the stated experimental range and model family, each doubling removes about 25% of the remaining gap to the fitted floor.
  • 2.6 How to read the model families: The GLM is an additive benchmark, while FFNs, Transformers, TabM models, and Transformer+TabM-mini differ in how they learn or combine rating-factor interactions.TabM uses shared ensemble-style adapters, Transformers use learned field comparisons, and the hybrid combines Transformer comparisons with a TabM-mini head.
  • 2.6 How to read the model families: Model complexity is treated as regime-dependent: simple, strongly regularized structures may dominate with little data, whereas additional capacity can help with larger datasets.The rationale is that larger datasets provide more observations for learning genuine structure rather than mainly fitting noise.

3 Literature Review

Prior work establishes scaling laws for modern deep learning while highlighting that tabular actuarial prediction has distinct data, noise, interpretability, and governance constraints. This paper addresses limited evidence on tabular scaling in large-data insurance settings and connects it to actuarial credibility and model selection.

  • Neural scaling laws relate generalization performance to compute, model capacity, and data volume, often through empirical power-law trends.
  • Scaling-law estimates require careful methodology because intermediate checkpoints, random-seed variability, and regime-dependent behavior affect fitted relationships.Prior work also shows that test error can be non-monotone in finite-sample capacity regimes.
  • Tabular actuarial problems commonly use heterogeneous features, with GLMs and tree ensembles remaining strong practical baselines.Tree-based methods often outperform standard neural networks on typical medium-sized tabular datasets because of their inductive biases.
  • Tabular Transformers tokenize features and use self-attention to learn contextual feature interactions across categorical and numerical inputs.Examples include TabTransformer, FT-Transformer, and SAINT.
  • Small-to-medium tabular benchmarks provide limited evidence about scaling behavior in the million-row regime common in insurance applications.The study uses approximately 4.5 × 10^6 rows and nested fractions from 0.05 to 1.00 to examine high-data scaling questions.

4 Data

The study uses an anonymized pre-COVID motor-insurance portfolio with approximately 4.5 million policy-period records and Poisson claim-frequency modeling. Data preparation applies leakage-avoidance, nested sampling, feature exclusions, and actuarial-style representation choices, while evaluation uses an IID split because temporal validation is unavailable.

  • The dataset contains approximately 4.5 million policy-period records, 60–70 raw rating-factor columns, and claim counts modeled with earned exposure as a Poisson offset.The response is y_i and the exposure offset is E_i, with predicted mean λ_i = E_i r_i.
  • The portfolio has approximately 1 million exposure-years, about 240,000 claims, and an overall frequency of 0.240 claims per exposure-year.Approximately 5% of policy-periods have a claim and 95% have none.
  • Preprocessing removes non-positive-exposure records, excludes split and sampling columns, and treats unknown codes as distinct categories.
  • The protocol uses a 90/10 IID train–test split and nested training fractions f ∈ {0.05, 0.10, 0.25, 0.50, 0.75, 1.00}.The full test set remains fixed across nested subsets.
  • Categorical predictors with more than 100 levels are excluded, while continuous predictors are quantile-binned on training data and represented with learned entity embeddings.This standardizes feature tokenization across model families and aligns with actuarial banding practice.
  • Figure 3 reports one-way claim frequency by selected rating factors, using actuarial age bands and anonymized territorial categories ordered by exposure size.

5 Experimental Methodology

The experiments compare model families across nested data sizes, configurations, and random seeds using held-out Poisson deviance. Fixed recipes, repeated seeds, seed-averaged predictions, and power-law fits provide a controlled framework for estimating data, parameter, and compute scaling.

  • Nested training subsets are formed by thresholding a pre-generated uniform variate and evaluated on one fixed held-out test set.This makes smaller training sets strict subsets of larger ones, so changes primarily reflect additional data rather than resampling noise.
  • Training fractions are indexed by f, while scaling uses the corresponding policy-period row count N_f as the data-size variable.Exposure-years scale linearly with N in expectation under the representative nested sampling design.
  • Models are trained with Poisson deviance, the canonical count-data loss and the primary held-out evaluation metric, where lower deviance is better.For a fixed test set, a decrease of Δ in mean deviance corresponds to an average held-out log-likelihood lift of Δ/2 nats per policy.
  • Each configuration is trained with five independent random seeds, and inference averages predictions across those seeds to reduce initialization variance.The paper distinguishes this seed-averaged ensemble from TabM’s jointly trained within-model ensemble of implicit submodels.
  • Architecture-specific hyperparameter dictionaries are fixed across training fractions and random seeds rather than retuned for each setting.The paper calls this fixed expert-selected recipe scaling; selections use prior experience and exploratory small-data runs.
  • Scaling laws use an irreducible-loss-plus-power-law form, with compute fits based on Pareto-optimal runs and data fits based on best-of-size envelopes.Exponents are estimated by log-space regression after subtracting the fitted loss floor; compute is interpreted as a per-epoch proxy.
  • The study focuses on claim frequency because low claim frequency leaves materially fewer observations for comparable severity scaling experiments.

6 Model Descriptions

The paper evaluates GLM, FFN, TabM, and Transformer families for exposure-scaled Poisson frequency modeling, introducing architectural adaptations for tabular interactions, pooling, conditional capacity, and self-supervision.

  • Baseline models: GLM provides an interpretable additive baseline, while FFNs learn nonlinearities and cross-feature interactions through dense mixing.
  • Ensemble architectures: TabM-mini and TabM use lightweight rank-1 adapters with shared MLP kernels, with TabM allowing implicit ensemble members to diverge across layers.Predictions are averaged across members for variance reduction.
  • Transformer foundations: Vanilla Transformers treat covariates as tokens and use self-attention to model cross-column interactions, with a learned CLS token producing the prediction summary.This replaces manual interaction construction with attention-based interaction modeling.
  • Transformer enhancements: MultiCLS introduces multiple learned pooling queries, exposing ncls as an explicit capacity-scaling parameter and addressing a hypothesized single-CLS bottleneck.Each CLS token summarizes learned interactions among the covariate tokens.
  • Transformer enhancements: Enhanced, TokenMoE, hybrid, and self-supervised variants add stabilization, value-side capacity, conditional expert routing, TabM ensembling, or auxiliary representation-learning signals.TokenMoE uses expert count as a scaling dimension while load balancing discourages expert collapse; swap-style self-supervision detects corrupted token positions during training.
  • Shared representation and objective: All models use unified categorical token representations, quantile-bin continuous variables, learned per-column embeddings, and exposure-scaled Poisson frequency training.The Poisson deviance is used for out-of-sample predictive evaluation because the response is a claim count with exposure-adjusted Poisson means.

7 Results

Across training fractions, model families show distinct scaling regimes: GLMs remain stable, supervised Transformers gain little from added size, while TabM and self-supervised Transformer variants scale more effectively with data and capacity.

  • Baselines: GLMs remain strong and stable, while larger FFNs become favorable only as training data increases.At small fractions, the smallest FFNs perform best; at larger fractions, the optimum shifts toward the largest FFN.
  • Transformer scaling: Purely supervised Transformers show weak parameter scaling, whereas TokenMoE+SSL improves more consistently with size and achieves the best Transformer-family full-data result.TokenMoE improves monotonicity with size, while swap-style self-supervision produces clearer gains at larger N.
  • Data-regime crossovers: At 5% to 25%, Transformer+TabM provides the lowest seed-averaged ensemble deviance in the main-sweep envelope.This indicates improved sample-efficiency for TabM-style adaptation within the transformer in these runs.
  • Data-regime crossovers: At 50%, a TabM variant is numerically best, but the gap is too small for a stable crossover; at 75% and 100%, TabM support strengthens and the gap widens.These comparisons use seed-level evidence alongside ensemble envelope differences.
  • Data scaling: TabM has stronger data scaling than supervised Transformers, with α ≈0.309 on the main sweep and α ≈0.409 when the largest full-data TabM-mini extension is included.The lower fit quality after adding the extension is consistent with capacity crossovers and piecewise scaling rather than one universal regime.
  • Parameter scaling: TabM converts parameters into generalization gains more effectively than FFNs and supervised Transformers, with β ≈0.148 versus β ≈0.028 and β ≈0.025.Its stronger scaling is not well explained by classic deep-ensemble variance averaging alone; seed-averaging gains are comparatively smaller for TabM-style models.
  • Compute allocation: The bivariate TabM/TabM-mini fit gives α ≈0.430 and β ≈0.170, implying P⋆∝C0.716 and N⋆∝C0.284 under C ∝NP.The authors interpret this allocation as a first-order per-epoch compute heuristic rather than an exact total-training-FLOPs optimum.
  • Regime diagnostics: At 5% data, compute scaling is nearly flat with γ5% ≈ 0.0066, while at full data it steepens to γ100% ≈ 0.0580.This pattern is consistent with data-limited behavior at small N and compute sensitivity emerging once sufficient data are available.

8 Conclusion

The paper finds power-law scaling in actuarial tabular data, with architecture-dependent returns and practical regime shifts as data volume grows. Its conclusions remain conditional on the dataset, evaluation protocol, and model families studied.

  • 8 Conclusion: TabM-style models show the strongest data scaling, with α ≈0.31–0.41, and achieve the lowest full-data deviance.Purely supervised Transformers show weaker returns unless augmented with additional inductive bias.
  • 8 Conclusion: The best model family depends on data regime: Transformer+TabM leads at 5%–25%, while TabM becomes better supported at 75%–100%.TabM is only marginally ahead on the 50% ensemble envelope once seed-level uncertainty is considered.
  • 8 Conclusion: Seed averaging consistently improves inference, especially for attention-based models, with gains of 0.001003 for supervised Transformers, 0.000018 for GLMs, and 0.000360 for TabM.Transformers have smaller generalization gaps yet worse test performance, suggesting representation or optimization capacity as a limiting factor.
  • 8 Conclusion: Likelihood-scale interpretation matters: deviance improvements correspond to held-out log-likelihood gains, but not directly to business-value multipliers.Business interpretation also depends on pricing, calibration, competitiveness, loss ratios, fairness, and governance.
  • 8 Conclusion: The empirical scaling claims use one motor insurance dataset, a fixed feature set, and primarily IID random splitting.Results may differ across business lines, jurisdictions, feature choices, targets, or time-based deployment evaluations.
  • 8 Conclusion: The study compares selected GLM, FFN, Transformer, and TabM families without a full comparable GBDT sweep or systematic within-family hyperparameter optimization.Therefore, conclusions concern scaling trends under a consistent recipe rather than a definitive leaderboard.

A.1 Quick reference: symbols and dimensions

The quick-reference appendix points readers to the notation used for data, losses, scaling quantities, and neural architecture descriptions. Models use an exposure-scaled Poisson objective.

  • A.1 Quick reference: symbols and dimensions: Table 12 collects core notation for the data, loss, and scaling-law quantities.
  • A.1 Quick reference: symbols and dimensions: Table 13 collects additional notation for the neural architecture descriptions.
  • A.1 Quick reference: symbols and dimensions: All models use the exposure-scaled Poisson objective unless noted otherwise.

B.1 Experiment Model Identifiers

This appendix defines the experiment model identifiers through their preprocessing, embedding, Transformer, routing, self-supervision, and ensemble components. The implementations use exposure-scaled Poisson prediction objectives.

  • B.1 Experiment Model Identifiers: GLMs use encoded feature vectors and a linear predictor producing exposure-scaled Poisson means.The predicted mean is λ̂_i = E_i exp(η_i).
  • B.1 Experiment Model Identifiers: FFN baselines apply multilayer nonlinear transformations to concatenated learned feature embeddings.ReLU is used as the pointwise nonlinearity, with optional dropout between layers.
  • B.1 Experiment Model Identifiers: Tabular Transformers quantile-bin continuous variables, integer-encode categorical variables, and embed each covariate as a token.Variants add positional embeddings, CLS tokens, layer-wise value embeddings, or expert banks.
  • B.1 Experiment Model Identifiers: Transformer readouts extract CLS states and map them to log-rates, either through concatenation or averaging rates from separate heads.The two readouts coincide when n_cls = 1.
  • B.1 Experiment Model Identifiers: Head-fix variants mix current, previous, and embedding value streams using learned softmax weights.They also use normalization, sink tokens, value gating, and residual stabilizers.
  • B.1 Experiment Model Identifiers: TokenMoE performs token-wise top-1 routing over expert embedding slices and adds load balancing to encourage expert utilization.Routing uses straight-through gates, with optional mixing between base and value components.
  • B.1 Experiment Model Identifiers: Swap-style self-supervision swaps token embeddings across rows and adds a binary cross-entropy objective to the supervised Poisson loss.TokenMoE models may additionally include the load-balancing term.
  • B.1 Experiment Model Identifiers: TabM and TabM-mini use multiplicative or rank-1 adapters to form multiple ensemble members whose exposure-scaled rates are averaged.

B.3 Model Variant Feature Matrix

The feature matrix provides a compact comparison of Transformer-derived experiment families and identifies which architectural components are central to each variant.

  • B.3 Model Variant Feature Matrix: Table 15 summarizes how the major Transformer-derived experiment families differ by component.
  • B.3 Model Variant Feature Matrix: A checkmark in the matrix indicates that a component is central to the corresponding variant.

B.4 Models Configurations

The appendix catalogs configurations for GLM/FFN, TabM, vanilla and Multi-CLS Transformers, Transformer–TabM hybrids, and TokenMoE variants, including self-supervised naming conventions and implementation aliases.

  • Tables 16–24 document configurations for GLM/FFN, TabM, vanilla Transformer, Multi-CLS Transformer, Transformer–TabM, TokenMoE, and TokenMoE+SSL families.
  • The TabM-mini xlarge 2 identifier denotes the TabM-mini xlarge architecture evaluated through a separate workflow.
  • Transformer enhanced variants retain corresponding Multi-CLS hyperparameters but add the head-fix attention block described in Section 6.
  • The Transformer Multi-CLS large largeffn noreg variant widens the FFN and disables dropout and weight decay.
  • TokenMoE runs use moe hard select = False and moe router hidden = 0, while TokenMoE+SSL uses swap-style self-supervision.
  • The identifier transformer tokenmoessl xlarge is an internal alias for transformer tokenmoe ssl xlarge.

B.5 Scaling by model size: full data split results

The full-data scaling appendix reports seed-averaged ensemble test Poisson deviance across model-size ladders and training fractions, covering baseline, progressively modified Transformer, and TabM families.

  • Across the tables, the best-performing model size can shift with data, while purely supervised Transformer size increases show diminishing returns.
  • Table 24 reports seed-averaged ensemble test Poisson deviance by training fraction and size for GLM, FFN, and vanilla Transformer ladders.
  • Table 25 reports size-by-fraction results for progressively modified Transformer architectures.
  • The TokenMoE+SSL xlarge result in Table 25 is available only at the full-data fraction.
  • Table 26 reports size-by-fraction results for TabM and TabM-mini, with the final two rows being full-data-only extensions.

C Chinchilla-Style Scaling: Technical Explainer

This explainer frames Chinchilla-style scaling as a compute-constrained allocation of labelled data and parameters, extending separate data- and parameter-scaling effects into a joint growth question.

  • N denotes labelled training rows, P denotes trainable parameters, and α and β denote the data and parameter exponents.
  • The additive bivariate scaling law separates reducible loss into finite-data and finite-parameter contributions.
  • At fixed P, doubling N multiplies the data term by 2^-α; at fixed N, doubling P multiplies the parameter term by 2^-β.
  • Chinchilla-style analysis asks how N and P should co-grow under a compute constraint, rather than directly comparing α and β.
  • At fixed compute C, increasing P forces N down and increasing N forces P down, making the problem one of constrained allocation.

C.2 Derivation

The derivation obtains the paper’s allocation rule by minimizing the scaling objective under a compute proxy, applying first-order conditions, and balancing marginal contributions.

  • The derivation begins by minimizing the scaling objective after ignoring constants in the compute proxy.
  • First-order conditions characterize the interior optimum of the constrained allocation problem.
  • Multiplying the first condition by N and the second by P produces the marginal balance condition.
  • This marginal balance condition is the allocation rule reported in the main text.

C.3 Resolving the apparent paradox

The compute-optimal allocation appears parameter-heavy because the smaller exponent corresponds to the slower-shrinking bottleneck term. This conclusion is conditional on the fitted additive law and per-epoch compute heuristic.

  • C.3 Resolving the apparent paradox: The larger isolated exponent receives the smaller growth share because its error term is easier to reduce, leaving the slower-shrinking term as the bottleneck.The optimizer balances marginal returns before the slower-shrinking term determines the allocation.
  • C.3 Resolving the apparent paradox: A β/α ratio below one implies that parameters grow faster than data along the compute-efficient path.The paper notes that this implication depends on which resource each exponent denotes.
  • C.3 Resolving the apparent paradox: Using α ≈0.309 for TabM gives β/α ≈0.48 and about 1.39× N⋆ growth per doubling of P⋆.This alternative remains below one and therefore still implies faster parameter than data growth with meaningful data growth.
  • C.3 Resolving the apparent paradox: The allocation result is only a per-epoch compute heuristic because early stopping makes realized compute depend additionally on Estop(P, N).An exact total-compute optimum would require modelling stopping time within the scaling law.
Loading 2609.03106v1…