PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction

Shivang Chopra, Fotis Iliopoulos, Zsolt Kira, Gaurav Menghani

arXiv:2609.37905v1cs.LGcs.AI

TL;DR

CTR models have often improved by increasing interaction capacity, but its predictive returns can diminish as capacity grows. The paper proposes estimator scaling and implements it in RECAP, a parameter-efficient recursive model that combines estimators from multiple sources. Across five CTR benchmarks, RECAP achieves state-of-the-art performance while approaching five-model ensemble performance at approximately a single-model parameter budget.

  • Problem

    Increasing a single CTR predictor’s interaction capacity can yield diminishing predictive returns, raising the question of how additional resources can improve prediction.

    Source: §1 · pp. 1–2

  • Method

    RECAP uses a parameter-efficient recursive backbone to combine estimators through cross-run distillation, trajectory averaging, and inference-time route aggregation.

    Source: §1 · pp. 2–3

  • Results

    Across five CTR benchmarks, RECAP achieves state-of-the-art performance while approaching five-model ensemble performance at approximately a single-model parameter budget.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Estimator-scaling gains are larger when additional estimators expose more non-shared predictive variation and diminish as their predictions become increasingly correlated.

    Source: §1 · p. 2

    EMA is a parameter-efficient approximation to trajectory-level prediction averaging, but that approximation need not hold globally, so trajectory scaling is evaluated empirically.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Click-Through Rate prediction, a core task in recommendation and advertising systems, relies on modeling interactions among sparse categorical features. Explicit cross networks are a central paradigm for CTR prediction, and recent progress has largely come from increasing the interaction capacity of a single predictor through deeper cross networks and more expressive cross operators. We revisit whether continually increasing interaction capacity remains the most effective way to improve predictive performance, and find that its benefits quickly exhibit diminishing returns even as capacity continues to grow. This motivates a complementary scaling direction that we call estimator scaling, where additional resources are used to incorporate multiple related estimators rather than only enlarging a single predictor. Through theoretical analysis, we show that the gains from estimator scaling are governed by the amount of non-shared predictive variation available across estimators. However, exploiting this variation naively can be expensive: independently trained models provide substantial estimator diversity but require deployment cost to grow with ensemble size. This motivates a parameter-efficient realization of estimator scaling that can incorporate diversity from multiple estimator sources without maintaining multiple full models. Building on this view, we introduce RECursive Averaged Predictor (RECAP), a parameter-efficient recursive CTR model that operationalizes estimator scaling at three levels: distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. Experiments across multiple benchmarks establish new state-of-the-art predictive performance on standard benchmarks, while placing the RECAP on a favorable performance-parameter Pareto frontier.

1 INTRODUCTION

CTR prediction depends on modeling feature interactions, but increasing a single predictor’s interaction capacity quickly yields diminishing returns. The paper proposes estimator scaling and introduces RECAP to combine related estimators efficiently, reporting improved performance–parameter trade-offs across benchmarks.

  • Independent-run averaging outperforms trajectory averaging and shared-model routes, while maintaining multiple full models incurs substantial deployment cost.The averaging gains track the non-shared predictive variation available from each estimator source.Source: Figure 1 · p. 2, §1 · p. 2
  • Increasing cross-network depth or adding interaction branches initially improves performance, but the marginal benefit quickly shrinks as interaction capacity grows.Source: §1 · p. 2, Figure 1 · p. 2
  • Estimator scaling uses additional training or inference resources to combine estimators of the same predictive function, with gains governed by their available non-shared predictive variation.Gains diminish as estimator predictions become increasingly correlated.Source: §1 · p. 2
  • RECAP combines a weight-shared recursive backbone with ensemble distillation, exponential moving weight averaging, and inference-time route aggregation to realize estimator scaling at three levels.The approach keeps the deployed model close to a single-model parameter budget.Source: §1 · pp. 2–3
  • Across multiple CTR benchmarks, RECAP improves the performance–parameter trade-off over strong cross-network baselines and approaches substantially larger ensembles at approximately a single-model deployment footprint.Source: §1 · p. 3

2 RELATED WORK

Prior CTR work scales feature-interaction capacity, while another line combines predictors through ensembling, averaging, and distillation. This paper frames estimator scaling as a complementary axis and uses recursive parameter sharing as a compact substrate for it.

  • CTR interaction models have expanded feature-interaction capacity through factorization-based and explicit-cross architectures, including high-order interactions, self-attention, low-rank crosses, and exponential cross networks.These approaches primarily enlarge or selectively allocate a predictor’s interaction capacity.Source: §2 · p. 3
  • Deep ensembles, trajectory averaging, model soups, and distillation combine learned predictors, with prior work highlighting model diversity and theoretically studying diversity’s role in ensemble performance.The paper distinguishes its contribution from proposing a new averaging operator in isolation: it studies estimator scaling across independent runs, trajectories, and inference-time routes.Source: §2 · p. 3
  • Recursive parameter reuse can decouple added computation from model size; RECAP uses that principle as a compact substrate for cross-run distillation, trajectory averaging, and route aggregation.Source: §2 · p. 3

3 FROM INTERACTION CAPACITY TO ESTIMATOR SCALING

Estimator scaling complements interaction-capacity scaling by aggregating related predictors rather than enlarging one predictor. Its benefit depends on predictive variation that averaging can remove, including residual variation exposed by additional estimator axes.

  • Interaction-capacity scaling enlarges one predictor, whereas estimator scaling incorporates more estimators while largely holding the model family fixed.Source: §3 · p. 4
  • Interaction-capacity scaling improves the family’s mean predictor, while estimator scaling reduces variation around that mean and can still help when mean-predictor gains diminish.Source: §3 · p. 4
  • BCE improvement is approximately proportional to reducible predictive variance, weighted by local loss curvature.Source: §3 · p. 4
  • An estimator axis’s standalone benefit depends on its accessible variation, while another axis’s marginal benefit depends on residual variation not already removed.Source: §3 · p. 5

4 RECAP: RECURSIVE AVERAGED PREDICTOR

RECAP realizes estimator scaling with a parameter-shared recursive CTR backbone and aggregates information from independent runs, training trajectories, and inference-time routes. It also trains across rollout depths so the model can be evaluated at different recursive budgets.

4 RECAP: RECURSIVE AVERAGED PREDICTOR

  • RECAP combines a shared recursive backbone with estimator information from independent runs, optimization trajectories, and inference-time routes, without deploying a separate full model for each.Source: §4 · p. 5

4.1 PARAMETER-EFFICIENT RECURSIVE INTERACTION BACKBONE

  • Recursive LCN and ECN towers share core interaction weights across steps, while route-specific adaptations expose alternative predictors over the shared backbone.Source: §4.1 · p. 5

4.2 ESTIMATOR SCALING AT THREE LEVELS

  • EMA averages model parameters across training checkpoints without adding deployed parameters or inference computation, and approximates trajectory-level prediction averaging only locally.Source: §4.2 · p. 6
  • Ensemble distillation transfers the cross-run mean-logit aggregate into one deployed predictor; in practice, the match is approximate.Source: §4.2 · pp. 6–7
  • Mean-logit aggregation combines logits from alternative recursive routes; increasing the number of routes raises inference compute but leaves the stored model unchanged.Source: §4.2 · p. 7

4.3 TRAINING ACROSS RECURSIVE BUDGETS

  • Training across multiple rollout budgets avoids specializing RECAP to a single inference depth, with route sampling exposing the shared backbone to multiple computation paths.Source: §4.3 · p. 7

5 EXPERIMENTS AND RESULTS

The experiments link estimator-averaging gains to non-shared predictive variation and show that RECAP approaches ensemble performance at approximately a single-model parameter budget.

5 EXPERIMENTS AND RESULTS

  • The experiments cover five public CTR benchmarks and report AUC, where higher is better, and Logloss, where lower is better.AUC is reported in percentage points.Source: §5 · p. 8

5.2 NON-SHARED PREDICTIVE VARIATION EXPLAINS ESTIMATOR-SCALING GAINS

  • Averaging gains increase with non-shared predictive variation; independent runs expose the most, trajectory checkpoints less, and inference-time routes the least.AUC shows the same ordering across estimator families.Source: §5.2 · pp. 8–9
  • Adding an estimator axis yields larger marginal AUC gains when more residual non-shared variation remains; combining trajectory, cross-run, and route scaling gives RECAP its strongest performance.Each source improves the recursive backbone individually, but overlapping variation makes their gains complementary rather than simply additive.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 ESTIMATOR SCALING IS NOT SPECIFIC TO THE RECURSIVE BACKBONE

  • EMA and ensemble distillation also improve FCN, while route aggregation adds a low-cost estimator-diversity source unavailable to that backbone.Table 3 reports that combining EMA and distillation yields the strongest FCN performance.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 RECAP SHIFTS THE PERFORMANCE–PARAMETER FRONTIER

  • 81.764 versus 81.771 AUC: RECAP approaches five-model FCN ensemble performance while using only a fraction of its deployed parameters.Figure 4 shows that additional towers yield diminishing returns, while seed ensembling improves AUC at substantially higher deployment cost.Source: Figure 4 · p. 9, §5.4 · p. 9

6 CONCLUSION

The conclusion presents estimator scaling as complementary to interaction-capacity scaling and introduces RECAP as a parameter-efficient way to use multiple estimator sources.

6 CONCLUSION

  • Across five CTR benchmarks, RECAP achieves state-of-the-art performance while approaching five-model ensemble performance at approximately a single-model parameter budget.The conclusion frames estimator scaling as complementary to interaction-capacity scaling, whose predictive returns can diminish.Source: §6 · p. 10
  • The gains from estimator scaling are governed by the non-shared predictive variation available to aggregation.Source: §6 · p. 10
  • RECAP combines cross-run distillation, trajectory averaging, and route-based estimator scaling within a parameter-efficient recursive backbone.Source: §6 · p. 10

A DIAGNOSIS EXPERIMENTS ON ADDITIONAL DATASETS

Diagnostic results on four additional CTR datasets reproduce the pattern that interaction-capacity scaling saturates while estimator scaling continues to help.

A DIAGNOSIS EXPERIMENTS ON ADDITIONAL DATASETS

  • Across KKBox, ML-1M, KDD12, and Avazu, greater interaction depth or breadth eventually yields diminishing returns, while averaging additional estimators continues to improve performance.Figures 5–8 report these dataset-specific diagnostics.Source: Figure 5 · p. 13, Figure 6 · p. 13, A DIAGNOSIS EXPERIMENTS ON ADDITIONAL… · p. 14
  • Across the four datasets, averaging gains are consistently associated with the amount of non-shared predictive variation exposed by each estimator source.The reported averaging improvements remain aligned with the variation available for aggregation.Source: A DIAGNOSIS EXPERIMENTS ON ADDITIONAL… · pp. 14–15
  • Which estimator source is strongest varies by dataset: independent runs lead on some, while trajectory averaging is comparatively stronger on others.The results motivate using multiple estimator axes rather than relying on a single diversity source.Source: A DIAGNOSIS EXPERIMENTS ON ADDITIONAL… · p. 15

B ADDITIONAL THEORY FOR ESTIMATOR SCALING

The appendix extends the estimator-scaling analysis to arbitrary estimator covariance and clarifies how covariance limits the variation averaging can remove.

B ADDITIONAL THEORY FOR ESTIMATOR SCALING

B.1 GENERAL COVARIANCE FORMULATION

  • Averaging gains depend jointly on estimator variance and covariance; substantial individual variation can yield little reducible variation when strongly shared.Source: §B.1 · p. 15
  • The general covariance formulation also specializes to the equicorrelated setting used in the main text.Source: §B.1 · p. 15

B.2 FROM REDUCIBLE VARIATION TO BINARY CROSS-ENTROPY

The appendix relates aggregation to expected binary cross-entropy through a local expansion, then distinguishes that approximation from an exact result for literal mean-logit aggregation.

B.2 FROM REDUCIBLE VARIATION TO BINARY CROSS-ENTROPY

  • The local derivation expands the aggregate logit around the estimator-family mean, with conditional expectation eliminating the first-order term.For BCE, the second derivative is σ(z)(1 − σ(z)) and is independent of the label.Source: §B.2 · p. 16
  • Locally, expected BCE improvement is proportional to curvature-weighted reducible predictive variation; with similar mean predictions, differences in benefit are primarily governed by Vns(c, M; x).Source: §B.2 · p. 16
  • The approximation is most accurate when estimator disagreement around µc(x) is modest; higher-order terms may matter for broader or heavy-tailed distributions.Source: §B.2 · p. 16

B.3 EXACT IDENTITY FOR LITERAL MEAN-LOGIT AGGREGATION

  • For literal mean-logit aggregation, BCE is no higher than the average BCE of its constituent logits on the same example.Source: §B.3 · pp. 16–17
  • That exact inequality does not quantify gains by reducible variance or directly characterize EMA and distilled students that only approximate ensemble predictions.The main text therefore uses the local variance formulation as a common framework across estimator sources.Source: §B.3 · p. 17

C.1 TWO-AXIS DECOMPOSITION

The appendix decomposes predictive variance across estimator axes to distinguish variation available to each averaging step from variation remaining for later steps.

C.1 TWO-AXIS DECOMPOSITION

  • For two estimator sources, the law of total variance separates variation accessible to averaging along A from residual variation across B after averaging A.Up to local BCE-curvature weighting, these terms correspond to standalone and subsequent-axis loss reductions.Source: §C.1 · p. 17
  • The allocation is order-dependent: reversing the averaging order changes each axis's attributed variation, although total predictive variance remains unchanged.Source: §C.1 · p. 17

C.2 THREE-AXIS DECOMPOSITION FOR RECAP

  • For RECAP, the three estimator sources are independent runs, trajectory checkpoints, and inference-time routes, analyzed in the nesting order R → T → S.The decomposition assigns route-accessible variation first, followed by residual trajectory and cross-run variation.Source: §C.2 · pp. 17–18
  • Gains across estimator sources need not add linearly: overlapping predictive variation reduces what remains for later averaging, while distinct variation can yield closer-to-additive gains.The decomposition is an output-space attribution convention, not RECAP's literal sequence of operations.Source: §C.2 · p. 18

D HYPERPARAMETER SENSITIVITY

On Criteo, RECAP remains robust across the tested method-specific hyperparameters, with the default configuration in regions of consistently strong performance.

D HYPERPARAMETER SENSITIVITY

  • RECAP is stable across a broad range of tested settings, and its default configuration lies in regions of consistently strong test AUC and Logloss performance.The table reports Criteo test AUC and Logloss; Logloss is measured after validation-fitted affine calibration.Source: Table 4 · p. 18, D HYPERPARAMETER SENSITIVITY · p. 19
  • Nonzero distillation weights consistently outperform no distillation, while adding teacher seeds improves performance with diminishing returns.Source: D HYPERPARAMETER SENSITIVITY · p. 19
  • Performance varies only modestly across EMA decay values, and increasing LoRA rank or route adapters beyond moderate values adds little improvement.Source: D HYPERPARAMETER SENSITIVITY · p. 19
Loading 2609.37905v1…