PRIME: Mitigating Subgroup Optimization Competition in Shared CTR Top Networks with Plug-in Residual Input-Conditioned Mixture of Expert

Heng Yao; Siyun Hou; Tianying Liu; Yulou Shu; Yong He; Chuan Yuan; Kaibin Qiu; Guowei Chen; Jiayu Zhao; Chao Yu; Ke Ding

Source-linked AI summary

PRIME: Mitigating Subgroup Optimization Competition in Shared CTR Top Networks with Plug-in Residual Input-Conditioned Mixture of Expert

Heng Yao, Siyun Hou, Tianying Liu, Yulou Shu, Yong He, Chuan Yuan, Kaibin Qiu, Guowei Chen, Jiayu Zhao, Chao Yu, Ke Ding

arXiv:2608.30449v1Aug 31, 2026cs.LGcs.IR

TL;DR

Shared CTR Top-NNs may force heterogeneous examples to update one mapping despite competing subgroup directions. PRIME adds Dense-anchored, input-conditioned low-rank residual experts, and across held-out Avazu and Criteo tests it improves performance across most architectures while preserving the Dense path. Its scope is bounded by backbone-dependent behavior, including degradation when routing occurs before distinctions emerge after feature crossing.

Problem
CTR models usually share one Dense Top-NN across heterogeneous examples, while whether this parameterization remains suitable under subgroup heterogeneity has received little direct examination.
Method
PRIME preserves the Dense prediction with zero-residual initialization and adds input-dependent routing over low-rank residual experts, with multi-bag aggregation and EMA load adjustment.
Results
PRIME improves mean AUC for 11 of 13 architectures on each dataset and reduces the semantic subgroup competition gap by 34.3%.
Takeaways & Limitations
Function-preserving conditional residuals provide a compact, budget-controlled route to conditional adaptation of shared CTR Top-NNs across heterogeneous backbones.
Takeaways & Limitations
PRIME’s effect depends on attachment and backbone: FwFM reverses improvement direction across datasets, and pre-cross routing degrades GDCN on Criteo.

Abstract

from arXiv · show

Click-through rate (CTR) models vary in feature-interaction design, yet their top networks usually remain a single multilayer perceptron shared by all examples. Heterogeneous user, item, and context subgroups therefore update the same parameters; weakly aligned learning signals make the aggregate gradient a compromise among competing directions. We study the competition on Avazu with 4 models and 4 semantic fields. Across all architectures, semantic subgroups show lower Top-NN gradient cosine similarity than random groups matched by sample size and label ratio, with reductions of 0.23-0.37. This competition motivates input-conditioned experts, but directly replacing an established Dense mapping changes its initial function, sharing pattern, and capacity, obscuring the source of gains. We introduce PRIME (Plug-in Residual Input-conditioned Mixture of Experts), a Dense-anchored mixture of low-rank residual experts. PRIME anchors the original prediction and uses zero-residual initialization to match the Dense baseline exactly at training onset. Input-dependent routing weights low-rank experts for example-specific logit corrections; multi-bag aggregation and EMA load biases stabilize conditional estimation. We evaluate PRIME on held-out Avazu and Criteo test sets across 13 CTR architectures and five paired seeds. Median paired AUC gains are +0.0022 and +0.0066, with LogLoss reductions of 0.0011 and 0.0081, respectively. On FiBiNET and DCNv2, PRIME outperforms APG in all ten seed-level AUC comparisons while using fewer parameters and lower inference latency on both backbones. These results show that function-preserving conditional residuals add input-dependent capacity while preserving the Dense path and its optimization stability. Code is available at https://github.com/YH-learning/PRIME.

1 Introduction

CTR models typically share a Dense Top-NN across heterogeneous examples, creating a potential parameter-sharing constraint when subgroup update directions diverge. The paper diagnoses this competition and motivates PRIME, which adds input-conditioned residual capacity while preserving the Dense path and controlling computation.

1.1 CTR Prediction and the Fully Shared Top-NN: CTR architectures commonly retain a shared Dense Top-NN despite differing feature-interaction modules.The shared decision path is updated by every training example.
1.1 CTR Prediction and the Fully Shared Top-NN: Heterogeneous semantic subgroups can impose competing update directions on the same shared parameters.Widening the MLP increases capacity but leaves every added unit active for every example.
1.2 Diagnosing Subgroup Gradient Competition: 0.26, 0.34, 0.23, and 0.37 are the semantic-versus-random gradient-alignment gaps for DNN, AutoInt, FiBiNET, and DCNv2, respectively.The random controls match subgroup size and positive-label ratio.
1.3 From Diagnosis to Function-Preserving Conditional Adaptation: The paper evaluates function preservation, input-conditioned specialization, and budget-controlled attachment across backbones.Multi-bag aggregation and load adjustment support the conditional parameterization.
1.3 From Diagnosis to Function-Preserving Conditional Adaptation: PRIME retains the Dense mapping as an anchor and uses input-dependent low-rank logit corrections to reorganize parameter sharing.Its design targets conditional specialization without changing the backbone interaction module or initial Dense function.

2 Related Work

Prior CTR work mainly improves feature interactions, while conditional parameterization and expert methods have addressed related sharing problems in other settings. PRIME targets the shared decision layer of single-task CTR by combining low-rank residual adaptation, residual initialization, and bag aggregation.

2 Related Work: CTR architectures have developed richer interaction operators but generally converge on a globally shared prediction parameterization.PRIME targets this common decision-layer endpoint rather than adding another interaction operator.
2 Related Work: Conditional parameterization makes network weights input- or task-dependent; APG is a closely related low-rank method for individual CTR examples.Other examples include HyperNetworks, STAR, and PEPNet.
2 Related Work: Recommendation expert architectures have focused mainly on multi-task or multi-scenario sharing, whereas PRIME addresses heterogeneous examples within one CTR objective.Latent semantic groups diagnose competition but do not supervise the router.
2 Related Work: PRIME combines low-rank adaptation, zero-residual initialization, and averaging in a single end-to-end optimization.These components restrict expert freedom, preserve the baseline mapping, and smooth conditional estimates.
2 Related Work: PRIME uses the original backbone prediction as an anchor and combines it with routed low-rank expert-bag predictions.The architecture caption specifies G=4 bags and α=0.5.

3 PRIME

PRIME extends a complete CTR backbone with input-conditioned, low-rank residual experts while preserving the original Dense prediction path. Its zero-residual initialization, bounded mixture, multi-bag aggregation, and gradient-free load adjustment target conditional specialization without replicating the backbone.

Problem formulation: Lower gradient cosine similarity weakens a subgroup’s effective descent and can make an update benefiting one subgroup increase another’s loss.A fully shared Top-NN therefore compromises among subgroup objectives.
Dense-anchored residual parameterization: PRIME preserves the complete baseline function and attaches a conditional adapter to the original embedding representation and prediction.The backbone, including interaction and parallel-logit paths, remains unchanged.
Input-conditioned routing: Input-dependent routing combines candidate residuals, so different inputs can receive different expert weights rather than forming a fixed ensemble.Replacing routing weights with constants or permuting them across examples tests the value of conditional routing.
Low-rank residual experts: Each low-rank expert projects the embedding into a q-dimensional subspace and maps the nonlinear response to a scalar logit correction using O(qd) parameters.Experts produce compact residuals instead of replicating a complete Top-NN or interaction module.
Aggregation and load balancing: PRIME averages predictions from multiple independently routed expert bags and convexly combines them with the Dense probability, using gradient-free load bias updates for utilization.The main experiments use α = 0.5; load adjustment changes subsequent routing biases without adding gradients to model parameters.
Functional preservation: Zero-initializing expert output projections and biases makes PRIME exactly function-equivalent to the Dense model at initialization.Projection matrices and routers may begin with different directions, while optimization first develops expert outputs before router specialization.
Budget control: The expert rank q controls PRIME’s added computation and trainable parameters, avoiding replication of E complete Top-NNs.Routers require Ed MACs and experts require Eq(d + 1) MACs, ignoring scalar activation costs.

4 Experiments

The experiments diagnose subgroup competition in shared Top-NNs, test whether PRIME reduces it beyond residual capacity alone, and evaluate accuracy, robustness, and efficiency across CTR architectures and datasets.

4.1 Research Questions and Experimental Protocol: The evaluation covers 13 CTR architectures on Avazu and Criteo, using shared preprocessing, paired configurations, validation-based model selection, and five held-out test seeds.Dense/PRIME pairs share splits, preprocessing, backbone configuration, and random seed.
4.2 Diagnosing Shared Top-NN Competition and Its Change under PRIME: Semantic subgroups are compared with matched random groups across four architectures, three checkpoints, and four semantic fields to measure Top-NN gradient competition.Controls match subgroup size and positive-label ratio.
4.2 Diagnosing Shared Top-NN Competition and Its Change under PRIME: 34.3%: the mean competition gap decreases from ΓDense = 0.3016 to ΓPRIME = 0.1981, with larger alignment gains for semantic subgroups.The comparison uses the same architectures, seeds, fields, and examples for Dense and PRIME.
4.3 Capacity-Matched Controls and the Contribution of Input Conditioning: PRIME reduces the competition gap beyond non-conditional residual capacity under the same anchor and zero-residual initialization, isolating a contribution from input-conditioned organization.The non-conditional residual reduces the gap by 0.0770 relative to Dense, while PRIME provides a further reduction of 0.0265.
4.3 Capacity-Matched Controls and the Contribution of Input Conditioning: Conditional routing and Dense anchoring matter: uniform and permuted routing reduce mean AUC by 0.0018 and 0.0017, while retaining the anchor contributes 0.0005.Multi-bag aggregation contributes a further 0.0005 mean AUC.
4.4–4.5 Final Comparisons and Held-Out Results: PRIME improves held-out performance across architectures while remaining competitive in deployment cost and outperforming APG on FiBiNET and DCNv2.On Avazu, macro-average AUC rises from 0.7588 to 0.7611 and LogLoss falls from 0.3701 to 0.3689; on Criteo, the median paired ΔAUC is +0.0066.

5 Discussion

PRIME preserves the shared Dense prediction while adding input-conditioned low-rank residuals, reorganizing parameter sharing around a common function. Its benefits depend on routing and residual attachment aligning with the shared decision path, with dataset-dependent boundary cases.

Mechanism: PRIME decomposes prediction into a shared Dense anchor and input-conditioned low-rank residual corrections, allowing heterogeneous examples to disagree locally without subgroup labels or replicated backbones.Common regularities remain in the anchor, while selected experts absorb input-specific deviations.
Optimization evidence: 34.3% reduction in the semantic subgroup competition gap supports conditional residuals as an optimization mechanism, not merely an increase in model capacity.Matched Dense controls, non-conditional residuals, and uniform or permuted routing underperform PRIME.
Function preservation: PRIME won 15 of 20 paired validation AUC-and-LogLoss comparisons, versus 4 of 20 and 8 of 20 for two alternatives that replace the Dense path.Zero-residual initialization and bounded mixing preserve a usable predictor during adaptation.
Attachment principle: Routing should observe the representation consumed by the shared decision mapping, while the residual should correct that same stage.This attachment alignment, rather than the interaction operator alone, distinguishes consistently positive backbones from boundary cases.
Boundary cases: FwFM gains 0.0163 mean AUC on Criteo but loses 0.0030 on Avazu, exposing dataset-dependent limits when no shared MLP Top-NN exists.In this setting, PRIME acts as an auxiliary conditional predictor rather than systematically reorganizing a contested shared mapping.
Boundary cases: GDCN improves by 0.0009 AUC on Avazu but declines by 0.0025 on Criteo when PRIME routes on pre-cross embeddings rather than decision-relevant crossed representations.Moving routing to an intermediate cross layer raises mean Criteo AUC from 0.8041 to 0.8063.
Implication: Architecture-aware attachment is therefore required: routing should expose subgroup distinctions that drive shared-path gradients, and residuals should target an explicit shared prediction path.The supported design principle links subgroup diagnostics to the location of conditional adaptation.

6 Conclusion

The paper identifies incompatible subgroup updates as a constraint in fully shared CTR top networks and addresses it with function-preserving conditional adaptation. Across held-out datasets and architectures, PRIME improves performance broadly while reducing subgroup competition.

Constraint: Semantic Avazu subgroups produce lower Top-NN gradient alignment than size- and label-matched random groups, motivating conditional parameter organization.The comparison controls for subgroup size and positive-label ratio.
Approach: PRIME anchors the Dense prediction, initializes residuals at zero, and combines bounded low-rank corrections through input-dependent routing, multi-bag aggregation, and EMA load adjustment.The design retains the shared path while introducing conditional specialization.
Results: 11 of 13 architectures improve in mean AUC on each of the held-out Avazu and Criteo datasets.On FiBiNET and DCNv2, PRIME also beats parameter- and MAC-matched Dense controls and wins all ten Avazu seed-level comparisons against APG.
Conclusion: 34.3% reduction in the semantic subgroup competition gap connects stronger predictions with more compatible shared updates.Routing controls attribute the gain to input-conditioned expert organization rather than capacity alone.

Ethical Considerations

The study uses public, de-identified CTR benchmark data without human recruitment or personally identifiable information.

The experiments use public, de-identified CTR benchmark data and involve no human recruitment or personally identifiable information.

A Reproducibility and Design Map

The reproducibility map fixes training and model-selection procedures, records PRIME’s design settings, and links its hypotheses to targeted controls and held-out comparisons. The APG comparison uses matched experimental conditions and five paired seeds.

Training protocol: All main experiments use Adam with learning rate 10^-3, batch size 4,096, validation-AUC checkpoint selection, and at most 100 epochs with early stopping.Embedding dimension is 10; backbone-specific dropout, normalization, and regularization remain fixed within Dense/PRIME pairs.
Configuration: The frozen PRIME configuration uses G=4 bags, M=8 experts per bag, rank q=16, Dense weight α=0.5, and correction bound cmax=2.Routing temperature is T=1, with moving-load coefficient μ=0.99 and bias step size ηb=10^-3.
Design map: The design tests conditional correction, function preservation, budget control, conditional aggregation, and objective preservation through routing, anchoring, adapter, bag, and auxiliary-loss controls.These controls separate input–expert correspondence, Dense-path preservation, capacity, aggregation, and load-adjustment effects.
Objective preservation: EMA load bias is evaluated separately from CTR gradients, with an auxiliary-loss control testing whether similar accuracy requires another loss coefficient.
Design map: PRIME’s core parameterization and efficiency constraints are separated from multi-bag aggregation and load adjustment, which serve as complementary stabilization choices.
APG comparison: The APG comparison holds data split, embedding dimension, optimizer, batch size, early stopping, and five paired random seeds constant across configurations.
APG comparison: PRIME achieves higher AUC in all ten paired seed-level comparisons and uses fewer parameters with lower inference latency than APG across FiBiNET and DCNv2.The smallest margin is +0.0002 on FiBiNET at seed 190034.

C Routing Diagnostics and Dense-Weight Sensitivity

PRIME’s routing remains meaningfully distributed across experts, while the frozen α=0.5 setting is retained as a balanced choice across architectures.

Routing diagnostics: 2.49±0.47 effective experts remain active across 16 Avazu runs, with normalized routing entropy of 0.432 ± 0.085.The maximum mean load is 0.425±0.116, above the uniform value of 0.125, indicating specialization alongside multi-expert usage.
Dense-weight sensitivity: α=0.5 is retained because architectures favor complementary settings across validation AUC comparisons.DNN and FiBiNET favor 0.5, whereas AutoInt and DCNv2 favor 0.25.

D GDCN Attachment Controls on Criteo

GDCN attachment controls show that routing on decision-relevant crossed representations improves PRIME, while retaining the Dense anchor and bounded probability mixture remains important.

Router and expert attachments: 0.8063 mean AUC results when routing moves from pre-cross embeddings to an intermediate cross layer, versus 0.8041 for pre-cross routing.With the expert source fixed, Post/Pre reaches 0.8061 AUC while Pre/Post falls to 0.8035.
Router and expert attachments: 0.4086 router entropy and 0.9021 maximum expert load accompany Pre/Post, compared with 0.9843 and 0.5418 for Post/Pre.These controls identify pre-cross router conditioning, rather than insufficient expert depth, as a major contributor to the original configuration’s weaker Criteo result.
Mechanism controls: Detaching the router gradient lowers mean AUC, while direct logit residuals produce the weakest result.Post/Pre is slightly stronger than Post/Post, so exact coincidence between router and expert inputs is unnecessary.
Efficiency: PRIME adds 0.6%–0.8% parameters over Dense, with training-step overhead ranging from 8.4% on FiBiNET to 16.9% on DCNv2.On FiBiNET, PRIME lowers peak training memory by 36.0% and remains faster at inference; it is also faster and substantially more memory-efficient than APG on DCNv2.

F Development-Stage Structural Controls

Development controls indicate that preserving the original Dense function is central: residual PRIME improves both metrics far more often than replacements that alter the baseline mapping.

Design implication: PRIME retains the shared Dense mapping while adding input-dependent low-rank logit corrections, separating conditional capacity from the backbone’s interaction mechanism.This design motivates the function-preservation requirement identified by the development controls.
Structural controls: 15 of 20 paired Avazu comparisons improve both validation AUC and LogLoss with PRIME.A win requires simultaneous improvement in both metrics.
Structural controls: 4 of 20 direct hidden-layer subspace replacements and 8 of 20 multi-layer bagged replacements achieve simultaneous improvement in both metrics.These alternatives modify the Dense path and begin optimization from a different function than the original model.