PaperlayerView exact revision on arXiv

Source-linked AI summary

LoopCTR: Unlocking the Loop Scaling Power for Click-Through Rate Prediction

Jiakai Tang, Runfeng Zhang, Weiqiu Wang, Yifei Liu, Chuan Wang, Xu Chen, Yeqiu Yang, Jian Wu, Yuning Jiang, Bo Zheng

arXiv:2604.19550v1cs.IR

TL;DR

Transformer-based CTR models face growing computation and storage overhead when scaled by stacking parameters, conflicting with industrial deployment constraints. LoopCTR reuses shared layers recursively, using a sandwich architecture, Hyper-Connected Residuals, Mixture-of-Experts, and process supervision. It achieves state-of-the-art performance, while zero-loop inference can outperform all baselines and oracle analysis reveals 0.02–0.04 AUC of untapped headroom.

  • Problem

    Stacking more parameters in Transformer-based CTR models increases computational and storage overhead under stringent industrial deployment constraints.

  • Method

    LoopCTR scales computation through recursive reuse of shared layers in a sandwich architecture with Hyper-Connected Residuals, Mixture-of-Experts, and process supervision at every loop depth.

  • Results

    LoopCTR achieves state-of-the-art performance across benchmarks, while zero-loop inference outperforms all baselines and oracle analysis reveals 0.02–0.04 AUC of untapped headroom.

  • Takeaways & Limitations

    Train-multi-loop, infer-zero-loop deployment encodes iterative-refinement benefits into shared parameters while reducing inference cost.

  • Takeaways & Limitations

    The gap between realized and optimal per-sample loop selection remains considerable, leaving adaptive inference as future work.

Abstract

from arXiv · show

Scaling Transformer-based click-through rate (CTR) models by stacking more parameters brings growing computational and storage overhead, creating a widening gap between scaling ambitions and the stringent industrial deployment constraints. We propose LoopCTR, which introduces a loop scaling paradigm that increases training-time computation through recursive reuse of shared model layers, decoupling computation from parameter growth. LoopCTR adopts a sandwich architecture enhanced with Hyper-Connected Residuals and Mixture-of-Experts, and employs process supervision at every loop depth to encode multi-loop benefits into the shared parameters. This enables a train-multi-loop, infer-zero-loop strategy where a single forward pass without any loop already outperforms all baselines. Experiments on three public benchmarks and one industrial dataset demonstrate state-of-the-art performance. Oracle analysis further reveals 0.02--0.04 AUC of untapped headroom, with models trained with fewer loops exhibiting higher oracle ceilings, pointing to a promising frontier for adaptive inference.

1 Introduction

LoopCTR introduces computation scaling through recursive reuse of shared layers, addressing Transformer CTR models’ parameter and deployment costs. Its architecture and training strategy support strong zero-loop inference, with experiments showing state-of-the-art performance and remaining oracle headroom.

  • 1 Introduction: LoopCTR scales computation by recursively reusing shared model layers instead of stacking additional parameters.This decouples computation from parameter growth and improves parameter efficiency.
  • 1 Introduction: Conventional Transformer blocks limit iterative refinement, while multi-loop inference introduces proportional latency and runtime overhead.These expressiveness and efficiency bottlenecks hinder deployment under low-latency requirements.
  • 1 Introduction: LoopCTR combines Entry, Loop, and Exit Blocks with Hyper-Connected Residuals, Mixture-of-Experts, and process supervision.The Loop Block recursively processes representations, while process supervision encodes multi-loop benefits into shared parameters.
  • 1 Introduction: Zero-loop inference matches or surpasses full multi-loop inference, while more training loops consistently improve performance.A single forward pass can therefore retain the benefits of iterative refinement without executing inference loops.
  • 1 Introduction: 0.02–0.04 AUC of untapped headroom remains in oracle analysis, with fewer-loop training producing higher oracle ceilings.The authors identify adaptive inference as a promising direction.

2 Preliminary

CTR prediction estimates click probability from user, item, behavior, context, and cross features. LoopCTR organizes these inputs into token sequences and processes them with a sandwich architecture designed for efficient representation and prediction.

  • 2 Preliminary: CTR prediction estimates the click probability for a user-item pair from heterogeneous behavioral, profile, item, context, and cross features.The complete input is x = (x_u, x_v, S_s, S_l, x_c, x_×).
  • 2 Preliminary: Short-term and long-term behavior sequences are treated as sequential features, while user, item, context, and cross features are global features.Short-term item tokens preserve recent-interest detail; long-term behavior is compressed into query-token representations.
  • 2 Preliminary: LoopCTR uses an Entry Block for feature encoding, a recursively reused Loop Block for latent reasoning, and an Exit Block for score prediction.The architecture separates feature encoding, iterative reasoning, and final click-probability production.
  • 2 Preliminary: Learnable query tokens compress long-term behavior through cross-attention, reducing downstream complexity while retaining salient preference information.The compression is optimized jointly with the entire model.
  • 2 Preliminary: At serving time, CTR models score hundreds to thousands of candidate items and display the highest-ranked items.This deployment setting makes computational efficiency relevant to practical use.

3 Methodology

LoopCTR separates feature encoding, recursive reasoning, and prediction in a sandwich architecture, using shared layers repeatedly while supporting zero-loop inference. Hyper-Connected Residuals, MoE components, asymmetric attention, process supervision, and serving-oriented caching address expressiveness and efficiency.

  • Sandwich Architecture: The sandwich architecture separates feature encoding in the Entry Block, iterative latent reasoning in the Loop Block, and score prediction in the Exit Block.The Loop Block recursively applies the same shared-parameter layer, while the Exit Block produces the final click probability.
  • Entry Block: The Entry Block uses group-specific projections and independent self-attention to align heterogeneous feature groups while enabling parallel computation.Each behavior sequence and individual global token forms a separate group before group-specific projection and attention.
  • Loop Block: The Loop Block applies one shared-parameter layer across L iterations with an asymmetric mask: sequential tokens attend only to sequential tokens, while global tokens attend to the entire input.This pattern lets global tokens aggregate sequential context without letting global features dominate sequential representations.
  • Hyper-Connected Residuals: Hyper-Connected Residuals replace fixed single-stream residuals with n parallel streams and input-dependent fusion for adaptive information flow across recursive applications.The coefficients governing mixing, layer contribution, and residual fusion combine static parameters with hidden-state-dependent adjustments.
  • MoE-Augmented Transformer: Mixture-of-Experts layers expand parameter capacity while activating sparse expert subsets, using shared routing in attention and load balancing in FFN components.In attention, Query and Key projections remain shared, while Value and output projections are replaced with MoE layers using the same router.
  • Training Objective: Process supervision averages binary cross-entropy losses across loop depths, enabling meaningful zero-loop predictions that bypass the Loop Block and remove its latency overhead.At each depth, the current representation passes through the Exit Block to produce a prediction.

4 Experiments

Experiments across three public benchmarks and one industrial dataset evaluate LoopCTR against DNN and Transformer-based baselines, then analyze loop counts and component contributions. LoopCTR achieves strong accuracy with efficient zero-loop inference, while training loops improve realized performance and expose substantial oracle headroom.

  • Experimental Setup: Experiments cover Amazon, TaobaoAds, KuaiVideo, and an InHouse production-log dataset, comparing DNN-based, Transformer-based, and unified sequence-feature baselines.The evaluation uses AUC, GAUC, and NE as standard CTR metrics.
  • Overall Performance: LoopCTR establishes state-of-the-art performance across all benchmarks, with LoopCTR(1/3) exceeding OneTrans by 0.0039 AUC on Amazon and DIN by 0.0020 AUC on KuaiVideo.The reported advantage holds across datasets with varying scale and domain.
  • Overall Performance: LoopCTR(0/3) surpasses all baselines on AUC and NE across every dataset, achieving 13.38M FLOPs and 9.26ms latency on InHouse.The zero-loop result is only 0.0013 AUC below the best multi-loop variant on Amazon.
  • Overall Performance: At identical FLOPs, LoopCTR(3/3) leads StackCTR on AUC across all four datasets, including 0.8726 versus 0.8690 on Amazon.The overall pattern favors shared parameters, although StackCTR occasionally matches or exceeds LoopCTR on TaobaoAds.
  • Loop Scaling: Increasing training loops from 0 to 3 consistently raises AUC, including 0.8662→0.8728 on Amazon and 0.6966→0.7007 on InHouse.The analysis reports 0.02–0.04 AUC of total oracle headroom above the L=0 baseline.
  • Loop Scaling: Inference loops show diminishing returns: on Amazon with L=3, AUC rises from 0.8715 at i=0 to 0.8728 at i=1, then plateaus or slightly declines.This pattern supports zero-loop or single-loop inference across datasets.
  • Loop Scaling: Fewer training loops yield higher oracle ceilings, with Amazon oracle AUC rising from 0.8858 at L=3 to 0.8885 at L=1.On InHouse, oracle AUC is 0.7195 at L=3 versus 0.7306 at L=1.
  • Ablation Study: All four LoopCTR components contribute positively, with HCR causing a 0.0201 Amazon AUC drop when removed and MoE causing a 0.0060 KuaiVideo drop.Process supervision and heterogeneous token projection contribute consistently on both datasets.

5 Related Work

Prior CTR research progressed from feature-interaction and self-attention models toward industrial Transformer systems that scale depth, width, and input length. Looped Transformer work instead reuses shared-weight blocks recursively, but existing approaches focus on language or reasoning tasks and require multiple inference loops.

  • Transformer-based CTR Prediction: Transformer-based CTR systems pursue scaling along depth, width, and input length after earlier feature-interaction and self-attention approaches.The cited prior work spans industrial-scale Transformer systems for CTR prediction.
  • Looped Transformers: Universal Transformer introduced recursively applied shared-weight blocks with Adaptive Computation Time, followed by theoretical and practical work on looped Transformers.These efforts emphasize language modeling, algorithmic reasoning, or length generalization rather than CTR prediction.

6 Conclusion

LoopCTR introduces loop scaling through recursive reuse of shared layers, combining a sandwich architecture with Hyper-Connected Residuals, MoE, and process supervision. The resulting strategy supports strong prediction quality with lower inference cost while leaving adaptive loop-selection headroom.

  • LoopCTR decouples computation scaling from parameter growth by recursively reusing shared model layers.
  • The sandwich architecture combines Hyper-Connected Residuals, Mixture-of-Experts, and process supervision to encode multi-loop benefits into shared parameters.
  • Train-multi-loop, infer-zero-loop achieves state-of-the-art prediction quality with substantially lower inference cost.
  • The gap between realized and optimal per-sample loop selection suggests untapped potential for adaptive inference strategies.

A.1 Full Loop Scaling Results

Table 3 reports loop-scaling results across training loop counts, evaluating multiple inference depths for models trained with loops. Oracle inference selects the optimal loop depth separately for each test sample.

  • Evaluation setup: Table 3 evaluates training loop counts L ∈ {0, 1, 2, 3}.Models trained with L > 0 are tested using inference loop counts i ∈ {0, 1, 2, 3}.
  • Evaluation setup: For L = 0, only zero-loop inference is applicable.
  • Evaluation setup: Oracle inference selects the optimal loop depth per sample.This provides an upper-bound reference for comparing fixed inference depths.

A.2 Loss Landscape Analysis

More training loops produce broader, smoother low-loss regions, while loop-depth representations become increasingly aligned without becoming identical. Oracle analysis indicates substantial unrealized gains from adaptive loop selection.

  • Loss landscape: As training loop count increases from L=1 to L=3, the Amazon loss landscape develops a broader, smoother low-loss basin.The L=1 model has the smallest basin and a visible secondary local minimum, whereas L=3 has a wider region around the optimum.
  • Loss landscape: Broader, flatter minima provide a geometric explanation for why more training loops yield higher realized performance.
  • Per-loop diagnostics: Cosine similarity between adjacent loop depths increases during training but remains below 1.0.Later loops are more similar than earlier loops, suggesting that iterative refinement converges with depth while retaining distinct representations.
  • Oracle analysis: The oracle upper bound remains well above realized inference, including 0.8858 versus 0.8728 AUC on Amazon.This gap motivates adaptive inference strategies that allocate loop depth per sample.

B.2 Parameter Sensitivity Analysis

LoopCTR performs best with sparse expert activation and a moderate total expert count. The 2/4 configuration offers the strongest reported balance between expert diversity and training stability.

  • Activated experts: Activating 2 of 4 experts yields the best AUC on both Amazon and KuaiVideo.Using 1 expert or all 4 experts degrades performance, indicating the value of sparse expert selection.
  • Total experts: Amazon AUC increases from 0.8719 to 0.8726 when total experts rise from 2 to 4.
  • Total experts: KuaiVideo AUC increases from 0.7441 to 0.7448 when total experts rise from 2 to 4.
  • Total experts: Increasing the total expert count from 4 to 5 provides no additional gain with 2 activated experts.The 2/4 configuration is reported as the best trade-off between expert diversity and training stability.

B.3 Expert Routing Analysis

LoopCTR’s shared MoE layers adapt their expert routing across loop iterations rather than repeating identical computation. Attention routing shifts from early specialization toward balanced utilization, while FFN routing evolves more gradually.

  • Attention MoE: Attention routing shifts from concentration on experts 2 and 4 at iteration 1 to near-uniform utilization by iteration 3.Experts 2 and 4 receive 44.0% and 42.6% initially, while iteration 3 reaches approximately 22–28% per expert.
  • Attention MoE: Early attention iterations use specialized expert pathways, whereas later iterations use more balanced computation as representations are refined.
  • FFN MoE: FFN routing begins comparatively balanced at approximately 16–34% per expert but still changes across loop iterations.
  • FFN MoE: FFN expert 3 dominates iterations 1–2 before declining in iteration 3 as expert 4’s share increases.Expert 3 receives 33.9% and 32.6% in iterations 1–2.

C Hyper-Connected Residuals Analysis

Hyper-Connected Residuals replace standard residual addition with input-dependent, multi-stream mixing for the attention and FFN sub-layers. The implementation uses two streams and adds only negligible residual-mixing overhead relative to the sub-layers.

  • Design: Hyper-Connected Residuals use input-dependent coefficients to control how multiple streams flow through attention and FFN sub-layers.
  • Benefits: The design provides parallel multi-stream computation and flexible blending for looped architectures.
  • Configuration: Experiments use n=2 HCR streams, with one stream for attention and one for the FFN sub-layer.

D.2 Complexity Analysis

LoopCTR’s complexity separates cached user-side work from per-item computation and scales predictably with loop count. Shared parameters keep model size independent of loop depth, while zero-loop deployment removes the Loop Block entirely and yields strong efficiency.

  • Serving decomposition: Online serving caches user-side sequence processing across candidate items, leaving item-side, context, and cross-feature computations online.
  • Loop Block: The Loop Block uses sparse k-out-of-E routing, costing O(kTd2) for value/output projections instead of O(ETd2) with dense expert activation.
  • Loop Block: HCR contributes O(Tn2d) per sub-layer, an overhead dominated by attention and FFN computation when n=2.
  • Full forward pass: Training cost is Centry + L · Cloop + (L+1) · Cexit, while inference cost with i loops is Centry + i · Cloop + Cexit.
  • Full forward pass: Zero-loop inference bypasses the Loop Block, reducing cost to Centry + Cexit.
  • Parameter sharing: Loop Block parameters are shared across iterations, so model parameter count is independent of L and nonzero inference loop count.
  • Efficiency comparison: LoopCTR(0/3) uses 13.38M FLOPs and 9.26ms latency on InHouse, 160× fewer FLOPs and 84× lower latency than HSTU.
  • Deployment scaling: Each additional inference loop adds a fixed cost, making FLOPs and latency scale linearly for deployment trade-offs.
Loading 2604.19550v1…