TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

Yuchen Jiang; Jie Zhu; Xintian Han; Hui Lu; Kunmin Bai; Mingyu Yang; Shikang Wu; Ruihao Zhang; Wenlin Zhao; Shipeng Bai; Sijin Zhou; Huizhi Yang; Tianyi Liu; Wenda Liu; Ziyan Gong; Haoran Ding; Zheng Chai; Deping Xie; Zhe Chen; Yuchao Zheng; Peng Xu

Source-linked AI summary

TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

Yuchen Jiang, Jie Zhu, Xintian Han, Hui Lu, Kunmin Bai, Mingyu Yang, Shikang Wu, Ruihao Zhang, Wenlin Zhao, Shipeng Bai, Sijin Zhou, Huizhi Yang, Tianyi Liu, Wenda Liu, Ziyan Gong, Haoran Ding, Zheng Chai, Deping Xie, Zhe Chen, Yuchao Zheng, Peng Xu

arXiv:2602.06563v2Feb 6, 2026cs.IR

TL;DR

Recommendation-model scaling has been limited by architectural inefficiencies, hardware under-utilization, and TokenMixer bottlenecks in deep configurations. TokenMixer-Large introduces mixing and reverting, deeper-model residual mechanisms, and Sparse-Pertoken MoE to support larger models. It scales to billion-parameter regimes and reports gains across ByteDance recommendation scenarios.

Problem
Existing recommendation architectures and TokenMixer face architectural, hardware-utilization, deep-model, and MoE-sparsification limitations that constrain practical scaling.
Method
TokenMixer-Large combines tokenization, mixing and reverting, channel mixing, and Sparse-Pertoken MoE with efficient MoE operators and Token-Parallel execution.
Results
TokenMixer-Large outperforms SOTA models with a +1.14% AUC gain in CTCVR versus MLP-500M, while Sparse-Pertoken MoE matches dense performance using half the activated parameters.
Takeaways & Limitations
TokenMixer-Large has delivered offline and online gains across multiple ByteDance scenarios while serving hundreds of millions of users.
Takeaways & Limitations
TokenMixer historically used shallow configurations, such as two layers in RankMixer, making deeper models difficult to stabilize and improve.

Abstract

from arXiv · show

While scaling laws for recommendation models have gained significant traction, existing architectures such as Wukong, HiFormer and DHEN, often struggle with sub-optimal designs and hardware under-utilization, limiting their practical scalability. Our previous TokenMixer architecture (introduced in RankMixer paper) addressed effectiveness and efficiency by replacing self-attention with a ightweight token-mixing operator; however, it faced critical bottlenecks in deeper configurations, including sub-optimal residual paths, vanishing gradients, incomplete MoE sparsification and constrained scalability. In this paper, we propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation. By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation even as model depth increases. Furthermore, we incorporate a Sparse Per-token MoE to enable efficient parameter expansion. TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments, respectively. Currently deployed in multiple scenarios at ByteDance, TokenMixer-Large has achieved significant offline and online performance gains, delivering an increase of +1.66\% in orders and +2.98\% in per-capita preview payment GMV for e-commerce, improving ADSS by +2.0\% in advertising and achieving a +1.4\% revenue growth for live streaming.

1 Introduction

Recommendation-model scaling has exposed weaknesses in architectural design and hardware utilization. TokenMixer-Large extends TokenMixer with deeper-model and sparse-MoE improvements, scaling to billion-parameter models with reported online gains.

Existing scaling efforts expanded model size but often overlooked hardware-aligned co-design, producing suboptimal performance.
TokenMixer improves efficiency by replacing attention with lightweight token mixing and increasing Model FLOPs Utilization through hardware-aware co-design.
TokenMixer-Large addresses TokenMixer’s semantic misalignment, fragmented operators, deep-model gradient limitations, and incomplete MoE sparsification.
15B and 7B parameters were reached offline for Douyin ads and e-commerce, while online experiments reached 7B and 4B, respectively.
2% ADSS and 2.98% GMV gains were reported online across ByteDance scenarios serving hundreds of millions of users.

2 Related Work

Related recommendation architectures evolved from early feature-interaction models toward Transformer-based and lightweight token-mixing designs. This progression reflects efforts to improve interaction modeling while reducing computational cost and better using GPU hardware.

Classical models such as Wide and Deep, DeepFM, XDeepFM, PNN, DCN, DCNv2, and FCN model feature interactions but retain inefficient CPU-era operators.
Transformer-based models brought attention mechanisms to recommendation, while RankMixer and MLP-Mixer simplified attention into lightweight token mixing.
Token mixing reduces the quadratic computational cost associated with standard attention.

3 Methodology

TokenMixer-Large combines semantic tokenization, mixing-and-reverting blocks, inter-layer residuals, auxiliary loss, Sparse-Per-token MoE, and hardware-aware operators for scalable recommendation models.

Overall architecture: TokenMixer-Large converts sparse features into dense, dimension-aligned semantic tokens, adds a global token, and aggregates outputs by mean pooling for prediction.Semantic groups are compressed and aligned with separate DNN mappings before combining with the global token.
Mixing and reverting: Mixing-and-reverting restores token dimensional consistency, allowing residual connections to propagate across layers despite changing intermediate token dimensions.The two-layer structure mixes information among original tokens and then restores the mixed-token dimension.
Deep-network stability: Inter-layer residuals and auxiliary loss strengthen deep-network gradient and feature flow, while Pre-Norm with RMSNorm addresses numerical explosion and gradient instability.Inter-residual connections are typically inserted every 2 or 3 layers, excluding the final layer.
Sparse parameter expansion: Sparse-Per-token MoE splits per-token SwiGLUs into sparsely activated sub-experts, enabling sparse training and serving while activating only half the parameters with near-dense performance.The design first enlarges the model and then sparsifies the per-token SwiGLUs.
System efficiency: TokenMixer-Large uses fused high-performance operators, FP8 inference, and Token Parallel to reduce overhead and improve hardware utilization across training and serving.Token Parallel partitions per-token weights and computation across devices, while fused kernels reduce operator scheduling overhead.
System efficiency: 4-way Token Parallelism improves production serving throughput by 29.2% over a non-parallelized baseline, rising to 96.6% when communication overlaps computation.The reported experiment uses global batch size 320.

4 Experiments

Experiments evaluate TokenMixer-Large against SOTA baselines, analyze its residual and scaling designs, and ablate its principal modules. Results show stronger effectiveness, scaling behavior, and online gains across recommendation scenarios.

SOTA comparison: +1.14% AUC gain in CTCVR over the MLP-500M baseline, with TokenMixer-Large outperforming all compared SOTA models.Sparse-Pertoken MoE matches dense performance while activating only half the parameters.
Residual design: TokenMixer-Large satisfies standard residual, original token residual, and token semantic alignment properties while achieving the best performance against RankMixer.These properties address semantic mismatch and loss of original-token information across layers.
Scaling laws: AUC increases with parameters and FLOPs for both RankMixer and TokenMixer-Large, while TokenMixer-Large achieves the best performance and a steeper improvement slope.The comparison uses a logarithmic x-axis for parameters and FLOPs.
Scaling laws: Balanced expansion across width, depth, and SwiGLU scaling factor yields better returns beyond 1B parameters, while larger models require more training data for convergence.A 30M-to-90M model converges with 14 days of samples, whereas a 500M-to-2B model requires 60 days.
Ablation studies: Removing any TokenMixer-Large block module degrades performance, with mixing-and-reverting and Per-token SwiGLU having the largest effects.The ablated components include tokenization, mixing-and-reverting, Per-token SwiGLU, and residual connections.
Ablation studies: Sparse-Pertoken MoE components each improve performance, while shared expert, gate scaling, and down-matrix initialization add no extra parameters or FLOPs.Under matched total and activated parameters, standard MoE shows a significant performance drop relative to Sparse-Pertoken MoE.
Online performance: +1.66% in orders and +2.98% in per-capita preview payment GMV are achieved for e-commerce, alongside +2.0% ADSS in advertising and +1.4% revenue growth in live streaming.These online comparisons use RankMixer baselines for the respective scenarios.

5 Conclusion

TokenMixer-Large upgrades TokenMixer with deeper-layer modeling and Sparse-Pertoken MoE, supported by efficient operators and Token-Parallel execution. It reports offline and online gains across multiple ByteDance scenarios.

TokenMixer-Large analyzes limitations in TokenMixer and develops deep-layer modeling plus Sparse-Pertoken MoE for large-scale recommendation.The paper also implements efficient MoE operators and a Token-Parallel strategy for Pertoken experts.

A.1 First Enlarge Then Sparse

The paper uses a “First Enlarge, Then Sparse” strategy: build a strong dense model, refine its SwiGLU networks into fine-grained components, and sparsify activation. A 1:2 sparsity ratio retains nearly the dense model’s offline and online performance, while 1:4 causes a slight decrease.

The “First Enlarge, Then Sparse” strategy first designs a strong dense model, then sparsifies refined fine-grained SwiGLU networks.This approach seeks efficiency gains while minimizing performance loss.
1:2 sparsity achieves near-zero offline and online performance drop, while 1:4 sparsity causes a slight decrease.

A.2 Load Balance

The 1:2 Sparse-Pertoken MoE configuration maintains balanced expert utilization and matches a same-parameter dense model’s offline and online performance. Increasing sparsity from 1:2 to 1:8 deteriorates load balance, motivating future work.

1:2 sparsity matches the 4.6B dense model’s offline and online performance with the same total number of parameters.The example uses one shared expert and three routing experts, activating two experts per token.
The 1:2 configuration has relatively balanced overall expert load, while the always-active shared expert activates more frequently than routing experts.
1:2-to-1:8 sparsity increases deteriorate load balancing to some extent.Switch Transformer load balancing and Z-loss provide some benefits, but the 1:2 version has the highest ROI and no load-balancing problem in deployment.

A.3 Pure Model Design

TokenMixer-Large reduces reliance on fragmented feature operators as model scale grows, while its block reaches high hardware utilization. The supplied passages describe this operator effect and report an MFU of up to 60%.

DCN’s effect gain diminishes as the TokenMixer backbone becomes more complex.The baseline manually filters pooling results, feeds them into DCN, and uses the output as additional backbone tokens.
As TokenMixer-Large’s parameter scale expands, its own model can capture gains from small, numerous, IO-bound operators such as DCN, DHEN, and LHUC.Experiments considered both parallel and serial integrations of DHEN and LHUC.
60% MFU is achieved by the TokenMixer-Large block.

A.4 Normalization and Position

The paper replaces LayerNorm with RMSNorm and removes bias kernel matrices, reporting higher end-to-end throughput while preserving performance. It also studies normalization placement and presents the analysis in Table 10.

Normalization choice: RMSNorm replaces LayerNorm throughout TokenMixer-Large, while bias kernel matrices are removed because normalization makes them mathematically redundant.The change is described as inspired by Llama.
Efficiency: 8.4% higher end-to-end throughput was achieved while maintaining the same performance.
Normalization position: The study compares pre-norm, post-norm, and sandwich norm placements.The comparison is reported in Table 10.

A.5 Mixing Strategy

The paper studies token mixing, sparse MoE gate scaling, and initialization choices for deep TokenMixer-Large models. Results indicate that preserving complete token information, scaling router outputs with sparsity, and reducing FCdown initialization improve the design, while simply increasing expert-kernel variance does not.

Mixing Strategy: Different split-concat strategies preserve performance when each newly mixed token contains all original token information.Mixing only half of the original token information causes a significant performance drop.
Gate Value Scaling: Gate Value Scaling multiplies routed-expert contributions before combining them with a shared expert, strengthening gradient updates for selected experts.The formulation uses routing weights g(x), expert networks, and a shared expert network.
Gate Value Scaling: The optimal scaling factor α is inversely proportional to the model’s sparsity ratio, while values that are too large or too small can degrade performance.This relationship is reported from further experiments in Table 12.
Gate Value Scaling: Directly increasing expert-kernel initialization variance significantly degrades results instead of reproducing the benefit of Gate Value Scaling.The comparison is reported in Table 13.
Residual Design: Standard, interval, and residual-loss connections are combined with Rezero’s concept to address convergence challenges in deeper models.
Small Initialization: Reducing the final FCdown initialization scale to 0.01 makes F(x) + x closer to an early-training identity mapping and facilitates deep-model convergence.FCup and FCgate retain initialization scale 1; Table 14 finds FCdown-specific small initialization performs best.