TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

• TokenMixer-Large introduces Mixing & Reverting to redesign TokenMixer’s residual mechanism, and combines it with Pertoken-SwiGLU, residual normalization, interval residuals, and residual loss.The framework stacks TokenMixer-Large Blocks and includes tokenization and Sparse-Pertoken MoE.Source: §1 · p. 2
• Removing fragmented, memory-bound operators raises the advertising backbone’s MFU to 60%, while stacking TokenMixer Blocks subsumes their low-level benefits.The paper also says hardware-aware co-design improves hardware utilization.Source: §1 · pp. 1–2
• Interval residuals, an auxiliary loss, and down-matrix small initialization are introduced to improve convergence in deep models, where TokenMixer otherwise struggles with stable training.The paper identifies insufficient gradient updates as a limitation of deeper TokenMixer configurations.Source: §1 · p. 2
• Offline models reached 15B parameters for Douyin ads and 7B for e-commerce; online models reached 7B and 4B, with gains of 2% ADSS and 2.98% GMV.The paper says TokenMixer-Large is deployed across multiple online scenarios at ByteDance.Source: §1 · p. 2
• Sparse-Pertoken MoE adopts “Sparse Train, Sparse Infer” with efficient operators, FP8, and Token Parallel to reduce training and inference costs.This replaces RankMixer’s dense-train, sparse-infer ReLU-MoE approach, which did not reduce training costs and made inference expert counts difficult to predict.Source: §1 · p. 2

Related Work

• Wide and Deep [6] and later designs including DeepFM [13], XDeepFM [17], PNN [22], DCN [30], DCNv2 [31], and FCN [6] model low- and high-order feature interactions but retain inefficient, high-latency CPU-era operators.The passage frames these early-stage designs as underusing GPU computational power.Source: §2 · p. 2
• DHEN [37] and Wukong [36] validated recommendation scaling laws through a unified-design, stacked-layers iteration pattern.Source: §2 · p. 2
• Transformer [29] supports GPU-era versatility and scaling, while AutoInt [25] and HiFormer [12] use attention, RankMixer [40] and MLP-Mixer [27] use lighter token mixing, and LONGER [3], HSTU [35], and MTGR [14] explore sequential and generative recommendation.Token mixing reduces the quadratic computational cost of standard attention.Source: §2 · p. 2

3.1 Overall Architecture

• The framework tokenizes sparse features, uses mixing-and-reverting blocks with Sparse-Pertoken MoE, and mean-pools output tokens for task prediction.The architecture comprises tokenization, token and channel mixing, and sparse per-token MoE.Source: §3.1 · p. 2

3.2.1 Semantic Group-wise Tokenizer.

• Semantic groups are compressed and aligned using group-specific DNN mappings, while a global token aggregates global information and propagates it to other tokens.The final input combines the global token with enhanced tokens.Source: §3.2.1 · p. 3, §3.2.2 · p. 3

3.3.1 Mixing & Reverting.

• Each stacked block combines mixing, per-token SwiGLU, and normalization; mixing followed by reverting restores dimensions for residual connections and stable gradient flow at depth.TokenMixer-Large upgrades per-token FFN to per-token SwiGLU and replaces LayerNorm with RMSNorm and Post-Norm with Pre-Norm to address computational overhead and training instability.Source: §3.3 · p. 3, §3.3.1 · p. 3, §3.3.2 · p. 4

3.3.4 Inter-Residual & Auxiliary Loss.

• Inter-residual connections, typically added every two or three layers but not at the final layer, strengthen information flow; joint loss on lower- and higher-layer logits forms a lightweight auxiliary loss.These mechanisms address vanishing gradients and insufficient training of lower-layer parameters as depth increases.Source: §3.3.4 · p. 4

3.4 Sparse-Pertoken MoE

• Sparse-Pertoken MoE first enlarges per-token SwiGLUs, then sparsely activates sub-experts for sparse training and serving, with shared experts assigned per token and a constant added before router g(·).The constant addresses insufficient SwiGLU gradient updates caused by the router's sum-to-one probability constraint.Source: §3.4 · p. 4, §3.4.1 · p. 4, §3.4.2 · p. 4

3.4.4 Down-Matrix Small Init.

• The MoE version nearly matches a fully activated dense model while activating half the parameters, and the changes balance token loads without an added load-balancing loss.The final SwiGLU down-projection is initialized with reduced variance to keep the residual branch near zero early in training and improve stability.Source: §3.4.4 · pp. 4–5

3.5.1 High Performance custom operators.

• 1.7x speedup without compromising model accuracy is achieved by FP8 serving, with bfloat16 retained for training.Custom operators permute inputs into expert-first layout, compute expert FFNs with a single grouped kernel, and unpermute outputs; grouped FFN is the primary latency contributor.Source: §3.5.2 · p. 5, §3.5 · p. 5, §3.5.1 · p. 5

3.5.3 Token Parallel.

• 29.2% throughput improvement over a non-parallelized baseline is achieved with 4-way token parallelism at global batch size 320, rising to 96.6% with communication-computation overlap.Token Parallel reduces communication overhead from 4L to 2L + 1 for L layers by keeping block outputs sharded by token until a final all-to-all restores batch sharding.Source: §3.5.3 · p. 5

4.1 Experiment Settings.

• Experiments use sampled Douyin data and compare established recommendation baselines under hybrid distributed training, evaluating effectiveness with AUC/UAUC and efficiency with dense parameters, FLOPs, and MFU.The E-Commerce dataset has about 400 million records per day and over 500 features; training uses 64 GPUs for E-Commerce and 256 for Ads and Live Streaming, with Adagrad learning rates of 0.01 for dense and 0.05 for sparse parameters.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, §4.1.3 · p. 6

4.2 Comparison With SOTA Methods

• +1.14% AUC in CTCVR versus the MLP-500M baseline, with TokenMixer-Large outperforming all SOTA models; Sparse-Pertoken MoE matches its dense counterpart while activating half the parameters.Per-token designs including Wukong, Group Transformer, and RankMixer also use significantly lower FLOPs than other baselines at batch size 2048.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Detailed Comparison With Rankmixer(TokenMixer)

• TokenMixer-Large satisfies standard-residual, original-token-residual, and token-semantic-alignment properties, addressing RankMixer’s residual semantic mismatch and achieving the best performance in the comparison.RankMixer mixes tokens before adding them to residual inputs from before mixing, which breaks semantic alignment and can prevent original-token information from propagating effectively.Source: Table 3 · p. 6, §4.3 · pp. 6–7

4.4.1 Scaling Laws of SOTA Models.

• TokenMixer-Large achieves the best performance and a steeper improvement slope than RankMixer as parameters and FLOPs grow.Both models’ AUC increases with parameter and FLOP growth.Source: Figure 5 · p. 7, §4.4.1 · p. 7

4.4.2 Scaling Laws of TokenMixer-Large on Different Scenarios.

• Scaling beyond 1B parameters benefits from balanced increases in width, depth, and SwiGLU expansion, while larger models require more training data to converge.The paper reports convergence with 14 days of samples for 30M–90M models and 60 days for 500M–2B models; offline scales reached 15B, 7B, and 4B in Feed Ads, E-Commerce, and Live Streaming, versus online scales of 7B, 4B, and 2B, respectively.Source: §4.4.2 · p. 7, Table 4 · p. 7

4.5.1 Ablation Study for TokenMixer-Large Block.

• Removing any tested TokenMixer-Large block module degrades performance, with mixing and reverting and per-token SwiGLU having the largest impact.The model uses parameterless mixing-and-reverting operations and grouped GEMMs; the paper also reports studies of normalization placement and mixer splitting strategy.Source: §4.5.1 · pp. 7–8, Table 5 · p. 8

4.5.2 Ablation Study for Sparse-Pertoken MoE.

• Sparse-Pertoken MoE outperforms standard MoE when total and activated parameter counts are aligned, with token-specific expert routing avoiding early router-learning difficulties.The shared expert, Gate Value Scaling, and Down-Matrix Small Init each contribute positively without adding parameters or FLOPs. Optimal gate scaling is 2 at a 1:2 sparsity ratio and 4 at 1:4; initializing the final down matrix to 0.01 times the up/gate matrices improves performance and training stability.Source: §4.5.2 · p. 8, Table 6 · p. 8

4.6 Online Performance

• +1.66% orders and +2.98% per-capita preview payment GMV for e-commerce, +2.0% ADSS for advertising, and +1.4% revenue growth for live streaming.Online baselines were RankMixer-150M for e-commerce, RankMixer-1B for advertising, and RankMixer-500M for live streaming.Source: Table 7 · p. 8, §4.6 · p. 8

5 Conclusion

• TokenMixer-Large is an upgraded TokenMixer whose design analyzes the original architecture’s limitations and investigates deep-layer modeling and Sparse-Per-token MoE.Source: §5 · p. 8
• The engineering implementation includes efficient MoE operators and a Token-Parallel strategy designed specifically for per-token experts.Source: §5 · p. 8
• TokenMixer-Large delivers offline and online gains across multiple ByteDance scenarios, serving hundreds of millions of users.Source: §5 · p. 8

A.1 First Enlarge Then Sparse

• The approach first designs a dense model with the best performance, then sparsifies it to gain efficiency while minimizing performance degradation.All FC networks in TokenMixer-Large’s SwiGLU are refined into fine-grained versions before sparse activation.Source: §A.1 · p. 10
• Conventional MoE expansion keeps activated experts constant while increasing total parameters, which the authors equate to single-dimension widening of Pertoken SwiGLU’s hidden expansion.They argue that this widening has an upper limit that is easily reached.Source: §A.1 · p. 10
• 1:2 sparsity achieves near-zero offline and online drops, while 1:4 sparsity causes a slight decrease.The 1:2 version was selected for online deployment as the highest-ROI option.Source: §A.1 · p. 10
• Greater sparsity reduces the SwiGLU GEMM component, so further sparsity gains depend on a larger model and a higher GEMM ratio.Source: §A.1 · p. 10
• The appendix leaves open whether sparsity greater than 1:8 can maintain performance without loss; this remains under exploration.Source: §A.1 · p. 10
• Figure 6 illustrates the dense model and its 1:2- and 1:4-sparsity MoE forms.Source: §A.1 · p. 10

A.2 Load Balance

• At 1:2 sparsity, the e-commerce 4.6B Sparse-Pertoken MoE achieves the same offline/online performance as 4.6B dense with the same total parameter count.It uses one shared expert and three routing experts, activating two experts in total.Source: §A.2 · p. 10
• At 1:2, the overall load is relatively balanced, while the always-active shared expert (index 3) has higher activation than routing experts (indices 0–2).Figure 7 illustrates token activation for the 1:2 and 1:8 versions.Source: §A.2 · p. 10, Figure 7 · p. 10
• As sparsity rises from 1:2 to 1:8, load balancing deteriorates to some extent.The authors report that Switch Transformer [11] and Z-loss [41] provide some benefits, but defer further exploration because the launched 1:2 version has the highest ROI and no load-balancing problem.Source: §A.2 · p. 10, Figure 7 · p. 10

• +0.09% gain is reported for DCN on a 150M-parameter model, while removing DCN from a 700M-parameter model causes no effect loss.The baseline manually pools sequence results into DCN and adds its output as tokens to TokenMixer.Source: Table 9 · p. 10, §A.3 · p. 10
• As TokenMixer-Large scales, gains from integrating DHEN and LHUC can be fully captured by the model itself, including gains from small, numerous, IO-bound operators.The experiments tested both parallel and serial configurations.Source: §A.3 · pp. 10–11
• TokenMixer-Large Block achieves up to 60% Model FLOPs Utilization (MFU).Source: §A.3 · p. 11

A.4 Normalization and Position

• 8.4% higher end-to-end throughput with the same performance followed replacing LayerNorm with RMSNorm and removing bias kernel matrices.The authors note that normalization makes the bias of the kernel matrix mathematically redundant.Source: §A.4 · p. 11
• Post-norm delivers better effectiveness but risks gradient explosion, whereas pre-norm is less effective but ensures stable training.Source: §A.4 · p. 11

A.5 Mixing Strategy

• Different split-concat strategies do not affect performance when each newly mixed token contains all original token information, but mixing only half causes a significant drop.Source: §A.5 · p. 11

A.6 Gate Value Scaling Analysis

• Gate Value Scaling places a constant factor α before the router function g(x), amplifying router output to enhance gradient updates for selected experts.Source: §A.6 · p. 11
• The optimal α is closely related to model sparsity: setting it inversely proportional to the sparsity ratio yields the best results.Source: §A.6 · p. 11
• Reciprocal sparsity scaling compensates for less frequent expert updates as SwiGLU kernels are split more finely, without increasing activated parameters (topk), to keep FC-layer gradients consistent with the dense model.Source: §A.6 · p. 11

A.7 Down-Matrix Small Initialization

• Small initialization specifically on FCdown yields the best performance across tested scales and layer positions.Source: Table 14 · p. 12, §A.7 · p. 12

A.6 Gate Value Scaling Analysis

• Directly increasing Expert-kernel initial variance did not improve performance and instead significantly degraded results, unlike Gate Value Scaling.Source: §A.6 · p. 11, Table 13 · p. 12

A.7 Down-Matrix Small Initialization

• Reducing FCdown initialization to 0.01 while keeping FCup and FCgate at 1 makes F(x) + x more identity-like early in training and constrains intermediate activation and gradient magnitudes.Source: Figure 8 · p. 12, §A.7 · p. 12
• To address deep-model convergence, the method combines standard residual connections, interval residual connections, and residual loss with the ReZero concept [2].Source: §A.7 · p. 12