PaperlayerView exact revision on arXiv

Source-linked AI summary

RankMixer: Scaling Up Ranking Models in Industrial Recommenders

Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, Zuotao Liu

arXiv:2507.15551v3cs.IR

TL;DR

Industrial recommendation scaling must improve model capacity without violating tight latency and QPS constraints or relying on GPU-inefficient feature-crossing modules. RankMixer addresses this with hardware-aware token mixing, per-token FFNs, and scalable routing, achieving broad production-scale gains. Its deployment demonstrates substantially larger models at similar serving cost and improved user engagement metrics.

  • Problem

    Industrial recommendation systems need larger models but must satisfy strict latency and high-QPS requirements while overcoming low-MFU CPU-era feature-crossing designs.

  • Method

    RankMixer uses multi-head token mixing, per-token FFNs, and dynamic routing for a hardware-aware, parallel feature-interaction architecture with Sparse-MoE scaling.

  • Results

    RankMixer scales ranking-model parameters by over 100× while maintaining shorter inference latency than the previous baseline in Douyin production deployment.

  • Takeaways & Limitations

    Full-traffic deployment increased active days by 0.3% and App duration by 1.08% without increasing inference cost.

Abstract

from arXiv · show

Recent progress on large language models (LLMs) has spurred interest in scaling up recommendation systems, yet two practical obstacles remain. First, training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands. Second, most human-designed feature-crossing modules in ranking models were inherited from the CPU era and fail to exploit modern GPUs, resulting in low Model Flops Utilization (MFU) and poor scalability. We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture. RankMixer retains the transformer's high parallelism while replacing quadratic self-attention with multi-head token mixing module for higher efficiency. Besides, RankMixer maintains both the modeling for distinct feature subspaces and cross-feature-space interactions with Per-token FFNs. We further extend it to one billion parameters with a Sparse-MoE variant for higher ROI. A dynamic routing strategy is adapted to address the inadequacy and imbalance of experts training. Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset. By replacing previously diverse handcrafted low-MFU modules with RankMixer, we boost the model MFU from 4.5\% to 45\%, and scale our ranking model parameters by 100x while maintaining roughly the same inference latency. We verify RankMixer's universality with online A/B tests across two core application scenarios (Recommendation and Advertisement). Finally, we launch 1B Dense-Parameters RankMixer for full traffic serving without increasing the serving cost, which improves user active days by 0.3\% and total in-app usage duration by 1.08\%.

1 Introduction

Scaling recommendation models requires architectures that balance larger capacity with strict industrial latency and throughput constraints. RankMixer addresses this need with hardware-aware, highly parallel feature interaction components and demonstrates large-scale production gains.

  • Industrial recommendation systems must balance model effectiveness against tight latency constraints and extremely high QPS demands.
  • CPU-era ranking architectures often use heterogeneous handcrafted cross-feature modules that achieve poor GPU parallelism and single-digit MFU.
  • Hardware-aligned designs should maximize GPU MFU and throughput while preserving heterogeneous feature spaces and personalized cross-feature interactions.
  • RankMixer combines multi-head token mixing with per-token FFNs to efficiently model heterogeneous and cross-feature interactions.Its design follows a hardware-aware model-design philosophy and uses dynamic routing to improve Sparse-MoE scalability.
  • Over 100× parameter scaling was achieved with shorter inference latency than the previous baseline in Douyin’s production recommendation system.The architecture decouples parameter growth from FLOPs and FLOPs growth from actual cost through high MFU and engineering optimization.
  • RankMixer was deployed for full-traffic Douyin Feed Recommendation, increasing active days by 0.3% and App duration by 1.08%.

2 Related Work

Related work develops increasingly explicit and automated feature-interaction architectures for recommendation, while scaling-law research examines how model and data scale affect performance across deep-learning domains.

  • DLRMs model feature interactions with neural networks over input embeddings, making dense interaction layers central to recommendation performance.
  • Recommendation models include explicit cross operators, attention-based architectures, and systems that combine multiple interaction operators, often increasing latency and memory use.
  • Scaling-law research studies power-law relationships between performance and factors such as model size, data volume, and computational capacity across NLP, vision, multimodality, and recommendation.

3 Methodology

RankMixer tokenizes heterogeneous feature embeddings and repeatedly processes them with multi-head Token Mixing and per-token FFNs. Sparse-MoE routing further increases capacity while targeting roughly constant computation cost.

  • Overall architecture: RankMixer tokenizes the input into T feature tokens and refines their representations through L successive blocks before mean pooling.Each token represents a coherent feature vector, and the pooled final representation is used for task predictions.
  • Multi-head Token Mixing: Multi-head Token Mixing splits each token into heads and recombines corresponding parts across tokens for global feature interactions without self-attention’s extra computation.The design preserves token count for residual connections and is motivated by heterogeneous recommendation feature spaces.
  • Input layer and feature tokenization: Semantic tokenization groups related features, concatenates their embeddings, and partitions them into fixed-dimension tokens.This balances distinct feature-space representation against the under-utilization caused by too many tokens or dominance caused by too few.
  • Per-token FFN: Per-token FFNs assign dedicated transformations to tokens, expanding parameters while keeping computational complexity unchanged and supporting separate feature-subspace modeling.Unlike shared FFNs or MMoE experts, RankMixer splits inputs and parameters simultaneously.
  • Sparse MoE in RankMixer: Sparse-MoE replaces dense per-token FFNs so capacity grows at roughly constant computation, using ReLU routing and dual-router dense-training/sparse-inference training.ReLU routing gives high-information tokens more experts, while the inference router is regularized toward the sparsity budget.

4 Experiments

Experiments use trillion-scale Douyin recommendation logs with hundreds of embedded feature types, standardized training settings, and AUC, UAUC, parameter, FLOPs, and MFU metrics.

  • Datasets and Environment: The offline dataset contains over 300 numerical, ID, cross, and sequential features from trillions of daily records.The features involve billions of user IDs and hundreds of millions of video IDs, converted into embeddings.
  • Evaluation Metrics: Evaluation uses AUC and UAUC for finish or skip prediction, while parameter count, FLOPs, and MFU measure efficiency.An AUC increase of 0.0001 is regarded as confidently significant.
  • Baselines: RankMixer is compared with DLRM-MLP, DCNv2, RDCN, MoE, AutoInt, Hiformer, DHEN, and Wukong.These baselines represent vanilla MLP, explicit feature-cross, attention, MoE, and hybrid interaction designs.
  • Datasets and Environment: Training uses hundreds of GPUs with asynchronous sparse-part updates and synchronous dense-part updates.RMSProp is used for dense parameters and Adagrad for sparse parameters, with consistent optimizer settings across models.

4.2 Comparison with SOTA methods

At approximately 100 million parameters, RankMixer outperforms established recommendation ranking architectures while maintaining comparatively moderate computational requirements.

  • Performance Comparison: RankMixer significantly outperforms other state-of-the-art models across multiple objectives and metrics.The comparison is summarized in Table 1 for approximately 100 million-parameter recommendation models.
  • Performance Comparison: Simply scaling DLRM to 100 million parameters yields only limited gains, motivating architectures tailored to recommendation data.Classic cross-structure models show an imbalance between parameter size and computational cost.
  • Efficiency Comparison: RankMixer achieves the best performance while retaining relatively moderate FLOPs among models scaled to 100 million parameters.This reflects a balance between model capacity and computational load.
  • Performance Comparison: Under similar parameter settings, RankMixer performs better than Hiformer and Wukong while requiring fewer computations.The comparison covers commonly used state-of-the-art scaling-up models.

4.3 Scaling Laws of different models

RankMixer exhibits the steepest scaling curves with respect to both parameter count and FLOPs, and its quality depends primarily on total parameters across scaling directions.

  • Scaling Laws: RankMixer shows the steepest scaling law for both parameters and FLOPs and remains consistently superior to other models.Figure 2 presents scaling curves using parameter size and FLOPs.
  • Scaling Laws: Wukong’s computational cost rises faster than its parameter curve, widening its AUC-versus-FLOPs gap relative to RankMixer and Hiformer.Hiformer is slightly inferior to RankMixer, while DHEN shows limited scalability.
  • Scaling Directions: RankMixer quality correlates primarily with total parameters, while increasing depth, width, or feature tokens yields almost identical performance.Larger hidden dimensions achieve higher MFU than stacking more layers because they create larger matrix-multiplication shapes.

4.4 Ablation Study

Ablations show that token mixing, residual connections, normalization, and feature-subspace routing are central to RankMixer performance; dense-training with ReLU routing preserves sparse-model accuracy.

  • Component Ablation: Removing Multi-Head Token-Mixing, residual connections, or LayerNorm significantly decreases RankMixer-100M performance.Token mixing supplies global information, while residual connections and LayerNorm improve training stability.
  • Routing Ablation: All-Concat-MLP and All-Share routing underperform Multi-Head Token-Mixing, highlighting the value of local feature-subspace modeling.All-Share uses the same entire input vector for every per-token FFN, while All-Concat-MLP processes concatenated tokens through a large MLP.
  • Sparse-MoE Ablation: Dense-training with ReLU-routed SMoE preserves almost all accuracy of the 1 B dense model as the activated expert ratio decreases.The Figure 3 comparison considers activation ratios of 1, 1/2, 1/4, and 1/8 of experts.
  • Routing Ablation: Self-Attention performs slightly worse than Multi-Head Token-Mixing and incurs higher computational cost for routing across heterogeneous feature spaces.The comparison indicates difficulty learning similarity across hundreds of distinct feature subspaces.

4.5 Sparse-MoE Scalability and Expert Balance

Sparse-MoE scaling preserves accuracy and improves throughput under aggressive sparsity when dense training, sparse inference, and ReLU routing are combined. This configuration also addresses expert imbalance and supports future larger deployments.

  • Scalability: > 8× parameter capacity and memory footprint with nearly no AUC loss under DTSI plus ReLU routing.The configuration also provides +50% inference throughput improvement over the comparison setting.
  • Scalability: Vanilla SMoE degrades monotonically as fewer experts are activated, reflecting expert-imbalance and under-training issues.
  • Scalability: Load-balancing loss reduces vanilla SMoE degradation but remains worse than DTSI plus ReLU because the main issue lies in expert training rather than routing.
  • Expert balance and diversity: Dense training gives most experts sufficient gradient updates, preventing expert starvation and dying experts.

4.6 Online Serving cost

RankMixer maintains stable inference latency despite a roughly 70× parameter increase by reducing FLOPs per parameter, increasing MFU, and using half-precision hardware efficiently.

  • Online Serving cost: 70× more parameters than the 16M-parameter baseline remained at stable inference latency through hardware-aligned design and optimization.
  • Online Serving cost: Latency is modeled as #Param × FLOPs/Param ratio ÷ MFU × Theoretical Hardware FLOPs.
  • Online Serving cost: A two-order-of-magnitude parameter increase is counteracted by a 3.6× lower FLOPs/Param ratio, 10× higher MFU, and 2× hardware FLOPs from quantization.
  • Online Serving cost: RankMixer achieves a 70-fold parameter increase with only around a 20-fold FLOPs increase, yielding a 3.6× efficiency gain in FLOPs/Param ratio.
  • Online Serving cost: Large GEMM shapes, parallel topology, fused per-token FFNs, and reduced memory-bandwidth overhead raise MFU and shift RankMixer from memory-bound to compute-bound.
  • Online Serving cost: Half-precision inference doubles theoretical peak GPU hardware FLOPs, benefiting RankMixer’s matrix-multiplication-heavy computation.

4.7 Online Performance

RankMixer was tested in feed recommendation and advertising, where online A/B experiments reported statistically significant business-metric uplifts and broad generalization across application scenarios.

  • Online Performance: RankMixer was evaluated online in feed recommendation and advertising, covering two core personalized-ranking scenarios.
  • Online Performance: The experiments monitored Active Days, app Duration, finish/like/comment behavior, advertising ΔAUC, and ADVV revenue.
  • Online Performance: 0.7% AUC improvement was observed when RankMixer-1B replaced the dense part of the 16M-parameter DLRM-plus-DCN baseline.
  • Online Performance: RankMixer delivered statistically significant uplifts across all reported business-critical metrics in the evaluated personalized-ranking applications.
  • Online Performance: Low-active users achieved over 1.7412% Active Days improvement, indicating strong generalization across user-activeness groups.

5 Conclusion

RankMixer combines heterogeneous feature-interaction modeling with a highly parallelizable serving architecture and was fully deployed on Douyin Feed ranking. The deployment increased active days and app duration.

  • 5 Conclusion: RankMixer was fully deployed on Douyin Feed ranking with heterogeneous feature-interaction designs and a highly parallelizable architecture.
  • 5 Conclusion: 0.3% increase in active days and 1% increase in App duration followed full deployment on the Douyin app.
Loading 2507.15551v3…