Source-linked AI summary
Task-Blind No MORE: Multi-Task Information Flow in Unified Ranking Backbones
Yuchen Wang, Feng Niu, Qing Tan, Junting Lu, Baoxin Wu, Jun Gao
TL;DR
Unified ranking backbones combine sequence and feature modeling, but leave multi-task information flow in post-backbone towers. MORE carries shared and task-specific anchors through the backbone; it leads both two-stage and unified baselines across seven tasks and is deployed with request-level scoring latency reduced by about 30%.
Problem
Unified ranking backbones keep multi-task modeling in post-backbone heads, so their internal sequence reading and feature interaction do not account for task-specific preferences.
Source: §1 · pp. 1–2Method
Results
MORE leads all seven tasks against two-stage and unified baselines and converts scaling from 0.1B to 0.9B parameters into multi-task GAUC gains more efficiently.
Source: §5 · p. 7Takeaways & Limitations
Request-level shared computation reduces P99 latency about 30%, and MORE is deployed in production on Momo.
Source: §3.5 · p. 5, §5 · p. 7
Abstract
from arXiv · showhide
Industrial ranking models for recommendation have scaled feature interaction and sequence modeling separately; recent architectures such as HyFormer and MixFormer unify both in a stackable backbone. Real-world recommender systems, however, nearly always require multi-task learning, yet existing unified architectures confine multi-task modeling to shallow post-backbone towers, leaving the backbone without task-aware information flow. We propose MORE (Multi-task cO-evolving Ranking modEl), which embeds multi-task information flow inside the backbone, enabling task-specific signals to co-evolve with sequence and feature representations at every layer rather than in a post-hoc fusion. It introduces Anchor Tokens that persist across backbone layers: Shared Anchors encode cross-task commonalities, while Private Anchors capture task-specific priors. In each block, Anchor Tokens (1) read task-conditioned signals from behavior sequences, (2) mix with non-sequential features under a task-boundary mask, and (3) refine per-task representations through independent branches; as blocks stack, each task obtains a differentiated representation refined through all backbone layers. Experiments on large-scale industrial datasets show that MORE consistently outperforms baselines across all tasks under comparable parameter and FLOPs budgets, and scales well with model size. Online A/B tests on Momo, a leading Chinese social discovery platform with tens of millions of monthly active users, yield 3% improvement in usage duration, 3.6% in interaction rate, and 2% in deep-chat rate. MORE is deployed in production with request-level shared computation reducing scoring latency by about 30%.
1 Introduction
Unified ranking backbones combine sequence modeling and feature interaction, but keep multi-task learning in post-backbone heads. MORE moves task-aware information flow into the backbone, where task signals co-evolve with sequence and feature representations.
1 Introduction
- Industrial ranking methods scaled feature interaction and behavior sequences separately before unified backbones placed both in one stackable architecture.The cited unified examples are OneTrans [30], HyFormer [11], and MixFormer [10].Source: §1 · p. 1
- Task-blind backbones cannot condition sequence reading or feature aggregation on each task, and post-backbone towers route an already-compressed shared representation.The introduction characterizes this as a structural bottleneck rather than a routing-capacity problem.Source: §1 · p. 2
- MORE moves multi-task modeling into the backbone through persistent Shared and Private Anchors that co-evolve with sequence and feature representations.Shared Anchors encode cross-task semantics, while Private Anchors encode task-specific priors.Source: §1 · p. 2
- MORE gains positive GAUC on all seven tasks under comparable FLOPs budgets, scales more efficiently, and improves online engagement on a social platform.The authors report that MORE is deployed in production.Source: §1 · p. 2
2 Related Work
Prior work scaled feature interaction and sequence modeling along separate routes, then unified them in shared backbones; task differentiation, however, remained outside those backbones. MORE's overview depicts a task-aware alternative built from persistent anchors and three block modules.
2 Related Work
- Figure 2 presents MORE as Anchor Token Construction followed by stacked blocks and task-specific prediction heads, with sequence reading, semantic mixing, and task enhancement in each block.The caption assigns sequence modeling, shared semantics and task-boundary awareness, and Private-Anchor discrimination to those modules, respectively.Source: Figure 2 · p. 3
- Earlier feature-interaction and sequence-modeling approaches remained decoupled, while multi-task learning stayed in post-backbone prediction heads.The passage describes feature-crossing methods and long-history sequence models as separate routes.Source: §2 · pp. 2–3
- Unified backbones combine sequence modeling and feature interaction, but their internal sequence reading and feature interaction remain task-agnostic.HyFormer [11], MixFormer [10], and OneTrans [30] are cited as unified designs.Source: §2 · p. 3
- MMoE, PLE, AITM, and PEPNet differentiate tasks in post-backbone towers, while cited early-layer attempts [14] leave task-specific sequence updates and explicit isolation unaddressed.Source: §2 · p. 3
3.1 Problem Statement
The problem statement defines ranking inputs as user behavior histories and non-sequential features, with scores required for multiple tasks. Model parameters are trained with a weighted binary cross-entropy objective.
3.1 Problem Statement
- For a user, the model receives a behavior-history sequence and non-sequential features comprising user profiles, candidate attributes, and context.Source: §3.1 · p. 3
- The ranking model must output prediction scores for T tasks, with click, like, comment, greet, and chat given as examples.Source: §3.1 · p. 3
- Training minimizes weighted binary cross-entropy, using a task-specific loss weight λ_t.Source: §3.1 · p. 3
3.2 Framework Overview
MORE stacks blocks that update Anchor Tokens through task-aware sequence reading, selective semantic mixing, and task enhancement. Task-specific towers use the resulting private and shared representations, while request-level shared computation supports deployment efficiency.
3.2 Framework Overview
- Each MORE block sequentially performs cross-attention sequence reading, masked semantic mixing between Anchor Tokens and F, and independent task-specific enhancement of Private Anchors.Source: §3.2 · p. 3
- After L blocks, each task tower receives its corresponding Private Anchor together with shared semantics from Shared Anchors and non-sequential tokens.Source: §3.2 · p. 3
- MORE uses request-level shared computation to maintain industrial deployment efficiency.Source: §3.2 · p. 4
3.3 Input Representation
MORE represents behavior histories and candidate-specific non-sequential features as tokens in a shared space, then derives persistent Anchor Tokens to carry multi-task information through the backbone. Behavior tokens distinguish combinations of behavior types, while Shared and Private Anchors encode cross-task and task-specific information, respectively.
3.3 Input Representation
- Behavior tokens encode each item’s behavior-type combination as a distinct token, helping preserve per-task structure for later sequence reading.Each token combines item, action-combination, and temporal embeddings.Source: §3.3 · p. 4
- MORE embeds user profiles, candidate attributes, and context into non-sequential tokens that supply semantics to Anchor Tokens during mixing and task enhancement.Source: §3.3 · p. 4
- Anchor Tokens are initialized from a mean-pooled behavior summary and flattened non-sequential semantics, projected and split by an MLP.Source: §3.3 · p. 4
- Shared Anchors represent cross-task semantics, while one Private Anchor per task represents task-specific semantics.A learnable task-prior embedding is added to strengthen task discrimination.Source: §3.3 · p. 4
3.4 MORE Block
MORE blocks update behavior sequences and persistent Anchor Tokens through task-aware sequence reading, selective mixing with non-sequential features, and task-specific enhancement. Shared information can flow across tokens, while task boundaries keep Private Anchors isolated from one another.
3.4.1 Task-Aware Sequence Reading.
- Shared and Private Anchors read the shared behavior sequence for general and task-relevant signals, respectively, with a sequence register carrying compressed sequence summaries across blocks.Independent gates let each anchor extract signals suited to its role.Source: §3.4.1 · p. 4
3.4.2 Selective Semantic Mixing.
- Selective Semantic Mixing enriches sequence-aware anchors with non-sequential semantics while a task-boundary mask makes each Private Anchor visible only to itself.F and Shared Anchors remain visible to all tokens.Source: §3.4.2 · p. 4
3.4.3 Task Enhancing.
- Independent per-task MLP branches inject discriminative information from non-sequential features into each Private Anchor through FiLM-style affine modulation and a residual connection.Source: §3.4.3 · p. 5
- Stacked blocks recurrently update hidden states, and each task tower receives its Private Anchor alongside shared semantics from F and Shared Anchors.This combines cross-task context with task-specific prediction representations.Source: §3.4.4 · p. 5
3.5 Request-Level Training and Inference
MORE groups candidates from one request so user-side sequence computation can be shared across them. The sequence state is candidate-independent, allowing request-level shared computation without modifying the backbone and reducing online scoring latency by about 30%.
3.5 Request-Level Training and Inference
- Request-level shared computation encodes the user-side sequence once and broadcasts it, while per-candidate features and Anchor Tokens are organized along the item dimension.At inference, candidates are split into mini-batches that share user-side computation.Source: §3.5 · p. 5
- MORE’s sequence state evolves independently of candidate information, while candidate-specific differences reside in F and Anchor Tokens.Source: §3.5 · p. 5
- Request-level shared computation reduces online scoring latency by about 30% without modifying the backbone.Source: §3.5 · p. 5
4 Experiment
MORE leads all seven offline tasks while maintaining a favorable accuracy-efficiency trade-off. Ablations highlight the contributions of its token-processing components, task-boundary control, and task-enhancing branches.
4.1 Experimental Setup
- The offline evaluation uses 90 days of Nearby Feed logs for training and the immediately following day for evaluation.Inputs include recent multi-type behavior histories, user profiles, candidate content attributes, and scenario context.Source: §4.1.1 · p. 5
4.2 Overall Performance
- MORE leads all tasks in Table 1, including Greet (+1.58%), Like (+1.47%), and Deep Chat (+0.33%).The gains are especially strong on interaction and social tasks where behavior-sequence signals require task-specific reading.Source: §4.2 · p. 6, Table 1 · p. 6
- MORE uses 132M parameters and 1.85G per-example FLOPs, below HyFormer’s 153M parameters and the reference baseline’s 1.92G FLOPs.It adds 0.12G FLOPs over OneTrans (1.73G) while outperforming it on all tasks.Source: §4.2 · p. 6
4.3 Ablation Study
- Removing either Token Process component produces consistent GAUC drops of 0.04%–0.10% across reported tasks.The components enable the model to acquire multi-task awareness from the initial stage.Source: §4.3 · p. 6
- Removing Task Enhancing reduces GAUC by 0.11%–0.17% across tasks through loss of discriminative information injected into Private Anchors.The component uses lightweight independent per-task branches.Source: §4.3 · p. 6
- Boundary Gating Mask removal causes the largest single-component average degradation, a 0.34% drop on Interaction.Without the mask, high-frequency task gradients dominate shared mixing weights; task-boundary control matters more than reading or enhancing capacity alone.Source: §4.3 · pp. 6–7
- Removing all six components yields task-agnostic BaseArch and decreases average GAUC by 0.33%.BaseArch retains multiple prediction heads for multi-task scoring.Source: §4.3 · p. 7
4.4 Scaling Analysis
MORE remains above a task-agnostic unified backbone at comparable parameter and compute budgets, with its advantage widening as capacity grows. In Momo Nearby Feed, online testing reports gains across six metrics and approximately 30% lower P99 scoring latency than per-candidate serving.
4.4 Scaling Analysis
- MORE scales more effectively across parameters and compute, remaining above the task-agnostic unified backbone at comparable budgets.The approaches yield comparable GAUC at the smallest configuration, but the gap widens as capacity grows.Source: §4.4 · p. 7
4.5 Online A/B Test
- MORE achieves positive improvements across all six online metrics, including usage duration (+3.0%), comment (+3.6%), and deep chat (+2.0%).The two-week test compares MORE with the production ranking model in Momo Nearby Feed.Source: §4.5 · p. 7, Table 3 · p. 7
- Request-level shared computation reduces P99 scoring latency by approximately 30% versus per-candidate serving.Sharing user-side sequence computation across candidates within each mini-batch amortizes the per-candidate backbone cost.Source: §4.5 · p. 7
5 Conclusion
MORE embeds task-aware sequence reading, semantic mixing, and per-task enhancement throughout the backbone. It leads both two-stage and unified baselines on all seven tasks, with reported online gains and about 30% lower P99 latency from request-level shared computation.
5 Conclusion
- MORE embeds task-aware sequence reading, selective semantic mixing under a task-boundary mask, and per-task enhancement in every backbone block.Shared and Private Anchor Tokens persist across blocks, carrying cross-task semantics and per-task priors.Source: §5 · p. 7
- MORE leads both two-stage and unified baselines on all seven tasks and converts added capacity into multi-task GAUC gains more efficiently when scaling from 0.1B to 0.9B parameters.The reported scaling comparison is against task-agnostic alternatives.Source: §5 · p. 7
- Online A/B tests report consistent improvements from browsing to deep social engagement, while request-level shared computation lowers P99 latency about 30%.MORE is deployed in production on Momo.Source: §5 · p. 7