Source-linked AI summary
Unified Heterogeneous Feature Interaction via Effective-Rank Scaling for Recommender Systems
Ao Zhang, Haochuan Li, Boya Du, Jian Yang, Qihang Fang, Yulin Xu
TL;DR
Prior ranking architectures often separate sequence encoding from feature interaction, limiting direct access to event-level states, while multi-scenario CVR modeling must transfer patterns and adapt to the target context. HiRank unifies behavioral modeling, long-term retrieval, and heterogeneous feature interaction with scenario-aware composite tokens. It achieves the highest AUC and GAUC on both reported traffic sources, while online Orders/u increases 2.27% versus the production baseline.
Problem
Prior ranking systems often encode behavior sequences before feature interaction, leaving downstream modules without direct access to event-level states, while CVR relevance varies by candidate and scenario.
Source: §1 · pp. 1–2Method
HiRank unifies main-sequence modeling, long-term memory retrieval, and heterogeneous feature interaction, then uses scenario-aware aggregation and composite projections for high-order interaction.
Source: §1 · p. 2Results
HiRank achieved the highest AUC and GAUC on Homepage and Main Search, while online Orders/u increased 2.27% versus the production baseline.
Source: §4.2 · p. 7, §4.5 · p. 8Takeaways & Limitations
HiRank was deployed in Taobao’s recommendation system, which serves hundreds of millions of daily active users.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Large-scale industrial ranking models must jointly capture heterogeneous feature interactions, long-term user behaviors, and multi-scenario dependencies under stringent online serving constraints. Existing architectures typically process sequential and non-sequential features with separate modules, limiting bidirectional information exchange. Directly applying homogeneous self-attention to all semantic tokens can also be inefficient when sequential and non-sequential signals exhibit different effective-ranks (eRanks). Meanwhile, shared representations can overlook scenario-specific patterns and induce negative transfer between scenarios. In this paper, we propose HiRank, a unified framework for multi-scenario recommendation that jointly models Heterogeneous feature Interactions via effective-Rank scaling. HiRank integrates gated sequential attention with heterogeneous feature interaction within a unified backbone, enabling bidirectional fusion between sequential and non-sequential representations. HiRank further uses scenario-aware aggregation to compress heterogeneous low-order tokens into a compact representation. Distinct learned projections transform the full compact representation into scenario-aware composite tokens with complementary global feature views and more uniform information density, enabling more effective high-order self-attention. Extensive experiments on industrial-scale datasets demonstrate favorable performance scaling with parameter count and consistent improvements over competitive baselines. Online A/B tests further show a 2.27% increase in orders per user.
1 Introduction
HiRank addresses the separation between behavioral sequence modeling and feature interaction by uniting them with long-term retrieval in a multi-scenario framework. Scenario-aware aggregation and composite projections support high-order interaction, and production deployment showed statistically significant increases in orders and GMV per user.
Motivation
- Prior ranking research separates behavioral sequence modeling from feature interaction, often compressing history before interaction and thereby limiting access to event-level states in the same forward pass.Sequence modeling extracts candidate-relevant interests, while feature interaction learns dependencies across user, item, and context fields.Source: §1 · p. 1
HiRank
- HiRank unifies main-sequence modeling, long-term memory retrieval, and heterogeneous feature interaction in a framework for multi-scenario recommendation.It retains explicit main-sequence states, selected long-term sequences as memories, and semantic tokens for non-sequential features and pooled behavior summaries.Source: §1 · p. 2
- Behavior-dependent queries attend to the main sequence before retrieving long-term evidence, complemented by candidate-conditioned retrieval from the candidate’s perspective.The two pathways provide complementary behavioral evidence for joint modeling with main-sequence states and semantic feature tokens.Source: §1 · p. 2
- Scenario-aware aggregation and distinct learned projections form multiple composite tokens from the complete compact representation for subsequent attention-based interaction.Each token can access the full aggregated context while learning a distinct projection, targeting complementary representation directions and a more balanced spectrum.Source: §1 · p. 2
Contributions and deployment
- Production deployment showed statistically significant increases in orders per user and gross merchandise value per user.The system was deployed in Taobao’s recommendation system, serving hundreds of millions of daily active users.Source: §1 · p. 2
2 Related Work
Prior work has explored architectures that jointly process sequences and non-sequential features, as well as methods for adapting shared models across scenarios. HiRank’s related-work positioning connects scenario-dependent evidence selection with a shared interaction backbone.
2.1 Unified Sequence and Non-Sequence Modeling
- Feature-interaction models emphasize heterogeneous-token operators, while sequence-centric designs emphasize event-level processing; joint modeling must coordinate both representations throughout the backbone.The cited sequence-centric examples include HSTU and ULTRA-HSTU, with the latter exploring semi-local attention and separate transducers for behavioral streams.Source: §2.1 · p. 2
- Unified sequence and non-sequence models use separate streams with bidirectional exchanges, common token streams, or query-mediated interaction; HiRank instead assigns distinct roles to evolving main-sequence states and long-term memories.InterFormer exchanges selected summaries across streams, OneTrans uses a common causal stream, and HyFormer and MixFormer use evolving query representations to revisit behavioral evidence.Source: §2.1 · p. 2
2.2 Multi-Scenario Modeling
- Multi-domain and multi-task methods balance shared patterns with domain- or task-specific representations, including expert mixtures, parameter sharing, and progressive separation.Examples include MMoE, PLE, and STAR, which respectively use task-specific expert combinations, separate shared and task-specific representations, or combine central and domain-specific parameters.Source: §2.2 · p. 3
- HiRank uses scenario-aware queries to select evidence from shared behavioral and non-sequential tokens, then transforms the aggregate into composite tokens before task-specific prediction.This connects scenario-dependent evidence selection with shared feature-token interaction in the ranking backbone.Source: §2.2 · p. 3
3 Methodology
The methodology formulates industrial e-commerce ranking as multi-scenario, multi-objective prediction from user, candidate, context, and behavioral-history inputs. Training across Taobao traffic sources requires capturing shared behavior while adapting to scenario-dependent feature relevance.
Problem Formulation
- The ranking setting jointly predicts multiple conversion-related outcomes, including purchase conversion and add-to-cart, for candidate items in an industrial e-commerce recommender.The task set is denoted T = {cvr, cart, ...}.Source: §3 · p. 3
- For a user and candidate item, non-sequential feature blocks describe user, item, and request context, while a separate input represents the user’s historical behavior sequences.Source: §3 · p. 3
- Samples come from multiple Taobao traffic sources, so the model must capture shared behavioral patterns while adapting to scenario-dependent feature relevance.Figure 1 presents an overview of HiRank.Source: §3 · p. 3
3.1 HiRank Overview
HiRank’s overview combines main-sequence states, candidate and semantic feature tokens, and read-only long-term memories. Layered token mixing feeds interaction context back into later sequence modeling and retrieval, followed by scenario-aware aggregation and composite-token interaction.
3.1 HiRank Overview
- Each HiRank block maintains a user main-sequence state, a candidate token, read-only long-term memories, and semantic feature tokens.Source: §3.1 · p. 3
- Blocks apply self-attention to the main sequence and a separate shared-parameter cross-attention path for candidate reads, then mix updated states and memory readouts with feature tokens.After the first-layer query initialization, later retrieval queries and mixed sequence and feature states are propagated across layers, allowing earlier feature interactions to guide later sequence modeling and memory access.Source: §3.1 · p. 3
- The final High-Rank Attention Block aggregates the shared token bank with scenario-conditioned queries, projects the compact readout into composite tokens, and sends the result to task-specific towers.An optional user-item decoupled variant shares computation across candidates within a request.Source: §3.1 · p. 3
3.2 Features and Tokenization
HiRank represents non-sequential features and pooled behavior summaries as semantic tokens, while preserving event-level states for the main sequence and selected long-term histories.
3.2.1 Semantic feature tokens.
- Non-sequential features and pooled behavior summaries are grouped by user attributes, item attributes, and request context to form semantic feature tokens.Constituent feature blocks are independently projected into d-dimensional tokens.Source: §3.2.1 · p. 3
- Feature tokens are initially assigned to user-side or candidate-side slots, but joint mixing can give either set cross-side information.A separate projection of candidate item features initializes the candidate state.Source: §3.2.1 · p. 3
3.2.2 Main sequence and multiple long-term memories.
- The main behavior sequence is projected event-wise to preserve event-level representations, while selected histories become separate long-term memories with valid-length masks.The main-sequence state changes across layers; projected long-term memory event states remain fixed during the forward pass.Source: §3.2.2 · p. 3
3.3 HiRank Block
HiRank couples gated sequence modeling, complementary long-term retrieval, and heterogeneous token interaction in a shared backbone. Interaction outputs feed subsequent sequence modeling and retrieval.
3.3.1 Gated Sequential Modeling.
- Gated self-attention models the main sequence alongside candidate-to-sequence cross-attention, using pointwise aggregated attention and multiplicative gating.Pointwise aggregation avoids softmax’s unit-sum constraint, while gating models interactions between local features and aggregated history.Source: §3.3.1 · p. 3
3.3.2 Complementary Memory Retrieval.
- Main-sequence-derived interest queries and candidate-conditioned queries retrieve complementary evidence from the same long-term memories.The candidate token initializes the candidate-conditioned query set; updated retrieval representations become queries for the next block while memory event states persist.Source: §3.3.2 · p. 4
3.3.3 Heterogeneous Token Interaction.
- The block concatenates sequence states, retrieval readouts, feature tokens, and the candidate state, then mixes tokens head-wise and applies per-token transformations.Mixing integrates evidence across token positions within channel groups; the restored layout supports token-specific channel recombination, and updated retrieval outputs become later queries.Source: §3.3.3 · p. 4
3.4 High-Rank Attention
High-Rank Attention compresses backbone tokens with scenario-aware queries, then applies heterogeneous attention to composite tokens and maps the shared state to task predictions.
3.4.1 Scenario-Aware Compression.
- Scenario-aware compression replaces projections over all backbone tokens with a compact state derived by cross-attending scenario-initialized scene tokens to the final-layer output.The scene tokens provide queries, and the final-layer representation provides keys and values.Source: §3.4.1 · p. 5
3.4.2 Heterogeneous Attention.
- Three independently parameterized projections turn the compressed state into composite queries, keys, and values for multi-head attention.Each composite token can combine all compressed-state channels, and projection matrix size depends on compressed-state and composite-token dimensions rather than backbone token count.Source: §3.4.2 · p. 5
- Multi-head attention models dependencies among composite tokens, and a residual update integrates those interactions into the compact prediction state.Source: §3.4.2 · p. 5
3.4.3 Task Prediction.
- Task-specific MLPs map the shared state to scalar logits for prediction.The training objective weights tasks and regularizes model parameters.Source: §3.4.3 · p. 5
3.5 User-Item Decoupling
The optional decoupled variant shares user-side processing within a request while retaining candidate-specific item-side processing. Directional interaction lets candidate representations incorporate shared user evidence across layers.
3.5 User-Item Decoupling
- Request-level batching shares user-side processing across candidates in a recommendation request while preserving user-to-item interaction.The variant maintains candidate-independent user states and candidate-specific item processing.Source: §3.5 · p. 5, Figure 2 · p. 5
3.5.1 Token Decoupling.
- The user-side stream contains the main-sequence state, interest-query retrieval tokens, and features shared across candidates; the item-side stream contains candidate-conditioned retrieval and candidate-varying features.Source: §3.5.1 · p. 5
3.5.2 Directional Interaction.
- Masked head-wise interaction permits user-to-item information flow while blocking item-to-user feedback, preserving shared user evidence in candidate representations throughout the backbone.The design extends computation reuse to behavioral modeling and feature interaction while retaining candidate-aware representation learning.Source: §3.5.2 · p. 5
4 Experiments
The experiments evaluate HiRank on industrial Taobao CVR prediction, using comparable dense-parameter budgets and shared inputs across models. HiRank achieves the strongest AUC and GAUC on Homepage and Main Search, with component and scaling analyses framed by the research questions.
4.1.1 Dataset.
- The industrial-scale Taobao dataset uses purchase conversion as the main task and cart addition as auxiliary supervision, with production training pooled across scenarios to mitigate sparse purchase feedback.It covers major traffic sources including main search and homepage recommendations, as well as smaller scenarios.Source: §4.1.1 · p. 6
4.1.2 Tasks and Metrics.
- Offline evaluation reports global AUC, user-weighted GAUC, PCOC, and RI, with RI measuring global-AUC improvement relative to the baseline’s margin over random guessing.PCOC closer to one indicates better agreement between predicted and observed conversions.Source: §4.1.2 · p. 6
4.1.3 Baseline Methods.
- The comparisons include DCNv2 + DIN as the production baseline, feature-interaction alternatives, and multi-task or multi-scenario baselines, with shared features, histories, examples, and tuning budgets.Base + PLE changes the production baseline’s prediction heads, while MDL is evaluated as a complete model.Source: §4.1.3 · p. 6
4.1.4 Implementation Details.
- Models use the same inputs and training data, with validation CVR AUC tuning under comparable dense-parameter budgets; HiRank’s default backbone has two layers and one High-Rank Attention block.The configuration uses token width 256, four long-term memories, 32 mixer heads, a 48-event main sequence, and 24 composite tokens.Source: §4.1.4 · pp. 6–7
4.2 RQ1: Performance Evaluation
- The ablation table compares HiRank components, with its first two variants replacing corresponding modules with DIN-style attention and boldface marking each metric’s best result.The stated research questions also cover component contributions, eRank evolution, model scaling, and online tests.Source: §4 · p. 6, Table 3 · p. 7
- 0.0009 AUC and 0.0186 GAUC gains over the respective strongest baselines on Homepage, and 0.0005 AUC and 0.0009 GAUC gains on Main Search; HiRank leads both metrics on both sources.HiRank also achieves the PCOC closest to one on Homepage.Source: §4.2 · p. 7
- HiRank combines attention with token mixing to balance adaptive dependency modeling and efficient information exchange, while main-sequence-derived queries complement candidate-aware long-term retrieval.Scenario-aware compressed representations are projected into composite views for higher-order interaction; Figure 3 tracks pooled eRank across mixing blocks, memory retrieval, and High-Rank Attention.Source: §4.2 · p. 7, Figure 3 · p. 7
4.3 RQ2: Ablation and Analysis
HiRank’s ablations support complementary roles for behavioral modeling and heterogeneous feature interaction, while effective-rank diagnostics and scaling tests examine representation diversity and capacity. Online tests report gains over the production baseline on Taobao Homepage traffic.
4.3.1 Ablation Study.
- Removing token interaction and High-Rank Attention together yielded the largest drops: 0.0020 in AUC and 0.0489 in GAUC.Full HiRank achieved the best AUC and GAUC among evaluated variants; replacing gated attention or complementary retrieval with DIN-style attention also reduced both metrics.Source: §4.3.1 · pp. 7–8
4.3.2 Effective Rank Analysis.
- Mean eRank increased from 96.51 to 112.80 and 128.36 to 134.80 after the two complete mixing blocks, and from 24.52 to 54.88 for retrieved representations versus input queries.Across High-Rank Attention, mean eRank changed from 228.58 to 224.09, a relative decrease of 1.96%; the authors describe this as consistent with largely retaining the compact state’s spectral diversity.Source: §4.3.2 · p. 8
4.4 RQ3: Scaling Validation
- At comparable costs, HiRank Block sequence length had the largest tested AUC effect, while High-Rank Attention gained with wider composite tokens and more tokens.For HiRank Block, T=96 yielded +0.23 pp and T=24 yielded −0.51 pp; for High-Rank Attention, d_t=1024 yielded +0.08 pp, N_t=96 yielded +0.05 pp, and K=4 yielded +0.06 pp.Source: §4.4 · p. 8, Figure 4 · p. 8
4.5 RQ4: Online A/B Tests
- Orders/u rose 2.27% and GMV/u rose 2.86% versus the production baseline in the Taobao Homepage A/B test.The test also reports increases of 0.45% in PV and 2.13% in DAC; further results include 0.85% more users clicking recommendations and 1.81% more orders per 1,000 impressions.Source: §4.5 · p. 8, Table 4 · p. 8
5 Conclusion
HiRank unifies multi-behavior sequence modeling and heterogeneous feature interaction, with iterative updates to sequence states and long-term retrieval queries. The conclusion reports deployment on Taobao Homepage feeds and improvements over the production baseline in offline evaluations and online A/B tests.
5 Conclusion
- HiRank combines multi-behavior sequence modeling with heterogeneous feature interaction in a unified framework.Its backbone iteratively updates main-sequence states and complementary long-term retrieval queries so feature context can guide subsequent behavior modeling.Source: §5 · p. 8
- Scenario-aware aggregation and composite projections construct complementary latent views for high-order interaction.A user–item decoupled variant also enables reuse of user-side computation across candidates within a request.Source: §5 · p. 8
- HiRank was deployed on Taobao Homepage feeds, where offline evaluations and online A/B tests improved over the production baseline.Ablations support complementary roles for key components, while effective-rank diagnostics characterize representation diversity and scaling experiments show gains from longer sequences and increased model capacity.Source: §5 · p. 8