Source-linked AI summary
A Hybrid Cross-Stage Coordination Pre-ranking Model for Online Recommendation Systems
Binglei Zhao, Houying Qi, Guang Xu, Mian Ma, Xiwei Zhao, Feng Mei, Sulong Xu, Jinghe Hu
TL;DR
Pre-ranking methods that rely only on downstream ranking information can limit coordination across the recommendation stream. HCCP combines upstream and downstream samples with objectives for consistency and long-tail precision. The authors report gains over state-of-the-art methods, including in JD E-commerce.
Problem
Existing pre-ranking work focuses on consistency with ranking and downstream information, leaving the question of how to use whole-stream unexposed data to make pre-ranking a better bridge.
Source: §1 · p. 2Method
HCCP combines multi-level unexposed samples from upstream and downstream with consistency and long-tail objectives, including a Margin InfoNCE loss.
Source: §1 · p. 2Results
14.9% UCVR and 1.3% UCTR increases over state-of-the-art methods are reported in the JD E-commerce recommendation system.
Source: §5 · p. 8Takeaways & Limitations
HCCP improves cross-stage coordination by jointly optimizing consistency and long-tail precision with multi-level samples from the entire stream.
Source: §5 · p. 8Because of data sensitivity, the reported table displays only absolute increase values.
Source: §4.1.4 · p. 7
Abstract
from arXiv · showhide
Large-scale recommendation systems often adopt cascading architecture consisting of retrieval, pre-ranking, ranking, and re-ranking stages. With strict latency requirements, pre-ranking utilizes lightweight models to perform a preliminary selection from massive retrieved candidates. However, recent works focus solely on improving consistency with ranking, relying exclusively on downstream stages. Since downstream input is derived from the pre-ranking output, they will exacerbate the sample selection bias (SSB) issue and Matthew effect, leading to sub-optimal results. To address the limitation, we propose a novel Hybrid Cross-Stage Coordination Pre-ranking model (HCCP) to integrate information from upstream (retrieval) and downstream (ranking, re-ranking) stages. Specifically, cross-stage coordination refers to the pre-ranking's adaptability to the entire stream and the role of serving as a more effective bridge between upstream and downstream. HCCP consists of Hybrid Sample Construction and Hybrid Objective Optimization. Hybrid sample construction captures multi-level unexposed data from the entire stream and rearranges them to become the optimal guiding "ground truth" for pre-ranking learning. Hybrid objective optimization contains the joint optimization of consistency and long-tail precision through our proposed Margin InfoNCE loss. It is specifically designed to learn from such hybrid unexposed samples, improving the overall performance and mitigating the SSB issue. The appendix describes a proof of the efficacy of the proposed loss in selecting potential positives. Extensive offline and online experiments indicate that HCCP outperforms SOTA methods by improving cross-stage coordination. It contributes up to 14.9% UCVR and 1.3% UCTR in the JD E-commerce recommendation system. Concerning code privacy, we provide a pseudocode for reference.
1 Introduction
Pre-ranking selects a smaller set from massive retrieval candidates, but downstream-only alignment can leave it a poor bridge between retrieval and ranking. HCCP combines information across the stream to improve consistency and long-tail precision; experiments report gains in JD E-commerce.
1 Introduction
- Pre-ranking reduces massive retrieved candidate sets to a smaller subset for ranking, making it a pivotal selection stage in the cascade.The passage gives approximate set sizes of 10^5 at retrieval, 10^3 after pre-ranking, and 10 at final exposure.Source: §1 · p. 1
- Vector-product-based models are commonly used in pre-ranking to balance effectiveness and efficiency.Source: §1 · p. 1
- Downstream-only alignment can worsen sample selection bias and the Matthew effect because downstream input derives from pre-ranking output.The paper argues pre-ranking should instead bridge retrieval and ranking and improve precision on high-quality mid-tail items.Source: §1 · p. 2
- HCCP integrates upstream and downstream information through hybrid sample construction and joint optimization of consistency and long-tail precision.Its Margin InfoNCE loss is designed for hybrid unexposed samples and to mitigate sample selection bias.Source: §1 · p. 2
- 14.9% UCVR and 1.3% UCTR increases are reported for HCCP over state-of-the-art baselines in the JD E-commerce recommendation system.The paper also reports extensive offline and online experiments.Source: §1 · p. 2
2 Related Work
Prior work improves pre-ranking architectures, ranking consistency, and negative sampling, but the cited approaches leave challenges around tail-item accuracy and adaptation to retrieval distributions. HCCP combines full-stream data to address these issues.
Pre-ranking
- Prior pre-ranking approaches improve model architectures, distill knowledge, or align with ranking, but lightweight models can remain inconsistent and ranking-log-only methods can neglect tail accuracy or retrieval-distribution shifts.The cited methods include point-wise, MSE, and ListNet score alignment, plus COPR's rank alignment and chunk-based sampling.Source: §2 · p. 2
Learning-To-Rank (LTR) Algorithm
- Point-wise LTR assigns item scores independently, pairwise methods predict relative item orders, and listwise methods consider whole-sequence quality.Some studies combine these losses to preserve calibration and ranking quality.Source: §2 · p. 2
Sample Selection Bias (SSB) Problem
- Prior SSB approaches sample negatives from predefined distributions, in batches, or across batches, while other methods target long-tail learning with network or loss modifications.In-batch negative diversity is constrained by batch size; HCCP instead leverages entire-stream data to address SSB and retrieval-distribution shifts.Source: §2 · pp. 2–3
HCCP architecture
- Figure 2 depicts HCCP's three components: hybrid sample construction, consistency learning for top items, and hybrid optimization for long-tail precision and SSB mitigation.The caption identifies consistency and hybrid optimization as jointly supporting cross-stage coordination.Source: Figure 2 · p. 3
3 METHODOLOGY
HCCP builds hybrid multi-level samples from the full stream and applies hybrid objective optimization to use unexposed information across pre-ranking models.
3 METHODOLOGY
3.1 Overall Architecture
HCCP uses a three-tower pre-ranking base and combines full-stream hybrid sampling with objectives for downstream consistency and long-tail precision. Its unexposed samples are handled with contrastive learning and Margin InfoNCE.
3.1.1 Base Model.
- The base model uses user, item, and cross-feature towers, with an MMOE user tower for multi-task learning and BCE calibration tasks on exposed items.Item embeddings are precomputed and updated near real time; online requests calculate user embeddings and cross-feature scores in real time.Source: §3.1.1 · p. 3
3.1.2 Innovation.
- Hybrid sampling captures multi-level data from downstream and upstream, while its objective is tailored to samples without user feedback to address SSB and improve precision.The design makes pre-ranking a bridge between retrieval and ranking rather than an isolated stage.Source: §3.1.2 · p. 3
- Ranking sequences are sampled non-uniformly, with higher sampling rates for top-ranked items, then rearranged using exposure feedback to incorporate diversity, context, and user interest.Upstream samples also include later-order pre-ranking items, in-batch negatives, and pool-sampling negatives; sequence samples are treated as hard negatives relative to the latter negative types.Source: §3.1.2 · p. 3
- List-wise global and local consistency modules use ranking sequences to align pre-ranking with downstream, while contrastive learning and Margin InfoNCE distinguish unexposed negatives from potential positives.The approach targets long-tail precision and SSB mitigation; the appendix discusses the loss's efficacy in selecting potential positives.Source: §3.1.2 · p. 3
3.2 Hybrid Multi-level Sample Construction
Hybrid multi-level sample construction combines downstream ranking data with upstream retrieval data that did not reach downstream stages. It organizes these samples into list-wise training data to guide pre-ranking.
3.2 Hybrid Multi-level Sample Construction
- The hybrid dataset combines impression feedback with unexposed ranking, later pre-ranking, in-batch, and pool samples in a list-wise format.The list-wise organization keeps samples from one request together during learning.Source: §3.2 · pp. 3–4
3.2.1 Ranking Sequence.
- Ranking candidates are sampled more often from higher-ranked items than from lower-ranked items to account for differences in ranking accuracy.Impressions are retained, and sampling rates can be adjusted to shift consistency from coarse- to fine-grained.Source: §3.2.1 · p. 4
- The sampled ranking sequence is rearranged so clicked items precede unclicked impressions, which precede unexposed items, while retaining local order within each group.Task-specific ordering within the exposure sequence supports local ranking ability.Source: §3.2.1 · p. 4
3.2.2 Pre-ranking Sequence with Latter Order.
- Later pre-ranking items that do not reach ranking are sampled as hard negatives, though they may include potential positives and help distinguish long-tail items.These items are usually discarded in prior pre-ranking training.Source: §3.2.2 · p. 4
3.2.3 In-batch Negatives.
- List-wise in-batch negatives are sampled from other users’ impressions, with a pool of negatives whose size scales as the batch size times the impression count.The method masks items belonging to the current user’s impression, ranking, and pre-ranking collections before sampling.Source: §3.2.3 · p. 5
3.2.4 Pool Sampling Negatives.
- Pool-sampled negatives come from items not retrieved, typically easy negatives that can improve discrimination when retrieval expands.Source: §3.2.4 · p. 5
3.3 Hybrid Objective Optimization
Hybrid objective optimization combines downstream consistency learning with long-tail precision optimization. Its Margin InfoNCE loss distinguishes hard and easy negatives among unexposed samples.
3.3.1 Consistency with Downstream Learning.
- Global consistency applies ListMLE to ranking sequence space, learning the downstream ranking order rather than only pairwise relative order.Minimizing the loss gives each item the highest top-one probability in the subsequence with later order.Source: §3.3.1 · p. 5
- Local consistency aligns impression order in exposure space while retaining calibration loss, adding a local ranking objective for CxR estimation tasks.The loss similarly favors each item having the highest top-one probability in the later-order subsequence.Source: §3.3.1 · p. 5
3.3.2 Long-tail Precision Optimization.
- The long-tail objective uses multi-level unexposed samples to address sample-selection bias and support discrimination when retrieval supplies many long-tail items.The design combines traditional calibration and local consistency with negative contrastive learning for CxR estimation tasks.Source: §3.3.2 · p. 5
- Margin InfoNCE separates positives, hard negatives N2~N3, and easy negatives N4~N5, accounting for their differing relevance to users.Hard negatives may contain potential positives; easy negatives are less aligned with user preferences and should be farther from positives.Source: §3.3.2 · p. 5
4 EXPERIMENT
The experiments evaluate HCCP on public Taobao and JD production data, using offline pre-ranking and ranking-consistency metrics alongside online measures. The section specifies its baselines and implementation choices, while noting data limitations that affect comparison.
4.1.1 Dataset.
- JD production training uses billions of clicked requests sampled from a week of homepage traffic, with validation on next-week pre-ranking candidates.The public Taobao dataset has 26 million impression logs over eight days and lacks multi-level unexposed data, limiting simulation of industrial settings.Source: §4.1.1 · p. 6
4.1.2 Metrics.
- Pre-ranking quality is measured with in-scenario and all-scenario click and purchase hitrates at K=10, 100, 1000, and 2000, covering top, middle, and tail items.AUC evaluates impression ranking on the public dataset; MAP@K and NDCG@K measure consistency with ranking, but ranking accuracy is insufficient for tail consistency at K=2000.Source: §4.1.2 · p. 6
4.1.3 Baselines.
- The baselines span impression-only methods, methods using ranking orders, and methods incorporating unexposed samples; HCCP includes ablations of upstream samples and negative-loss choices.The final HCCP variant is compared with versions removing upstream samples, ranking/pre-ranking negatives, sampled negatives, or the margin distinction.Source: §4.1.3 · p. 6
4.1.4 Implementation Detail.
- HCCP samples ranking candidates non-uniformly across seven chunks and uses 35 ranking-sequence items and 15 pre-ranking items per request.In-batch sampling uses a 0.05 rate; pool sampling adds 10 easy negatives per request.Source: §4.1.4 · pp. 6–7
- Table values are restricted by data sensitivity: only absolute increase values are displayed.The public Taobao dataset also lacks unexposed data, limiting comparison between HCCP and Rethink in that setting.Source: §4.1.4 · p. 7
- The implementation averages two losses: InfoNCE on N2~N3 final weighted scores and Margin InfoNCE on user-item dot products for N2~N5.Performance is somewhat sensitive to hyperparameters, especially global-consistency loss weight and Margin InfoNCE margin.Source: §4.1.4 · p. 7
4.2 Offline Performance Comparison (Q1)
Across offline and online evaluations, HCCP improves pre-ranking performance by combining cross-stage information and balancing ranking consistency with long-tail precision. The experiments also assess the approach’s hybrid samples, loss, and serving cost.
4.2.1 Performance on Taobao Dataset.
- On Taobao, HCCP achieved a slight improvement, but the dataset’s lack of unexposed data limits comparison with Rethink.The authors therefore conduct further experiments on the JD production dataset.Source: §4.2.1 · p. 7
4.2.2 Performance on JD Production Dataset.
- On the JD production dataset, HCCP surpassed baseline methods in hitrate metrics for top items at K=10 and 100 and mid- and tail-items at K=1000 and 2000.The paper reports that MAP and NDCG decline as K increases to 1000 and cautions that ranking consistency alone does not fully measure pre-ranking performance.Source: §4.2.2 · p. 7
4.3 Online A/B Test
- 14.9% UCVR and 1.3% UCTR improvements were reported for HCCP in a seven-day JD homepage A/B test.HCCP was deployed to serve main user traffic; mid- and tail-frequency items showed notable CVR improvement.Source: §4.3 · p. 8
4.4.1 Ablation studies (Q2).
- HCCP(w/o Up) enhanced all metrics over Base+ListNet and significantly outperformed COPR on consistency metrics.This variant trains global consistency on exposed and unexposed ranking sequences.Source: §4.4.1 · p. 8
- N2~N5 samples were important, but direct incorporation did not maximize utility; removing the margin led to declines on ISP@1000, ISPH@1000, and ASPH@2000.The paper attributes the declines to hard negatives and proposes Margin InfoNCE to distinguish negatives; more hard negatives increase long-tail learning difficulty.Source: §4.4.1 · p. 8
4.4.2 Performance on long-tail items (Q3).
- HCCP significantly improved mid- and long-tail item precision while enhancing top-item precision, compared with HCCP(w/o Up) and COPR.The paper reports that these consistency-focused variants were less effective at mitigating long-tail bias.Source: §4.4.2 · p. 8, Figure 7 · p. 8
4.4.3 Complexity and Time Analysis(Q4).
- HCCP’s list-wise organization reduced offline training complexity compared with HCCP(Point), while its serving TP99 latency cost increased by 0.73%.The augmented data volume reached 4.75 billion instances, compared with 39.5 million for Base.Source: §4.4.3 · p. 8, Table 4 · p. 8
5 CONCLUSION
The conclusion presents HCCP as a hybrid pre-ranking model that jointly optimizes consistency and long-tail precision using multi-level samples. It reports JD deployment gains and identifies all-scenario positive feedback as future work.
5 CONCLUSION
- HCCP jointly optimizes consistency and long-tail precision through multi-level sample construction and hybrid objective optimization.The model is intended to improve cross-stage coordination within the entire stream.Source: §5 · p. 8
- In a seven-day JD E-commerce deployment, HCCP outperformed SOTA methods, contributing up to 14.9% UCVR and 1.3% UCTR.Source: §5 · p. 8
- Future work will extend training data to all scenarios with all-scenario positive feedback information.Source: §5 · p. 8
A Appendix
The appendix analyzes how Margin InfoNCE affects gradients for potential positives among hard negatives, comparing its behavior with BCE.
A Appendix
- In BCE calibration, unclicked impressions that reflect a clear user tendency not to choose are treated as negatives.Source: A Appendix · p. 9
- The appendix compares predicted-score gradients for the same negative sample under BCE and Margin InfoNCE.The functions are shown with parameter choices that control the Margin InfoNCE curve’s position and slope.Source: A Appendix · p. 9, Figure 8 · p. 9
- As a negative sample’s predicted score approaches 0, Margin InfoNCE slows its rate of descent relative to BCE, resulting in higher scores for potential positives among hard negatives.The appendix states that this behavior helps select potential positives.Source: A Appendix · p. 9
1: function Cal_Margin_InfoNCE_Loss( )
The pseudocode computes cosine-based logits, applies a margin and weighted update, then forms a contrastive loss from positive and negative logits.
1: function Cal_Margin_InfoNCE_Loss( )
- The function computes cosine similarity by normalizing logits with user- and item-vector norms.Source: §1 · p. 10
- It adjusts logits with a margin and combines them with the original logits using a weighted update whose beta decreases during training.Source: §1 · p. 10
- The loss pairs positive logits with hard and easy negatives, applies temperature-scaled log-softmax, and sums the losses.Source: §1 · p. 10