Source-linked AI summary
Soft Curriculum Learning for Optimizing Fresh and Generalized Recommendations
Arnab Bhadury, Siyan Zheng, Anlan Yu, Palaksh Rungta, Jiawei Li, Changping Meng, Dapeng Hong, Chuan He, Onkar Dalal
TL;DR
Popularity feedback loops cause recommendation models to overlearn head items, making generalization to tail content difficult. The paper introduces Soft Curriculum Learning, which adjusts loss weights for industrial continuous training instead of rejecting data. Online tests report higher user satisfaction and cold-start engagement, alongside retrieval engagement gains.
Problem
Popularity feedback loops generate disproportionate positive training data for head items, making robust generalization difficult and suppressing fresh, cold-start, and tail content.
Source: §1 · p. 1Method
Soft Curriculum Learning uses cyclic, annealed loss weighting and difficulty estimates to adjust examples' influence without rejecting data in industrial retrieval and ranking training.
Source: §1 · p. 2, §4.2 · p. 3, §6 · p. 6Results
Online tests reported +0.54% overall User Satisfaction and +5.70% cold-start content engagement, while SASRec retrieval reported +7.7% unique retrieval engagements.
Source: §5.2.1 · p. 5, §6 · p. 6Takeaways & Limitations
The case study demonstrates an industrial implementation of Soft Curriculum Learning for continuously trained retrieval and ranking models, with reported online gains in satisfaction and fresh-content consumption.
Source: §6 · p. 6A global curriculum that permanently shifts to hard examples can fail to capture day-to-day data changes in continuous training and prevent adaptation to emerging popular trends.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Large-scale recommender systems, particularly short-form video platforms, are often bottlenecked by massive popularity feedback loops. In such environments, as models recommend popular items, they generate an overwhelming amount of skewed training data for "head" items. This creates a self-reinforcing cycle where retrieval and ranking models memorize "head" item patterns at the expense of generalizing across the vast "tail" of the catalogue. While Curriculum Learning (CL) offers a powerful mechanism to break this feedback loop by systematically exposing models to progressively more difficult and less frequent examples, its adoption in industrial recommendation has been hampered by hardware utilization inefficiencies or the needs for complicated pre-processing techniques because dynamic data rejection algorithms tend to starve hardware accelearators (TPUs/GPUs) by becoming largely CPU-bound. In this work, we introduce a scalable Soft Curriculum Learning framework designed specifically for continuous training setups within industry-scale retrieval and ranking models. By utilizing loss annealing and in-graph weight adjustments rather than rigid data filtering, we break the popularity feedback and enable dynamic curriculum pacing without sacrificing system throughput. We demonstrate empirical evidence through applications across sequence-based retrieval models (such as SASRec), two-tower retrieval models, and large-scale continuous ranking models. Online A/B tests on our short-video platform demonstrate substantial lifts in both overall user satisfaction and fresh content consumption, all without degrading model throughput.
1 Introduction
Popularity bias creates a self-reinforcing cycle in which models overlearn popular-item patterns and suppress fresh, cold-start, and tail content. The paper adapts curriculum learning for industrial recommendation with a scalable soft approach that adjusts loss weights rather than rejecting data.
1 Introduction
- Popular-item exposure generates disproportionate positive training data, reinforcing head-item recommendations while suppressing fresh uploads, cold-start items, and niche tail content.The resulting feedback loop drives memorization rather than robust generalization.Source: §1 · p. 1
- Curriculum learning begins with easy, popular examples and gradually shifts toward hard, sparse examples to counter popularity bias and prevent embeddings from collapsing into locally popular optima.The introduction frames this as a way to learn discriminative patterns for lower-viewed items.Source: §1 · pp. 1–2
- Soft Curriculum Learning uses cyclic, annealed loss weighting on TPU cores instead of discarding examples, and is evaluated on sequence retrieval, two-tower retrieval, and continuous ranking models.The paper reports online and offline gains in engagement and fresh-content consumption.Source: §1 · p. 2
- The work presents a case study of adapting curriculum learning to production-grade recommendation and a blueprint for future industry deployments.This addresses curriculum learning's under-exploration in large-scale industrial retrieval and recommendation.Source: §1 · p. 2
2 Related Work
Related work connects curriculum learning to optimization, feedback-loop mitigation, and ranking, while identifying large-scale continuous industrial training as a deployment challenge. This paper adapts those foundations to that setting.
2 Related Work
- Bengio et al. formalized curriculum learning, showing that progressively harder examples can speed convergence and guide optimization toward better local minima.Prior work on counterfactual risk minimization [11] and popularity de-biasing addresses amplification of logged-feedback biases.Source: §2 · p. 2
- Prior recommender and ranking studies applied curricula to conversation response ranking, open-domain answer re-ranking [7], and fair ranking.These works report improved retrieval effectiveness, MRR, or gradual mitigation of data skewness and historical exposure bias.Source: §2 · p. 2
- The paper builds on theoretical and offline ranking foundations but adapts curriculum learning to bypass computational bottlenecks in continuous, large-scale industrial training.The authors describe curriculum learning for information retrieval and recommendation as less explored than in other deep-learning fields.Source: §2 · p. 2
3 Difficulties of Curriculum Learning in Practice
Industrial recommendation faces two curriculum-learning challenges: dynamic data rejection can starve accelerators, and a single global schedule can become mismatched to continuously shifting data.
3 Difficulties of Curriculum Learning in Practice
- Dynamic data rejection runs filtering and preprocessing on CPUs before accelerator training, creating a potential throughput bottleneck in massive, continuously trained systems.The paper contrasts this deployment setting with the comparatively successful one-time training paradigm.Source: §3 · p. 2, §3.1 · p. 2
3.1 CPU-Bound Bottleneck
- Rejecting many easy or popular candidates can empty the CPU-refilled queue faster than it is replenished, starving accelerators and potentially requiring a costly parallel mini-batch-building pipeline.The challenge grows when a strict curriculum focuses training on tail items.Source: §3.1 · p. 2
3.2 The Pitfalls of Global Curriculum Pacing
- A global curriculum that permanently shifts from easy to hard examples can miss day-to-day distribution changes in continuous training and prevent adaptation to newly emerging popular trends.The authors discuss this pacing limitation with empirical results and performance comparisons in Section 5.1.3.Source: §3.2 · p. 2
4 Methodology: Soft Curriculum Learning
Soft Curriculum Learning keeps all examples in the training pipeline and varies their loss weights according to difficulty and progress. Cyclic pacing and two weight strategies adapt the curriculum to continuous training.
4 Methodology: Soft Curriculum Learning
- The Soft Curriculum Learner is designed to circumvent CPU bottlenecks, mitigate catastrophic forgetting, and combat popularity bias.It applies curriculum weights rather than filtering training data.Source: §4 · p. 2, §4.1 · p. 2
4.1 Loss Annealing
- Dynamic loss annealing multiplies the loss vector by a progress- and difficulty-dependent coefficient directly on the TPU or GPU core, adding virtually no overhead.All data passes through the pipeline without rejection, keeping it accelerator-compute-bound.Source: §4.1 · pp. 2–3
4.2 Quantifying Task Difficulty
- Example difficulty is estimated either from engagement, with higher-engagement items assigned lower difficulty, or from SNGP prediction uncertainty, where greater uncertainty indicates greater difficulty.The uncertainty measure is based on Spectral-normalized Neural Gaussian Process variance.Source: §4.2 · p. 3
4.3 Cyclic Curriculum Pacing
- The cyclic curriculum periodically resets rather than progressing continuously, and defines training progress within each sub-day interval using the examples processed in that interval.Progress p is normalized to [0, 1] within the interval.Source: §4.3 · p. 3
4.4 Curriculum Weight Strategies
- Hard-Example Warmup initially down-weights hard examples and lets their penalty decay, while Bidirectional Difficulty Scaling shifts priority from easy to hard examples over training.The second strategy reverses the weighting: easy examples are favored at the start, and hard examples by the end.Source: §4.4 · p. 3
5.1 Ranking Evaluation
The evaluation compares curriculum-weighting variants with production models offline and online, then contrasts soft curriculum with day-level global filtering.
5.1 Ranking Evaluation
- Replacing static inverse-popularity weights with dynamic curriculum weights improved the weighting strategy; removing the popularity penalty alone yielded neutral offline and negative online results.Offline performance was measured with unweighted AUC because AUC under dynamic sample weights is not comparable.Source: §5.1.1 · p. 3
5.1.1 Offline Evaluation.
- Both SNGP- and engagement-based curriculum methods surpassed baseline across tasks; bidirectional difficulty outperformed hard-example warm-up, with the largest marginal gains on cold and new content.The Eng-Bi AUC comparison is categorized by video age.Source: §5.1.1 · p. 3, Table 2 · p. 4
5.1.2 Online A/B Testing.
- Soft curriculum significantly improved primary engagement metrics regardless of difficulty heuristic; Bidirectional engagement-based Soft CL had the most substantial gains, increasing User Satisfaction by +0.54% and cold-start user engagement by +5.7%.Source: §5.1.2 · p. 4
5.1.3 Comparison with Global Curriculum Learning.
- Day-level global filtering had distinctly negative offline and online results despite lower CPU cost; permanently removing easy examples degraded overall generalization.The comparison evaluates filtering easy examples after one or three days.Source: §5.1.3 · p. 4, Table 5 · p. 4, Table 6 · p. 4
5.2 Retrieval Evaluation
The retrieval evaluation covers SASRec-style and Two-Tower models using engagement-based curriculum weighting. SASRec showed a marginal offline MAP decline but higher online engagement, while the Two-Tower treatment achieved higher offline MAP than its baseline.
5.2 Retrieval Evaluation
- The retrieval evaluation covers SASRec-like sequence-based and Two-Tower architectures; SNGP uncertainty is computed at serving time, so retrieval experiments report engagement-based metrics only.For SASRec, MAP@K measures mean average precision across user-video predictions, with ranks drawn from the global video population; Two-Tower uses in-batch candidates and a separately named MAP@K_TwoTower metric.Source: §5.2 · p. 4, §5.2.1 · p. 4, §5.2.2 · p. 5
5.2.1 SASRec Style Model.
- All evaluated SASRec curriculum variations had marginally lower offline MAP than baseline.The authors explain that head-heavy evaluation penalizes slight losses on popular items while barely reflecting improved tail capability.Source: §5.2.1 · pp. 4–5, Table 7 · p. 5
- 7.7% more unique engagements were recorded for the SASRec-style retrieval model during a two-week online A/B test of Eng-Hard Soft CL.Engagement gains persisted across age buckets, indicating accelerated discovery of cold-start content.Source: §5.2.1 · p. 5, Table 8 · p. 5
5.2.2 Two-Tower Style Model.
- The Two-Tower treatment achieved higher MAP@K_TwoTower than baseline, unlike the SASRec-style retrieval model.The authors attribute this contrast to the Two-Tower model’s feature-rich architecture and progressive introduction of harder examples.Source: §5.2.1 · p. 4, §5.2.2 · p. 5, Table 9 · p. 5
5.3 Production Challenges: Mitigating A/A Variance via Batch Difficulty Normalization
Soft Curriculum Learning can increase A/A variance because batch-to-batch difficulty composition changes the aggregate gradient magnitude. Normalizing each batch’s mean curriculum weight to one stabilizes updates and mitigates that variance.
5.3 Production Challenges: Mitigating A/A Variance via Batch Difficulty Normalization
- A/A variance is the statistical fluctuation in metrics when identical model configurations are evaluated.Low A/A variance matters because high variance can obscure genuine improvements in later model iterations.Source: §5.3 · p. 5
- Dynamic difficulty weights can inflate A/A variance when randomly sampled batches contain different proportions of hard tail and easy head items, causing aggregate gradient magnitudes to fluctuate.The resulting fluctuation acts like noise in the effective learning rate.Source: §5.3 · p. 5
- Normalizing curriculum weights so each mini-batch has mean weight one stabilizes gradient updates and significantly mitigates A/A variance.The normalization keeps the average applied weight consistent despite variation in batch composition.Source: §5.3 · p. 5
6 Conclusion
Soft Curriculum Learning uses cyclic schedules and loss annealing to support retrieval and ranking, with online gains in user satisfaction and cold-start engagement and increased retrieval exploration.
6 Conclusion
- Cyclic schedules and loss annealing bypass CPU bottlenecks, while popularity quantiles or SNGP variance dynamically adjust item difficulty in retrieval and ranking.Source: §6 · p. 6
- +0.54% overall User Satisfaction and +5.70% cold-start content engagement were reported in online A/B tests.The conclusion describes these gains alongside improved distribution of cold-start content without sacrificing overall ecosystem engagement or hardware efficiency.Source: §6 · p. 6
- +7.7% unique retrieval engagements in online A/B testing came despite lower offline MAP on head-heavy test sets for the SASRec-style model.The conclusion attributes the increase to devoting model capacity to long-tail, harder examples.Source: §6 · p. 6
- Ranking models showed broad offline AUC improvements across all multi-task prediction heads as harder examples were gradually introduced.The conclusion also reports richer, more robust representations across the shared embedding space in Two-Tower retrieval and deep ranking models.Source: §6 · p. 6