Source-linked AI summary
Empowering News Recommendation with Pre-trained Language Models
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang
TL;DR
Personalized news recommendation requires effective modeling of rich news text, but traditional methods may not capture its deep semantics. The paper incorporates pre-trained language models into news encoders and reports improved offline performance plus substantial click and pageview gains after deployment on Microsoft News.
Problem
Traditional news-text modeling methods may not adequately capture deep semantic information needed for personalized news recommendation.
Method
The paper uses pre-trained language models as news encoders, pools their hidden states into news embeddings, and fine-tunes them for recommendation.
Results
8.53% click and 2.63% pageview gains were achieved in English-speaking markets, with 10.68% click and 6.04% pageview gains in 43 other global markets.
Takeaways & Limitations
Incorporating PLMs can improve news modeling and recommendation performance across monolingual and multilingual settings and deployed online markets.
Abstract
from arXiv · showhide
Personalized news recommendation is an essential technique for online news services. News articles usually contain rich textual content, and accurate news modeling is important for personalized news recommendation. Existing news recommendation methods mainly model news texts based on traditional text modeling methods, which is not optimal for mining the deep semantic information in news texts. Pre-trained language models (PLMs) are powerful for natural language understanding, which has the potential for better news modeling. However, there is no public report that show PLMs have been applied to news recommendation. In this paper, we report our work on exploiting pre-trained language models to empower news recommendation. Offline experimental results on both monolingual and multilingual news recommendation datasets show that leveraging PLMs for news modeling can effectively improve the performance of news recommendation. Our PLM-empowered news recommendation models have been deployed to the Microsoft News platform, and achieved significant gains in terms of both click and pageview in both English-speaking and global markets.
1 INTRODUCTION
News recommendation depends on modeling rich news text to understand candidate content and infer user interests. The paper proposes using pre-trained language models to improve this modeling and reports gains offline and online.
- News modeling is central because it represents candidate-news content and supports inferring user interests from clicked news.
- Traditional shallow NLP models may fail to capture deep semantic information because they learn primarily from recommendation-task supervision.
- PLMs are pre-trained with self-supervision on large unlabeled corpora to encode universal text information before downstream fine-tuning.
- The paper explores fine-tuning PLMs for news modeling and reports consistent offline improvements across English and multilingual recommendation datasets.
- 8.53% click and 2.63% pageview gains were observed in English-speaking markets, while 10.68% click and 6.04% pageview gains were observed across 43 other global markets.
2 METHODOLOGY
The methodology integrates a PLM-based news encoder into a standard personalized recommendation framework. PLM hidden states are pooled into news embeddings, which feed user modeling, candidate matching, and click prediction.
- 2 METHODOLOGY: The general framework uses a news encoder, user encoder, and click prediction module for personalized candidate-news ranking.
- 2.2 PLM Empowered News Recommendation: The resulting news embeddings support user modeling and candidate matching based on users’ clicked-news histories.
- 2.2 PLM Empowered News Recommendation: The PLM-based news encoder transforms token sequences through Transformer layers and uses attention to summarize hidden states into unified news embeddings.
- 2.2 PLM Empowered News Recommendation: Negative sampling constructs labeled samples from impression logs, while cross-entropy training jointly tunes recommendation-model and PLM parameters.
3 EXPERIMENTS
Experiments evaluate PLM-based news recommendation on English and multilingual datasets, then examine model size, pooling, embedding structure, and online deployment. Results consistently favor PLM-enhanced modeling, with significant offline gains and substantial online improvements in clicks and pageviews.
- 3.1 Datasets and Experimental Settings: The experiments use MIND, an English dataset with click logs from 1 million users, and a multilingual dataset spanning seven markets and languages.The multilingual dataset contains 200,000 impression logs per market, with the last week reserved for testing.
- 3.2 Offline Performance Evaluation: PLM variants consistently improve basic news recommendation models on MIND, with improvements reported as statistically significant at p < 0.001.The comparison includes EBNR, NAML, NPA, LSTUR, and NRMS with BERT, RoBERTa, and UniLM variants.
- 3.2 Offline Performance Evaluation: Multilingual PLM methods outperform MUSE-based methods, while InfoXLM outperforms Unicoder for multilingual news recommendation.The authors attribute these differences to stronger multilingual semantic modeling and InfoXLM’s contrastive pre-training strategies.
- 3.3 Influence of Model Size: Larger PLMs with more parameters usually yield better recommendation performance, but the authors prefer base models because huge models are cumbersome online.The size comparison uses BERT-Base, BERT-Medium, BERT-Small, and BERT-Tiny with NAML and NRMS on MIND.
- 3.4 Influence of Different Pooling Methods: Attention pooling outperforms average pooling, while CLS pooling performs worst because it cannot exploit all PLM hidden states.The authors therefore select attention as the pooling mechanism for learning news embeddings.
- 3.5 Visualization of News Embedding: NRMS-UniLM produces more discriminative news embeddings and clearer clusters than NRMS, supporting more separated news representations for user-interest modeling and news matching.The comparison uses t-SNE visualizations of embeddings learned by the shallow and PLM-empowered models.
- 3.6 Online Flight Experiments: Online experiments report 8.53% higher click and 2.63% higher pageview in English-speaking markets, plus 10.68% higher click and 6.04% higher pageview across 43 other markets.NAML-UniLM served English-speaking markets, while NAML-InfoXLM served users in other multilingual markets.
4 CONCLUSION
The paper shows that pre-trained language models can improve personalized news recommendation across English and multilingual datasets. Deployed PLM-empowered models also improved click and pageview results across markets with different languages.
- PLMs can effectively improve news modeling for personalized news recommendation.
- Offline experiments covered both English and multilingual news recommendation datasets.
- PLM-empowered news recommendation models were deployed to a commercial news platform.
- Online flight results showed significant improvement in both click and pageview across many multilingual markets.