PaperlayerView exact revision on arXiv

Source-linked AI summary

NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application

Chuhan Wu, Fangzhao Wu, Yang Yu, Tao Qi, Yongfeng Huang, Qi Liu

arXiv:2102.04887v2cs.CL

TL;DR

Large PLMs could support news applications, but their computational demands challenge low-latency services, and general-corpus models may not fit the news domain. NewsBERT jointly learns teacher and student models and uses momentum distillation; experiments report improved performance on three news tasks with much smaller models.

  • Problem

    Large PLMs pose computational challenges for low-latency news services, while general-corpus pretraining and distillation may be mismatched with news.

    Source: §1 · p. 1

  • Method

    NewsBERT jointly learns teacher and student models, distills outputs and representations, and uses teacher gradients as momentum in student updates.

    Source: §1 · p. 2

  • Results

    NewsBERT consistently improves performance across three news intelligence tasks on two real-world datasets using much smaller models.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reported results support NewsBERT as an efficient approach for PLM-based intelligent news applications.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Pre-trained language models (PLMs) like BERT have made great progress in NLP. News articles usually contain rich textual information, and PLMs have the potentials to enhance news text modeling for various intelligent news applications like news recommendation and retrieval. However, most existing PLMs are in huge size with hundreds of millions of parameters. Many online news applications need to serve millions of users with low latency tolerance, which poses huge challenges to incorporating PLMs in these scenarios. Knowledge distillation techniques can compress a large PLM into a much smaller one and meanwhile keeps good performance. However, existing language models are pre-trained and distilled on general corpus like Wikipedia, which has some gaps with the news domain and may be suboptimal for news intelligence. In this paper, we propose NewsBERT, which can distill PLMs for efficient and effective news intelligence. In our approach, we design a teacher-student joint learning and distillation framework to collaboratively learn both teacher and student models, where the student model can learn from the learning experience of the teacher model. In addition, we propose a momentum distillation method by incorporating the gradients of teacher model into the update of student model to better transfer useful knowledge learned by the teacher model. Extensive experiments on two real-world datasets with three tasks show that NewsBERT can effectively improve the model performance in various intelligent news applications with much smaller models.

1 Introduction

NewsBERT addresses the challenge of applying large pre-trained language models to news applications with demanding latency needs and a domain mismatch in existing general-corpus models. It jointly learns teacher and student models and adds momentum distillation; experiments report improved performance with smaller models.

1 Introduction

  • Online news applications need to serve many concurrent users with low latency, while large PLMs demand substantial computational resources.BERT-Base has about 110M parameters and 12 Transformer layers.Source: §1 · p. 1
  • News articles contain rich text, making language understanding relevant to applications such as topic classification, headline generation, and news recommendation.The paper notes that efficient PLM incorporation could improve news-reading experiences.Source: §1 · p. 1
  • Existing PLM distillation typically trains teacher and student successively, leaving teacher learning experience unused; general-corpus pretraining may also mismatch news.These gaps may make existing approaches suboptimal for news intelligence.Source: §1 · pp. 1–2
  • NewsBERT improves performance across news intelligence tasks with much smaller models and outperforms many PLM-distillation baselines.The experiments cover two real-world datasets and three tasks.Source: §1 · p. 2
  • NewsBERT jointly learns teacher and student models, sharing top-layer parameters and distilling output probabilities and hidden representations.This lets the student learn from the teacher's learning experience while the teacher accounts for the student's learning status.Source: §1 · p. 2
  • Momentum distillation uses teacher gradients to boost student gradients and transfer knowledge learned by the teacher.The method incorporates teacher gradients into student updates in a momentum manner.Source: §1 · p. 2

2 Related Work

Prior work compresses PLMs through varied distillation techniques, but generally trains teacher and student models successively and may rely on corpora mismatched with news. NewsBERT instead combines joint learning with momentum distillation.

2 Related Work

  • PLM distillation methods use signals such as soft targets, hidden states, embeddings, or attention representations, and some target downstream retrieval.The related work also describes approaches that distill at pre-training, fine-tuning, or downstream-task stages.Source: §2 · p. 2
  • Earlier methods usually train the teacher and student successively, do not exploit the teacher's learning experience, and may use corpora shifted from news texts.The paper therefore describes off-the-shelf distilled models as potentially suboptimal for intelligent news applications.Source: §2 · p. 3
  • NewsBERT addresses these gaps with joint teacher-student learning and momentum distillation that incorporates teacher gradients into student gradients.The student can exploit knowledge produced during the teacher's learning process.Source: §2 · p. 3

3 NewsBERT

NewsBERT jointly trains teacher and student models, adding momentum distillation to transfer teacher gradients; the paper also describes its use in personalized news recommendation.

3.1 Teacher-Student Joint Learning and Distillation Framework

  • NewsBERT jointly trains a large teacher and lightweight student, sharing pooling and prediction layers so the student can learn from the teacher’s learning experience.The student uses fewer Transformer layers; its inference speed is approximately K times faster than the teacher.Source: §3.1 · pp. 3–4
  • Hidden losses align student-layer and news-embedding representations with the teacher, while soft-label distillation and task losses train the student and classification loss trains the teacher.The teacher loss is classification-only, limiting how heavily the student influences the teacher.Source: §3.1 · pp. 3–4

3.2 Momentum Distillation

  • Momentum distillation mixes teacher-block gradients into corresponding student-layer gradients, with β controlling the strength of the teacher-gradient contribution.The teacher-block gradient is computed as the average of the gradients from its layers.Source: §3.2 · p. 5

3.3 Applications of NewsBERT for News Intelligence

  • For personalized recommendation, shared NewsBERT encodes clicked and candidate news; shared user and click modules predict scores, with distillation and MSE losses constraining the models.The user embedding is matched with the candidate-news representation to predict a click probability for ranking and display.Source: §3.3 · p. 5

4 Experiments

Across two real-world datasets, NewsBERT outperforms the compared baselines, while its joint teacher-student framework improves student performance and its momentum setting balances teacher guidance against gradient mismatch. The experiments also examine dataset and model settings, component ablations, and cross-task retrieval performance.

4.1 Datasets and Experimental Settings

  • MIND supports topic classification and personalized recommendation, while NewsRetrieval supplies Bing search queries and clicked news for cross-task retrieval evaluation.The experiments use the first eight UniLM layers as the teacher and initialize students with its first 1, 2, or 4 layers; reported results average five runs.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 Performance Evaluation

  • NewsBERT outperforms all compared baselines in topic classification and news recommendation, with significant improvements at p < 0.01 for models with the same number of layers.NewsBERT-4 has only a 0.24% accuracy gap from the teacher, and the one-layer model is about 12.0x faster.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • NewsBERT also performs better than TinyBERT and TwinBERT in cross-task news retrieval after recommendation-task distillation and retrieval-data fine-tuning.Directly fine-tuned general UniLM performs worse than models distilled on recommendation data.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Effectiveness of Teacher-Student Joint Learning and Distillation Framework

  • The joint framework consistently improves student models of different capacities, and some jointly trained teacher variants outperform the original UniLM-8.The authors suggest reciprocal learning may improve both models.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.4 Ablation Study

  • Removing momentum distillation considerably reduces performance; distillation loss and hidden loss also contribute to NewsBERT.The authors associate these components with transferring teacher output behavior and aligning teacher-student hidden representations.Source: §4.4 · pp. 8–9, Figure 6 · p. 9

4.5 Hyperparameter Analysis

  • Performance is not optimal with β too small and declines when β > 0.2; the authors recommend β from 0.1 to 0.2.Small β weakens teacher guidance, while large β can let inconsistent teacher gradients move student updates away from an appropriate direction.Source: §4.5 · p. 9, Figure 7 · p. 9

5 Conclusion

NewsBERT distills pretrained language models for intelligent news applications through joint teacher-student learning and momentum distillation. Experiments on two real-world datasets and three tasks show improved performance with considerably smaller models.

5 Conclusion

  • NewsBERT combines joint teacher-student learning with momentum distillation and improves performance across three news tasks using considerably smaller models.The framework lets the student learn from the teacher’s learning experience and incorporates teacher gradients into student updates.Source: §5 · p. 9

Related papers

Loading 2102.04887v2…