Source-linked AI summary
Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization
Siyao Li, Deren Lei, Pengda Qin, William Yang Wang
TL;DR
Exact n-gram rewards can miss semantic similarity in abstractive summarization, while RL addresses exposure bias and non-differentiable metrics. The paper uses BERTScore as a distributional semantic reward, and human judgments on Gigaword and CNN/Daily Mail report improved relevance and fluency.
Problem
Common sentence-level ROUGE rewards count exact n-gram matches and are not robust to different words with similar meanings.
Source: §1 · p. 1Method
The paper uses BERTScore as a distributional semantic reward in deep RL and designs novel objective functions.
Source: §1 · p. 1, §2 · p. 2, §3.4 · p. 3Results
Human evaluations on Gigaword and CNN/Daily Mail find that DSR and DSR+XENT improve generated-summary relevance and fluency.
Source: §5 · p. 4Takeaways & Limitations
Pure DSR does not require XENT and can limit exposure bias originating from it while producing fewer repetitions and improving fluency.
Source: §5 · p. 4, §6 · p. 5, §7 · p. 5The DSR model’s high training-time ROUGE-L does not generalize well to the test set, and ROUGE-L is not the target evaluation metric.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) has been a commonly-used strategy for the abstractive summarization task to address both the exposure bias and non-differentiable task issues. However, the conventional reward Rouge-L simply looks for exact n-grams matches between candidates and annotated references, which inevitably makes the generated sentences repetitive and incoherent. In this paper, instead of Rouge-L, we explore the practicability of utilizing the distributional semantics to measure the matching degrees. With distributional semantics, sentence-level evaluation can be obtained, and semantically-correct phrases can also be generated without being limited to the surface form of the reference sentences. Human judgments on Gigaword and CNN/Daily Mail datasets show that our proposed distributional semantics reward (DSR) has distinct superiority in capturing the lexical and compositional diversity of natural language.
1 Introduction
Abstractive summarization must generate concise paraphrases, but long generation suffers from exposure bias and exact n-gram rewards miss semantic similarity. The paper introduces distributional semantic rewards and novel objectives for RL-based summarization.
1 Introduction
- Long sentence generation suffers from exposure bias as decoding errors accumulate.Abstractive summaries paraphrase long articles in fewer words and may use tokens outside the article’s vocabulary.Source: §1 · p. 1
- Sentence-level rewards based on exact n-gram matches are not robust to different words with similar meanings.Prior deep RL methods provide sentence-level feedback after complete sentences, but commonly used automatic metrics lack semantic-level reward.Source: §1 · p. 1
- Distributional semantic reward (DSR) is proposed for RL-based abstractive summarization, alongside several novel objective functions.The proposal draws on contextualized word representations’ capacity to reflect distributional semantics.Source: §1 · p. 1
- DSR outperforms conventional objectives while increasing sentence fluency and avoiding exposure bias introduced by XENT.The paper also reports greater token diversity and fewer unnecessary repetitions.Source: §1 · p. 1
2 Methodology
Sequence models trained with XENT are evaluated with discrete metrics, creating exposure-bias and non-differentiability issues. The paper uses BERTScore as a distributional semantic reward for deep RL, omitting IDF weighting for stated dataset-use and consistency reasons.
2 Methodology
- Sequence models face exposure bias and non-differentiable task metrics when trained with XENT but evaluated using discrete NLP metrics.RL addresses these issues, though its poor sample efficiency and slow convergence typically lead to starting from an XENT-pretrained policy.Source: §2 · pp. 1–2
- The conventional RL approach fine-tunes with a self-critical strategy based on the target evaluation metric.The supplied equation passage gives the policy log-probability term used in this approach.Source: §2 · p. 2
- BERTScore is introduced as a distributional semantic reward for deep RL, using contextualized token embeddings to measure sentence similarity.The described BERTScore aggregates cosine similarities between tokens with pretrained BERT embeddings.Source: §2 · p. 2
- DSR omits IDF weighting because its calculation would require the test set, and ROUGE does not use similar weighting.The authors omit IDF for consistency with ROUGE.Source: §2 · p. 2
3 Experimental Setup
The experiments compare distributional-semantic reward objectives with a ROUGE-based baseline on Gigaword and CNN/Daily Mail. Models are initialized from an XENT-pretrained sequence-to-sequence system.
3.1 Datasets
- Gigaword has 3.8M training and 189k development examples, with 15k randomly sampled for testing; CNN/Daily Mail has 287k training, 13k validation, and 11k test examples.CNN/Daily Mail articles are truncated to 400 tokens and summaries to 100 tokens.Source: §3.1 · p. 2
3.2 Pretrain
- The sequence-to-sequence attention model is pretrained with XENT, then its best parameters initialize RL; it uses 256-dimensional hidden states, 128-dimensional embeddings, and a pointer mechanism.The pointer mechanism handles out-of-vocabulary words.Source: §3.2 · p. 2
3.3 Baseline
- The baseline combines XENT with a self-critical ROUGE-L reward, using the ROUGE-L F score to calculate the reward loss.The ROUGE-L component is scaled by γ.Source: §3.3 · pp. 2–3
3.4 Proposed Objective Functions
- The proposed objectives use self-critical BERTScore reward alone or combine it with ROUGE or XENT, with XENT included to make generated phrases more readable.The paper identifies the pure DSR objective as having no teacher forcing.Source: §3.4 · p. 3
4 Results
Results compare ROUGE-L and FBERT on Gigaword and CNN/Daily Mail. DSR objectives improve reported scores, though training-time ROUGE-L does not generalize well for DSR.
4 Results
- The models are evaluated with FBERT and ROUGE-L; for CNN/Daily Mail, scores are calculated after concatenating the reference abstract sentences or their contextualized embeddings.The text points to Table 1 for results, but its values are not included in the supplied passages.Source: §4 · p. 3
- DSR without XENT improves the pretrained model on both FBERT and ROUGE-L; DSR+ROUGE achieves the best ROUGE on Gigaword and similar performance to ROUGE on CNN/Daily Mail.DSR and DSR+XENT obtain the best BERTScores.Source: §4 · p. 3
- DSR's high training-time ROUGE-L does not generalize well to the test set, and ROUGE-L is not the target evaluation metric.Source: §4 · p. 3
5 Human Evaluation
Human judgments compared DSR variants with ROUGE+XENT on summary relevance and fluency, finding significant improvements across Gigaword and CNN/Daily Mail.
5 Human Evaluation
- DSR and DSR+XENT significantly improve judged summary relevance and fluency over ROUGE+XENT.Pure DSR performs better on Gigaword and comparably to DSR+XENT on CNN/Daily Mail.Source: §5 · p. 4, Table 2 · p. 4
- Judges compared generations with the article context but without the ground-truth summary, permitting ties on relevance and fluency.The evaluation used 500 randomly sampled items.Source: §5 · pp. 3–4
6 Analysis
The analysis finds that DSR generates the most diverse outputs and fewest repetitions, while examples also show that some semantically similar substitutions can reduce quality.
Diversity
- DSR achieves the highest diversity, measured as the percentage of out-of-article n-grams in generated sentences.Table 3 reports unigrams for Gigaword and 5-grams for CNN/Daily Mail.Source: §6 · p. 4, Table 3 · p. 4
- When optimizing DSR to improve generation diversity, semantically similar words may be generated and harm summarization quality; the new token “wins” reduces scores of both metrics.Table 4 marks generated words absent from context blue and repeated words red. Its first two examples represent more diverse DSR tokens, while examples 3 and 4 show possible problems.Source: §6 · p. 4, Table 4 · p. 5
Repetition
- DSR produces the fewest repetitions among the evaluated rewards while achieving high fluency without XENT.Table 3 reports the average percentage of repeated n-grams.Source: Table 3 · p. 4, §6 · p. 5
7 Conclusion
The conclusion reports improved performance on Gigaword and CNN/Daily Mail, fewer repetitions, and better fluency with distributional semantic reward.
7 Conclusion
- On Gigaword and CNN/Daily Mail, DSR improves performance, reduces repetitions, and improves fluency, aligning with Wieting et al. (2019) on semantic similarity for machine translation.Source: §7 · p. 5