Source-linked AI summary
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
Byeongchang Kim, Hyunwoo Kim, Gunhee Kim
TL;DR
Abstractive summarization is challenging, and existing formal-document datasets may favor extractive methods through structural and textual biases. This paper introduces the Reddit TIFU dataset and multi-level memory networks (MMN), finding that the dataset is highly abstractive and MMN outperforms state-of-the-art abstractive methods.
Problem
Abstractive summarization often performs worse than extractive methods, partly because formal-document datasets contain structural patterns that extractive methods can exploit.
Method
The paper collects the Reddit TIFU dataset from informal user-generated posts and proposes MMN, which stores source information at multiple abstraction levels.
Results
MMN outperforms state-of-the-art abstractive summarization methods on Reddit TIFU, Newsroom abstractive subset, and XSum, while Reddit TIFU is highly abstractive.
Takeaways & Limitations
Reddit TIFU provides an informal-text benchmark less reliant on locational and near-copy biases, and MMN is effective across evaluated abstractive summarization datasets.
Takeaways & Limitations
Exploring data from other online forums, including Quora, Stack Overflow, and other subreddits, remains future work.
Abstract
from arXiv · showhide
We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model. First, we collect Reddit TIFU dataset, consisting of 120K posts from the online discussion forum Reddit. We use such informal crowd-generated posts as text source, in contrast with existing datasets that mostly use formal documents as source such as news articles. Thus, our dataset could less suffer from some biases that key sentences usually locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Second, we propose a novel abstractive summarization model named multi-level memory networks (MMN), equipped with multi-level memory to store the information of text from different levels of abstraction. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the Reddit TIFU dataset is highly abstractive and the MMN outperforms the state-of-the-art summarization models.
1 Introduction
The paper targets extractive bias in formal-document summarization datasets by introducing informal Reddit TIFU data and a multi-level memory model. It reports that MMN outperforms state-of-the-art abstractive summarization methods across three datasets.
- Motivation: Existing formal-document datasets may favor extractive methods because key sentences and similar summary candidates often appear in predictable locations.This bias can encourage models to memorize keywords or phrases and show limited abstraction.
- Dataset: Reddit TIFU contains 122,933 post-summary pairs from casual, conversational Reddit posts that lack strict formatting and rarely duplicate the gold summary.Each post-summary pair is created by the same user, although posts come from many users.
- Model: MMN stores source information at word, sentence, paragraph, and document abstraction levels, then sequentially reads these representations to generate a summary sentence.The model is motivated by the need to understand entire documents while identifying salient words, phrases, and sentences.
- Model: MMN replaces recurrent sequence-to-sequence encoding with convolutional memory that captures long-term information and representations across multiple ranges.Different receptive fields support representations spanning sentences, paragraphs, and the whole text.
- Results: Quantitative evaluation and Amazon Mechanical Turk user studies show MMN outperforms state-of-the-art abstractive methods on Reddit TIFU, Newsroom’s abstractive subset, and XSum.The paper presents this as evidence for the model’s effectiveness across multiple datasets.
2 Related Work
Prior abstractive summarization research relies heavily on formal documents and RNN-based sequence-to-sequence architectures. This work distinguishes Reddit TIFU through informal, user-generated text and MMN through explicit multi-level convolutional memory.
- Neural Abstractive Summarization: RNN-based sequence-to-sequence models dominate abstractive summarization, but their sequential representations can limit access to distant information and multiple representation ranges.The paper contrasts these limitations with convolutional memory.
- Neural Memory Networks: MMN replaces an RNN encoder with explicit multi-level convolutional memory covering sentences, paragraphs, and the whole text.Its stated novel features include dilated memory writing, normalized gated tanh units, and non-RNN output generation.
- Summarization Datasets: Most existing summarization datasets use formal sources, especially news articles, while Reddit TIFU uses posts from an online discussion forum.The paper positions Reddit TIFU as a dataset built from informal online text.
- Summarization Datasets: Reddit TIFU is larger than Rotten Tomatoes and uses post-summary pairs written by the same user, whereas Rotten Tomatoes combines reviews with an editor-written consensus.The paper reports 3.7K Rotten Tomatoes posts compared with 80K TIFU-short posts in Table 1.
- Neural Memory Networks: Earlier memory-network summarization work addressed extractive summarization by storing embeddings of individual sentences, whereas MMN targets abstractive summarization.The paper identifies this as a key distinction from closely related memory-network approaches.
3 Reddit TIFU Dataset
Reddit TIFU is constructed as a large-scale abstractive summarization dataset from informal Reddit posts, with body text paired with short-title and long-TL;DR summaries. Its text exhibits weaker lead bias and stronger abstractness than news-based datasets, making extractive shortcuts less effective.
- Dataset construction: Reddit TIFU uses Reddit TIFU posts collected from January 2013 through March 2018, pairing each body with its title and TL;DR as short and long summaries.The dataset contains TIFU-short and TIFU-long sets, respectively.
- Dataset construction: Posts follow subreddit rules requiring contextual bodies, descriptive titles, and TL;DR summaries covering the incident and its consequences.These posting rules provide the source-summary structure used to construct the corpus.
- Abstractive properties: Reddit TIFU shows a relatively uniform distribution of gold-summary bigrams across source-text positions, unlike CNN/DailyMail and Newsroom.The comparison is based on the density histogram of relative bigram locations.
- Abstractive properties: The Lead baseline obtains the lowest scores on TIFU, indicating that summaries are harder to produce by selecting only early source sentences or words.Lead selects the first few sentences or words, so its score reflects the strength of locational bias.
- Abstractive properties: TIFU has the lowest Ext-Oracle ROUGE scores among the compared datasets, showing that source sentences similar to gold summaries rarely occur in the dataset.Ext-Oracle selects sentences with the highest average F1 ROUGE-1/2/L and serves as an extractive upper bound.
- Abstractive properties: High PG/Lead and PG/Oracle ratios indicate that TIFU is potentially an excellent benchmark for evaluating abstractive summarization systems.The ratios quantify difficulty for extractive methods and suitability for abstractive methods, respectively.
4 Multi-level Memory Networks (MMN)
MMN replaces a conventional RNN encoder with explicit multi-level convolutional memories that represent source text at different abstraction levels. It attends to these memories while generating the summary one word at a time.
- Multi-level Memory: MMN memorizes source text at multiple abstraction levels and generates summary words sequentially from relevant memory information.Its memory levels can correspond to word-, sentence-, paragraph-, and document-level representations.
- Memory Writing with Dilated Convolution: Dilated convolutions expand receptive fields without increasing computational cost proportionally, allowing higher layers to represent wider input ranges.With dilation rate d = 1 the operation becomes regular convolution; larger dilation widens the represented context.
- Normalized Gated Tanh Units: Normalized gated tanh units extend gated tanh units with weight and layer normalization to preserve activation variances.The authors report that NGTU significantly improves performance compared with the residual-block scaling heuristic used in earlier work.
- Multi-level Memory: The model uses outputs from multiple CNN layers as separate memories, with receptive fields controlling the represented context.For example, smaller and larger receptive fields can produce sentence-level and document-level embeddings.
- State-Based Sequence Generation: During decoding, the model attends to each memory level, combines the resulting representations, and selects the highest-probability next word until EOS.The decoder also uses causal dilated convolutions and global conditioning from a whole-document embedding.
- Training: Training uses softmax cross-entropy with label smoothing, which replaces extreme target probabilities with a smoothed distribution.The smoothing parameter is set to 0.1, with target probability 1 − ϵ and non-target probability ϵ/V.
5 Experiments
The experiments evaluate MMN against abstractive and extractive baselines on Reddit TIFU and two abstractive benchmarks, using automatic metrics, ablations, qualitative examples, and AMT preferences. MMN outperforms the compared abstractive methods, while Reddit TIFU shows especially strong abstractive characteristics.
- Experimental Setup: The evaluation measures perplexity and ROUGE-1/2/L, where lower perplexity and higher ROUGE indicate better summarization.
- Experimental Setup: MMN is compared with three state-of-the-art abstractive systems, an attention-based seq2seq model, heuristic extractive methods, and component-removal variants.The variants remove dilated convolution, multi-level memory, or the normalized gated tanh unit.
- Quantitative Results: MMN outperforms state-of-the-art abstractive methods on both ROUGE and perplexity for TIFU-short/long.Table 3 reports these results on the TIFU-short/long dataset.
- Quantitative Results: Abstractive methods outperform extractive methods on TIFU, and their performance gaps are much larger there than in other datasets.The extractive Ext-Oracle is not effective because retrieving existing source sentences is insufficient for this highly abstractive dataset.
- Quantitative Results: MMN outperforms RNN- and convolutional-based methods in all ROUGE scores on Newsroom-Abs and XSum, including T-ConvS2S.The comparison uses scores from the original papers for systems other than MMN.
- Human and Qualitative Evaluation: Human annotators significantly prefer MMN summaries to baseline summaries in AMT tests, while the gold summary receives the most votes.The tests use randomly sampled examples and three turkers per example.
- Human and Qualitative Evaluation: Qualitative examples show baselines focusing mainly on keywords, whereas MMN considers keywords and whole context through multi-level memory.
6 Conclusions
The paper concludes that Reddit TIFU provides an abstractive dataset for informal online text and that MMN is effective for this task. It identifies broader online forums as a future data source.
- The authors introduce Reddit TIFU for abstractive summarization on informal online text.
- Experiments show that Reddit TIFU is uniquely abstractive and MMN is highly effective.
- Future work can explore online forums such as Quora, Stack Overflow, and other subreddits.
A Implementation Details
The implementation uses Xavier initialization, Adam optimization, weight normalization, learning-rate decay, gradient clipping, and dataset-specific training durations.
- Optimization: All parameters are initialized with Xavier initialization, and Adam uses β1 = 0.9, β2 = 0.999, and ϵ = 1e-8.
- Optimization: The models apply weight normalization, use learning rate 0.001, and clip gradients at 0.3.
- Optimization: Every 4 epochs, the learning rate is divided by 10 until reaching 0.0001.
- Training: Training lasts up to 12 epochs for TIFU-short and 60 epochs for TIFU-long.Table 6 summarizes hyperparameters across TIFU-short/long, Newsroom, and XSum experiments.
B Novel N-gram Ratios
Novel N-gram ratios quantify how much reference summaries use N-grams absent from source texts, providing a measure of extractive bias. Reddit TIFU is compared with other summarization datasets.
- Metric: The novel N-gram ratio is the number of reference-summary N-grams absent from the source divided by the total number of reference-summary N-grams.Higher ratios indicate fewer identical N-grams between source and reference summary.
- Dataset Comparison: CNN/DailyMail, New York Times, and Newsroom exhibit low novel 1-gram ratios.
- Dataset Comparison: Datasets more favorable to abstractive methods have higher novel N-gram ratios.The comparison includes Newsroom-Ext, Newsroom-Mix, and Newsroom-Abs.
- Dataset Comparison: Newsroom-Abs and XSum have higher novel N-gram ratios than the compared datasets.
- Dataset Comparison: Table 7 compares novel N-gram ratios between Reddit TIFU and other summarization datasets.
C More Examples
Figure 6 presents selected source texts, reference summaries, and generated summaries from MMN and baseline systems. The examples suggest that MMN uses both keywords and broader context, whereas baselines focus mainly on keywords.
- C More Examples: Figure 6 compares source texts, reference summaries, and generated summaries from MMN and baseline systems.Each example set contains all three elements for qualitative comparison.
- C More Examples: MMN considers keywords together with the whole context when generating summaries.The paper attributes this behavior to MMN's multi-level memory.
- C More Examples: Baselines generate summaries by focusing mostly on selected keywords.This contrast is illustrated in the selected generation examples.
- C More Examples: The examples include informal narratives about a first motorcycle ride and a lost phone, as well as an underage-drinking incident.These source-text excerpts illustrate the varied narrative content used in the qualitative examples.