Source-linked AI summary
LCSTS: A Large Scale Chinese Short Text Summarization Dataset
Baotian Hu, Qingcai Chen, Fangze Zhu
TL;DR
Automatic summarization lacks large datasets, especially for constructing full-text summaries. The paper builds LCSTS from naturally annotated Sina Weibo posts, manually assesses summary relevance, and evaluates RNN-based generation, reporting promising results and a baseline for future work.
Problem
Automatic text summarization is difficult, and large summarization datasets are scarce, particularly for full-text summary construction.
Method
The paper mines verified Sina Weibo content into LCSTS, manually labels 10,666 pairs for relevance, and applies RNN encoder-decoder models.
Results
RNN-based summary generation achieves promising results, with context outperforming no-context models and character-based input outperforming word-based input.
Takeaways & Limitations
LCSTS provides a large-scale resource and an RNN baseline for Chinese short text summarization research.
Takeaways & Limitations
Treating each short text as one sequence may be unreasonable because many short texts contain several sentences, motivating hierarchical RNNs.
Abstract
from arXiv · showhide
Automatic text summarization is widely regarded as the highly difficult problem, partially because of the lack of large text summarization data set. Due to the great challenge of constructing the large scale summaries for full text, in this paper, we introduce a large corpus of Chinese short text summarization dataset constructed from the Chinese microblogging website Sina Weibo, which is released to the public {http://icrc.hitsz.edu.cn/Article/show/139.html}. This corpus consists of over 2 million real Chinese short texts with short summaries given by the author of each text. We also manually tagged the relevance of 10,666 short summaries with their corresponding short texts. Based on the corpus, we introduce recurrent neural network for the summary generation and achieve promising results, which not only shows the usefulness of the proposed corpus for short text summarization research, but also provides a baseline for further research on this topic.
1 Introduction
The paper addresses difficult automatic summarization by mining naturally annotated Sina Weibo posts, creating LCSTS and using it to support neural summary generation.
- Motivation: Sina Weibo posts are naturally annotated resources because authors often provide concise summaries under the 140-character limit.Posts from informative accounts such as People’s Daily offer short, author-written summaries alongside the original text.
- Background: Automatic summarization requires systems to understand text and reorganize important information into coherent, informative, short summaries.Traditional abstractive systems commonly extract textual elements and then rewrite or paraphrase them.
- Dataset contribution: LCSTS is introduced as a large-scale Chinese short text summarization dataset mined from Sina Weibo.The dataset targets research constrained by limited large-scale summarization data.
- Contributions: The paper provides large-scale training data, a human-labeled test set, dataset analysis, and an RNN encoder-decoder baseline.The contributions include manual quality checking of 10,666 instances and recurrent neural summary generation.
2 Related Work
Prior summarization datasets were generally small, while related work also explored headline generation and naturally annotated web resources for language-processing tasks.
- Automatic summarization: Most existing summarization datasets contain fewer than 1000 documents or examples, limiting labeled-data availability.DUC2002, for example, contains 567 documents with two human summaries per document.
- Headline generation: Headline generation produces one sentence describing or entitling a text and is related to short-text summarization.A cited English financial-news dataset contains 1.3 million headlines but was not publicly available.
- Naturally annotated resources: Naturally Annotated Web Resources are user-generated pages, blogs, and microblogs containing marks created unintentionally for communication.Researchers can mine these resources for knowledge or useful data.
- This work: This work applies naturally annotated resources to construct a large-scale Chinese short text summarization dataset.The stated purpose is to facilitate research on text summarization.
3 Data Collection
The dataset is collected from verified Sina Weibo organizations through seeded crawling, rule-based filtering, cleaning, and extraction of short-text–summary pairs.
- User selection: Accounts are filtered using human-written rules requiring blue verification and more than 1 million followers.Verified organizations are selected because they are more likely to produce clean, formal, informative text.
- Quality control: The process includes substantial human intervention at each step to maintain crawled-text quality.The source emphasizes manual involvement alongside automated crawling and filtering.
- User selection: The collection begins with 50 popular organization users spanning domains including politics, economics, military, movies, and games.These users serve as seeds for discovering additional accounts.
- Pair extraction: The authors crawl selected users’ weibos, then filter, clean, and extract short text–summary pairs using about 100 rules.Five people developed the rules by carefully investigating raw text.
4 Data Properties
LCSTS combines a large training collection with human-labeled relevance data and a separately constructed test set, while its summaries are generally relevant but include difficult abstractive cases.
- Dataset partitions: 2,400,591 short text–summary pairs form Part I and support supervised training for summary generation.This is the main content of LCSTS.
- Dataset partitions: 10,666 Part II pairs receive human relevance scores from 1 to 5, where 1 is least relevant and 5 most relevant.Five volunteers annotate randomly sampled pairs, with one annotator per pair.
- Length analysis: Figure 3 compares length distributions for original and segmented short texts and summaries using medians and quartiles.The red line marks the median and box edges mark the quartiles.
- Data quality: Less than 20% of the data receives scores of 1 or 2, and these lower-quality pairs can be filtered using a trained classifier.Scores of 3–5 generally indicate informative, concise, relevant summaries, whereas scores of 1–2 are often headlines or comments.
- Dataset partitions: Part III contains 1,106 pairs and uses pairs scored 3, 4, or 5 as the short-text summarization test set.Three annotators label the same 2000 texts, and commonly scored pairs are extracted.
5 Experiment
The experiments evaluate GRU-based RNN encoder-decoder summarization with character- and word-based inputs, comparing decoding with and without context. Context improves results for both input types, while character-based input avoids many unknown-word tokens seen with word-based input.
- Models and evaluation: Two architectures decode either from the encoder’s final hidden state without local context or from all encoder hidden states with context.The context-based architecture follows the attention approach described by Bahdanau et al.
- Results: Generated summaries from the context-based RNN are described as close to human-written summaries when enough training data is provided.Figure 7 illustrates examples of generated summaries.
- Results: RNNs with context outperform RNNs without context for both character-based and word-based inputs.The authors interpret the result as evidence that encoder hidden states can represent summary-word context.
- Results: Character-based input outperforms word-based input, while context-based word models produce many UNKs for rare names and organizations.The paper gives the company name “愿景光电子” as an example replaced by an UNK token.
6 Conclusion and Future Work
The paper concludes that its large-scale dataset and RNN-based methods produce promising results, while identifying sequence structure and rare words as important areas for future improvement.
- Conclusion: The authors report promising results from constructing a large-scale Chinese short text summarization dataset and applying RNN-based methods.They frame this work as a starting point for deep models on the task.
- Future work: Treating each short text as one sequence may be unreasonable because many short texts contain several sentences.The authors suggest hierarchical RNNs as one possible direction.
- Future work: Rare-word handling remains important for summary generation, especially with word-based inputs.The paper connects this issue to neural generative modeling research.
- Future work: The authors also plan to construct a large-document summarization dataset from naturally annotated web resources.This extends the dataset-construction direction beyond short texts.