Source-linked AI summary
Deep Keyphrase Generation
Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, Yu Chi
TL;DR
Existing keyphrase extraction methods often rank text-derived candidates, leaving absent keyphrases and deeper semantics unaddressed. The paper uses an RNN encoder-decoder with copying to generate keyphrases, and experiments show strong performance on present phrases alongside absent-phrase generation. The authors note that evaluation against human annotators remains future work.
Problem
Existing extraction approaches cannot identify absent keyphrases and their occurrence-based features cannot reveal the full semantics underlying document content.
Method
An RNN encoder-decoder compresses document semantics and uses a copying mechanism to retain important source phrases while generating keyphrases.
Results
Across six datasets, the model significantly outperforms six supervised and unsupervised extraction baselines, recalls up to 20% of absent keyphrases, and generates topically relevant absent phrases.
Takeaways & Limitations
Deep keyphrase generation can produce representative phrases for both present and absent content, with potential use in information retrieval and document browsing.
Takeaways & Limitations
The authors have not yet compared predictions with human annotators or used human judges to evaluate predicted-phrase quality.
Abstract
from arXiv · showhide
Keyphrase provides highly-condensed information that can be effectively used for understanding, organizing and retrieving text content. Though previous studies have provided many workable solutions for automated keyphrase extraction, they commonly divided the to-be-summarized content into multiple text chunks, then ranked and selected the most meaningful ones. These approaches could neither identify keyphrases that do not appear in the text, nor capture the real semantic meaning behind the text. We propose a generative model for keyphrase prediction with an encoder-decoder framework, which can effectively overcome the above drawbacks. We name it as deep keyphrase generation since it attempts to capture the deep semantic meaning of the content with a deep learning method. Empirical analysis on six datasets demonstrates that our proposed model not only achieves a significant performance boost on extracting keyphrases that appear in the source text, but also can generate absent keyphrases based on the semantic meaning of the text. Code and dataset are available at https://github.com/memray/OpenNMT-kpg-release.
1 Introduction
The paper argues that conventional keyphrase extraction misses absent phrases and deeper document semantics. It proposes an RNN encoder-decoder with copying to generate both absent and in-text keyphrases.
- Keyphrases condense a document’s main semantic meaning and support understanding, organization, and access to content.
- Existing extraction methods generate phrase candidates and rank them using textual patterns or occurrence-based features.Candidate generation commonly uses n-grams or noun phrases, while ranking often uses features such as TF-IDF and PageRank.
- Absent keyphrases do not match contiguous source-text subsequences, yet they constitute large portions of keyphrases across four datasets and cannot be extracted by prior approaches.
- Human annotators rely on semantic understanding and contextual importance, so useful keyphrase prediction must capture meaning beyond exact source words.For example, “Latent Dirichlet Allocation” may suggest “topic modeling” or “text mining.”
- The proposed RNN model compresses semantic information into a dense vector and adds copying to locate important source spans.This combination supports generation regardless of whether keyphrases appear in the text while retaining important in-text information.
- The model recalls up to 20% of absent keyphrases and significantly outperforms six supervised and unsupervised extraction baselines.
2 Related Work
Prior work largely extracts and ranks candidate phrases, while neural approaches introduced end-to-end sequence modeling and copying mechanisms. However, sequence-labeling approaches cannot predict absent keyphrases.
- Typical extraction pipelines generate many candidates using part-of-speech patterns, n-grams, or noun phrases before scoring them.
- Candidate scoring has used supervised and unsupervised learning, including binary classification and features designed to estimate phrase importance.
- Language models have separately measured phrase phraseness and informativeness, while word alignment has addressed vocabulary gaps between documents and keyphrases.
- A recurrent neural network for tweets used sequence labeling but could not predict keyphrases absent from the source text.
- Encoder-decoder models provide end-to-end variable-length sequence modeling, with attention locating relevant inputs and copying transferring source content.
3 Methodology
The method uses an RNN encoder-decoder with attention and copying to generate keyphrases from source text, including phrases absent from the input.
- 3.1 Problem Definition: Each source document is paired separately with each target keyphrase before training the sequence-to-sequence model.A sample with M_i keyphrases becomes M_i source–target pairs sharing the same source sequence.
- 3.2 Encoder-Decoder Model: The encoder compresses a variable-length source sequence into hidden states and a context vector representing the whole input.The decoder then uses this representation to generate a variable-length keyphrase sequence word by word.
- 3.3 Details of the Encoder and Decoder: A bidirectional GRU encoder, forward GRU decoder, and attention mechanism provide the model’s main sequence-processing architecture.Attention computes the context vector as a weighted sum of encoder hidden representations, with weights based on soft alignment.
- 3.4 Copying Mechanism: The generative component predicts from the vocabulary, while the copying component considers only unique words appearing in the source text.This design supports both semantic generation and source-based word reuse.
- 3.4 Copying Mechanism: The copying mechanism combines generated-word probabilities with probabilities of copying words from the source text.Its copy component uses positional attention over source words, helping preserve important in-text terms and predict source words outside the fixed vocabulary.
4 Experiment Settings
The experiments evaluate keyphrase generation on four established scientific-publication datasets and the newly constructed KP20k dataset, using large-scale training data and standard baselines and metrics.
- Datasets: 567,830 articles were collected after removing duplicates and overlaps, with 527,830 used for model training and 40,000 held out.The held-out articles form the KP20k test set and a validation set.
- Datasets: The evaluation combines four widely adopted scientific-publication datasets with the newly created KP20k testing dataset.The source text is the concatenation of each publication’s title and abstract.
- Training: 2,780,316 text–keyphrase pairs were used for training, with text formed from titles and abstracts and keyphrases taken from author-assigned keywords.Preprocessing includes tokenization, lowercasing, and replacing digits with a special symbol.
- Baselines: The study compares attention-only RNN and attention-plus-copying CopyRNN models with four unsupervised and two supervised baseline algorithms.The baselines are Tf-Idf, TextRank, SingleRank, ExpandRank, KEA, and Maui.
- Evaluation: Macro-averaged precision, recall, and F1 measure performance, with Porter stemming used when matching predicted and reference keyphrases.Dataset-specific splits include held-out testing sets and five-fold cross-validation for supervised baselines on NUS.
5 Results and Analysis
Across present-keyphrase, absent-keyphrase, and cross-domain evaluations, CopyRNN generally benefits from combining semantic generation with source-text information, though transfer to news remains constrained.
- 5.1 Predicting Present Keyphrases: CopyRNN significantly outperformed the RNN model and all baselines on present-keyphrase extraction, exceeding the best baselines by more than 20% on average.The comparison used F-measure at top 5 and top 10 predictions across five datasets.
- 5.1 Predicting Present Keyphrases: RNN predictions were often too general, whereas CopyRNN generated more detailed source-related phrases such as “video metadata” and “integrated ranking”.The example also includes a CopyRNN error involving “rich content”.
- 5.2 Predicting Absent Keyphrases: CopyRNN recalled around 8% and 15% of absent keyphrases at top 10 and top 50 predictions, respectively, on average across five datasets.It outperformed RNN in this task, although the improvement was smaller than for present-keyphrase extraction.
- 5.2 Predicting Absent Keyphrases: Both models could paraphrase source meaning to recover absent phrases, while CopyRNN additionally captured detailed information from highlighted text segments.For example, “video indexing” was predicted although “indexing” did not appear in the source text.
- 5.3 Transferring the Model to the News Domain: On DUC-2001 news articles, CopyRNN extracted some correct keyphrases but outperformed only TextRank and KeyCluster, lagging behind three other baselines.The model was trained on scientific publications and evaluated on news.
- 5.3 Transferring the Model to the News Domain: CopyRNN recalled 766 keyphrases in the news-domain experiment, including correctly predicted names of persons and places; 14.3% contained out-of-vocabulary words.The domain shift introduced more unknown words and increased reliance on positional and syntactic features.
6 Discussion
The CopyRNN model generates both present and absent keyphrases and may support information retrieval and user browsing by producing representative short text from longer documents.
- CopyRNN predicts topically relevant keyphrases that are absent from the source text as well as present keyphrases.
- The model maps long paper abstracts to representative short keyphrase text chunks.
- CopyRNN could potentially improve information retrieval by generating high-quality index terms.
- CopyRNN could assist user browsing by summarizing long documents into short, readable phrases.
- The model was tested on scientific publications and news articles, including direct application to news articles without adaptive training.
- The encoder-decoder framework may extend core-content discovery beyond text to images and videos.
7 Conclusions and Future Work
The paper proposes an RNN-based generative model for scientific-text keyphrase prediction that uses deep semantic meaning and copying to handle rare phrases. Experiments support generating both present and absent keyphrases, while future work targets human evaluation and target-keyphrase correlation.
- The paper proposes an RNN-based generative model for predicting keyphrases in scientific text.
- Figure 1 compares predicted keyphrases from RNN and CopyRNN, with bold phrases marking correct predictions.
- The model summarizes text using deep semantic meaning and incorporates copying to handle rarely occurring phrases.
- Comprehensive empirical studies demonstrate effectiveness for generating both present and absent keyphrases across different text types.
- The authors evaluated the model only with offline experiments and proposed comparison with human annotators and judges as future work.
- The current model does not fully consider correlation among target keyphrases, motivating exploration of multiple-output optimization.
Erratum
The erratum corrects the reported scores from micro-averaged to macro-averaged values and states that the correction does not affect the paper’s conclusions.
- The authors mistakenly reported micro-averaged scores instead of macro-averaged scores and updated all reported scores.
- The difference between micro-averaged and macro-averaged scores was marginal, so the correction did not affect the submitted conclusions.