Source-linked AI summary
Simple Unsupervised Keyphrase Extraction using Sentence Embeddings
Kamil Bennani-Smires, Claudiu Musat, Andreea Hossmann, Michael Baeriswyl, Martin Jaggi
TL;DR
Keyphrase extraction needs fast, nonredundant outputs without large labeled corpora or broader document collections. The paper introduces EmbedRank, which uses sentence embeddings to extract and diversify phrases from a single document, and reports stronger benchmark F-scores than graph-based methods on two of three datasets, alongside human preference for diverse selections.
Problem
Existing systems face training-data, domain-generalization, speed, redundancy, and corpus-dependence limitations in keyphrase extraction.
Method
EmbedRank uses sentence embeddings to rank phrases from a single document by informativeness and diversify selections through embedding-based MMR.
Results
EmbedRank clearly outperforms complex graph-based state-of-the-art methods on two of three common datasets under traditional F-score evaluation, while users highly prefer diverse keyphrases.
Takeaways & Limitations
Embedding-based selection supports single-document extraction while explicitly balancing phrase informativeness with diversity.
Takeaways & Limitations
Word-score aggregation can over-generate redundant phrases and reduce the diversity of extracted results.
Abstract
from arXiv · showhide
Keyphrase extraction is the task of automatically selecting a small set of phrases that best describe a given free text document. Supervised keyphrase extraction requires large amounts of labeled training data and generalizes very poorly outside the domain of the training data. At the same time, unsupervised systems have poor accuracy, and often do not generalize well, as they require the input document to belong to a larger corpus also given as input. Addressing these drawbacks, in this paper, we tackle keyphrase extraction from single documents with EmbedRank: a novel unsupervised method, that leverages sentence embeddings. EmbedRank achieves higher F-scores than graph-based state of the art systems on standard datasets and is suitable for real-time processing of large amounts of Web data. With EmbedRank, we also explicitly increase coverage and diversity among the selected keyphrases by introducing an embedding-based maximal marginal relevance (MMR) for new phrases. A user study including over 200 votes showed that, although reducing the phrases' semantic overlap leads to no gains in F-score, our high diversity selection is preferred by humans.
1 Introduction
The paper targets fast, nonredundant keyphrase extraction from single documents without relying on labeled corpora or external document collections. It proposes EmbedRank and reports strong benchmark performance plus human preference for diverse outputs.
- Keyphrases support search, summarization, and categorization, while Web and social-media applications require fast, disjoint outputs.
- Supervised methods require large annotated corpora and generalize poorly beyond their training domains.
- EmbedRank extracts keyphrases unsupervised from only the current document using sentence embeddings.
- EmbedRank clearly outperforms complex graph-based state-of-the-art methods on two of three common datasets under traditional F-score evaluation.
- A user study found that users highly prefer keyphrases with diversity, an aspect not captured by F-score.
- The paper evaluates EmbedRank across related work, method components, diversity enhancement, experiments, and real-world examples.
2 Related Work
Related work distinguishes corpus-dependent and corpus-independent unsupervised extraction, reviews graph and embedding approaches, and identifies redundancy and limited diversity as weaknesses of word-score aggregation. EmbedRank addresses these weaknesses by comparing document and phrase embeddings in a shared semantic space.
- Unsupervised keyphrase extraction is divided into corpus-dependent and corpus-independent approaches.
- Corpus-independent methods use only the target document; most existing systems are graph-based, with KeyCluster and TopicRank as notable exceptions.
- Graph-based systems build word cooccurrence graphs, rank words, aggregate scores into phrases, and select candidates using part-of-speech patterns.
- Aggregating individual word scores can over-generate redundant phrases and produce phrases that are not informative as wholes.
- MMR is used because it combines relevance and diversity in a simple, controllable, and interpretable way.
- TopicRank clusters phrases by shared words, whereas multipartite graphs encode topical information structurally.
- EmbedRank represents documents and candidate phrases as vectors, enabling document proximity for informativeness and candidate distance for diversity.
- Sent2Vec produces sentence embeddings from word n-gram features and offers faster inference than Paragraph Vector or Skip-Thought.
3 EmbedRank: From Embeddings to Keyphrases
EmbedRank extracts linguistically filtered candidate phrases, embeds them and the document in a shared vector space, and ranks phrases by semantic proximity. The method uses cosine similarity as an informativeness proxy, supports Doc2Vec and Sent2Vec variants, and exposes redundancy that motivates later diversification.
- EmbedRank method: EmbedRank first extracts candidate phrases consisting of zero or more adjectives followed by one or more nouns.
- Embedding and ranking: It then embeds the document and each candidate phrase in the same vector space and ranks candidates by their distance to the document.
- Embedding and ranking: The document embedding applies noise reduction by retaining only adjectives and nouns from the input document.
- Embedding variants: The experiments compare Doc2Vec-based EmbedRank d2v with Sent2Vec-based EmbedRank s2v using pretrained models.
- Embedding variants: The embedding-space visualization uses multidimensional scaling with cosine distance on 700-dimensional embeddings.
- Embedding variants: Sent2Vec embeddings have 700 dimensions, Doc2Vec embeddings have 300, and Sent2Vec inference uses a single linear pass through the text.
- Embedding and ranking: Cosine similarity between candidate and document embeddings serves as a measure of phrase informativeness.
- Selecting candidates: The method selects ten highlighted keyphrases in the example, which align with the document title.
4 EmbedRank++: Increasing Keyphrase Diversity with MMR
EmbedRank++ adapts MMR to select keyphrases that balance informativeness with dissimilarity, addressing redundancy in fixed-size outputs. Its trade-off parameter λ tunes the balance between relevance and diversity.
- 4 EmbedRank++: Increasing Keyphrase Diversity with MMR: Four of ten keyphrase slots can be occupied by redundant phrases, hindering diversification when a fixed number of phrases is extracted.Examples include topological shape/topological shapes and molecular equivalence variants.
- 4 EmbedRank++: Increasing Keyphrase Diversity with MMR: MMR combines keyphrase informativeness with dissimilarity among selected keyphrases, following the analogy of search-result diversification.The method adapts MMR from information retrieval and text summarization to keyphrase extraction.
- 4 EmbedRank++: Increasing Keyphrase Diversity with MMR: When λ = 1, MMR produces a relevance-ranked list, whereas λ = 0 produces a maximal-diversity ranking.The parameter controls the relevance–diversity trade-off.
- 4 EmbedRank++: Increasing Keyphrase Diversity with MMR: The adapted formulation uses the document embedding for relevance and candidate-phrase embeddings for dissimilarity among selected keyphrases.C denotes candidate keyphrases, K extracted keyphrases, doc the document embedding, and Ci and Cj candidate-phrase embeddings.
- 4 EmbedRank++: Increasing Keyphrase Diversity with MMR: Normalized cosine similarity makes relevance and diversity equally important at λ = 0.5, with an analogous transformation applied to candidate-phrase similarities.EmbedRank++ uses this MMR-based final selection to return relevant and diverse keyphrases.
5 Experiments and results
Experiments compare EmbedRank with corpus-independent graph-based methods across three datasets, showing strong performance on short and medium documents but weaker results on long documents. A diversity-oriented variant lowers F-score yet is preferred by users.
- 5.2 Performance Comparison: EmbedRank outperforms competing methods on two of three datasets in precision, recall, and Macro F1 score.The comparison uses Inspec, DUC2001, and NUS, with results reported at multiple extraction cutoffs.
- 5.2 Performance Comparison: Long documents favor Multipartite, while positional information brings EmbedRankpositional to TopicRank-level performance and near Multipartite-level performance.The authors attribute this pattern to explicit positional and topic information in long documents.
- 5.2 Performance Comparison: Short and medium documents capture semantic information effectively through the topology of the embedding space.The paper contrasts this with the stronger role of explicit topic representation for long documents.
- 5.2 Performance Comparison: Sent2Vec embeddings significantly outperform Doc2Vec on DUC2001 and NUS, while making almost no difference on Inspec.The authors link the Inspec result to its small texts and high proportion of single-word keyphrases.
- 5.3 Keyphrase Diversity and Human Preference: EmbedRank++ reduces redundancy but decreases F-score relative to EmbedRank when λ = 0.5 is used to balance informativeness and diversity.The variant was evaluated on the same three datasets using the adapted MMR equation.
- 5.3 Keyphrase Diversity and Human Preference: Users largely prefer EmbedRank++ keyphrases despite the lower F-scores, based on 214 votes across 20 Inspec and 20 DUC2001 documents.The study is described as preliminary and does not support a strong claim about F-score’s usefulness.
6 Discussion
EmbedRank supports fast, corpus-free extraction for real-time news analysis and produces disjoint keyphrases that form succinct summaries. Phrase-level embeddings also enable grouping semantically similar candidate phrases before presentation.
- 6 Discussion: EmbedRank can extract keyphrases in environments such as news articles without requiring an external corpus.The extraction is described as fast enough for real-time computation and visualization.
- 6 Discussion: Disjoint keyphrases improve readability by creating a succinct summary of the original article.This practical benefit is presented alongside real-time extraction for news content.
- 6 Discussion: Phrase-level analysis enables grouping candidates with similar embeddings and selecting a representative phrase for users.Possible representatives include the highest-scoring or longest phrase in a group.
7 Conclusion
EmbedRank and EmbedRank++ provide simple, scalable, entirely unsupervised keyphrase extraction from single documents using sentence embeddings. Their corpus-independent design supports Web-scale use while modeling both informativeness and diversity.
- EmbedRank and EmbedRank++ extract keyphrases from single documents without requiring the larger corpus to which a document belongs.
- Sentence embeddings replace traditional graph representations while modeling both informativeness and diversity.
- Sent2Vec-based EmbedRank consistently improves the state of the art on short and medium length documents and offers faster inference for Web-scale settings.