A Review of Keyphrase Extraction

Eirini Papagiannopoulou; Grigorios Tsoumakas

Source-linked AI summary

A Review of Keyphrase Extraction

Eirini Papagiannopoulou, Grigorios Tsoumakas

arXiv:1905.05044v2May 13, 2019cs.CLcs.IR

TL;DR

Reliable evaluation remains a major challenge in keyphrase extraction. This paper organizes prior work and compares methods, finding that simple unsupervised approaches are strong baselines while deep learning achieves state-of-the-art results.

Problem
Reliable evaluation measures and approaches remain a major unresolved issue in keyphrase extraction.
Method
The paper comprehensively organizes prior work and conducts quantitative and qualitative empirical studies of keyphrase extraction methods.
Results
Simple unsupervised methods such as TfIdf2 are strong baselines, while deep learning methods achieve state-of-the-art results.
Takeaways & Limitations
Simple unsupervised methods should be considered in empirical studies of keyphrase extraction.
Takeaways & Limitations
Partial-match evaluation can fail to properly assess the syntactic correctness of returned keyphrases.

Abstract

from arXiv · show

Keyphrase extraction is a textual information processing task concerned with the automatic extraction of representative and characteristic phrases from a document that express all the key aspects of its content. Keyphrases constitute a succinct conceptual summary of a document, which is very useful in digital information management systems for semantic indexing, faceted search, document clustering and classification. This article introduces keyphrase extraction, provides a well-structured review of the existing work, offers interesting insights on the different evaluation approaches, highlights open issues and presents a comparative experimental study of popular unsupervised techniques on five datasets.

datasets.

This section is associated with keyphrase extraction, review and survey work, with emphasis on unsupervised keyphrase methods.

The section concerns keyphrase extraction and reviews or surveys of unsupervised keyphrase methods.

1 | INTRODUCTION

This article reviews keyphrase extraction as the automatic extraction of representative phrases that concisely summarize documents and support information-management tasks. It organizes existing methods and evaluation practices, catalogs resources, and reports an empirical comparison of commercial APIs and unsupervised methods.

Motivation: Keyphrase extraction automatically identifies representative phrases that concisely summarize documents for indexing, search, clustering, classification, summarization, and publishing workflows.Keyphrases enable semantic and faceted search, query expansion, document clustering and classification, human inspection, sentence extraction, and publishing-related recommendation and analysis.
Method landscape: Unsupervised methods are popular because they are domain independent and require no labeled training data, whereas supervised methods have stronger modeling capabilities and typically higher accuracy (Kim et al., 2013; Caragea et al., 2014; Meng et al., 2017).Manual labeling introduces subjectivity and substantial time and monetary costs.
Related work: Hasan and Ng (2014) identify four keyphrase-extraction error types: evaluation errors, redundancy errors, infrequency errors, and overgeneration errors.
Review contributions: The review systematically categorizes unsupervised and supervised methods, adds 37 methods beyond Hasan and Ng (2014), and traces their evolution, features, and class-imbalance issues.It also provides a timeline and discusses the main feature types used in supervised methods.
Evaluation and resources: The article surveys evaluation approaches and measures, lists popular datasets with their sources and properties, and catalogs commercial APIs and free software.Evaluation measures are discussed alongside their popularity in the literature.
Empirical study: It conducts a quantitative and qualitative empirical study of commercial APIs and state-of-the-art unsupervised methods to assess effects of evaluation approaches, measures, and ground-truth standards.The literature search covered Google Scholar, Springer Link, IEEE Xplore, ACM Digital Library, and DBLP, focusing mainly on high-quality journals and conference proceedings.

2 | UNSUPERVISED METHODS

This section reviews unsupervised keyphrase extraction methods that select and rank candidate lexical units using statistical, graph-based, topical, co-occurrence, and semantic information. It covers approaches based on knowledge graphs, embeddings, language-model divergences, and foreground–background corpus scoring.

2 | UNSUPERVISED METHODS: Unsupervised systems select candidate lexical units, rank them, and form keyphrases from top-ranked words or high-scoring phrases.Candidate selection may exclude stopwords or restrict words by part of speech.
2 | UNSUPERVISED METHODS: Graph-based methods are the most popular unsupervised approaches, while statistics-based methods retain research attention and semantics is increasingly incorporated.The review characterizes methods as statistics-based, graph-based, topic-based, citation or neighbor-information-based, semantic, and language-model-based.
2.1 | Statistics-based Methods: TfIdf is the common baseline, while later methods extend statistical scoring with candidate filtering, term position and frequency, compound-term boosting, and contextual information.
2.1 | Statistics-based Methods: Won et al. (2019) show that combining simple textual statistical features can achieve results that compete with state-of-the-art methods.Their method selects candidates using morphosyntactic patterns and combines term frequency, inverse document frequency, relative first occurrence, and length.
2.2 | Graph-based Ranking Methods: Graph-based ranking builds a graph whose nodes are candidate phrases and whose edges connect related candidates, then ranks nodes with algorithms such as PageRank, Positional Function, or HITS.PageRank has been successfully used for graph-based keyphrase extraction, and TextRank applies POS filtering and co-occurrence edges before iterative ranking.
2.2 | Graph-based Ranking Methods: Graph-based variants incorporate co-occurrence information differently: SingleRank weights edges by co-occurrence counts, whereas RAKE combines word frequency and degree to score phrases.RAKE partitions text using stopwords and delimiters and can identify keyphrases containing interior stopwords.
2.2 | Graph-based Ranking Methods: Graph-based methods combine statistical, positional, and word co-occurrence information; SGRank and PositionRank thereby improve overall performance.SGRank uses filtered n-grams, modified TfIdf, statistical heuristics, and graph-based ranking, whereas PositionRank uses position-biased weighted PageRank over word co-occurrences.
2.2.1 | Incorporating Information from Similar Documents/Citation Networks: ExpandRank extends SingleRank with neighboring documents, while CiteTextRank uses citation contexts to enrich keyphrase extraction.ExpandRank builds a graph over candidate words from the target document and its k nearest neighbors, weighted by document similarity and co-occurrence; CiteTextRank combines global, citing, and cited contexts with PageRank.
2.2.2 | Topic-based Methods: Topic-based methods extract phrases representative of document topics using clustering or LDA, with TopicRank selecting candidates across topics and topic-specific PageRank methods weighting topical importance.MultipartiteRank biases candidates appearing earlier, Single Topical PageRank reduces TPR’s cost to one PageRank per document, and combining differing topic models improves performance when their topics differ substantially.
2.2.3 | Graph-based Methods with Semantics: Topic-based methods can produce topics that are too general and vague, while co-occurrence methods lose information when words do not co-occur within the chosen window.LDA-based methods also commonly model unigrams although human interpretation often groups terms into phrases; phrase mining can address this limitation by identifying significant phrases and assigning their constituent terms a shared latent topic.
2.2.3 | Graph-based Methods with Semantics: Knowledge-graph methods address missing graph-of-words links and statistical information overload by clustering keyterms, integrating DBpedia subgraphs, and ranking phrases with Personalized PageRank, frequency, and position.Shi et al. (2017) connect semantically similar nouns and named entities through h-hop DBpedia graphs before applying Personalized PageRank; WikiRank instead combines TAGME concepts and candidate noun groups to maximize concept-score coverage.
2.2.3 | Graph-based Methods with Semantics: Key2Vec represents candidate keyphrases with domain-specific phrase embeddings, weights them by thematic similarity, and ranks a co-occurrence graph using weighted Personalized PageRank.The method derives a theme vector from phrases in the document’s opening sentence(s), connects candidates co-occurring within a window of 5, and weights edges using semantic similarity and co-occurrence frequency.
2.3 | Keyphrase Extraction based on Embeddings: Embedding-based methods rank candidate phrases by semantic similarity to document representations, using sentence embeddings in EmbedRank and document-specific GloVe reference vectors in RVA.EmbedRank represents candidates and documents in one vector space, whereas Papagiannopoulou and Tsoumakas (2018) train local GloVe embeddings on the document and rank candidates by cosine similarity to the title-and-abstract mean vector.
2.4 | Language Model-based Methods: Language-model methods compare foreground and background distributions, using phrase-level phraseness and informativeness based on Kullback-Leibler divergence.Tomokiyo and Hurst (2003) build unigram and n-gram models for the target document and document set; phraseness compares foreground unigram and n-gram models, while informativeness compares foreground and background models.
2.4 | Language Model-based Methods: The method applies n-gram language models to foreground and background corpora to assess phrases.
2.4 | Language Model-based Methods: Phrases receive a final score by summing phraseness and informativeness, after which they are sorted by that score.

3 | SUPERVISED METHODS

This section reviews supervised keyphrase extraction methods, spanning traditional feature-, sequence-, and graph-based approaches, deep neural models, and systematic feature families. It covers their features, comparative performance, semantic and cross-domain challenges, and responses to limited or unreliable supervision.

3.1 | Traditional Supervised Methods: Supervised extraction trains models on annotated documents to distinguish candidate keyphrases from non-keyphrases, generating positive and negative examples.Traditional methods also formulate extraction as learning to rank, which orders candidates by keyphrase scores rather than making hard classification decisions.
3.1 | Traditional Supervised Methods: KEA calculates TfIdf and first-occurrence features for candidate phrases; a later version adds a section-occurrence vector and morphological/linguistic features, using Naive Bayes.
3.1 | Traditional Supervised Methods: Ranking SVM represents candidate keyphrases with feature vectors and ranks them by transforming training data into ordered phrase pairs with preference labels.
3.1 | Traditional Supervised Methods: Supervised graph extensions can incorporate domain-controlled keyphrases that are absent from the document by linking them with document topics and applying graph co-ranking.TopicCoRank unifies a topic graph with controlled keyphrases and scores both through inner and outer recommendations.
3.1 | Traditional Supervised Methods: Yang et al. (2018) propose an event-oriented keyphrase-extraction model that segments documents into phrases, then applies ToLDA, treating each event as a topic and using user-defined seed keyphrases.
3.2 | Deep Learning Methods: Chen et al. (2018) propose CorrRNN, an alternative sequence-to-sequence architecture that captures correlations among keyphrases, using coverage and review mechanisms so phrases cover document topics and differ from each other.
3.2 | Deep Learning Methods: Deep learning methods reduce labeled-data requirements through synthetic keyphrases from TfIdf and TextRank, multi-task title generation, and topic-based adversarial transfer across domains.Ye and Wang (2018) mix synthetic and labeled data or share an encoder between keyphrase and title generation; Wang et al. (2018) use TANN with labeled resource-rich and unlabeled source/target data.
3.3 | Types of Features: Supervised methods organize input knowledge into statistical, positional, linguistic, context, stacking, and external-knowledge feature categories, while the taxonomy can expand with new methods and features.Tables 2 and 3 summarize popular features and representative methods by learning algorithm and incorporated knowledge.
3.3.1 | Statistical Features: Statistical features include Tf, Idf, TfIdf, phrase entropy, boolean score thresholds, frequent co-occurrences, and user-guided ToLDA topic distributions.These features appear across methods including KEA, MAUI, CeKE, SurfKE, PCU-ICL, Ranking SVM, MIKE, WINGNUS, and others.
3.3.2 | Positional Features: Positional features use phrase occurrence, section, title, abstract, and citation-context locations because early or title occurrences indicate greater phrase importance.Examples include Ranking SVM, MIKE, Shi et al. (2008), Gollapalli et al. (2017), and section-header frequency features in Nguyen and Kan (2007).
3.3.2 | Positional Features: Positional features include occurrence positions, spread, citation distance, and sentence boundaries to characterize a phrase’s role in the document.Popular supervised systems incorporate these features, including KEA, MAUI, CeKE, Ranking SVM, MIKE, WINGNUS, and PCU-ICL.
3.3.3 | Linguistic Features: Linguistic features use POS patterns, morphology, stems, capitalization, stopword and punctuation indicators, bibliography occurrence, and phrase length.POS features improve supervised extraction performance regardless of candidate-phrase selection, while phrase length is widely adopted.
3.3.4 | Context Features: Context features represent neighboring tokens, syntactic information, relative position, learned embeddings, and lexical n-gram or compound patterns.These features can be learned with graph structures and neural networks or combined with positional, linguistic, statistical, and external-knowledge signals.
3.3.5 | Stacking: Stacking ensembles use predictions from multiple learners as features for a second-level model, combining unsupervised and supervised outputs.Examples stack trees over TextRank and SGRank, or a linear SVM over a random forest.
3.3.6 | External Knowledge: External knowledge features use ontologies, Wikipedia statistics, training keyphraseness, graph-based semantic weights, expert bias, user interaction, and pretrained embeddings.These signals include Wikipedia-based Idf and keyphraseness, DBLP title overlap, posterior-regularized bias, reference keyphrases, and GloVe representations.
3.4 | Comparative Experimental Results: Deep-learning methods CopyRNN and CorrRNN achieve better F1 scores than Maui and KEA, while Maui outperforms KEA; Maui and KEA are unstable on some datasets.
3.4 | Comparative Experimental Results: WINGNUS shows promising results on the SemEval dataset, achieving relatively high F1 scores compared with the baselines.The comparison uses exact string matching between predicted and golden phrases.
3.4 | Comparative Experimental Results: CorrRNN’s mechanisms are credited with producing high performance across all datasets.
3.5 | Subjectivity and Class Imbalance: Unbalanced training data arises because human-unannotated candidate phrases are treated as negative examples, although unlabeled candidates are not reliable negatives.Subjectivity reflects differing author preferences and concept drift in keyphrase selection.
3.5 | Subjectivity and Class Imbalance: Positive Unlabeled Learning and multiple annotators can partially address subjectivity in supervised keyphrase extraction (Chuang et al., 2012b; Sterckx et al., 2016).Sterckx et al. (2016) reweight unlabeled candidates using classifier predictions and annotation uncertainty before training a final ranking classifier.
3.5 | Subjectivity and Class Imbalance: Evaluation requires reliable semantic approaches because golden keyphrase sets also contain subjectivity, despite exact string matching being the preferred approach.Sterckx et al. (2018) propose evaluation collections compiled from varied sources with multiple annotations.

4 | EVALUATION APPROACHES

The review distinguishes evaluation measures that assess accuracy from those that assess ranking quality, and surveys exact, manual, partial, and semantic matching approaches. It finds that exact matching is widely used but overly strict, while top-N precision/recall/F1 generally prevail over ranking measures in applications.

Evaluation measures: Precision, recall, and F1-measure assess extracted-keyphrase accuracy, whereas ranking measures assess the relative order of returned phrases.Ranking measures include MRR, MAP, Bpref, and ACEK.
Evaluation approaches: Exact matching determines correctness by string overlap, manual evaluation uses experts but is subjective and costly, and partial matching compares words across gold and extracted phrases.Partial matching requires stemming but cannot assess syntactic correctness, over-generation, or overlapping candidates.
Usage patterns: Precision/Recall/F1-measure at the top N phrases, with N = 5, 10, 15, 20, prevail over ranking-quality measures because top-of-ranking success matters more than actual keyphrase ordering in most applications.
Limitations: Exact phrase matching is overly strict: it penalizes variants such as “approximate similarity search” against gold phrases “approximate search” and “similarity search.”It also cannot identify semantic similarity and treats subset, superset, and small-variant predictions too harshly.
Semantic evaluation: Semantic evaluation is proposed as a solution, comparing mean word vectors for gold and returned keyphrases through cosine similarity.This approach evaluates semantic similarity rather than relying only on exact phrase identity.

5 | DATASETS AND SOFTWARE

The section surveys popular datasets used to develop and evaluate keyphrase extraction methods and catalogs free and commercial software with their features. It also reports that Google’s entity-recognition API can provide satisfactory results relative to state-of-the-art keyphrase extraction methods, while cautioning about the evaluation context.

5.1 | Datasets: Popular evaluation datasets cover full-text scientific publications, paper abstracts, and news documents, with metadata on creators, document counts, languages, annotation types, and evaluation usage.The survey also notes that 16 papers use less-known datasets that are unavailable publicly or were created specifically for method evaluation.
5.1 | Datasets: Dataset creation has used multiple annotation approaches, including author-assigned, reader-assigned, and professionally indexed keyphrases, with newer collections annotated by multiple annotators.Sterckx et al. (2018) created collections from online news and sports, lifestyle magazines, and newspaper articles, available for research upon request.
5.2.1 | Free Software: Free software packages implement either individual methods such as Maui, KEA, TextRank, YAKE, and RAKE or multiple extraction methods, with documented implementation languages and supported languages.Supported-language coverage ranges from English-only tools to multilingual systems; TextRank supports additional languages, while PKE and YAKE can use other languages through a language parameter.
5.2.2 | Commercial Software: Commercial offerings include dedicated keyphrase extraction APIs from Microsoft, Aylien, IBM, and Amazon, while Google and TextRazor provide entity-recognition services with relevance-related outputs.Amazon Comprehend directly analyzes English and Spanish, and Aylien’s keyphrase feature supports six languages.
5.2.2 | Commercial Software: Google’s entity-recognition feature returns satisfactory results compared to state-of-the-art keyphrase extraction methods on the surveyed datasets.The empirical analysis evaluates commercial APIs on five popular datasets containing domain-specific texts, so the comparison should not be generalized to all-purpose API performance.

6 | EMPIRICAL EVALUATION STUDY · 6.1 | Experimental Setup

The empirical study examines commercial APIs and unsupervised keyphrase extraction methods across datasets, evaluation approaches, measures, gold standards, and qualitative comparisons. Its experimental setup covers method selection, dataset diversity, input conditions, and top-ranked extraction metrics.

6 | EMPIRICAL EVALUATION STUDY: The setup is motivated by the persistent difficulty of reliably evaluating keyphrase extraction and analyzing shortcomings in existing evaluation measures and approaches.The broader study therefore compares systems, measures, evaluation approaches, and gold standards rather than relying on a single evaluation protocol.
6 | EMPIRICAL EVALUATION STUDY: The study evaluates commercial APIs and state-of-the-art or popular unsupervised methods using exact, partial, and manual evaluation approaches with F1-measure and MAP.It investigates whether evaluation approaches produce different performance comparisons, how systems and measures behave with different gold standards, and qualitative differences among approaches.
6.1 | Experimental Setup: The unsupervised comparison includes statistical methods KPM, YAKE, and TfIdf; graph methods SR, TR, MR, PR, and RAKE; and embedding-based RVA.TfIdf2 is additionally developed by combining statistical and positional heuristics, retaining TfIdf-ranked candidates with term frequency greater than 3 among the first 100 nouns or adjectives.
6.1 | Experimental Setup: The study uses Krapivin, Semeval2010, and NUS full-text computer-science datasets, plus Inspec abstracts and 500N-KPCrowd broadcast-news stories.These datasets cover full publications, short texts, and general-domain news; author- and reader-assigned keyphrases are both used as gold standards for Semeval and NUS.
6.1 | Experimental Setup: All methods use full-text articles except RVA, which derives word representations from full text but returns keyphrases from abstracts.APIs receive only abstracts to avoid additional costs, while 500N-KPCrowd APIs receive the first 1500 characters, containing at least 75% of keyphrases.
6.1 | Experimental Setup: The experiments use default parameters, NLTK preprocessing, PKE implementations for most unsupervised methods, and publicly available YAKE and RAKE implementations.The implementation sources include Boudin’s PKE toolkit and corresponding GitHub implementations for YAKE and RAKE.
6.1 | Experimental Setup: The study uses F1-measure as its main measure, with strict exact-match evaluation between golden and candidate keyphrases and looser partial-match evaluation between their word sets, and also provides Mean Average Precision.

6.2 | Commercial APIs

Commercial APIs perform differently across scientific-publication and news datasets. IBM generally leads on scientific data, while Aylien and Google lead on news, and Amazon is consistently weakest or near the bottom.

6.2 | Commercial APIs: Amazon is the worst-performing commercial API across all scientific-publication datasets and ranks last among the APIs on 500N-KPCrowd.The scientific comparison covers Semeval, NUS, Krapivin, and Inspec.
6.2 | Commercial APIs: IBM ranks first and Google second across four scientific-publication datasets, except where Aylien wins Semeval F1@10 and Google wins NUS.The datasets are Semeval, NUS, Krapivin, and Inspec; Table 7 reports F1@10 and F1@20 under exact-match evaluation on five datasets.
6.2 | Commercial APIs: Aylien and Textrazor are competitive on scientific publications, with Textrazor outperforming Aylien on Krapivin and at F1@20 on NUS and Inspec.Appendix B additionally reports MAP with exact-match evaluation and F1-score with partial-match evaluation.
6.2 | Commercial APIs: On the 500N-KPCrowd news dataset, Aylien and Google achieve similarly high F1-scores, followed by IBM, Textrazor, and Amazon.This comparison includes commercial keyphrase-extraction APIs and entity-extraction services on a non-scientific domain.

6.3 | Unsupervised Keyphrase Extraction Methods

This section examines unsupervised keyphrase extraction methods across evaluation settings, text lengths, and domains, considering their performance, matching criteria, and computational cost. It highlights the differing strengths of statistical and graph-based approaches, including the strong performance of MR and SingleRank among graph-based methods.

6.3.1 | Exact Match Evaluation Approach: KPM leads the first three full-text scientific-publication datasets under exact match, while TfIdf and TfIdf2 lead on news articles and SR leads on Inspec.The exact-match results also identify statistical methods as strongest on full texts, graph-based methods as strongest on short scientific texts, and TfIdf/TfIdf2 as first on news articles.
6.3.1 | Exact Match Evaluation Approach: Exact-match results attribute full-text statistical superiority to sufficient statistical information, whereas graph methods benefit from abstracts’ focused word co-occurrence.Full texts contain redundant information that hinders graph-based correlation modeling, while abstracts can avoid this noise; TfIdf2’s frequency threshold also harms short texts.
6.3.3 | General Remarks: MR remains the strongest graph-based method across both evaluation approaches, suggesting that combining topical, positional, and statistical information can improve extraction.The general remarks also propose a broadly adopted TfIdf2-like variant as a future baseline.
6.3.2 | Partial Match Evaluation Approach: Under partial match, KPM again leads the first three full-text datasets, whereas SR leads Inspec and the 500N-KPCrowd news dataset.TfIdf2 usually follows KPM on full-text scientific publications, while graph-based methods occupy the leading positions on Inspec and news articles.
6.3.2 | Partial Match Evaluation Approach: Partial-match findings likewise link full-text statistical strength to information volume and graph-based weakness to redundancy, while abstracts improve graph-based and RVA performance.The abstract version of RVA and graph-based methods is better than the full-text version, whereas TfIdf benefits from fuller text.
6.3.3 | General Remarks: Both evaluation strategies favor KPM and TfIdf2 over graph-based methods on full-text scientific publications, but partial matching ranks graph-based methods higher.Partial evaluation also produces higher graph-based rankings on 500N-KPCrowd and more often places recent methods above the TfIdf baseline.
6.3.3 | General Remarks: Statistical methods offer a good balance between performance and computation time, while graph-based methods achieve lower accuracy with higher computational cost in longer texts.The higher cost of graph-based methods is attributed to PageRank, which runs in most of them.
6.3.3 | General Remarks: For short texts, graph-based methods outperform statistical methods, with SingleRank the best option compared with TR, PR, and MR.Their computation cost is limited for short texts, while graph-of-words and co-occurrence information is sufficient without additional positional or topical information.
6.3.3 | General Remarks: For nonspecialized-domain documents, both statistical and graph-based methods perform well under exact and partial evaluation, leaving method choice to practitioners.The choice should reflect computational cost relative to text length.
6.3.3 | General Remarks: The subsequent evaluation analysis compares exact and partial F1-scores with manual F1-scores, assesses their closeness, tests score distributions, and recommends context-appropriate strategies.This analysis aims to provide more meaningful insights into evaluation-strategy selection.

6.4 | Exact vs Partial Matching

This section empirically compares exact and partial matching with manual evaluation, showing that averaging exact and partial scores best approximates manual evaluation and should be considered for method comparison.

Limitations of matching criteria: Partial matching can reward inappropriate phrases sharing words with golden keyphrases, while penalizing representative phrases that use different wording.Exact matching instead penalizes semantically similar predictions, whereas partial matching may reward word overlap without phrase appropriateness.
Experimental setup: The study evaluated top-10 predictions from KPM and MR on 50 randomly selected Krapivin articles using exact, partial, and manual F1@10 scores.Manual evaluation marked predictions as true positives when reviewers judged them relevant after reading the full article, focusing on abstracts and introductions when necessary.
Correlation and distribution: Manual-evaluation F1@10 scores correlate more strongly with exact than partial matching for both KPM and MR, although partial matching follows the manual score distribution more closely.Exact scores have a lower, more concentrated range, whereas partial scores have similar statistics, with slightly higher median and interquartile range values.
MSE analysis: Averaging exact and partial F1@10 scores achieves the lowest MSE against manual evaluation, motivating its use for empirical comparison.Partial matching has lower MSE than exact matching, while their average achieves the lowest MSE.
Implications: The analysis recommends automatic evaluation methods that account for semantic similarity between predicted and golden keyphrases.This recommendation accompanies the proposed use of averaged exact and partial matching for empirical comparisons.

6.5 | The Role of the Evaluation Gold Standards

Evaluation gold standards substantially change both keyphrase extraction performance estimates and method rankings. The union generally produces the highest scores and aligns most closely with reader annotations, whereas the intersection is overly strict and not recommended.

Evaluation implications: Different gold standards can produce different method rankings and accuracy estimates, so experiments should explicitly identify the evaluation keyphrase set.The study reports changed rankings and scores across gold standards for Semeval and NUS.
Ranking variation: Gold-standard choice can alter rankings substantially, especially on NUS, where fifth place shifts from MR under Union to PR under the authors’ keyphrases.The ranking differences are based on average F1@10 across documents.
Correlation analysis: The union is strongly correlated with reader-based scores but only moderately with author-based scores, while author and reader standards are weakly to moderately correlated.All strong and moderate correlations are statistically significant at significance level 0.05.
Score variation: The union of authors’ and readers’ keyphrases yields the highest F1@10 scores across methods, while the intersection produces lower scores in almost all cases.The same pattern holds for F1@20, MAP@10, and MAP@20.
Recommendations and limitations: The intersection is not recommended because it contains few keyphrases, may omit document topics, and reflects differing author and reader vocabularies and annotation motivations.Author sets contain fewer but sufficient phrases, whereas reader annotations provide a broader unbiased set depending on the annotation process and collective effort.

6.6 | Qualitative Analysis

The qualitative study uses MR on three Krapivin full texts to expose limitations of exact- and partial-match keyphrase evaluation. It shows that partial matching can better reflect actual success in some cases, yet can also score lower or overlook syntactic correctness.

Evaluation limitations: The study highlights that exact-match evaluation may misrepresent a method’s actual success rate, while partial matching can sometimes provide a closer assessment.The authors also show a case where partial matching produces a lower score than exact matching.
Evaluation limitations: Partial-match evaluation can be harmful because it may fail to assess the syntactic correctness of returned keyphrases.
Experimental setup: The analysis applies MR to keyphrases extracted from three full-text publications in the Krapivin dataset collection.
Exact versus partial matching: In one example, the partial-evaluation F1-score exceeds the exact-match score and is closer to the corresponding manually evaluated score.The example uses the publication “Exact algorithms for finding minimum transversals in rank-3 hypergraphs.”
Exact versus partial matching: Under exact matching, MR’s top-10 output contains 1 true positive, 9 false positives, and 3 false negatives, yielding precision 0.10 and recall 0.25.
Exact versus partial matching: Under partial matching, MR’s top-10 output contains 6 true positives.

7 FPs and 1 FNs. Consequently,

The example shows that exact-match evaluation can be too strict, while partial matching more closely reflects the MR method’s performance but introduces its own errors. Both approaches struggle with context-dependent unigrams and abbreviation equivalence.

Exact-match evaluation can be too strict, whereas partial matching is closer to the MR method’s actual performance.However, partial matching may count isolated words as true positives despite their trivial meaning without context.
2 TPs, 8 FPs, and 6 FNs were returned by MR under exact-match evaluation, whereas partial matching produced 2 TPs, 8 FPs, and 9 FNs.Both approaches identified pebbles and automata as the true positives.
Exact-match F1 can exceed partial-match F1 when unigram keyphrases are true positives and multiword keyphrases are false positives or negatives.The appendix also gives an example where partial matching is considered harmful compared with strict exact matching.
Neither evaluation approach recognizes the unigram mso as a true positive for the multiword keyphrase monadic second-order logic.

7 | CONCLUSIONS AND FUTURE DIRECTIONS

The article reviews and organizes keyphrase extraction methods, evaluates their performance and evaluation practices, and identifies limitations and promising future directions. Keyphrases support human inspection and machine tasks including search, classification, clustering, query expansion, and recommendation.

Review contributions: The review comprehensively organizes unsupervised, supervised, and recent deep learning methods by features, properties, strengths, and weaknesses to help readers navigate prior work.It also catalogs evaluation practices, software and APIs, and document collections with associated keyphrases.
Limitations: Supervised methods face subjective annotated datasets and imbalance between keyphrases and non-keyphrases, while state-of-the-art accuracy remains unsatisfactory.The paper characterizes keyphrase extraction as a challenging task with substantial progress still needed.
Empirical evaluation: Simple unsupervised methods such as TfIdf2 are strong baselines, deep learning methods achieve state-of-the-art results, and graph-based versus statistical methods favor short versus long documents, respectively.The study compares commercial APIs and unsupervised methods, while compiling supervised results from prior papers.
Evaluation practices: Exact and partial matching should be averaged, while future evaluations should incorporate semantic similarity between predicted and golden keyphrases.The study also finds that author- versus reader-supplied golden keyphrases significantly affect evaluation and should be explicitly reported.
Future directions: Exploiting deep learning and unsupervised language models for keyphrase extraction or generation is identified as a promising future direction (Yang et al., 2019; Cer et al., 2018).These models are described as part of the frontier of developments in mastering language.

A | QUALITATIVE RESULTS

The example shows that partial-match evaluation can substantially overestimate keyphrase extraction quality relative to strict exact matching. Although partial matching yields scores closer to manual judgments, it misses whether returned phrases are syntactically correct.

Under exact-match evaluation, the top-10 returned candidate keyphrases yielded precision = 0.10, recall = 0.33, F1 = 0.15, with 1 true positive, 9 false positives, and 2 false negatives.The true positive was the bigram phrase low-rank approximations.
Exact matching is unusually close to manual evaluation in this example, indicating the method’s actual success rate more accurately than partial matching.The evaluation compares stemmed gold and extracted keyphrase sets, but partial matching instead compares words found across those phrases.
Under partial-match evaluation, the top-10 returned candidate keyphrases by MR yielded 4 true positives, 10 false positives, and 1 false negative, with precision = 0.29, recall = 0.80, and F1 = 0.42.
Partial matching fails to evaluate syntactic correctness when general unigrams such as “algorithm” and “matrix” replace the gold bigram “matrix algorithm.”This makes the partial-match F1-score quite far from the method’s actual performance on the document, despite being closer to manual scores.

B | COMMERCIAL APIS EVALUATION

The commercial API evaluation compares exact- and partial-match performance across datasets and examines how these automated scores differ from manual evaluation. Average-match scores align more closely with manual scores than exact- or partial-match scores for KPM and MR methods.

Exact-match API evaluation: The evaluation reports MAP@10 and MAP@20 for all APIs and datasets under exact-match assessment.The results are presented in Table 17.
Partial-match API evaluation: The evaluation also reports F1@10 and F1@20 for all APIs and datasets under partial-match assessment.The results are presented in Table 18.
Evaluation comparison: Figure 6 compares manual F1 scores with exact, partial, and average scores across 50 manually evaluated documents.Exact, partial, and average scores are shown with solid, dashed, and dotted lines, respectively.
Evaluation comparison: Average-match scores are closer to manual F1 scores than exact- or partial-match scores for KPM and MR methods.Most manual-exact differences are positive, while many manual-partial differences are negative.