PaperlayerView exact revision on arXiv

Source-linked AI summary

One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases

Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, Adam Trischler

arXiv:1810.05241v4cs.CLcs.LG

TL;DR

Existing approaches struggle to adapt generated phrase counts and evaluation to texts with variable numbers of keyphrases. The paper proposes delimiter-separated Seq2Seq generation and variable-size metrics; CatSeqD outperforms CatSeq on all metrics, while the proposed metrics are reported to improve on fixed-k metrics.

  • Problem

    Beam-search approaches lack a dynamic output count and can diminish phrase diversity, while fixed-count evaluation mismatches the variable number of reference keyphrases.

    Source: §1 · pp. 1–2

  • Method

    ONE2SEQ concatenates multiple phrases with a delimiter for Seq2Seq generation, and the model adds semantic coverage and orthogonal regularization to enhance diversity.

    Source: §6.1 · p. 8, §7 · p. 9

  • Results

    CatSeqD outperforms CatSeq on all metrics, while the proposed variable-size metrics are reported to have improved empirical characteristics over fixed-k metrics.

    Source: §1 · p. 2, §6.1 · p. 8

  • Takeaways & Limitations

    The reported comparisons support ONE2SEQ as an alternative to ONE2ONE for keyphrase generation.

    Source: §5.1 · p. 7

    Exhaustive decoding is computationally expensive for ONE2SEQ, and less than 5% of its generated phrases are unique.

    Source: §3.3.1 · p. 5

Abstract

from arXiv · show

Different texts shall by nature correspond to different number of keyphrases. This desideratum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modeling and evaluation perspectives. We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. Generation diversity is further enhanced with two novel techniques by manipulating decoder hidden states. In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs. We further propose two evaluation metrics tailored towards the variable-number generation. We also introduce a new dataset StackEx that expands beyond the only existing genre (i.e., academic writing) in keyphrase generation tasks. With both previous and new evaluation metrics, our model outperforms strong baselines on all datasets.

1 Introduction

Keyphrase generation predicts phrases from source texts, but both the number of phrases and their lengths can vary. The section motivates modeling that controls output count and diversity, evaluation metrics for variable-size outputs, and a dataset that adds genre heterogeneity.

1 Introduction

  • Beam-search generation does not determine how many phrases each text needs, and parallel decoding can reduce inter-phrase diversity; fixed-count evaluation also mismatches variable reference counts.The number of reference keyphrases varies with factors such as text length, genre, and annotation granularity.Source: §1 · pp. 1–2
  • The proposed Seq2Seq model generates diverse keyphrases while controlling the number of outputs.This addresses the stated limitations of beam-search generation.Source: §1 · pp. 1–2
  • The proposed metrics adapt commonly used F1 to variable-size outputs and are reported to have improved empirical characteristics over fixed-k metrics.Source: §1 · p. 2
  • STACKEX adds a dataset with a marked genre difference, intended to bring added heterogeneity to keyphrase-generation evaluation.Source: §1 · p. 2

2 Related Work

Related work covers extraction and generation approaches, including Seq2Seq models and their use across natural-language-generation tasks.

2.1 Keyphrase Extraction and Generation

2.2 Sequence to Sequence Generation

3 Model Architecture

The model represents multiple keyphrases as one delimiter-separated Seq2Seq output, using decoder-state mechanisms to increase diversity and self-terminating decoding to produce a variable number of phrases.

3 Model Architecture

  • Seq2Seq generates multiple keyphrases from a source text using a delimiter-joined target sequence, optimized with negative log-likelihood.The proposed ONE2SEQ paradigm concatenates phrases with ⟨sep⟩ during training.Source: §3 · p. 3, §3.2.3 · p. 4

3.2 Mechanisms for Diverse Generation

  • Homogeneous decoder states at delimiters can produce similar following-token distributions; two plug-in components address this diversity problem.The model tends to produce identical tokens after delimiters, motivating semantic coverage and orthogonal regularization.Source: §3.2 · p. 3

3.2.1 Semantic Coverage

  • Semantic coverage adds a target encoder for generated tokens and trains its final phrase representation to maximize mutual information with the source encoding using random negatives.The target encoder state is also supplied as an extra decoder input.Source: §3.2.1 · pp. 3–4

3.2.2 Orthogonal Regularization

  • Orthogonal regularization stacks delimiter-position decoder states and penalizes their non-orthogonality, encouraging distinct states and more diverse following tokens.The loss is applied to all decoder states corresponding to delimiters.Source: §3.2.2 · p. 4

3.3 Decoding Strategies

  • Exhaustive decoding uses beam search to return ranked sequences, whereas self-terminating decoding can determine how many phrases to output.For ONE2SEQ, each beam-search sequence contains several phrases and requires further processing to produce a unique ordered phrase list.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.1 Exhaustive Decoding

  • Beam search costs O(Bm); exhaustive decoding is especially expensive for ONE2SEQ, where sequences are longer, and fewer than 5% of generated phrases are unique.Here B is beam width and m is maximum generated-sequence length.Source: §3.3.1 · p. 5

3.3.2 Self-terminating Decoding

  • Self-terminating decoding generates delimiter-separated phrases until </s>, implicitly estimating set size while conditioning each phrase on those previously generated.A single output sequence can be decoded with greedy search or beam search on only the top-scored sequence.Source: §3.3.2 · p. 5

4 Evaluating Keyphrase Generation

The section explains why fixed-cutoff evaluation can misrepresent keyphrase generation when documents have different numbers of ground-truth phrases. It introduces two metrics that adapt the number of evaluated predictions to each example.

4 Evaluating Keyphrase Generation

  • The section frames the metrics against evaluation using a ranked list of unique predictions and the ground-truth keyphrase set.The top k predictions are used when evaluation is conditioned on k.Source: §4 · p. 5
  • With fixed k, an oracle model can score below 1 on F1@k when |Ŷ:k| ≤ k < |Y|, a drawback common to existing studies.The section notes that ground-truth phrase counts vary substantially across examples.Source: §4 · p. 5
  • F1@O sets k to the number of oracle phrases, while F1@M evaluates all predicted phrases without truncation.F1@O uses k = |Y|; F1@M uses k = |Ŷ|.Source: §4 · p. 6
  • Both metrics reflect variable phrase counts, and either reaches 1.0 if and only if the predictions exactly match the ground truth.Because F1@O is independent of model outputs, it can compare existing models.Source: §4 · p. 6

5 Datasets and Experiments

The experiments compare CatSeq and CatSeqD with existing models on scientific-publication datasets and a new StackEx dataset, and test variable-number decoding. CatSeqD performs strongly across the reported settings, while greedy search leads beam search in the variable-number experiments.

5 Datasets and Experiments

  • Reported F1@O scores from Chan et al. and Chen et al. may not be fully comparable because different post-processing and filtering may have been used.The paper points to its appendices for the preprocessing and evaluation protocols.Source: §5 · p. 6

5.1 Experiments on Scientific Publications

  • The scientific-publication experiments use KP20K, INSPEC, KRAPIVIN, NUS, and SemEval; the smaller four datasets test transferability.KP20K uses article abstracts and titles as source text and author keywords as targets.Source: §5.1 · p. 7
  • CatSeqD outperforms existing abstractive models on most scientific-publication datasets and exceeds CatSeq on all reported metrics.CatSeq and CatSeqD are comparable to CopyRNN; CatSeqD also beats ONE2ONE baselines on F1@10 for KP20K and KRAPIVIN, where the average phrase count is five.Source: §5.1 · p. 7

5.2 Experiments on The STACKEX Dataset

  • CatSeqD performs best in general on StackEx and beats CatSeq by a large margin on absent-keyphrase generation.The dataset uses question titles as source text and user-assigned tags as target keyphrases.Source: §5.2 · pp. 7–8

5.3 Generating Variable Number Keyphrases

  • In variable-number generation, greedy search consistently outperforms beam search, while F1@O is generally lower than in earlier experiments.The authors suggest beam search may produce short, similar sequences and that over-generation may still benefit recall.Source: Table 4 · p. 7, §5.3 · p. 8

6 Analysis and Discussion

The analysis examines the effects of CatSeqD’s two added components and whether its generations are more diverse. Ablation, hidden-state clustering, and an example of predictions all support greater diversity for CatSeqD.

6.1 Ablation Study

  • Semantic coverage boosts CatSeq across datasets, orthogonal regularization alone hurts performance, and enabling both components yields a noticeable gain over CatSeq.The authors describe the combined effect as synergistic.Source: §6.1 · p. 8

6.2 Visualizing Diversified Generation

  • CatSeqD produces 89.70 unique predicted phrases versus 20.38 for CatSeq, and its decoder hidden states form clearer clusters after delimiters.The t-SNE figure compares CatSeq in the upper row with CatSeqD in the lower row, across steps following a delimiter.Source: §6.2 · p. 8, Figure 2 · p. 9

6.3 Qualitative Analysis

  • For an example with 29 ground-truth phrases, CatSeq predictions all start with “test,” while CatSeqD predictions are diverse.The authors say this supports their assumption that CatSeq decoder states after delimiters are less diverse without target encoding and orthogonal regularization.Source: §6.3 · pp. 8–9

7 Conclusion and Future Work

The conclusion presents a recurrent model that generates multiple keyphrases sequentially with added diversity modules, alongside new evaluation metrics. The paper reports competitive performance across datasets and leaves phrase ordering and order-invariant set generation for future work.

7 Conclusion and Future Work

  • The paper proposes a recurrent model that sequentially generates multiple keyphrases with two modules to enhance diversity, and new metrics for keyphrase generation.The model is reported to perform competitively across keyphrase-generation datasets, including the newly introduced dataset.Source: §7 · p. 9

A.1 The Encoder-Decoder Model

  • The encoder maps input token embeddings to contextual vectors with a bidirectional GRU, and the decoder uses a unidirectional GRU initialized from the final encoder state.The encoder and decoder states are regularized with dropout.Source: §A.1 · p. 11

A.2 Attentive Decoding

  • Attentive decoding uses the current decoder state to score source words, then combines those attention scores with source encodings to form the generative distribution.During evaluation, the previous predicted token is fed to the decoder.Source: §A.2 · p. 11

A.3 Pointer Softmax

  • Pointer softmax interpolates between generating from the vocabulary and pointing to source-text tokens at each step.Its switch is conditioned on the attention-weighted source representation and decoder state.Source: §A.3 · p. 11

B Experiment Results on KP20K Absent Subset

Absent-keyphrase prediction on scientific publication datasets favors recall-oriented models, while CatSeqD performs comparably to CopyRNN and generally better for top predictions. The authors caution that large-scale model comparisons are difficult.

B Experiment Results on KP20K Absent Subset

  • Exhaustive decoding with large beams can yield seemingly good absent-keyphrase recall while completely ignoring precision.Table 6 reports absent-keyphrase prediction performance on scientific publication datasets, with best and second-best scores highlighted by column.Source: B Experiment Results on KP20K Absent… · p. 11, Table 6 · p. 12
  • CatSeqD performs comparably to CopyRNN under exhaustive decoding and diverse mechanisms, and generally works better for top predictions on absent keyphrases.The authors argue that STACKEX is a better testbed for absent keyphrase generation.Source: B Experiment Results on KP20K Absent… · p. 11
  • Recall-oriented models are favored for absent keyphrase prediction, making CopyRNN with beam size 200 suited to this task setting.The reported comparison uses Recall@10/50 on the absent portions of five scientific paper datasets.Source: B Experiment Results on KP20K Absent… · p. 11

D Implementation Details

The experiments use fixed model dimensions and vocabulary settings, tune CatSeqD regularization coefficients on validation sets, and select checkpoints by validation F1@O.

D Implementation Details

  • The models use 100-dimensional random embeddings, 150-unit encoder and decoder GRUs, a 50,000-word vocabulary, and dropout of 0.1.The target encoder GRU also has 150 hidden units.Source: D Implementation Details · p. 12
  • CatSeqD selects λOR and λSC from [0.01, 0.03, 0.1, 0.3, 1.0] using validation sets; negative sampling uses 16 same-batch samples for a 17-way target-encoding classification loss.Table 7 lists the selected regularization coefficients.Source: D Implementation Details · p. 12, Table 7 · p. 12
  • Optimization uses Adam with learning rate 1e−3, and exhaustive decoding uses beam size 50 with a maximum sequence length of 40.Evaluation lowercases and stems ground-truth and generated keyphrases; checkpoints are selected by validation F1@O.Source: D Implementation Details · p. 12

E Dataset and Evaluation Details

The evaluation follows prior preprocessing protocols, separates present from absent keyphrases, and reports macro-averaged scores over eligible documents.

E Dataset and Evaluation Details

  • Documents and ground-truth keyphrases are segmented, lowercased, and digit-normalized, and examples with empty ground-truth keyphrases are removed.The authors follow the preprocessing and evaluation protocols of Meng et al. (2017).Source: E Dataset and Evaluation Details · p. 12
  • The evaluation statistics distinguish document and keyphrase counts from present/absent keyphrase and document counts.Scores are macro-averaged over documents with at least one ground-truth keyphrase of the relevant present or absent type.Source: Table 8 · p. 13, E Dataset and Evaluation Details · p. 13
  • Present and absent keyphrase performance is evaluated separately, with presence determined by checking whether a lowercased, stemmed keyphrase is a substring of the source text.Present-phrase Precision, Recall, and F1 are computed per document using only present ground-truth keyphrases.Source: E Dataset and Evaluation Details · p. 13

F Examples of KP20K and ST A C KEX with Model Prediction

The examples show model predictions alongside examples from KP20K and STACKEX; the STACKEX example uses a full question, including code, as input.

F Examples of KP20K and STACKEX with Model Prediction

  • The section points readers to Table 9 and Figure 3 for examples of model predictions.Source: F Examples of KP20K and ST A C KEX… · p. 13
  • The STACKEX example presents the entire question, including code, as input, while its three collected ground-truth keyphrases are all absent.The screenshot’s ground truth is javascript, jquery, and event handling.Source: Figure 3 · p. 15
  • The STACKEX example contrasts the three ground-truth keyphrases with longer CatSeq and CatSeqD prediction lists.Figure 3 identifies the example as coming from STACKEX; a separate example is identified as coming from the KP20K validation set.Source: Table 9 · p. 14, Figure 3 · p. 15

Related papers

Loading 1810.05241v4…