Source-linked AI summary
SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, Andrew McCallum
TL;DR
Scientific researchers need ways to identify which publications describe particular tasks, processes, and materials, and how those concepts relate. This paper presents SemEval 2017 Task 10, a benchmark for extracting keyphrases and their relations from scientific articles, and reports findings from 26 submitted systems. Systems used varied approaches, while keyphrase identification remained the most challenging subtask because many keyphrases were long and infrequent.
Problem
Researchers often need to determine which publications describe particular tasks, processes, and materials and how these concepts relate, but obtaining this information generally requires reading many publications.
Method
The paper establishes a corpus and shared task for identifying and classifying keyphrases and synonym and hyponym relations, with dependent subtasks that can support joint learning or structured prediction.
Results
26 systems were submitted; successful systems varied, and although most used RNNs with CRFs or CNNs, the best system in evaluation scenario 1 used an SVM with engineered lexical features, while keyphrase identification was the most challenging subtask.
Takeaways & Limitations
The corpus supports scientific-document information extraction and potential applications including article recommendation, citation-gap detection, reviewer identification, trend analysis, and joint learning.
Takeaways & Limitations
Because mention-level annotation is time-consuming and semantic relations are sparse, only paragraphs likely to contain relations were annotated.
Abstract
from arXiv · showhide
We describe the SemEval task of extracting keyphrases and relations between them from scientific documents, which is crucial for understanding which publications describe which processes, tasks and materials. Although this was a new task, we had a total of 26 submissions across 3 evaluation scenarios. We expect the task and the findings reported in this paper to be relevant for researchers working on understanding scientific content, as well as the broader knowledge base population and information extraction communities.
1 Introduction
SemEval 2017 Task 10 addresses mention-level keyphrase extraction and semantic relation extraction in scientific publications. The released corpus is intended to support scientific-content understanding and information-extraction research.
- Researchers often need to determine which publications describe particular tasks, processes, materials, and relationships, but answering these questions usually requires reading many papers.
- The task jointly targets mention-level keyphrase identification, classification, and semantic relation extraction, including hypernym and synonym relations.
- Keyphrases are difficult to identify because they vary across domains, lack clear signifiers and contexts, and can span many tokens.
- The corpus contains 500 double-annotated publications from Computer Science, Material Sciences, and Physics.
- The corpus may support joint learning and structured prediction because its three annotated subtasks depend on one another.
- Organizers expect the corpus and task to stimulate information-extraction methods and applications for making scientific publications easier to understand.
2 Task Description
The task evaluates mention-level keyphrase identification, type classification, and semantic relations between keyphrases. Its annotations represent boundaries, three keyphrase types, and synonym or hyponym relations.
- The task has three subtasks: keyphrase identification, keyphrase classification, and semantic relation extraction.
- Keyphrase types are PROCESS, TASK, and MATERIAL, with MATERIAL including corpora and physical materials.
- Relations are extracted between keyphrases with the same types and are labeled HYPONYM-OF or SYNONYM-OF.
- The corpus format pairs scientific text with stand-off keyphrase annotations based on character offsets and relation annotations.
3 Resources for SemEval-2017 Task
The SemEval corpus samples scientific paragraphs across three domains and provides mention-level annotations for keyphrases and relations. Its construction balances annotation feasibility with coverage of relation-dense material.
- 3.1 Corpus: 500 journal-article paragraphs were selected, evenly distributed among Computer Science, Material Sciences, and Physics.
- 3.1 Corpus: The corpus split contains 350 training documents, 50 development documents, and 100 test documents.
- 3.1 Corpus: 22% of training keyphrases contain at least five tokens, 93% are noun phrases, and 31% appear only once.These statistics make identification challenging while offering some information for candidate-generation heuristics.
- 3.2 Annotation Process: Because mention-level annotation is time-consuming and publications contain few semantic relations, only paragraphs likely to contain relations were annotated.
- 3.2 Annotation Process: The planned knowledge-graph and distant-supervision selection strategy was abandoned because automatically detecting keyphrases was difficult and relations overlapped little across documents.
- 3.2 Annotation Process: Documents were double-annotated with a student and an expert, and disagreements were resolved in favor of the expert.
4 Evaluation
SemEval 2017 Task 10 compares systems under three levels of annotation support, from plain text alone to manually supplied keyphrases and types. Performance is measured by exact-match precision, recall, and F1.
- Scenario 1 provides only plain text and requires solving Subtasks A, B, and C.
- Scenario 2 provides manually annotated keyphrase boundaries and evaluates keyphrase classification and relations.
- Scenario 3 provides manually annotated keyphrases and types and evaluates relation extraction alone.
- Systems are evaluated against the gold standard using exact matches, with micro-averaged precision, recall, and F1 across publications and genres.
5 Pilot Task
The pilot task exposed weaknesses in fixed-count, exact-match keyphrase extraction and motivated a more semantically grounded design for future evaluation.
- The task required returned phrases to exactly match annotator- or author-assigned keyphrases, allowing only minor syntactic variations.
- Annotator-provided keyphrases matched the paper text only 85% of the time, while author-provided keyphrases matched 81% of the time.
- Pilot-task performance varied with the number of keywords extracted, so the organisers recommended against fixing a keyphrase-count threshold.
- The organisers recommended accounting for keyphrase synonyms rather than requiring exact matches, and planned to assign keyphrase types in future task design.
6 Existing Resources
Existing resources differed from the proposed task in annotation level, semantic scope, domains, or availability. The paper therefore describes a mention-level, three-type, relation-aware resource and a sequence-to-sequence task formulation with separate output layers.
- The proposed corpus contains 100 noun phrases from Web of Science-derived titles and abstracts, annotated by three annotators using 14 fine-grained types, including PROCESS.
- Fine-grained annotation achieved Fleiss’ kappa of 0.28, indicating fair agreement, while substantially increasing annotation time; the task therefore uses three main types.
- Existing keyphrase-extraction corpora were not similar enough to the proposed task to justify reuse.
- The SemEval 2010 corpus used document-level, type-level keyphrases without process, task, or material classes or semantic relations.
- The proposed task frames extraction as sequence-to-sequence prediction over tokenised sentences and separates boundary, keyphrase-type, and relation classification.
- The upper-bound span-token-span conversion achieved F1 0.84, showing performance loss from sentence splitting and tokenisation alone.
7 Summary of Participating Systems
The competition evaluated scientific keyphrase and relation extraction across three scenarios, attracting broad participation and diverse approaches. Performance increased when teams received partial annotations, while neural and feature-rich methods achieved the strongest reported scores.
- Competition overview: 54 teams entered development and 26 participated in the final competition, applying neural networks, supervised classifiers, feature engineering, and rule-based methods.The final competition therefore included a broad range of modeling strategies.
- Evaluation Scenario 1: 0.04–0.43 F1 was achieved in Scenario 1 without annotation information, with the top three teams using RNN-based approaches.The best reported scores were 0.38, 0.42, and 0.43 for TTI COIN, TIAL UW, and s2 end2end.
- Evaluation Scenario 2: 0.43–0.64 F1 was achieved in Scenario 2 with partial Subtask A annotation, and partial annotation was critical for performance.BUAP’s 0.45 F1 for Subtask B exceeded s2 end2end’s 0.44 in Scenario 1, reinforcing the difficulty of keyphrase boundary identification.
- Evaluation Scenario 2: Lexical-feature classifiers and neural networks performed similarly at the top of Scenario 2, reaching F1 scores of 0.64 and 0.63.MayoNLP used an SVM with rich features, while UKP/EELECTION combined three neural approaches by majority voting.
- Evaluation Scenario 3: 0.1–0.64 F1 was reported in Scenario 3, where neural-network models performed better than other methods for relation extraction.The best method used a CNN, while another two-phase neural approach reached an F1 of 0.54.
- Evaluation Scenario 3: Most Scenario 3 teams outperformed teams from the other scenarios in relation prediction when given partial information for Subtasks A and B.The authors connect this pattern to the importance of Subtask A and B accuracy for Subtask C.
8 Conclusion
The task attracted 26 submitted systems using varied approaches, with keyphrase identification remaining the most challenging subtask.
- 26 systems submitted to SemEval 2017 Task 10, covering keyphrase identification, classification, and relation extraction from scientific articles.
- Most participating systems used RNNs, often combined with CRFs and CNNs.
- An SVM with a well-engineered lexical feature set performed best in evaluation scenario 1.
- Keyphrase identification was the most challenging subtask because the dataset contained many long and infrequent keyphrases.
- Systems relying on remembering keyphrases did not perform well.