PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Knowledge Graphs by Employing Natural Language Processing and Machine Learning Techniques within the Scholarly Domain

Danilo Dessì, Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, Enrico Motta

arXiv:2011.01103v1cs.CLcs.AIcs.LG

TL;DR

Scientific knowledge graphs describe publication metadata but lack explicit representations of the knowledge in research papers, while extracting and managing scholarly content still requires manual effort. The paper combines NLP and machine-learning tools to extract and integrate entities and relations into a scientific knowledge graph. Its evaluation reports good-quality graphs and better overall performance from integrating different tools, while noting that semantic constraints on triples remain incomplete.

  • Problem

    Scientific knowledge graphs lack explicit representations of research knowledge in papers, and extracting and managing that content requires manual effort.

    Source: §1 · pp. 2, 4

  • Method

    The paper combines NLP and machine-learning methods to extract entities and relations from publications and integrates their outputs into a knowledge graph.

    Source: §1 · p. 4

  • Results

    The evaluation found that the solution automatically produces good-quality scientific knowledge graphs and that integrating different tools yields better overall performance.

    Source: §6 · p. 23

  • Takeaways & Limitations

    The authors conclude that their pipeline can automatically produce a detailed scientific knowledge graph for the Semantic Web literature.

    Source: §6 · p. 23

    The current version does not fully use semantic characterization of research entities to verify triples, so semantically incorrect entity-type combinations can occur.

    Source: §6 · p. 23

Abstract

from arXiv · show

The continuous growth of scientific literature brings innovations and, at the same time, raises new challenges. One of them is related to the fact that its analysis has become difficult due to the high volume of published papers for which manual effort for annotations and management is required. Novel technological infrastructures are needed to help researchers, research policy makers, and companies to time-efficiently browse, analyse, and forecast scientific research. Knowledge graphs i.e., large networks of entities and relationships, have proved to be effective solution in this space. Scientific knowledge graphs focus on the scholarly domain and typically contain metadata describing research publications such as authors, venues, organizations, research topics, and citations. However, the current generation of knowledge graphs lacks of an explicit representation of the knowledge presented in the research papers. As such, in this paper, we present a new architecture that takes advantage of Natural Language Processing and Machine Learning methods for extracting entities and relationships from research publications and integrates them in a large-scale knowledge graph. Within this research work, we i) tackle the challenge of knowledge extraction by employing several state-of-the-art Natural Language Processing and Text Mining tools, ii) describe an approach for integrating entities and relationships generated by these tools, iii) show the advantage of such an hybrid system over alternative approaches, and vi) as a chosen use case, we generated a scientific knowledge graph including 109,105 triples, extracted from 26,827 abstracts of papers within the Semantic Web domain. As our approach is general and can be applied to any domain, we expect that it can facilitate the management, analysis, dissemination, and processing of scientific knowledge.

1. Introduction

Scientific knowledge graphs commonly describe publication metadata but lack explicit, machine-readable representations of the knowledge expressed in papers. The paper proposes an NLP and machine-learning architecture to extract and integrate research entities and relations, and evaluates the hybrid approach against alternatives.

1. Introduction

  • Current scientific knowledge graphs lack explicit representations of research knowledge and its relationships, which are often left in natural-language text or topic metadata.The paper argues that richer representations could support semantic queries, navigation among claims, recommendation, and hypothesis generation.Source: §1 · p. 3
  • An open challenge is automatically generating scientific knowledge graphs that represent publication content, including approaches, claims, applications, data, and results.Such a knowledge base could support more granular content-aware services for exploring research.Source: §1 · p. 4
  • Different extraction tools may label the same entities differently and disagree about relations, making coherent integration of their outputs an open issue.The paper also notes that extracting knowledge from publications still requires manual intervention and that structured research descriptions may require manual population.Source: §1 · p. 4
  • The proposed architecture combines NLP and machine-learning tools to extract entity-relation triples and integrates their outputs using Semantic Web best practices.Its stated hypothesis is that supervised and unsupervised methods together can yield high recall while retaining good precision.Source: §1 · p. 4
  • The authors compare the hybrid approach with supervised-classification-only methods and NLP tools such as OpenIE, and evaluate triples using precision, recall, and F-measure.They also report creating a manually annotated gold standard for the task.Source: §1 · pp. 4–5

2. Problem Statement

The problem is to generate a large-scale knowledge base from research papers, representing relevant domain entities and their relationships. Each extracted triple is associated with the papers from which it came.

2. Problem Statement

  • Given a large collection of research papers, the goal is to generate a knowledge base containing relevant domain entities and their relationships.Source: §2 · p. 5
  • The paper models this task as γ: D → T, mapping scientific documents to triples (s, p, o), with subjects and objects from entity set E and predicates from relation-label set L.Source: §2 · p. 5
  • Each triple is associated with the papers it was extracted from so that support for the claim can be assessed in the source collection.Source: §2 · p. 5
  • The resulting graph is proposed as a resource for research problems such as detecting communities, tracking trends, and forecasting research dynamics.Source: §2 · p. 5

3. Methodology

The methodology builds a scientific knowledge graph from scientific text through extraction, refining, and triple selection. Selection first uses triples associated with many papers and then enriches that set with other semantically consistent triples.

3. Methodology

  • The workflow extracts entities and triples from text using an ensemble of NLP and machine-learning tools.Source: §3 · p. 5
  • Entity refining merges and cleans the entities produced by extraction.Source: §3 · p. 5
  • Triple refining merges triples from different tools and maps their relations to a common vocabulary.Source: §3 · p. 5
  • Triple selection starts with triples associated with a good number of papers, then enriches that set with semantically consistent triples.Source: §3 · p. 6

3.1. Extraction of Entities and Relations

The extraction stage combines tools with different roles: identifying scientific entities and relations, adding computer-science topics, and expanding relation candidates. Their outputs are subsequently constrained to triples whose entities match the identified domain entities.

3.1. Extraction of Entities and Relations

  • The Extractor Framework detects six entity types and seven predefined relation types; the authors discard Conjunction triples as too generic.It scores text spans for entity types and span pairs for relations, then assigns the highest-scoring types to form triples.Source: §3.1 · pp. 6–7
  • The CSO Classifier identifies computer-science topics using syntactic and semantic modules and can enhance topic sets with relevant super-topics.The super-topic functionality is not used during extraction because those topics cannot be mapped to other entities in the sentence; a similar process is used in knowledge-graph enhancement.Source: §3.1 · p. 7
  • OpenIE generates triples from clauses, and the pipeline keeps its triples only when both entities match those found by the Extractor Framework and CSO Classifier.Source: §3.1 · pp. 7–8
  • The Stanford Core NLP PoS tagger identifies verbs between entity pairs and creates triples from them, aiming to capture additional relations at the cost of some noisy relations.Source: §3.1 · p. 8
  • For each abstract sentence, the pipeline obtains entities and triples from the Extractor Framework, adds CSO topics, applies OpenIE to matching entities, and expands relations with the PoS tagger.Source: §3.1 · p. 8

3.2. Entities Manager

The Entities Manager reduces noisy, ambiguous, and overly generic entities, then merges alternative labels for the same concept. It combines text preprocessing with frequency-based filtering and lemmatization or CSO labels.

3.2.1. Entities Refiner Module

  • Noisy, ambiguous, and overly generic entities are refined; the module also preserves CSO topics and author keywords through a whitelist.The paper gives “approach” and “method” as examples of abstract entities that add noise.Source: §3.2.1 · pp. 8–9
  • The refiner cleans entities, splits compound expressions, expands acronyms within each abstract, and filters generic terms using their frequency across three document sets.The sets cover Semantic Web publications, Computer Science papers outside Semantic Web, and papers from other domains.Source: §3.2.1 · p. 9

3.2.2. Entities Merger Module

  • The merger combines singular and plural forms with a lemmatizer and uses CSO alternative labels to merge entities referring to the same research topic.For a CSO-known entity, it replaces the entity with the longest alternative label.Source: §3.2.2 · pp. 9–10

3.3. Relations Manager

The Relations Manager selects predicates for entity pairs and maps relation labels to a consistent vocabulary. It combines relation evidence from three extraction sources, then clusters and manually consolidates semantically similar verbs.

3.3. Relations Manager

  • The module aims to select the best predicate for each related entity pair and map relations into a defined table.Its inputs include triples from the Extractor Framework, OpenIE, and the PoS tagger.Source: §3.3 · p. 10, §3.3.1 · p. 10

3.3.1. Best Relation Finder Module

  • For Extractor Framework triples, the most frequent relation for each entity pair is selected; OpenIE and PoS relations are consolidated using word embeddings.For OpenIE and PoS, the relation nearest the average embedding is selected; the resulting triples also record the number of papers supporting each entity pair.Source: §3.3.1 · p. 10

3.3.2. Mapper Module

  • Hierarchical clustering groups semantically similar verb relations using (1 − cosine similarity) as distance, with the dendrogram cut at an empirically set average Silhouette-width threshold of 0.65.The clusters were manually revised, and their relations were mapped to a representative label such as “produces.”Source: §3.3.2 · p. 11

3.4. Triples Selection

Triples are selected using tool output and paper-level support to define a valid set. A classifier then evaluates additional triples against that set, allowing some triples with lower support to be included.

3.4.1. Valid Triples

  • The prototype treats Extractor Framework and OpenIE triples as valid, along with PoS-tagged triples associated with at least 10 papers.These sets are combined to form the valid triples set, Tvalid.Source: §3.4.1 · p. 12

3.4.2. Consistent Triples

  • Triples outside Tvalid may still capture valid claims with limited paper support, including newer discoveries that have not yet gained broad support.The classifier is trained using valid triples to identify triples consistent with that set.Source: §3.4.2 · p. 12
  • A classifier evaluates triples outside Tvalid by predicting the relation for each entity pair and comparing it with the triple’s actual relation.Exact matches are added to Tvalid; otherwise, the triple is included when the average cosine and Wu-Palmer similarity exceeds the empirically set threshold of 0.5.Source: §3.4.2 · p. 12

3.5. Knowledge Graph Enhancement

The Knowledge Graph Enhancement step infers additional triples from hierarchical relations in CSO and converts the resulting triples to RDF.

3.5. Knowledge Graph Enhancement

  • A triple about a topic can be extended to its CSO supertopic when no triple already connects the other entity to that supertopic.For example, the system can infer that an NLP system uses artificial intelligence from a triple stating it uses named-entity recognition.Source: §3.5 · pp. 12–13
  • Table 1 presents examples of triples detected by the pipeline and examples it discarded.Source: Table 1 · p. 13
  • The CSO Triples Integrator performs the inference step, after which the triples are converted to RDF.Source: §3.5 · p. 13

4. Results and Discussion

On 26,827 Semantic Web abstracts, the framework produces a 109,105-triple graph, and its full combination achieves the strongest reported recall and F-measure. The authors validate the triples against a manually annotated gold standard and discuss graph examples and domain-transfer constraints.

4.1. The Semantic Knowledge Graph

  • The graph contains 109,105 triples, but TEF’s large output is mostly low-support; at support ≥5, PoS + Cons. has 1,268 triples versus TEF’s 393 and TOIE’s 45.TEF has 80,030 triples supported by one paper; the reported graph combines triples from TEF, TOIE, and PoS + Cons.Source: §4.1 · p. 14, Figure 2 · p. 14

4.2. Gold Standard Creation

  • The 818-triple gold standard samples Semantic Web subtopics, includes discarded triples for recall assessment, and uses five-researcher majority labels (agreement 0.747 ± 0.036).The resource is intended both to evaluate the pipeline and support further evaluations.Source: §4.2 · p. 15

4.3. Precision, Recall, F-measure Analysis

  • Among basic methods, T′ PoS + Cons. Triples has the best precision (84.7%), while EF reaches 84.3% precision but only 54.4% recall.The paper reports that the basic methods generally have good precision but suffer in recall.Source: §4.3 · p. 16, Table 2 · p. 17
  • The full framework combining OpenIE, EF, and T′ PoS + Cons. Triples achieves 80.2% recall, 81.2% F-measure, and 78.7% precision.Table 2 identifies the final row as the full pipeline.Source: §4.3 · p. 16, Table 2 · p. 17

4.4. Examples and considerations about the Scientific Knowledge Graph

  • Ontology-alignment triples describe its context and functions, but examples also expose ambiguous merged relations and a predicate the authors consider incorrect.They cite missing links between a distributed and heterogeneous ontology and its broader heterogeneous-ontology concept as another issue.Source: §4.4 · p. 17
  • The supervised-machine-learning subgraph links structured data models and rich semantics, but its generic “uses” and “includes” relations leave adoption unclear.The authors suggest their predicate taxonomy may need finer granularity.Source: §4.4 · p. 19, Figure 5 · p. 20
  • The graph is presented as useful for publication navigation, graph embeddings, paper recommendation, and research-trend detection.The authors describe these as potential applications of the knowledge base.Source: §4.4 · p. 20

4.5. Applicability in other domains

  • Applying the pipeline beyond computer science requires replacing the computer-science-trained extractor and CSO ontology with domain-appropriate resources.The authors state that the current pipeline can be used across computer-science subfields without this substitution.Source: §4.5 · pp. 20–21

5. Related Work

Prior work extracts entities and relations from text and builds scholarly knowledge resources, but integrating useful, domain-specific knowledge across publications remains challenging. This paper combines open-domain extraction with scholarly information and synthesizes knowledge across papers.

Related Work

  • Earlier work used PoS tags, context, and existing knowledge graphs to detect entities and develop ensemble extraction methods.The authors build on the ensemble idea with a software pipeline for organizing extracted entities and relations in a scientific knowledge graph.Source: §5 · p. 21
  • The pipeline combines open-domain information-extraction tools with scholarly-domain information to synthesize knowledge across multiple papers into one graph.The authors contrast this with FRED, which processes one text at a time and can produce unpredictable, generic knowledge that is difficult to use for scholarly modeling.Source: §5 · p. 22
  • FRED links text elements to ontologies and extracts relations, but its generic output and single-text processing limit its fit for domain-specific scholarly modeling.The authors instead combine open-domain extraction with scholarly-domain information and information from multiple texts.Source: §5 · p. 22
  • Prior scholarly extraction included syntactic-pattern approaches and deep-learning work that built a graph from 110,000 papers.Against Yi et al., this work uses textual and statistical similarity for entity ambiguity and seeks more precise, fine-grained action relations.Source: §5 · p. 22
  • The literature also includes PoS-based entity detection, relation-extraction challenges, and approaches to harvesting entities and relations from text.Other Semantic Web efforts include graph-based word-sense disambiguation and entity linking, while ensemble methods draw on context and existing knowledge graphs.Source: §5 · pp. 21–22

6. Conclusions

The paper presents a pipeline that merges outputs from NLP and machine-learning tools into a scientific knowledge graph, with evaluation reporting good quality and improved overall performance from tool integration. The authors identify limitations in semantic constraints, relation coverage, synonym recognition, scalability, and evaluation across tasks.

Conclusions

  • Integrating multiple tools yielded better overall performance, and evaluation found the pipeline could automatically produce good-quality scientific knowledge graphs.The graph integrates knowledge from many publications to represent detailed Semantic Web research knowledge.Source: §6 · p. 23
  • The prototype does not enforce semantic constraints on triples and extracts only one relationship between any two entities.The authors also identify conjunction handling and cross-document relations, such as citations, as areas for future work.Source: §6 · p. 23
  • The pipeline may fail to recognize synonyms absent from existing knowledge bases, such as “Radial Basis Function Neural Network” and “RBFNN.”The authors propose using semantic similarity between word and graph embeddings to detect and merge synonyms.Source: §6 · p. 23
  • The current implementation has scalability bottlenecks: the Extractor Framework requires substantial disk space, and single-threaded Stanford Core NLP processing takes a long time.The authors suggest multithreading the server to speed extraction.Source: §6 · p. 23
  • The authors identify extrinsic evaluation on different tasks, including recommender systems and graph-embedding strategies, as a next step.They propose assessing whether these AI tasks benefit from the knowledge base.Source: §6 · p. 24
Loading 2011.01103v1…