Source-linked AI summary
The CSO Classifier: Ontology-Driven Detection of Research Topics in Scholarly Articles
Angelo A. Salatino, Francesco Osborne, Thiviyan Thanapalasingam, Enrico Motta
TL;DR
Research-paper topic classification can support retrieval and analytics, but practical unsupervised methods for granular topic sets are lacking. The CSO Classifier uses paper metadata to assign ontology topics, and the authors report better recall and F-measure than alternatives on manually annotated papers.
Problem
Method
The CSO Classifier uses a paper’s title, abstract, and keywords to identify explicit CSO topics, infer semantically related topics, and add their super-areas without requiring labelled examples.
Source: §1 · p. 2Results
75.3% recall and 74.1% F-measure: the CSO Classifier outperformed all other methods on these metrics.
Source: §6.1 · p. 12Takeaways & Limitations
The classifier and its evaluation materials are publicly available, including a gold standard of 70 manually annotated papers from three Computer Science fields.
Source: §1 · p. 2The approach assumes research concepts can be represented as nouns or adjectives followed by nouns.
Source: §4.2.2 · p. 6
Abstract
from arXiv · showhide
Classifying research papers according to their research topics is an important task to improve their retrievability, assist the creation of smart analytics, and support a variety of approaches for analysing and making sense of the research environment. In this paper, we present the CSO Classifier, a new unsupervised approach for automatically classifying research papers according to the Computer Science Ontology (CSO), a comprehensive ontology of re-search areas in the field of Computer Science. The CSO Classifier takes as input the metadata associated with a research paper (title, abstract, keywords) and returns a selection of research concepts drawn from the ontology. The approach was evaluated on a gold standard of manually annotated articles yielding a significant improvement over alternative methods.
1 Introduction
The paper introduces an unsupervised classifier that assigns research papers topics from CSO, addressing the need for ontology-based classification without labelled examples. It evaluates the approach against alternatives and reports improved performance.
1 Introduction
- Ontology-based classification offers formally defined, human-readable topics, while topic-detection methods tend to produce noisier, less interpretable results.Topic classification supports applications such as digital-library categorisation, metadata enrichment, recommendations, analytics, and trend detection.Source: §1 · p. 1
- Because CSO was not yet routinely used and labelled examples were lacking, the authors developed an unsupervised approach for classifying papers with its topics.The approach uses word embeddings produced from a large text collection but does not require labelled examples.Source: §1 · p. 2
- The classifier takes article metadata, typically title, abstract, and keywords, and returns CSO topics by finding explicit mentions, adding semantically related topics, and including their super-areas.Source: §1 · p. 2
- The CSO Classifier significantly improves over alternative approaches, including Springer Nature’s previous classifier, on a gold standard of manually annotated papers.The gold standard contains 70 papers across Semantic Web, Natural Language Processing, and Data Mining, annotated by 21 domain experts; evaluation materials and implementation are publicly available.Source: §1 · p. 2
- The paper’s contributions include a CSO classifier and a 70-paper gold standard annotated by 21 domain experts across three research fields.The evaluation data, Python implementation, and word embeddings are publicly available.Source: §1 · p. 2
2 Literature Review
Prior work assigns papers to predefined categories or detects topics from text, keywords, citations, and other signals. The review identifies a gap in practical unsupervised classification using granular topic vocabularies.
2 Literature Review
- Research-paper topics have been characterized either by assigning preexisting categories, which yield clean, formally defined topics when good domain characterization is available, or by bottom-up topic detection from text.Examples include an unsupervised keyword-and-abstract approach, deep-learning subject classification applied to papers annotated with STW (~5K classes) and MeSH (~27K classes), and supervised annotation with ACM topics.Source: §2 · pp. 2–3
- Topic-detection approaches use clustering, LDA, keywords, citations, titles, and word embeddings, but generating topics from scratch can make results noisier and less interpretable.LDA represents topics as word distributions that can be difficult to map to classification topics.Source: §2 · p. 3
- Available scholarly repositories often use keywords or coarse-grained topic representations, while practical unsupervised approaches for classifying papers against granular topic sets remain lacking.The CSO Classifier was designed to address this gap in Computer Science.Source: §2 · p. 3
3 The Computer Science Ontology
The Computer Science Ontology is a large-scale, automatically generated ontology of research areas, with many topics and semantic relationships. Its relations support representing connections among topics and linking concepts to external resources.
3 The Computer Science Ontology
- CSO was generated with Klink-2 from 16 million publications, mainly in Computer Science, and its current version contains 14K semantic topics and 162K relationships.Source: §3 · p. 4
- CSO extends SKOS and includes relations for super-areas, equivalent topics, contributions between topics, and identity with external resources.The ontology connects research concepts to DBpedia and is available through a portal for downloading, exploring, and providing feedback.Source: §3 · p. 4
- CSO supports applications including proceedings classification, book recommendation, research-trend forecasting, scholarly-data exploration, and research-paper annotation.Source: §3 · p. 4
4 CSO Classifier
The CSO Classifier takes a paper’s title, abstract, and keywords and returns relevant concepts from CSO. It combines syntactic and semantic components to identify and enrich topics.
4 CSO Classifier
- The classifier analyses title, abstract, and keywords to output relevant concepts from CSO through syntactic and semantic modules.Source: §4 · pp. 4–5
- Figure 1 is identified as the workflow of the CSO Classifier.Source: Figure 1 · p. 5, §4 · p. 5
- The syntactic module finds explicitly mentioned CSO concepts; the semantic module uses part-of-speech tagging and word embeddings to infer related topics, after which their results are combined and enriched with super-areas.The paper describes this workflow using a sample paper as a running example.Source: §4 · p. 5
4.1 Syntactic Module
The syntactic module maps text n-grams to CSO concepts using string similarity, returning a broad set of pertinent topics beyond a paper’s keywords.
4.1 Syntactic Module
- The module removes English stop words, extracts unigrams, bigrams, and trigrams, and matches them to CSO labels using Levenshtein similarity.Topics with similarity at or above msm are selected; the prototype sets msm to 0.94.Source: §4.1 · p. 5
- For the running example, the syntactic module returns topics including microblogging, data mining, twitter, and network topology.The paper’s keyword field contains only “social networks”, “anonymity”, and “privacy”.Source: §4.1 · p. 6, Table 2 · p. 6
4.2 Semantic Module
The semantic module uses word embeddings to find CSO topics related to a paper even when they are not explicitly mentioned. It extracts and identifies candidate concepts, ranks them, then selects a cutoff using the elbow method.
4.2 Semantic Module
- The semantic module targets topics related to the paper but not explicitly mentioned, using word embeddings to compare document terms with CSO concepts.Its four stages are entity extraction, concept identification, ranking, and selection.Source: §4.2 · p. 6
4.2.1 Word Embedding generation
- The module uses word2vec embeddings trained on titles and abstracts from 4,654,062 English Computer Science papers in MAG.CSO topic labels and frequent n-grams were treated as single tokens during preprocessing.Source: §4.2.1 · p. 6
4.2.2 Entity extraction
- Entity extraction assumes research concepts are nouns or adjectives followed by nouns, using a grammar-based chunk parser to find those phrases.The grammar is <JJ.*>*<NN.*>+; this reduces the chunks analyzed and avoids combinations that usually produce false positives.Source: §4.2.2 · pp. 6–7
4.2.3 CSO concept identification
- For each extracted chunk’s unigrams, bigrams, and trigrams, the module retrieves up to 10 similar words and adds their matching CSO topics.Retrieved words must have cosine similarity higher than 0.7; unavailable multiword terms use averaged token embeddings.Source: §4.2.3 · p. 7
4.2.4 Concept ranking
- The relevance score multiplies how often a topic is identified by the number of unique n-grams that identify it; explicitly mentioned topics receive the maximum score found.This weighting addresses large candidate sets that can include marginally related topics.Source: §4.2.4 · p. 7
4.2.5 Concept Selection
- The elbow method selects the first 18 topics in the example by finding a cutoff in the curve of relevance scores.Figure 3 illustrates the score distribution and elbow.Source: §4.2.5 · p. 7, Figure 3 · p. 8
4.3 Combined output and enhancement
The classifier combines syntactic and semantic topics, filters generic semantic matches, and enriches the result with CSO super-topics. The union maximized F-measure in the evaluation, though other combinations trade precision against recall.
4.3 Combined output and enhancement
- It removes semantic topics among the 3,000 most frequent embedding-vocabulary words because generic terms tend to match many n-grams and produce false positives.Examples of such terms include “language”, “learning”, and “component”.Source: §4.3 · p. 8
- The classifier enriches the combined topics by inferring their direct CSO super-topics, such as “artificial intelligence” for “machine learning”.By default it adds only direct super-topics, but inference can extend up to Computer Science.Source: §4.3 · p. 8
- The example’s inferred topics include authentication, world wide web, and security of data.Table 4 lists topics obtained from the enhancement process.Source: §4.3 · p. 8, Table 4 · p. 8
- The classifier unions the syntactic and semantic results because this strategy maximized F-measure in the evaluation.The syntactic topics tend to be more accurate, while adding semantic topics allows better recall.Source: §4.3 · p. 8
5 Creation of the Gold Standard
The authors built a gold standard because no manually annotated dataset was available for the recently released CSO. It comprises 70 papers annotated by domain experts and supports evaluation and further research.
5.1 Data preparation
- The gold standard contains 70 papers from Semantic Web, Natural Language Processing, and Data Mining, selected as the most cited papers published in 2007–2017.The fields contributed 23, 23, and 24 papers, respectively.Source: §5.1 · p. 9
5.2 Data collection
- Twenty-one researchers annotated papers, with each paper assigned to at least three experts and disagreements resolved by majority vote.Experts assessed candidate topics as relevant or not relevant and could add missing CSO topics; each paper received an average of 18 ± 9 topics.Source: §5.1 · p. 9, §5.2 · p. 9
5.3 Gold Standard
- Candidate topics combined outputs from three classifiers, yielding an inclusive set averaging 41.8 ± 17.5 topics per paper.The final gold standard used majority rule and averaged 14.4 ± 7.0 topics per paper; annotations were semantically enriched with direct super-areas.Source: §5.2 · p. 9, §5.3 · p. 10
- The three annotators achieved average Fleiss’ Kappa of 0.451 ± 0.177, indicating moderate inter-rater agreement.Agreement was measured across the three annotators on each paper.Source: §5.3 · p. 10
6 Evaluation
The CSO Classifier was evaluated against thirteen alternatives using precision, recall, and F-measure. It achieved the strongest overall results, while the comparisons also show trade-offs between explicit and inferred topics and between precision and recall.
6 Evaluation
- The evaluation compared fourteen approaches on the annotated gold standard using average precision and recall across papers and their harmonic mean, F1.Source: §6 · pp. 10–11
- The alternatives included TF-IDF and LDA variants, sliding-window W2V-W, STM, and syntactic, semantic, intersection, and union-based CSO methods.Source: §6 · p. 10
6.1 Results
- LDA- and TF-IDF-based approaches did not exceed 30.1% f-measure; W2V-W reached 41.2% precision and 16.7% recall.Source: §6.1 · p. 11
- STM and SYN achieved 80.8% and 78.3% precision, respectively, but their recall was 58.2% and 63.8%.Source: §6.1 · p. 11
- SEM improved recall and f-measure over SYN, while INT raised precision to 79.3% but lowered recall to 59.1%.Source: §6.1 · p. 12
- 75.3% recall and 74.1% f-measure: CSO-C outperformed all other methods and performed significantly better than each (p<10^-7).Source: §6.1 · p. 12
- Selecting the first 10 topics from SEM's ranked list peaked at 63.6% f-measure, below the elbow method's 71.5%.Figure 4 tracks average precision, recall, and f-measure across candidate-set sizes; precision decreases as recall increases.Source: §6.1 · p. 12, Figure 4 · p. 12
- Including CSO-C slightly lowered general inter-rater agreement to 0.392 ± 0.144, still yielding moderate agreement with most human experts.Source: §6.1 · p. 12
7 Conclusions and future work
The CSO Classifier analyzes paper text syntactically and semantically to assign CSO topics, and outperformed alternatives in recall and f-measure on 70 manually annotated articles. Future work will improve the approach, extend it to other fields, and integrate it into editorial and recommendation applications.
7 Conclusions and future work
- On 70 manually annotated articles, the classifier outperformed alternatives in recall and f-measure; its code and related materials are freely available.Source: §7 · p. 13
- Planned research includes combining NLP, machine learning, and semantics and adapting the approach to Engineering and Medicine.Source: §7 · p. 13
- The authors plan to integrate the classifier into Smart Topic Miner and Smart Book Recommender to support Springer Nature editorial processes.Source: §7 · p. 13