Source-linked AI summary
A Pathway for Assessing Grey Literature: Leveraging AI to Extract Conference Metadata and Organiser Information from Calls for Papers
Angelo Salatino, Francesco Osborne, Alexis Vizcaino, Aliaksandr Birukou, Enrico Motta
TL;DR
Grey literature such as Calls for Papers is difficult to track systematically because it is heterogeneous and poorly indexed. COCI extracts and standardises CfP metadata, linking organisers and concepts to external services, to support scientometric analysis beyond publisher-based venues.
Problem
Grey literature, including Calls for Papers, remains largely overlooked because its heterogeneous content is difficult to track systematically and is seldom standardised in databases.
Method
COCI uses a multi-stage AI pipeline to extract CfP entities, disambiguate organisers against OpenAlex, map topics and conference series, and validate structured metadata.
Results
COCI extracts granular structured metadata from heterogeneous CfPs, including conference details, organisers, roles, affiliations, topics, and standardised entity links.
Takeaways & Limitations
The structured CfP data supports scientometric analysis of scholarly events beyond traditional publisher-based venues and can guide discovery of additional grey literature.
Takeaways & Limitations
Harvesting reliable data remains challenging because formal proceedings may be distributed across different platforms, as illustrated by STI-ENID.
Abstract
from arXiv · showhide
Despite its importance, grey literature, including Calls for Papers (CfPs), remains largely overlooked in Metascience and Scientometric analysis due to its unstructured, highly heterogeneous format, which traditional tools struggle to process at scale. However, Large Language Models now offer a pivotal opportunity to devise innovative tools for systematically harvesting and processing such data. In this paper, we introduce COCI, an AI-based framework that automates the extraction of granular, structured metadata from raw CfP text. COCI employs a multi-stage pipeline for entity extraction, followed by author disambiguation against OpenAlex and semantic mapping of topics and conference series. This process identifies key data points, including conference editions, geographic locations, and comprehensive lists of organisers, along with their specific roles and affiliations. By structuring this previously inaccessible information, COCI establishes a foundation for the systematic analysis of grey literature, enabling new research opportunities and shifting the scholarly focus towards non-publisher-based events.
1. Introduction
COCI addresses the limited use of Calls for Papers in Metascience and Scientometric analysis by structuring information extracted from raw CfP text. It identifies conference metadata, organiser details, thematic topics, and conference URLs to facilitate grey-literature analysis and further discovery.
- Motivation and contribution: COCI processes raw Calls for Papers to identify and structure conference series, editions, acronyms, and geographic locations.The framework targets information typically absent from analyses focused on published literature, journal issues, and conference proceedings.
- Motivation and contribution: COCI extracts comprehensive organiser lists, including affiliations and roles such as research, workshop, and publicity chairs.These organiser attributes are structured alongside other conference metadata.
- Motivation and contribution: The extracted metadata is standardised and presented through an intuitive user interface, including conference URLs that support further discovery.The system also captures thematic topics of interest, enabling analysis of grey literature.
2. Related Work
Related work frames CfP analysis as an information-extraction problem involving unstructured event metadata, entity disambiguation, semantic mapping, and conference-series alignment. It also situates CfPs within expanding grey literature that captures emerging research directions but is poorly indexed and difficult to track systematically.
- Information extraction: Information extraction derives structured entities, attributes, and relations from unstructured or semi-structured text, including event metadata encoded in proceedings titles and CfPs.Such methods underpin digital libraries, knowledge graphs, and analytics on scientific communities.
- Entity disambiguation: Author and entity disambiguation is foundational because bibliometric accuracy depends on correctly linking authors and institutions across datasets.Persistent identifiers such as ORCID and infrastructures such as OpenAlex improve reliability, but affiliation variants and inconsistent metadata remain challenges.
- Semantic mapping: Semantic mapping aligns free-text terms with structured conceptual frameworks using lexical similarity, string matching, co-occurrence statistics, and controlled vocabularies.Examples include OpenAlex Concepts and the Computer Science Ontology.
- Conference-series alignment: Bibliometric research aligns conference series across data sources to track longitudinal research trends, supported by infrastructures such as DBLP and ConfIDent.DBLP provides a curated registry of computer-science proceedings and conference series, while ConfIDent models events as persistent, citable entities.
- Grey literature: Grey literature such as preprints, technical reports, and CfPs captures emerging research directions before formal publication but is seldom indexed in standardised databases.Its informal nature and scattered metadata complicate systematic tracking.
3. Approach
COCI uses a five-stage pipeline to transform unstructured Calls for Papers into structured, externally linked metadata about conferences, organisers, and topics. It combines GPT-4o extraction with OpenAlex-based disambiguation, semantic mapping, conference-series matching, and data visualisation.
- Pipeline overview: COCI’s workflow comprises five stages: LLM prompt engineering, OpenAlex author disambiguation, conference-series matching, topic mapping, and data visualisation.The pipeline addresses the limited precision of raw LLM outputs for scholarly databases by adding post-extraction enrichment and alignment.
- Data limitations: Proceedings harvesting is challenging because STI-ENID proceedings appear on different platforms across editions, including Zenodo, Orvium, and the official ENID-Europe website.This variation in publication location complicates systematic collection despite the availability of formal proceedings.
- Entity extraction: GPT-4o extracts event names, acronyms, series, years, locations, topics, and comprehensive organiser information from variable CfP text.A prompt template was refined iteratively to handle CfPs spanning different scientific disciplines.
- Author disambiguation: Organisers are linked to OpenAlex profiles through a precision-oriented multistage search, enriching records with ORCID, OpenAlex, and ROR identifiers.The strategy first locates the organiser’s institution, then filters authors associated with that institution.
- Topic mapping: 384-dimensional embeddings from all-MiniLM-L6-v2 map unstructured topics to standardised OpenAlex concepts using semantic similarity rather than literal string matching.This approach accommodates variations such as hyphenation and pluralisation without treating equivalent concepts as mismatches.
- Conference matching: Conference-series names are semantically matched against DBLP, the AIDA Dashboard, and ConfIDent using sentence-transformer embeddings and precomputed indices.The system retrieves nearest-neighbour records for each external service.
4. Evaluation
The evaluation focuses on COCI’s technical performance rather than its user interface. It processes 40 CfPs from diverse fields and manually inspects the extracted metadata.
- Technical evaluation: COCI’s evaluation assesses the quality of its extracted metadata through technical-performance analysis.The study does not evaluate the user interface because the current version primarily supports functional code execution and may be redesigned.
- Evaluation corpus: 40 CfPs were processed and their outputs manually inspected.The CfPs were drawn from a diverse range of fields, including Computer Science, Engineering, and Materials Science.
5. Vision and Future Plan
The project targets conferences beyond formal proceedings, where absent publishers and standardised bibliographic records make reliable data collection difficult. Given heterogeneous conference outputs, calls for papers provide a natural starting point because they contain rich information for research and analysis.
- Rationale: Conferences outside the proceedings-based model lack centralised publishers and standardised bibliographic records, complicating reliable data collection.This context creates a significantly higher data-gathering challenge than conventional proceedings-based events.
- Rationale: Conference participation and outputs are highly heterogeneous, ranging from invitation-only events to video recordings and repository deposits.Examples include VideoLectures.net recordings and materials deposited in Zenodo Communities.
- Future direction: Calls for papers constitute a natural starting point because harvesting and processing them can support a wide range of research and analytical opportunities.CfPs contain a rich body of information that becomes useful once systematically harvested and processed.
6. Conclusions
The paper introduces COCI, an AI-based framework for extracting and standardising structured metadata from heterogeneous Calls for Papers. COCI is released with its source code and evaluation data to support reproducible research.
- Contributions: COCI automates granular metadata extraction from heterogeneous Calls for Papers through entity extraction, OpenAlex-linked author disambiguation, and semantic mapping.The framework standardises topics and conference series names using external knowledge bases and semantic methods.
- Open science practices: The COCI software, source code, and 40 Calls for Papers used in evaluation are publicly available through the project’s GitHub repository.A preserved release provides the exact code and data version used in the experiments for reproduction.
- Author contributions: The authors report contributions across conceptualisation, methodology, software, validation, funding acquisition, and writing under the CRediT system.Angelo Salatino is credited for conceptualisation, methodology, software, validation, and original drafting, while the other authors contributed to funding acquisition or review and editing.
- Declarations: The authors declare no competing interests and acknowledge Springer Nature for funding the research.These statements appear in the paper’s competing-interests and funding-information disclosures.