Source-linked AI summary
A Pathway for Assessing Grey Literature: Leveraging AI to Extract Conference Metadata and Organiser Information from Calls for Papers
Angelo Salatino, Francesco Osborne, Alexis Vizcaino, Aliaksandr Birukou, Enrico Motta
TL;DR
Grey literature, including Calls for Papers (CfPs), is largely overlooked in metascience and scientometric analysis because its metadata is heterogeneous and difficult to process at scale. COCI applies a multi-stage AI framework to extract and standardise information from CfPs. It extracts conference and organiser metadata, providing a foundation for scientometric analyses of events beyond traditional publisher-based venues.
Problem
Grey literature such as CfPs remains largely overlooked in metascience and scientometric analysis, while its heterogeneous and often unindexed information lacks consistent structure.
Source: §1 · p. 1, §2.1 · p. 2, §2.5 · p. 4Method
Results
COCI extracts structured CfP metadata including conference editions, locations, organisers, their roles and affiliations, and topics.
Source: §1 · pp. 1–2, §6 · p. 8Takeaways & Limitations
COCI provides a foundation for scientometric analyses of scholarly events beyond traditional publisher-based venues.
Source: §6 · p. 8For STI-ENID, proceedings locations vary by edition across platforms, making harvesting more challenging.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Despite its importance, grey literature, including Calls for Papers (CfPs), remains largely overlooked in Metascience and Scientometric analysis due to its unstructured, highly heterogeneous format, which traditional tools struggle to process at scale. However, Large Language Models now offer a pivotal opportunity to devise innovative tools for systematically harvesting and processing such data. In this paper, we introduce COCI, an AI-based framework that automates the extraction of granular, structured metadata from raw CfP text. COCI employs a multi-stage pipeline for entity extraction, followed by author disambiguation against OpenAlex and semantic mapping of topics and conference series. This process identifies key data points, including conference editions, geographic locations, and comprehensive lists of organisers, along with their specific roles and affiliations. By structuring this previously inaccessible information, COCI establishes a foundation for the systematic analysis of grey literature, enabling new research opportunities and shifting the scholarly focus towards non-publisher-based events.
1. Introduction
COCI uses AI to extract structured metadata from unstructured Calls for Papers, a form of grey literature that is difficult to analyse at scale. Its standardised data, including conference and organiser information, supports further discovery of research artefacts.
Introduction
- COCI addresses the difficulty of analysing scattered, heterogeneous grey literature by automating fine-grained metadata extraction from raw Calls for Papers.The introduction identifies Calls for Papers as grey literature that often lacks bibliographic control and is distributed through informal channels.Source: §1 · p. 1
- COCI identifies conference series and editions, acronyms, locations, organisers, their affiliations and roles, and thematic topics, then presents the data in a standardised format.Organiser roles include research, workshop, and publicity chairs.Source: §1 · pp. 1–2
- Conference URLs extracted by COCI can enable AI agents to discover event materials such as preprints, presentations, and videos.COCI is released as an open-source project to support reproducibility and community-driven research.Source: §1 · p. 2
2. Related Work
Related work frames Calls for Papers as rich but poorly structured scholarly evidence and reviews the challenges of extracting, disambiguating, and aligning their metadata. It also positions grey literature as a source of emerging research directions that standard publication-focused analyses may miss.
2.1. Information Extraction from Scholarly Text
- Traditional scholarly IE works with stable templates but degrades on heterogeneous CfPs, while LLM extraction handles flexible schemas yet remains prompt-sensitive and prone to hallucination (Li et al., 2025).For CfPs, extraction must interpret informal phrasing, infer implicit information such as event-series names, and adapt to new schemas.Source: §2.1 · p. 2
2.2. Author and Entity Disambiguation
- Author and affiliation disambiguation remains error-prone despite identifiers and learning methods, as name and affiliation changes, multidisciplinary work, and evolving institutions complicate linkage.Persistent identifiers and OpenAlex can improve author linking, while ROR and string-plus-geographic matching support affiliation alignment; quality checks include trusted-source cross-validation and anomaly detection.Source: §2.2 · pp. 2–3
2.3. Controlled Vocabulary Mapping
- Lexical matching aligns free-text topics with controlled vocabularies but struggles with paraphrases, while embedding-based semantic mapping can identify concept relationships beyond surface wording (Ampudia Vicente, 2025; Reimers and Gurevych, 2019).Hybrid approaches combine lexical matching with semantic similarity (Salatino et al., 2019b).Source: §2.3 · p. 3
2.4. Conference Series and Event Alignment
- Syntactic conference matching can be precise but miss complex naming variants, while hybrid semantic-syntactic alignment combines broad matching with precision (Schraagen, 2014; Salatino et al., 2019b).The proposed approach uses SentenceTransformers to identify close matches with high recall, then applies syntactic similarity for high precision.Source: §2.4 · pp. 3–4
2.5. Analysing Grey Literature
- CfPs and other grey literature can reveal emerging directions and early disciplinary intersections, but their informal, scattered, and unindexed metadata makes them difficult to track.The work emphasizes non-publisher-based events, including conferences without formal proceedings, as evidence for research trends and nascent collaboration networks.Source: §2.5 · p. 4
3. Approach
COCI turns raw Calls for Papers into structured conference and organiser metadata through a multi-stage pipeline that combines LLM extraction with post-processing and external validation. It links organisers, topics, and conference series to external services and presents enriched records for analysis.
3. Approach
- COCI uses a five-stage workflow for LLM extraction, author disambiguation, conference-series matching, topic mapping, and data visualisation.Because STI-ENID proceedings locations vary by edition and platform, harvesting them is particularly challenging.Source: §3 · p. 4
3.1. Prompt Generation
- GPT-4o extracts conference details and every listed organiser’s name, affiliation, country, and role, with a JSON schema enforcing machine-readable output.The prompt template was refined to handle variation across scientific disciplines and specifies formats for nested organiser data, years, and topics.Source: §3.1 · p. 5
3.2. Author Matching
- COCI links organisers to OpenAlex profiles using institution-first search, affiliation checks, name-similarity heuristics, and profile verification or institution inference.It discards suspiciously homogeneous affiliations, uses publication counts and Levenshtein similarity to rank candidates, and may infer an institution from publications over the past decade.Source: §3.2 · p. 5
3.3. Topics of Interest
- COCI maps extracted topics to OpenAlex concepts using 384-dimensional embeddings and cosine similarity with a 0.6 threshold, avoiding reliance on literal string matching.The method uses the all-MiniLM-L6-v2 SentenceTransformers model to map ad-hoc keywords to standardised concepts while accommodating variations such as hyphenation and pluralisation.Source: §3.3 · pp. 5–6
3.4. Conference Series Matching
- COCI matches conference series to DBLP, AIDA, and ConfIDent using semantic search, then validates matches with Levenshtein similarity and cross-references identifiers across services.Source: §3.4 · p. 6
3.5. Visualisation
- The Streamlit interface displays conference metadata, an organiser table with matched profiles and affiliations, external records, and OpenAlex-mapped topics, and supports Excel export.A star indicates that a CfP affiliation aligns with an OpenAlex author profile.Source: §3.5 · pp. 6–7
4. Evaluation
After iterative prompt and validation refinements, COCI successfully processed all 40 CfPs across a diverse range of disciplines. The evaluation focused on technical performance and identified inconsistencies involving affiliations, author matching, and organiser roles.
Evaluation
- The evaluation assessed technical performance; user-interface evaluation was deferred because the current interface was intended for functional code execution and likely to be redesigned.Source: §4 · p. 7
- After iterative refinement, COCI successfully processed all 40 CfPs drawn from a diverse range of academic disciplines.The outputs were manually inspected as part of an iterative test-and-learn evaluation.Source: §4 · p. 7
- A distribution heuristic flags affiliations as unreliable when unique organisers exceed four times the number of unique organisations, addressing hallucinated uniform affiliations in STI 2026.The source lacked committee affiliations, but the LLM attributed every member to the University of Antwerp.Source: §4 · p. 7
- An OpenAlex error matched organiser Andrea Giovanni Nuzzolese to Isabel F. Cruz because the database listed his name as her alternative name.The authors attribute this error to the external database rather than the matching algorithm.Source: §4 · p. 7
- Prompt engineering stabilised role identification after the LLM hallucinated organisers’ roles or defaulted to “main track” instead of interpreting roles stated in the text.Source: §4 · p. 7
5. Vision and Future Plan
COCI uses Calls for Papers as an entry point into heterogeneous conference grey literature, with a vision of expanding event analysis and recognizing researchers’ community contributions.
Vision and Future Plan
- Because events range from open-call to invitation-only and lack centralized publishers or standardized records, COCI treats CfPs as a natural starting point for heterogeneous grey literature.Conference outputs may also include video recordings or materials in online repositories.Source: §5 · pp. 7–8
- COCI could follow CfP URLs to harvest further grey literature and assess events’ scope, organizer diversity and seniority, and prior misconduct via RetractionWatchDatabase and PubPeer.The proposed assessments could help evaluate the health and scientific quality of conferences.Source: §5 · p. 8
- To expand coverage scalably without exhaustive web crawling, COCI could track CfP registries and professional mailing lists, supplemented by community submissions.Examples named are WikiCFP and the ISSI Society.Source: §5 · p. 8
- By mapping organisers’ professional backgrounds and ties to journals, editorial boards, and proceedings editors, COCI could support formal credit for researchers’ community contributions.The envisioned recognition system is analogous to Publons’ model for peer review.Source: §5 · p. 8
6. Conclusions
COCI turns heterogeneous Calls for Papers into structured metadata for scientometric analysis of scholarly events beyond publisher-based venues. The authors plan to expand it with continuous harvesting, event organization for longitudinal and comparative analysis, and downstream applications.
6. Conclusions
- COCI automates entity extraction, links authors to external knowledge bases such as OpenAlex, and standardizes topics and conference series through semantic mapping.The resulting structured dataset enables scientometric analyses of scholarly events beyond traditional publisher-based venues.Source: §6 · p. 8
- Planned extensions include continuous CfP crawling across registries and mailing lists, systematic event organization for longitudinal and comparative analysis, and downstream platforms and applications.The planned analyses cover conference series, topics, and communities.Source: §6 · p. 8
Open science practices
- The open-source framework, evaluation data from 40 CfPs, and v1.0.0 preservation release are publicly available on GitHub to support reproduction of the experiments.The release provides the exact code and data version used in the experiments.Source: §6 · pp. 8–9