PaperlayerView exact revision on arXiv

Source-linked AI summary

OC-AND: A Citation-Enriched Benchmark for Author Name Disambiguation

Francesca Cappelli, Giovanni Colavizza, Silvio Peroni

arXiv:2610.13462v1cs.DL

TL;DR

Widely adopted AND benchmarks, mostly derived from AMiner, have become the de facto evaluation standard, but systematic limitations compromise their suitability for citation-aware and graph-based evaluation. OC-AND remaps WhoIsWho’s author clusters and partitions onto OpenCitations bibliographic and citation records through a reproducible curation pipeline. BOND experiments show that OC-AND provides a substantially different and more challenging disambiguation environment while remaining compatible with an existing AND pipeline.

  • Problem

    Widely adopted AND benchmarks, mostly derived from AMiner, have become the de facto evaluation standard, but systematic limitations compromise their suitability for citation-aware and graph-based evaluation.

    Source: Background & Summary · p. 1

  • Method

    OC-AND remaps WhoIsWho’s author clusters and partitions onto OpenCitations bibliographic and citation records through a reproducible curation pipeline.

    Source: Background & Summary · p. 2, Conclusions · p. 13

  • Results

    BOND experiments show that OC-AND provides a substantially different and more challenging disambiguation environment while remaining compatible with an existing AND pipeline.

    Source: Conclusions · p. 14

  • Takeaways & Limitations

    OC-AND offers a reproducible benchmark for evaluating AND with open bibliographic metadata and citation information.

    Source: Conclusions · pp. 13–14

    The DOI validator uses publication-year agreement alone, a conservative baseline that leaves richer validation to settings with additional reliable metadata.

    Source: Stage 2: Crossref score optimisation… · p. 4

Abstract

from arXiv · show

Author Name Disambiguation benchmarks are predominantly derived from closed infrastructures that may not reflect conditions in open bibliographic environments. We present OC-AND, an open citation-enriched benchmark for AND constructed by remapping the WhoIsWho dataset with bibliographic and citation metadata from OpenCitations. The dataset is produced through a reproducible five-stage pipeline involving DOI verification, metadata validation, OpenCitations enrichment, and author-centric transformation. OC-AND retains the WhoIsWho ground-truth files and partition structure unchanged, while its publication files are restricted to publications that could be matched to a validated DOI and an OpenCitations record. It introduces realistic characteristics of open environments: heterogeneous metadata completeness, absent abstracts and affiliations, asymmetric citation coverage, and sparse graph connectivity. The dataset contains author records with associated citations, enabling evaluation of AND methods in settings closer to real-world open scholarly infrastructure scenarios. OC-AND is released as open data with complete provenance documentation and pipeline code, supporting reproducible research and the development of citation-aware disambiguation approaches.

Background & Summary

Widely used AND benchmarks are tied to closed or managed metadata environments and often lack integrated citation information and open-data heterogeneity. OC-AND remaps WhoIsWho onto OpenCitations while retaining the original ground-truth clusters and partition assignments.

Background & Summary

  • Existing benchmarks are mostly derived from AMiner and have limited suitability for citation-aware evaluation in open bibliographic settings.The paper identifies weakly integrated citation information and consistently populated metadata as key differences from real-world open data.Source: Background & Summary · pp. 1–2
  • S2AND unifies historical datasets but uses Semantic Scholar’s dense proprietary metadata layer, leaving direct evaluation in real-world bibliographic settings open.The paper notes that Semantic Scholar’s internal pipelines systematically manage metadata completeness.Source: Background & Summary · p. 2
  • OC-AND remaps WhoIsWho onto OpenCitations records while retaining its ground-truth clusters and train/validation partitions.The benchmark preserves open-environment characteristics, including absent abstracts and affiliations, asymmetric citation coverage, and sparse graph connectivity.Source: Background & Summary · p. 2

Methods

Bridge4AND constructs OC-AND through five stages, from DOI recovery and validation to OpenCitations enrichment and author-centric transformation. It is designed to add open metadata while preserving reproducibility and structural consistency.

Stage 1: Gold standard construction and DOI recovery

  • WhoIsWho lacks DOIs, so annotators manually verified 500 high-ambiguity validation records, confirming active DOIs for 410 and marking 90 unresolved.The sample was drawn from the validation partition; the resulting gold-standard records were checked against Crossref and split for threshold calibration and validation.Source: Stage 1: Gold standard construction… · pp. 2–3

Stage 2: Crossref score optimisation and preliminary metadata validation

  • Crossref ranks title-and-year queries with BM25; a training-split grid search selects the accuracy-maximizing score cutoff, followed by a publication-year consistency check.The year-only validator is a conservative baseline because other fields may be incomplete or inconsistently curated.Source: Stage 2: Crossref score optimisation… · pp. 3–4

Stage 3: Large-scale DOI validation

  • The calibrated threshold and validator are applied across WhoIsWho, and only validated DOI assignments proceed to OpenCitations enrichment.Validated, rejected, and unresolved records are written to separate outputs.Source: Stage 3: Large-scale DOI validation · p. 5

Stage 4: OpenCitations metadata and citation enrichment

  • OpenCitations Meta supplies bibliographic records, while the Index API supplies incoming citations and outgoing references; publications without Meta records are excluded.Missing properties remain empty, and title-derived keywords preserve schema compatibility.Source: Stage 4: OpenCitations metadata and… · p. 5

Stage 5: Author-centric transformation

  • The author-centric transformation keeps WhoIsWho label files and partition structure unchanged, but effective ground truth is limited to publications present in each publication file.Training publications pair with target names by identifier; validation input maps normalized OpenCitations author names to publication-ID lists.Source: Stage 5: Author-centric transformation · p. 5

Data Records

OC-AND is distributed in five JSON files across training and validation partitions. Its publication records, ground-truth files, and validation input serve different roles, while the ground-truth structure remains tied to WhoIsWho.

Data Records

  • OC-AND is distributed as five JSON files organized into training and validation partitions.Table 1 summarizes the files and their partition mapping.Source: Data Records · p. 6, Table 1 · p. 6

Publication Records

  • Publication files use a shared paper-centric schema, with unavailable abstracts and affiliations left empty and keywords derived from publication titles.Figure 3 shows an example record with empty abstract and affiliation fields.Source: Publication Records · p. 6, Table 2 · p. 6, Figure 3 · p. 7

Ground-Truth Cluster Files

  • Ground-truth files retain the WhoIsWho structure: author names map to cluster IDs and publication lists, and the files remain identical to their WhoIsWho counterparts.Publication identifiers without an OC-AND publication record are not removed from these files.Source: Ground-Truth Cluster Files · p. 6, Figure 4

Validation Inference Input

  • The validation input lists publications under OpenCitations Meta author strings, including co-authors and separate name variants, without cluster labels.A paper can appear under both target names when both are co-authors, while some ground-truth papers appear only under a variant string or no key.Source: Validation Inference Input · p. 7, Figure 5

Dataset Profile and Properties

OC-AND exchanges dense textual metadata for open-registry conditions, including sparse text fields and broad citation coverage. Its publication files include only records that passed DOI validation and were retrieved from OpenCitations.

Dataset Profile and Properties

Metadata Completeness and Richness Profile

  • Over 92.9% of OC-AND records have citation links, while abstracts and institutional affiliations—covered for approximately 78.27% and 72.79% of WhoIsWho records—are absent.The contrast reflects a relational layer counterbalancing textual sparsity.Source: Metadata Completeness and Richness… · p. 8, Table 3 · p. 8

Volumetric and Partition Structure

  • Publication files retain records only when a DOI was validated against Crossref and an OpenCitations record was retrieved; ground-truth clusters are restricted to those publications.The ground-truth files and target author names remain unchanged, while the validation input additionally includes all OpenCitations Meta author strings.Source: Volumetric and Partition Structure · p. 8, Table 4 · p. 9

Technical Validation

Technical validation measures DOI assignment performance and tests BOND on OC-AND as a drop-in benchmark. The operational DOI configuration achieved precision of 0.837 and recall of 0.958 on held-out validation.

DOI assignment accuracy

  • In the N = 500 gold-standard audit, 82.0% (410 records) were resolved to active DOIs and 18.0% (90 records) were unresolvable.Among 98 records with Chinese-only titles, 76 had valid DOIs absent from Crossref indexes.Source: DOI assignment accuracy · p. 9
  • On the calibration split, the accuracy-optimal threshold (τopt = 45.45) and F1-optimal threshold (τ = 41.01) yielded similar performance.Source: DOI assignment accuracy · p. 9
  • On the held-out validation split, the complete DOI configuration reached a precision of 0.837 and a recall of 0.958.The operational cutoff was τop = 35.00; on calibration it yielded recall 0.994 and precision 0.718 to retain lower-scoring valid DOI candidates for later validation.Source: DOI assignment accuracy · p. 10
  • The held-out validation split began with 200 cases: the score threshold removed 46 candidates and Φvalid removed 19 more, leaving 135 accepted DOIs, of which 113 were correct.Of the 22 false positives, 13 involved publications for which no DOI exists.Source: DOI assignment accuracy · p. 10, Figure 7 · p. 11
  • BOND was run on both WhoIsWho and OC-AND in text-only configurations with citation topology disabled; the OC-AND run tested compatibility as a drop-in replacement.The experiments were intended to validate the dataset for AND evaluation; Table 6 reports BOND performance on both datasets.Source: Functional validation via… · pp. 10–11, Table 6 · p. 11

Usage Notes

The usage notes define evaluation procedures and structural error measures, then report increased splitting error in the open-metadata setting. They also identify citation topology as a signal for future evaluation.

Evaluation setup

  • Because BOND bootstraps from scratch without supervised transfer of learned weights or representations, its training and validation partitions serve as independent validation gates rather than conventional overfitting controls.The gates are used to assess stability across distinct data samples.Source: Datasets Split Design · pp. 11–12

Evaluation Framework and Metrics of the Functional validation

  • Evaluation excludes publications without a prediction, computes metrics per target name, sums pairwise and B3 counts, and averages LE and SE.The ground truth is restricted to publications for which a prediction is produced.Source: Evaluation Framework and Metrics of… · p. 12
  • B3 measures instance-level cluster correctness and is suited to AND settings where author publication counts vary substantially.The paper distinguishes these instance-level metrics from structural metrics based on cluster topology.Source: Evaluation Framework and Metrics of… · pp. 12–13

Structural error metrics

  • LE measures predicted-cluster contamination by merged authors, while SE measures ground-truth authors fragmented across predicted clusters.The two structural measures distinguish aggressive merging from conservative fragmentation.Source: Evaluation Framework and Metrics of… · pp. 12–13
  • In open scholarly infrastructures, lumping errors are described as harder to detect and correct downstream than splitting errors.The notes connect contamination in author profiles to downstream operational cost.Source: Evaluation Framework and Metrics of… · p. 13

Structural Score and Composite Ranking

Metadata Sparsity and Model Performance

  • SE rose from 0.1348 to 0.7865 on the validation split; training SE was 0.7797, and the authors associate the shift with metadata sparsity.The comparable training result suggests the increase is not primarily driven by validation name-string retrieval.Source: Metadata Sparsity and Model Performance · p. 13

Exploiting Citation Topology

  • OpenCitations publication records expose incoming and outgoing citations, allowing researchers to test whether citation topology and bibliographic coupling can offset missing text-rich features.Source: Exploiting Citation Topology · p. 13

Conclusions

The paper introduces OC-AND, an open citation-enriched benchmark remapped from WhoIsWho through the reproducible Bridge4AND workflow. Its experiments show a changed and more challenging disambiguation environment, while citation topology remains unevaluated.

Conclusions

  • OC-AND remaps WhoIsWho onto OpenCitations bibliographic and citation information using a reproducible Bridge4AND curation workflow.The five-stage workflow preserves reproducibility and structural consistency.Source: Conclusions · pp. 13–14
  • Bridge4AND performs DOI recovery, Crossref-based validation, OpenCitations enrichment, and author-centric transformation while preserving WhoIsWho's ground-truth files and partition structure.Source: Conclusions · p. 14
  • The reconstructed records omit abstracts and affiliations, add DOI identifiers, and include citation relations for the large majority of records.Source: Conclusions · p. 14
  • BOND experiments found a substantially different and more challenging disambiguation environment, but did not evaluate citation topology because citation information was disabled during functional validation.OC-AND is intended as a resource for future evaluation of citation-aware AND methods.Source: Conclusions · p. 14

Code Availability

The Bridge4AND pipeline code is available through a GitHub repository and is implemented in Python 3.

Code Availability

Declaration of Use of Generative AI

The authors report no generative AI use for data generation or analysis, or for drafting the scientific content; AI tools supported language editing and Bridge4AND coding.

Declaration of Use of Generative AI

Loading 2610.13462v1…