PaperlayerView exact revision on arXiv

Source-linked AI summary

CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis

Rochana R. Obadage, Sarah M. Rajtmajer, Jian Wu

arXiv:2511.07790v1cs.DLcs.CL

TL;DR

Assessing reproducibility from published work remains challenging, and existing citation contexts offer a promising signal. The paper introduces CC30k, a labeled dataset of machine-learning citation contexts, and reports that traditional sentiment classifiers fail to capture reproducibility-oriented sentiment.

  • Problem

    Automatic reproducibility assessment remains challenging, and existing sentiment and citation-context datasets do not specifically incorporate reproducibility signals.

    Source: §1 · p. 1

  • Method

    The authors construct CC30k from citation contexts labeled for reproducibility-oriented sentiment and fine-tune large language models for classification.

    Source: §1 · p. 2

  • Results

    Traditional sentiment classifiers fail to capture reproducibility-oriented sentiment, while crowdsourced labels achieve 93.85% overall accuracy.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    CC30k fills a resource gap for studying reproducibility and replicability in AI research.

    Source: §7 · p. 14

    Fine-tuning improvements are not monotonic with more data for every model, with label noise a plausible contributor.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

Sentiments about the reproducibility of cited papers in downstream literature offer community perspectives and have shown as a promising signal of the actual reproducibility of published findings. To train effective models to effectively predict reproducibility-oriented sentiments and further systematically study their correlation with reproducibility, we introduce the CC30k dataset, comprising a total of 30,734 citation contexts in machine learning papers. Each citation context is labeled with one of three reproducibility-oriented sentiment labels: Positive, Negative, or Neutral, reflecting the cited paper's perceived reproducibility or replicability. Of these, 25,829 are labeled through crowdsourcing, supplemented with negatives generated through a controlled pipeline to counter the scarcity of negative labels. Unlike traditional sentiment analysis datasets, CC30k focuses on reproducibility-oriented sentiments, addressing a research gap in resources for computational reproducibility studies. The dataset was created through a pipeline that includes robust data cleansing, careful crowd selection, and thorough validation. The resulting dataset achieves a labeling accuracy of 94%. We then demonstrated that the performance of three large language models significantly improves on the reproducibility-oriented sentiment classification after fine-tuning using our dataset. The dataset lays the foundation for large-scale assessments of the reproducibility of machine learning papers. The CC30k dataset and the Jupyter notebooks used to produce and analyze the dataset are publicly available at https://github.com/lamps-lab/CC30k .

1 Introduction

Citation contexts can reveal authors’ perspectives on cited work and may signal its reproducibility, but automatic reproducibility assessment remains challenging. CC30k provides a labeled resource for studying reproducibility-oriented sentiment in machine-learning citation contexts.

1 Introduction

  • Automatic reproducibility assessment remains challenging in computer science and AI, while citation contexts have shown promise as a signal of cited-study reproducibility.Analyzing citation contexts may also help identify patterns, trends, and gaps in reproducibility.Source: §1 · pp. 1–2
  • CC30k contains 30,734 machine-learning citation contexts labeled Positive, Negative, or Neutral for reproducibility-oriented sentiment.Contexts were filtered for ambiguous citation marks, annotated through crowdsourcing, and assigned final labels by majority vote.Source: §1 · p. 2
  • The paper argues that traditional sentiment classifiers fail to capture reproducibility-oriented sentiment.Source: §1 · p. 2
  • The authors fine-tune large language models on CC30k for reproducibility-oriented sentiment classification.Source: §1 · p. 2
  • The paper compares CC30k with existing sentiment- and citation-analysis datasets and categorizes citation contexts by citation-mark format.Source: §1 · p. 2

2 Related Work

Existing sentiment and citation-context datasets do not specifically incorporate reproducibility or replicability signals. CC30k targets that gap by labeling reproducibility-oriented sentiments in citation contexts.

2 Related Work

  • Traditional sentiment datasets focus on web text such as reviews and social-media posts, rather than reproducibility signals in scholarly literature.Source: §2 · p. 2
  • Citation-context resources such as ACL-ARC, SciCite, and the Critical Citation Contexts Corpus study citation functions, intents, or critique, but not reproducibility-oriented sentiment.Source: §2 · p. 2
  • Table 1 compares CC30k with datasets for sentiment analysis and citation analysis.Source: Table 1 · p. 3

3 Methods

The methods begin from a small, imbalanced set of manually labeled citation contexts and build a larger dataset through collection, filtering, crowdsourcing, validation, and negative augmentation.

3.1 Overview

  • The earlier manually labeled dataset contained 1,937 contexts, only 23 with negative labels, and both LLMs and generic sentiment models struggled to label reproducibility-oriented sentiment accurately.Source: §3.1 · p. 3

3.2 Dataset Production

  • The authors collected data for 145 reproducibility studies and 130 original papers, primarily sourcing studies from Machine Learning Reproducibility Challenge competitions.Source: §3.2 · p. 4
  • Citation-context collection identified 13,314 citing papers and 41,244 contexts about original studies, collected in January 2024.Source: §3.2 · p. 4
  • After filtering ambiguous citation marks, crowdsourcing yielded 25,829 contexts labeled Positive, Negative, or Neutral, and stratified sampling selected 244 contexts for manual verification.Source: §3.2 · p. 4
  • To address class imbalance, the pipeline added 4,905 negative contexts using labels from human experts and a supervised machine-learning labeler.Source: §3.2 · p. 4

3.3 Citation mark analysis

The analysis examines citation-mark styles and filters contexts so that the cited paper can be identified unambiguously.

3.3 Citation mark analysis

  • A citation mark is an inline reference to a bibliography entry, appearing in styles such as parenthetical author-year text or a bracketed number.Source: §3.3 · p. 5
  • Contexts may contain several papers and citation marks; the authors exclude contexts with multiple, inconsistent, or ambiguous marks to keep cited papers identifiable to labelers.Source: §3.3 · p. 5
  • The dataset’s citation contexts commonly use MLA, APA, and IEEE text-based formats, and the analysis categorizes them by citation-mark count.Source: §3.3 · p. 5
  • Mixed or differently placed citation marks complicate parsing, and Table 3 lists identified formats while noting that formats (c) and (e) were excluded.Source: Table 3 · p. 5, §3.3 · p. 6

3.4 Cleansing citation contexts for crowdsourcing

The cleansing process resolves citation-mark formats and sentence structures to retain contexts that identify a cited paper unambiguously.

3.4 Cleansing citation contexts for crowdsourcing

  • Citation-mark styles and sentence structures were analyzed to produce clean contexts for crowdworkers.The process also resolved marks referring to the cited paper from multiple citation marks.Source: §3.4 · p. 6
  • The filtered dataset retains only contexts that uniquely identify the related cited paper.Table 4 summarizes scenarios involving citation marks that refer to uniquely identified cited papers.Source: §3.4 · p. 6, Table 4 · p. 6
  • Cleansing reduced the number of contexts from 41,244 to 25,829.Source: §3.4 · p. 6

3.5 Crowdsourcing annotation

Crowdworkers independently labeled contexts as Positive, Negative, or Neutral, and pilot-based screening was used to select annotators.

3.5.1 Task Design

  • Each of 25,829 citation contexts was labeled independently by three workers, with majority voting determining the final label.The job was posted in 26 batches of about 1,000 contexts each.Source: §3.5.1 · p. 6

3.5.2 Types of ROS labels and descriptions

  • Positive, Negative, and Neutral labels distinguish reproducibility signals, irreproducibility signals, and contexts without such indications.Positive examples include reuse of cited work’s data or code; negative examples include unavailable data or unsuccessful attempts to obtain results.Source: §3.5.2 · p. 6

3.5.3 Worker selection mechanism

  • The pilot used 20 contexts with expert ground-truth labels to assess potential workers.It contained 10 actual contexts and 10 dummy contexts with readily determined labels.Source: §3.5.3 · p. 6
  • Iterative batches built a pool of 138 workers, from which 16 were selected using pilot performance criteria.Selected workers labeled at least 15 pilot contexts, exceeded 90% accuracy, and correctly labeled every dummy context.Source: §3.5.3 · p. 7

3.6 Augmenting Negative Citation Contexts

To address scarce negative contexts, the authors augmented CC30k using model-generated candidates followed by manual and classifier-based verification.

3.6 Augmenting Negative Citation Contexts

  • Negative contexts represented only about 1% of the dataset, motivating the effort to add examples and balance positive labels.Source: §3.6 · p. 7
  • The ensemble classified about 600,000 contexts collected from 21,757 computer science papers published after 2017.It comprised five transformer models fine-tuned on crowdsourced labels.Source: §3.6 · pp. 7–8
  • 0.81 weighted average F1 was achieved by the ensemble, which produced 43,790 negative candidates.Source: §3.6 · p. 8
  • 4,905 new negative contexts were added to CC30k through the augmentation process.The total comprised 1,055 manually verified negatives and 3,850 high-confidence classifier candidates.Source: §3.6 · p. 8, Figure 4 · p. 8
  • The label_type field records whether labels were crowdsourced, human-validated after augmentation, or machine-labeled through augmentation.Source: §3.6 · p. 8

4 Data Records

CC30k is distributed as a structured dataset, and its crowdsourced portion shows skew in citation counts and sentiment proportions across cited papers.

4.1 Labeling Outcomes

  • The paper reports average word counts and sentence counts by sentiment category, with their distributions shown in Figure 6.Source: §4 · p. 8
  • 2.36% of cited papers receive more than 2,000 citation contexts, while most receive no more than 100.The distribution of citing-paper counts across cited papers follows a similar skew.Source: §4.1 · p. 8, Figure 5 · p. 10
  • Most cited papers receive less than 5% negative ROS citations, while positive citation-context fractions have a wider distribution.Source: §4.1 · p. 10, Figure 5 · p. 10

4 Data Records

  • The final dataset contains 30,734 contexts and is stored as a CSV with 37 columns of labels and associated metadata.The fields include metadata about citing and cited papers.Source: §4 · p. 8, Table 5 · p. 9, Table 6 · p. 10

5 Data Quality Assessment

The crowdsourced labels were audited using stratified sampling and show high agreement with manual validation. The audit reports 93.85% overall accuracy and 94% macro- and weighted-average F1, while ordinal Krippendorff’s α is lower in the extremely skewed label distribution.

5 Data Quality Assessment

  • The manual validation compared majority-voted labels with ground truth in a stratified sample of 244 contexts.The sample contained 10 majority-voted contexts per batch of 1,000, balanced across Positive, Negative, and Neutral labels.Source: §5 · pp. 10–11, Table 8 · p. 11
  • 93.85% overall accuracy was obtained for the crowdsourced labels, with all 15 majority-vote mislabels occurring in 2/3-agreement cases.No mislabels were found when all three workers agreed.Source: §5 · p. 11
  • 94% macro- and weighted-average F1 and 99.35% overall annotator agreement further indicate high crowdsourced label quality.Ordinal Krippendorff’s α was 0.29, interpreted as fair agreement; the paper attributes the lower value primarily to extreme label skew toward Neutral.Source: §5 · p. 12, Table 9 · p. 12

6 Potential Uses of the Dataset

Off-the-shelf sentiment classifiers perform poorly on reproducibility-oriented sentiment, while fine-tuning LLMs on CC30k generally improves classification performance. CC30k may also support citation-graph studies of reproducibility practices and concerns.

6.1 Baseline performance of off-the-shelf sentiment classifiers

  • All five evaluated off-the-shelf sentiment classifiers scored below 0.41 macro-averaged F1 on CC30k’s crowdsourced contexts.The paper presents this as evidence that generic sentiment models have difficulty with this domain-specific task.Source: §6.1 · p. 12

6.2 Fine-tuning large language models

  • Fine-tuning on CC30k generally improves weighted-average F1 over base direct inference by 5% to 27%, with the gains varying by model, prompting method, and training size.The best reported settings differ by model: Qwen1.5–7B reaches 0.695 with 9k samples and few-shot prompting; LLaMA 3–8B reaches 0.671 with 3k and few-shot prompting; GPT-4o with RAG reaches 0.786 with 3k and zero-shot prompting.Source: §6.2 · p. 12, Figure 7 · p. 13
  • GPT-4o with 3k RAG samples scores 0.786 in zero-shot versus 0.703 in few-shot inference, showing that zero-shot can match or exceed few-shot with RAG.The paper suggests retrieved supporting passages may remove the need for additional in-prompt examples.Source: §6.2 · p. 13
  • Performance gains are not monotonic with added training data for every model, and the paper identifies noisy supervision as a plausible contributor.Qwen’s few-shot score rises from 3k to 9k samples and then drops at 15k; LLaMA shows a similar pattern.Source: §6.2 · p. 13

6.3 Other Usages

  • Linking CC30k contexts to citing and cited papers could support citation-graph analyses of reproducibility practices and concerns across AI and potentially other disciplines.The paper also notes possible use in identifying factors related to reproducibility and in reproducibility-aware data or software recommendation.Source: §6.3 · p. 13

7 Conclusion

CC30k provides 30,734 carefully labeled citation contexts focused on reproducibility-oriented sentiment in AI literature. The paper concludes that this dataset addresses a resource gap and can support further ROS analysis.

7 Conclusion

  • CC30k contains 30,734 reproducibility-oriented sentiment-labeled citation contexts and is presented as a resource for studying reproducibility and replicability in AI research.The authors cite its crowdsourcing pipeline and validation as support for the dataset’s reliability and utility.Source: §7 · p. 14
Loading 2511.07790v1…