PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger

arXiv:2606.02255v1cs.CLcs.AI

TL;DR

NLP research depends on human judgments, yet reporting of who annotates and how annotation quality is controlled remains inconsistent. The paper audits reporting at the task level with a taxonomy and LLM-assisted extraction pipeline, finding progress alongside persistent omissions, especially in model-evaluation studies.

  • Problem

    NLP research relies on human judgments, but papers often insufficiently document annotators, annotation procedures, training, and disagreement handling.

  • Method

    The paper audits annotation reporting at the task level using a unified taxonomy, a human-adjudicated gold standard, and an LLM-assisted extraction pipeline.

  • Results

    Annotation reporting has improved, but compensation, socio-demographics, training, adjudication, and agreement values remain substantially underreported, especially in model-evaluation studies.

  • Takeaways & Limitations

    A bare minimum of annotation-task details is needed for readers to assess whether human annotation is reliable, interpretable, and reproducible.

  • Takeaways & Limitations

    The taxonomy requires fine-grained interpretation of incomplete or inconsistent descriptions, making perfect agreement difficult.

Abstract

from arXiv · show

Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated gold standard of 41 papers and 72 annotation tasks, where the best model reaches human-comparable agreement with adjudicated labels, with Krippendorff's alpha of 0.606 versus 0.585 for human-human agreement. Using this pipeline, we construct Annotated-llm, a dataset covering ACL-venue papers from 2018-2025, with 2,667 extracted annotation tasks from 1,603 papers, and find that papers frequently report operational details such as recruitment strategies, annotator expertise, and annotation volume, but often omit details needed to assess annotation validity, including training, language proficiency, compensation, socio-demographics, adjudication, and agreement values, especially in model-evaluation studies. Our results show that annotation reporting in NLP has improved over time but remains uneven, and they establish a scalable framework and bare-minimum reporting recommendations for making human annotation more reliable, reproducible, and interpretable.

1 Introduction

The paper frames human annotation as foundational to empirical NLP while identifying limited evidence about annotators and reporting practices. It introduces a task-level audit, validates scalable extraction against adjudicated labels, and proposes reporting recommendations.

  • Motivation: NLP claims depend on who annotators are, what they do, how they are trained, and how disagreements are handled, yet empirical evidence about these practices remains scarce.The introduction highlights unresolved questions about crowdworkers versus experts, task adequacy, compensation, and the measurable effects of reporting initiatives.
  • Contributions: ANNOTATED treats each annotation task as a separate unit, preserving within-paper differences in annotators, instructions, purposes, and quality-control procedures.A task-level perspective avoids obscuring variation that paper-level analysis would miss.
  • Contributions: Gemini-3.1-Pro reaches human-comparable agreement with adjudicated labels at Krippendorff’s α = 0.606, enabling scalable meta-analysis of annotation practices.The validation uses ANNOTATEDGOLD, a human-adjudicated gold standard covering 41 papers and 72 human-annotation tasks.
  • Contributions: The large-scale audit extracts 2,667 annotation tasks from 1,603 ACL-venue papers published between 2018 and 2025.The analysis examines reporting across time, venues, NLP topics, and uses of human judgment.
  • Contributions: The paper proposes a scalable framework and bare-minimum reporting recommendations to make human annotation more reliable, reproducible, and interpretable.These recommendations are motivated by the large-scale audit of annotation-reporting practices.

2 Related Work

Prior NLP research emphasizes that human annotation, annotator characteristics, agreement measurement, evaluation design, and reproducibility shape annotation quality. Related AI4Science work examines AI support for scientific workflows, including LLM-based peer review, writing, graphics generation, and empirical annotation or evaluation.

  • Annotator effects, annotation quality, and reporting: Human annotation is central to NLP evaluation and dataset construction, motivating attention to agreement measurement, evaluation design, and reproducibility.Prior work also emphasizes that annotators are not interchangeable, highlighting the relevance of their demographic backgrounds, language competence, and cultural context.
  • AI4Science and meta-scientific analysis: AI4Science research studies how AI systems can support or transform scientific work, including literature analysis, discovery, experimentation, writing, figure generation, and evaluation.
  • AI4Science and meta-scientific analysis: Recent work explores LLM-based support for peer review, abstract generation, and scientific graphics, while NLP increasingly uses LLMs as annotators, evaluators, or assistants for scientific and empirical analysis.

3 Human Annotation Setup

The study annotates individual human annotation tasks rather than papers, using a unified taxonomy and a calibrated, consensus-based process to assess reporting practices. Tasks share instructions and data, while the taxonomy covers 25 categories across seven reporting aspects.

  • 3 Human Annotation Setup: Individual human annotation tasks—not papers—are the unit of analysis, defined as shared setups where annotators follow the same instructions and annotate the same data.Tasks may support dataset creation, human-performance estimation, or model-output evaluation.
  • 3 Human Annotation Setup: Annotators used a structured interface with integrated instructions, pilot annotations, group discussions, and calibration, favoring closed-list selection for consistent coding and aggregation.Some categories were coded only from explicit statements in papers, while others required informed judgment.
  • 3 Human Annotation Setup: The taxonomy organizes 25 reporting categories into seven aspects: task description, agreement, workload, recruitment and qualifications, compensation, socio-demographics, and quality control.It provides a unified framework for mapping heterogeneous human annotation tasks in ACL papers.
  • 3 Human Annotation Setup: Each paper was annotated by at least two independent annotators; disagreements were discussed, unresolved cases were adjudicated by a third annotator, and consensus labels formed the ANNOTATEDGOLD standard.This consensus subset was used to evaluate LLM extraction quality.

4 ANNOTATED Dataset

The ANNOTATED dataset combines a manually adjudicated gold-standard benchmark with a large-scale LLM-extracted corpus of human-annotation tasks from ACL venues published between 2018 and 2025. Keyword-based retrieval and filtering produced 1,603 papers and 2,667 annotation tasks, while the sampling strategy was evaluated against a stratified random sample and showed generally modest distributional differences.

  • Dataset construction: The corpus covers ACL, EMNLP, NAACL, TACL, EACL, and AACL publications from 2018–2025, retrieved by iteratively refined matching of 34 annotation-related title and abstract keywords.The time window spans three years before and after the ACL checklist’s introduction.
  • Dataset construction: ANNOTATEDGOLD retains 41 reliably annotatable papers from 61 initially sampled to form the manually annotated and adjudicated benchmark.Papers were excluded when annotation tasks remained unresolved after a third annotation or when at least two annotators identified more than three difficult-to-annotate tasks.
  • Dataset construction: 1,603 papers and 2,667 extracted annotation tasks comprise ANNOTATEDLLM after filtering 1,995 keyword-retrieved candidates for annotatable human-annotation content.The 61 papers sampled for ANNOTATEDGOLD were strictly excluded from ANNOTATEDLLM.
  • Sampling validation: Compared with a stratified random sample from the same venue-year scope, keyword-based sampling produced statistically significant differences for some category-value frequencies, but generally modest average differences across 67 observations.This comparison assesses potential sampling bias relative to all ACL-venue papers.

5 Experiment Setup

The experiment evaluates six proprietary and open-weight LLMs under a unified, schema-constrained extraction protocol, comparing task-aligned outputs with normalized gold labels. It quantifies annotation reporting using a task-sensitive REPORTAGE SCORE over universal and conditionally applicable attributes.

  • Extraction: Six LLMs—three proprietary and three open-weight—are evaluated under a unified extraction protocol.The proprietary models are Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, and GPT-4.1; the open-weight models are Qwen3.6-27B, gemma-4-31B-it, and gpt-oss-120b.
  • Extraction: The extraction framework encodes taxonomy rules and constrains outputs to a JSON-defined schema, while prompting continued scanning to reduce under-extraction in multi-experiment papers.The protocol includes exact allowed values, field-level decision rules, interdependency constraints, and a self-audit checklist.
  • Evaluation: LLM outputs are evaluated against gold labels by aligning corresponding annotation tasks and normalizing values to a canonical form before comparison.Task alignment uses short task descriptions recorded during annotation, including matches such as tweet sentiment labeling and tweet sentiment annotation.
  • REPORTAGE SCORE: The REPORTAGE SCORE measures the proportion of applicable annotation attributes explicitly reported for each task, using ten universal attributes plus conditional attributes when relevant.The universal attributes include guidelines, recruitment, training, expertise, language proficiency, education, annotator and item counts, compensation, and post-annotation filtering; agreement and adjudication can be excluded for single-annotator tasks.

6 Results

Model-assisted annotation slightly outperformed human–human agreement against the adjudicated gold standard, while large-scale results show that NLP papers report operational annotation details more consistently than validity-critical methodological information. Reporting has improved over time, but remains uneven across topics and intended uses, with model-evaluation studies documenting less detail than resource-creation studies.

  • 6.1 Agreement: Human–human exact-match agreement was 79.2%, with macro-averaged Krippendorff’s α of 0.585; reliability exceeded α > 0.8 for clearly defined, explicitly reported fields.Agreement was lower for inferred or ambiguous fields such as guideline availability, annotator expertise, recruitment, and quality-control procedures.
  • 6.1 Agreement: 79.9% vs. 79.2% macro-level agreement and 0.606 vs. 0.585 Krippendorff’s α show Gemini-3.1-Pro slightly outperforming human–human annotation against the gold standard.Gemini-3.1-Pro was selected for large-scale extraction after evaluation of six models; all annotations were adjudicated through discussion to create the consensus dataset.
  • 6.2 Reporting trends: 90.4% of tasks reported recruitment, 86.5% reported annotator expertise, and 86.0% reported annotation volume, whereas validity-relevant attributes were reported much less consistently.Socially oriented NLP papers were slightly more attentive to annotator sourcing and language background but still omitted broader methodological details needed to assess validity.
  • 6.2 Reporting trends: Reporting quality improved over time, but the pre-2022 upward trend suggests the ACL Responsible NLP Checklist may have formalized practices already emerging before its introduction.The analysis estimates both an immediate post-2022 level shift and a change in the post-2022 trend.
  • 6.2 Reporting trends: Resource-creation studies consistently reported substantially more methodological detail than model-evaluation studies, whose omissions included recruitment, compensation, training, and quality-control procedures.The gap persisted throughout the observation period despite overall improvements in reportage quality.

7 Conclusion and Recommendations

The study finds progress but persistent gaps in how NLP papers document human annotation, especially details needed to assess reliability, interpretability, and reproducibility. It recommends minimum, task-sensitive reporting standards and greater transparency in model-evaluation studies, supported by a scalable taxonomy and LLM-assisted pipeline.

  • Findings: NLP papers report recruitment source, annotator expertise, and annotation scale more consistently than training, qualification, compensation, and demographic information.The field has made clear progress, but its documentation remains uneven and incomplete.
  • Recommendations: Papers should report annotator source, annotator and item counts, annotators per item, training, language proficiency, expertise, compensation, quality control, and annotation-guideline access.These details enable readers to assess whether annotation is reliable, interpretable, and reproducible.
  • Recommendations: Reporting should be task-sensitive: demographics matter for socially grounded or subjective tasks, whereas agreement, adjudication, and quality control matter for benchmark-label and evaluation tasks.Different annotation uses require different evidence about validity and quality.
  • Findings and contribution: Model-evaluation studies document annotation less often than resource-creation studies, despite relying on annotation for central empirical claims.The proposed taxonomy and LLM-assisted pipeline provide a scalable basis for monitoring reporting practices that support reliability and reproducibility.

Limitations

The annotation taxonomy is limited by categories that require fine-grained interpretation of incomplete or inconsistently written paper descriptions, making perfect agreement difficult. These categories were retained because they capture information important for assessing annotation validity.

  • Fine-grained interpretation of incomplete or inconsistently written descriptions makes perfect agreement difficult for some taxonomy categories.
  • The taxonomy nevertheless retains difficult categories because they capture recruitment, expertise, guideline availability, compensation, and quality control.
  • The limitation concerns both the design and reliability of the annotation taxonomy.

Ethics Statement

The study uses publicly available papers and author-conducted manual annotation rather than new experiments with external human subjects. It nevertheless addresses ethical concerns about annotation labor, transparency, and responsible use of automated extraction.

  • The analysis uses publicly available scientific papers, with manual annotation conducted by the authors and collaborators on published research articles.
  • The findings raise ethical concerns because compensation, annotator demographics, training, and quality-control details are often insufficiently reported.
  • Because ambiguous procedures may cause extraction errors, the pipeline is validated against a human-adjudicated gold standard and used for aggregate trends rather than ranking or criticizing individual papers.Any released dataset and code should support community-level reporting practices and better documentation standards, not assign blame to individual authors.

Broader Impact

The work aims to improve transparency and reproducibility in NLP human annotation by providing tools to monitor reporting of annotation labor, quality control, and annotator characteristics. Clearer reporting can help assess whether annotation-based claims are reliable.

  • Broader Impact: The study’s methodological impact is to support clearer, more reproducible reporting of annotation labor, quality control, and annotator characteristics in NLP.It provides a task-level taxonomy, a reporting-completeness metric, and an LLM-assisted extraction pipeline for monitoring these practices.
  • Broader Impact: Clearer annotation reporting can help reviewers, readers, and future researchers assess whether annotation-based claims are reliable.

A Datasets

The dataset identified papers involving human annotation by matching 34 keywords in their titles and abstracts, including “human in the loop.”

  • A Datasets: 34 keywords were used to identify papers involving human annotation through title and abstract matches.
  • A Datasets: Paper inclusion was therefore based on keyword matches in titles and abstracts.
  • A Datasets: The keyword list included “human in the loop.”

A.1 Random-sample validation of keyword retrieval … E LLM Extraction Prompt

The study validates keyword retrieval as an efficient but composition-altering strategy, then operationalizes annotation reporting through a task-level score, taxonomy, annotator-demographic fields, and a rigid LLM extraction prompt. The resulting framework distinguishes applicable reporting attributes and enforces exact, non-inferred extraction of annotation details.

  • A.1 Random-sample validation of keyword retrieval: Keyword retrieval retained 82% of 1,995 candidate papers with annotation content, versus 36% of 3,000 papers in the stratified random sample.This confirms substantially greater retrieval efficiency for identifying papers containing human annotation content.
  • B REPORTAGE SCORE Calculation: REPORTAGE SCORE computes each annotation task’s proportion of reported applicable attributes across 21 schema categories, dynamically excluding inapplicable or redundant fields.The schema includes universal, socio-demographic, and conditional attributes, while incomplete fields such as an IAA metric without its value do not count as valid reportage.
  • C Annotator demographics: The gold annotation process involved 12 annotators: 2 professors, 2 postdoctoral researchers, 6 PhD students, and 2 master’s students, with reported English fluency and varied national origins.The annotator pool included 5 women and 7 men and individuals originating from Italy, Germany, India, China, and Russia.
  • D Details on Human Annotation: The annotation taxonomy records topic, task type, intended use, guideline reproducibility, agreement metrics and values, workload, recruitment, training, proficiency, expertise, compensation, and demographics.Intended-use labels distinguish resource creation, human performance, and model output evaluation, while agreement fields preserve metric-value alignment without inference.
  • E LLM Extraction Prompt: The LLM prompt requires exact allowed taxonomy values, task-specific experiment rows, and separate experiments when annotator groups, rounds, instructions, or objectives differ.It also instructs the model not to infer agreement values, to search prose, tables, and appendices, and to use explicit defaults such as “Not reported” when evidence is absent.

F Additional Analysis Details

Additional analyses show that socially oriented NLP studies differ mainly in annotator sourcing, language-background reporting, quality control, and disagreement handling—not in overall reporting quality or expert-annotator reliance. Across venues and annotation purposes, reporting trends converge after 2022, while annotation purpose predicts reporting behavior more strongly than publication year.

  • RQ1: Additional analysis: Socially oriented papers report slightly more recruitment information, use more crowdsourced or mixed pools, and report native-speaker status more often, but their overall reporting quality is not substantially higher.Native-language reporting is significantly associated with the social subset (χ2 = 20.50, p < 0.001), although the association is weak (Cramer’s V = 0.088).
  • RQ1: Additional analysis: Crowd-annotated social-subset papers use quality control measures more often than the rest of the dataset, at 72% versus 62.5%.This difference indicates more frequent screening or filtering of crowdworkers in socially related studies.
  • RQ1: Additional analysis: Socially oriented papers report training slightly more often (20% versus 18%) but do not rely more on expert annotators and more often omit expertise information.Their ordinal and scalar tasks more commonly use majority voting (50.0%) or label distributions (25.0%) to handle disagreement, rather than expert adjudication or consensus discussion.
  • RQ2: Additional analysis: After 2022, reporting trajectories flatten across ACL, EMNLP, and NAACL, reducing venue differences without a strong immediate quality increase after the Responsible NLP Checklist.The intervention year is approximate because ARR adoption and venue-specific rollout schedules were not synchronized.
  • RQ3: Additional analysis: Resource-creation studies show the strongest reporting of procedural details, human-performance studies are intermediate, and model-evaluation studies are consistently weakest across measures.The pattern extends beyond formal quality control to annotator transparency and documentation, with recruitment source and compensation especially weakly reported in model-evaluation studies.
Loading 2606.02255v1…