PaperlayerView exact revision on arXiv

Source-linked AI summary

Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?

R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, Jenny Huang

arXiv:1912.08320v1cs.CYcs.CLcs.DLcs.LG

TL;DR

Supervised ML depends on high-quality training data, yet contemporary research often gives limited attention to obtaining and validating it. This paper examines human-annotation reporting in Twitter-based ML application publications and finds substantial variation in documentation, motivating greater attention to annotation as part of research validity.

  • Problem

    Supervised ML requires high-quality training data, but contemporary ML research and education often gives less attention to obtaining and validating that dataset.

  • Method

    The authors conducted a structured, multistage content analysis of published reports about human-labeling projects, examining how annotation practices and training-data information were documented.

  • Results

    The examined Twitter-based ML application publications showed wide variation in documentation; about half using original human annotation used multiple overlap, and about 70% of those reported inter-annotator agreement metrics.

  • Takeaways & Limitations

    Human annotation should receive attention and care comparable to performance metrics because training-data quality is central to confidence in models and classifiers.

  • Takeaways & Limitations

    The findings are limited in scope because the sample is likely not representative of academic publications and may differ across application domains.

Abstract

from arXiv · show

Many machine learning projects for new application areas involve teams of humans who label data for a particular purpose, from hiring crowdworkers to the paper's authors labeling the data themselves. Such a task is quite similar to (or a form of) structured content analysis, which is a longstanding methodology in the social sciences and humanities, with many established best practices. In this paper, we investigate to what extent a sample of machine learning application papers in social computing --- specifically papers from ArXiv and traditional publications performing an ML classification task on Twitter data --- give specific details about whether such best practices were followed. Our team conducted multiple rounds of structured content analysis of each paper, making determinations such as: Does the paper report who the labelers were, what their qualifications were, whether they independently labeled the same items, whether inter-rater reliability metrics were disclosed, what level of training and/or instructions were given to labelers, whether compensation for crowdworkers is disclosed, and if the training data is publicly available. We find a wide divergence in whether such practices were followed and documented. Much of machine learning research and education focuses on what is done once a "gold standard" of training data is available, but we discuss issues around the equally-important aspect of whether such data is reliable in the first place.

1 INTRODUCTION

The paper examines how human-labeled training data is created and documented in supervised machine learning, arguing that this process is central to classifier quality and validity. It studies reporting practices in social-computing papers while noting that many papers provide insufficient information about their annotation processes.

  • 1 INTRODUCTION: Supervised machine learning depends on high-quality training data, but creating the “gold standard” is often treated as a black box.The paper contrasts this neglected process with the field’s emphasis on producing classifiers from already-clean datasets.
  • 1 INTRODUCTION: Human-labeled training-data creation involves structured human judgment resembling established content-analysis practices.The paper frames annotation as a methodological process rather than merely a preliminary step before model development.
  • 1 INTRODUCTION: The study systematically examined social-computing papers that classified Twitter data and recorded what they reported about their training data.The corpus included mostly ArXiV papers and a smaller set of Scopus publications.
  • 1 INTRODUCTION: Many papers did not identify their labelers or provide enough detail to determine whether important annotation practices were followed.The researchers frequently assigned “no information,” limiting assessment of training-data validity and, by extension, classifier validity.

2 LITERATURE REVIEW AND MOTIVATION

The paper connects machine-learning training-data creation to content analysis, open science, and documentation efforts, emphasizing reliability, operationalization, and transparency in human annotation. It distinguishes the paper’s concern with taken-for-granted training-data production from interpretability-focused uses of “black-boxing.”

  • 2 LITERATURE REVIEW AND MOTIVATION: Here, “black-boxing” means taking classifier-development inputs for granted, not explaining why the resulting model makes individual decisions.The paper distinguishes this usage from interpretability-oriented discussions of machine-learning systems.
  • 2 LITERATURE REVIEW AND MOTIVATION: Opening the training-data black box concerns who can question data and who must accept it, linking annotation to research validity and reproducibility.The paper treats black-boxing as a relational orientation toward routinely trusted sociotechnical systems.
  • 2 LITERATURE REVIEW AND MOTIVATION: Human-labeled training datasets often resemble structured content analysis, in which teams systematically convert unstructured material into structured or quantitative data.The paper uses coder, annotator, labeler, and reviewer as interchangeable terms for the human labor involved.
  • 2 LITERATURE REVIEW AND MOTIVATION: Established content-analysis practices include defined coding schemes, independent labeling of shared items, inter-annotator agreement, and reconciliation of disagreements.Reconciliation may involve majority voting or discussion-based procedures.
  • 2 LITERATURE REVIEW AND MOTIVATION: The paper situates its concerns alongside corpus-annotation standards, crowdwork quality controls, open science, and training-data documentation frameworks.Related documentation proposals include datasheets, model cards, data statements, nutrition labels, and similar tools addressing datasets, models, performance, bias, or impacts.

3 DATA AND METHODS

The study sampled Twitter-classification papers and used a multistage, collaborative content-analysis process to assess how they documented human-labeled training data. The researchers independently coded papers, reconciled disagreements, verified labels, and quantified agreement and reporting information.

  • 3 DATA AND METHODS: The corpus targeted papers creating original human-annotated training datasets for supervised classifiers trained on Twitter data.The sample covered application areas including political science, public health, NLP, sentiment analysis, cybersecurity, content moderation, hate speech, and demographic profiling.
  • 3 DATA AND METHODS: The corpus contained 494 ArXiV papers and 29 usable Scopus articles identified through Twitter or tweet searches.One Scopus paper was corrupted and excluded from analysis.
  • 3 DATA AND METHODS: The researchers used a six-person labeling team with computer-science or data-science coursework and varied machine-learning experience.Students worked for course credit in a university-sponsored research experience program under a research scientist’s leadership.
  • 3 DATA AND METHODS: Five students independently labeled the same sampled papers each week, after which the team leader identified disagreements and the team discussed them.A second round independently re-examined all labels for 164 papers, with at least two team members reviewing each paper.
  • 3 DATA AND METHODS: Mean percent total agreement measured whether all labelers initially assigned the same label, and round-two reconciliation followed the independent reviews.Round-one agreement averaged 66.67%, while round-two agreement averaged 84.80%; the lowest scores were 38.2% and 63.4%, respectively.
  • 3 DATA AND METHODS: The study scored reporting about annotation, crowdworkers, and inter-annotator metrics using raw and normalized information measures.Applicability depended on whether a project used crowdworkers or multiple annotators per item.

4 FINDINGS

The study operationalized supervised-ML and human-annotation categories, then examined how papers reported annotation sources, procedures, and documentation. Reporting varied substantially across these practices, including annotator recruitment, instructions, training, overlap, compensation, and dataset access.

  • Scope and definitions: The study defined an original classifier as one authors created using new or old data, excluding exclusive use of pretrained models.
  • Scope and definitions: Human-annotation analyses distinguished original labels obtained from people, external annotation data, and datasets combining both sources.The schema was modified to capture these three cases separately.
  • Annotator sources: 74 papers used original human annotation, with authors comprising the plurality of annotators and no information reported for almost a quarter.Experts or professionals appeared more often than expected, while crowdworkers comprised about 15% of papers.
  • Annotator sources: A slim majority of papers specified the number of annotators, typically describing either small closed teams or medium-to-large open groups.The observed groups commonly consisted of 2–6 people or 25–500 people, respectively.
  • Instructions and training: 43.2% of papers provided annotator definitions or examples, while the plurality supplied insufficient information to determine whether formal instructions existed.The coding distinguished definitions/examples from merely presenting a question and treated undocumented instructions as no information.
  • Instructions and training: 15% of papers involving human annotation reported a training session, although the authors note that some tasks may not require specialized training.Training required an interactive process involving feedback or dialogue, rather than one-way materials such as videos or schemas.
  • Compensation and access: Zero papers using crowdworkers mentioned compensation, while dataset-link reporting was separately assessed for papers producing original human-annotated data.The study did not follow dataset links or verify that linked data was actually available.

5 PAPER INFORMATION SCORES

Information scores vary substantially across papers, suggesting wide differences in human-annotation practices and documentation. Scores also differ across publication types and publishers, although the authors caution against broad generalization.

  • Information scores show substantial variation in human-annotation, training-data curation, and documentation practices.The authors present these results as an initial framework and limited evaluation of the literature.
  • The normalized information score distribution appears bimodal, with clusters near 0.25 and 0.7.Scores ranged from 0 to 1; six papers scored 0.
  • Scopus papers had the lowest median scores among the publication-status categories, while publication preprints and postprints had higher medians.Unpublished preprints had a wider IQR and standard deviation than the other categories.
  • Among combined corpora, journal articles had a median normalized score of 0.5 and a narrower IQR than conference proceedings and unpublished ArXiv preprints.The latter two categories had somewhat similar medians and IQRs.
  • In the ArXiv corpus, ACM and ACL papers tied for the highest publisher median at 0.667, while IEEE had a median of 0.226.Scopus publisher medians were lower than ArXiv publisher medians, but the authors caution that the Scopus sample was small.

6 CONCLUDING DISCUSSION

The study finds uneven documentation of human-annotation practices in Twitter-based ML application papers, while arguing that annotation deserves greater attention as part of research validity and integrity. Its conclusions are preliminary and scope-limited, and intensive annotation involves a tradeoff between richer understanding and fewer items processed.

  • Documentation of human-annotation practices varied widely across the examined Twitter-based ML application papers.
  • About half of papers using original human annotation used multiple overlap, and about 70% of those reported inter-annotator agreement metrics.
  • The roughly bimodal annotation-information scores suggest distinct groups providing substantially more versus less training-data information.
  • The authors frame training-data documentation as a research-validity and integrity issue because readers need information to assess confidence in classifiers.
  • Their intensive, collaborative, multistage annotation produced richer data and contextual understanding but reduced the number of items they could annotate.
  • The findings are preliminary and limited because the study examined only a sample of ML application publications, especially Twitter-classification papers.

APPENDIX

The appendix follows the references section.

  • The appendix appears following the references section.

7 APPENDIX

The appendix describes the corpus’s topical and disciplinary characterization, publication-version checks, and venue composition. It also documents that the corpus combines ArXiv materials with a smaller Scopus sample and includes many publication venues.

  • The researchers assigned topical, domain, disciplinary, and methodological keywords to characterize papers in the corpus.
  • Figure 4 separates keyword distributions for papers using original human annotation from papers reusing external datasets or lacking such annotation.
  • The corpus contained 164 papers, whose ArXiv versions were checked against published versions to distinguish preprints, postprints, and substantive differences.
  • The corpus represented 80 journals, conference proceedings, or workshops, with SocInfo contributing six papers and ASONAM four.

7.2 Methods and analysis details

The study used multiple rounds of independent annotation, discussion, schema revision, and agreement assessment to analyze machine-learning papers. Agreement improved substantially, but interpretation was complicated by task complexity, changing annotator coverage, and ambiguous categories.

  • Inter-annotator agreement: Five annotators independently examined each paper in round one, then discussed disagreements; agreement was calculated as the percentage of items receiving the same label from all annotators.Items irrelevant to a question were excluded from the calculation.
  • Inter-annotator agreement: The first round produced insufficient proportional agreement for confidence, partly because annotators answered 13 questions per paper and concepts were difficult to define consistently.The external human annotation question had the lowest second-round agreement and remained ambiguous.
  • Inter-annotator agreement: Agreement increased substantially between rounds, although the comparison was confounded because all five annotators reviewed every item initially, while only two or three did so later.The reduced annotator coverage makes the size of the improvement difficult to interpret directly.
  • Inter-annotator agreement: Because first-round reliability showed that annotators were not fully standardized instruments, the team used multiple annotators and group discussion rather than single-person or mechanical majority reconciliation.This approach was intended to support confidence in the final dataset.
  • Inter-annotator agreement: Disagreements often arose because annotators missed information in long papers, then changed answers after colleagues identified the relevant passage.This differs from disagreements caused solely by divergent interpretations of the same compact item.
  • Schema development: The coding schema was revised during round one, with difficult edge cases, broader compatible labels, and three added questions incorporated into the process.Changes included questions about external human annotation, dataset links, and related reporting details.

7.3 Software used

The researchers conducted computational analysis in Python 3.7 using standard data-processing, scientific-computing, and visualization libraries, with notebooks and datasets planned for public release.

  • Software used: Computational analysis used Python 3.7 with Pandas, SciPy, NumPy, Matplotlib, and Seaborn in Jupyter Notebooks.The notebooks used IPython kernels and were designed to run on Binder.
  • Software used: Datasets and Jupyter Notebooks for data collection and analysis were scheduled to become available upon publication.The materials were intended to support reproducible execution through Binder.

7.4 Coding schema, examples, and instructions

The coding schema operationalized which papers and datasets counted as machine-learning classification projects using human annotation, while providing examples and explicit rules for ambiguous cases.

  • Task inclusion: The schema distinguished original classifiers from papers that only analyzed data or reviewed classification studies.Classification required predicting cases among defined categories, not merely describing word-use statistics or predicting unrestricted numeric values.
  • Human annotation: Human annotation counted when people created labels for the classification purpose, including reused judgments, but not metadata reuse or evaluation-only annotation.Projects with no qualifying annotation skipped subsequent human-annotation questions.
  • Examples and instructions: The instructions included examples distinguishing human judgment from manual lookup, semi-automated labeling, and annotation that was not used to train the classifier.These examples helped resolve borderline cases consistently.
  • Human annotation: The schema treated mixed datasets as using original annotation when any new systematic human labeling was present, while excluding gap-filling and marking vague methods as unsure.Existing external human-labeled datasets were tracked separately and could cause later questions to be skipped.
  • Annotator sources: Annotator-source categories included Mechanical Turk, other crowdwork platforms, paper authors, academic experts, other sources, no information, and unsure.Professional expertise was coded separately from task-specific training.

6. Number of human annotators:

The schema separately recorded whether papers reported annotator training, formal instructions, and concrete labeling examples, while distinguishing interactive training from merely providing guidance.

  • Missing information: When papers did not provide enough information to determine the answer, annotators recorded no information rather than inferring a value.This option was explicitly available for the relevant reporting questions.
  • Training: Interactive training required task-specific feedback; prescreening, prior expertise, or accuracy-based assignment alone did not qualify.The coding options included whether some training was mentioned or whether the paper provided no information.
  • Training: A paper could satisfy the minimum training-information standard by describing careful instruction, definitions, and agreement-based resolution of remaining differences.The example involved four volunteers and more than 90% agreement.
  • Instructions: Formal instructions were coded separately from training and could consist of definitions or examples beyond the question text.The coding schema also included an option for no instructions beyond the question text.
  • Examples and instructions: The example guidance illustrated labeling relevant crisis information such as injuries, infrastructure damage, urgent needs, and donation requests versus non-relevant content.These examples showed how operational criteria could be expressed for annotators.

9. Prescreening for crowdwork platforms

The coding scheme records crowdworker prescreening options and several additional annotation-quality and dataset-availability practices. It distinguishes qualification approaches, annotator overlap, agreement reporting, compensation disclosure, and dataset links.

  • Prescreening may involve no screening, platform performance or skills qualifications, location requirements, project-specific screening, no information, or uncertainty.
  • Annotator overlap is coded according to whether annotators labeled all items, some items, or an unclear subset of shared items.
  • Inter-annotator agreement is recorded through reported metrics such as Krippendorff’s alpha, Cohen’s kappa, or F1 score, but only when overlap occurred.
  • The scheme separately checks whether crowdworker compensation is disclosed, leaving the field blank when crowdworkers were not used.
  • It also records whether the paper provides a link to the dataset used.
Loading 1912.08320v1…