PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

Toufique Ahmed, Premkumar Devanbu, Christoph Treude, Michael Pradel

arXiv:2408.05534v1cs.SEcs.HCcs.LG

TL;DR

Human-subject studies are costly in software engineering, even though evaluating tool outputs can require judgments of their usefulness to people. This paper compares six LLMs with human annotations across ten tasks from five datasets and finds that LLMs can match human agreement in many studied settings. Model-to-model agreement and per-sample confidence are proposed as guides for selecting tasks and samples, but confidence does not support replacing all human raters across the datasets studied.

  • Problem

    Human-subject studies are costly, while software engineering evaluations may need human judgments of whether tool outputs are useful.

    Source: §I · p. 1, §II · p. 2

  • Method

    The study applies six LLMs to annotation tasks from five prior-work datasets and examines model-to-model agreement and output confidence as guides to task and sample selection.

    Source: §I · p. 2, A. Tasks & Datasets · p. 3, D. Evaluation Methodology · p. 4

  • Results

    For most studied settings, human–model agreement is fairly consistent with human–human agreement, while model agreement and output confidence indicate when model judgments align with human ratings.

    Source: §I · p. 2, §VII · p. 10

  • Takeaways & Limitations

    Model-to-model agreement can help assess task suitability, and model confidence can help identify samples for selective human–LLM annotation.

    Source: §I · p. 2

    Across datasets, no confidence split consistently identified samples for replacing all humans, except in semantic similarity; the study considers discrete multiple-choice LLM responses.

    Source: §V · p. 9, §VII · p. 10

Abstract

from arXiv · show

Experimental evaluations of software engineering innovations, e.g., tools and processes, often include human-subject studies as a component of a multi-pronged strategy to obtain greater generalizability of the findings. However, human-subject studies in our field are challenging, due to the cost and difficulty of finding and employing suitable subjects, ideally, professional programmers with varying degrees of experience. Meanwhile, large language models (LLMs) have recently started to demonstrate human-level performance in several areas. This paper explores the possibility of substituting costly human subjects with much cheaper LLM queries in evaluations of code and code-related artifacts. We study this idea by applying six state-of-the-art LLMs to ten annotation tasks from five datasets created by prior work, such as judging the accuracy of a natural language summary of a method or deciding whether a code change fixes a static analysis warning. Our results show that replacing some human annotation effort with LLMs can produce inter-rater agreements equal or close to human-rater agreement. To help decide when and how to use LLMs in human-subject studies, we propose model-model agreement as a predictor of whether a given task is suitable for LLMs at all, and model confidence as a means to select specific samples where LLMs can safely replace human annotators. Overall, our work is the first step toward mixed human-LLM evaluations in software engineering.

I. INTRODUCTION

Evaluating software engineering tools often requires human judgments of whether outputs are useful, but human-subject studies are costly and difficult to conduct. This paper tests whether LLMs can substitute for some annotation effort and proposes agreement and confidence as guides for choosing tasks and samples.

I. INTRODUCTION

  • Human-subject studies are costly, yet tool outputs often need human evaluation because their usefulness depends on human cognition.Studies may need representative developers and artifact samples, with multiple ratings per sample.Source: §I · p. 1
  • Six LLMs are applied to ten annotation tasks from five prior-work datasets to compare model responses with human judgments.The tasks span software engineering artifacts and research settings.Source: §I · p. 2
  • For identifier name–value consistency, mean human-to-human agreement is 0.52 and mean human-to-model agreement is 0.49.The authors interpret this as evidence that LLMs can meaningfully replace humans on some annotation tasks.Source: §I · p. 2
  • Model-to-model agreement has a Spearman correlation of 0.65 with LLMs’ ability to meaningfully replace humans, suggesting a relatively cheap task-suitability proxy.The proposed proxy is intended to help decide whether to involve LLMs in an evaluation.Source: §I · p. 2
  • For code-summary similarity judgments, confidence-selected LLM ratings replaced 50% of one annotator’s effort without a statistically significant change in overall inter-rater agreement.The authors propose model confidence to select samples where human annotation may not be required.Source: §I · p. 2

II. BACKGROUND

Reliable qualitative annotations generally require multiple independent raters, but human interpretation makes agreement effort-intensive and time-consuming. The paper frames LLMs as a possible complement and focuses on annotations with pre-existing categories, leaving open coding for future work.

II. BACKGROUND

  • Multiple independent annotators can reduce the risk that findings reflect one person’s bias, but achieving high inter-rater reliability takes substantial manual effort and time.Inter-rater reliability is used as a benchmark for annotation objectivity and is commonly measured with metrics such as Cohen’s κ or Krippendorff’s α.Source: §II · p. 2
  • LLMs are presented as a potential way to provide more consistent annotations and scale qualitative analysis, not as an established outcome of this study.The background describes these possibilities as potential benefits.Source: §II · p. 2
  • The paper considers structured annotation with pre-existing categories, while annotation without such categories, or open coding, is left for future work.The categories may come from related work or an initial research phase.Source: §II · pp. 2–3

III. METHODOLOGY

The methodology applies LLMs to annotation tasks drawn from five prior-work datasets and compares their responses with human annotations. The datasets cover code-summary assessment, name–value consistency, causal relations, function similarity, and static-analysis warnings.

III. METHODOLOGY

  • The study selects five prior-work datasets containing ten annotation tasks intended to represent diverse software engineering research.The methodology aims to use the original studies’ instructions where possible.Source: A. Tasks & Datasets · p. 3, §III · p. 3

A. Tasks & Datasets

  • Experienced programmers rated 420 human- and model-generated summaries, with each sample seen by at least three raters; LLMs received the original guidelines and questions.The summary ratings address accuracy, adequacy, conciseness, and similarity, using four response options and few-shot examples.Source: A. Tasks & Datasets · p. 3
  • The name–value dataset uses 11 raters’ 1–5 judgments on 40 samples, while the causality task samples 1,000 sentences from a dataset exceeding 10,000.Causality labels are binary; the LLMs are prompted to perform the original annotation task.Source: A. Tasks & Datasets · p. 3
  • For functional similarity, the study retrieves both function bodies for 786 of 857 manually classified pairs and evaluates goals, operations, and effects.The original pairs came from JavaDoc comments across 11 open-source repositories.Source: A. Tasks & Datasets · p. 3
  • The static-analysis dataset labels warnings open, closed, or unknown; the experiments use 200 of its 1,306 samples, selected uniformly at random.Some models’ rate limits and cost management motivate using the 200-sample subset.Source: A. Tasks & Datasets · p. 4

B. Models under Consideration

The study includes three closed models, GPT-3.5, and two open models. The selected models were described as leading or latest in their respective families at the time of the work.

B. Models under Consideration

  • The six models are GPT-4, Claude-3.5-Sonnet, Gemini-1.5-Pro, GPT-3.5, Llama3 (70B), and Mixtral (8x22B).The first three were selected as the best in their organizations’ families at the time; the two open models were the latest from their families.Source: B. Models under Consideration · p. 4

C. Research Questions

The research questions examine when LLMs can substitute for human annotation, how to identify suitable tasks and samples, and how much effort can be saved without reducing agreement.

C. Research Questions

  • RQ1 compares human-human, human-model, and model-model agreement to assess whether model annotations can replace some human ratings.The study also examines how model-model agreement varies across datasets.Source: C. Research Questions · p. 4
  • RQ2 asks whether model-model agreement can indicate when LLMs are unsuitable for a task, amid task-dependent agreement levels.The section motivates this question with varying agreement across datasets and tasks.Source: C. Research Questions · p. 4
  • RQ3 asks whether an LLM answer for a particular sample is likely to agree with a human, so successful samples can be selected for delegation.The motivation notes that selective replacement may reduce human effort even if a model does not perform well on every sample.Source: C. Research Questions · p. 4
  • RQ4 asks how much human annotation effort can be saved without sacrificing inter-rater agreement.It follows the questions about deciding whether and when to delegate work to LLMs.Source: C. Research Questions · p. 4

D. Evaluation Methodology

The methodology compares human, model, and mixed-rater agreement, then tests model agreement and confidence as guides for delegation and measures savings against agreement changes.

D. Evaluation Methodology

  • For RQ1, Krippendorff’s α is computed for each human-human, human-model, and model-model pair using only samples both annotators rated.The metric accommodates missing ratings, varying numbers of annotators and values, and different annotation types across datasets.Source: D. Evaluation Methodology · p. 4
  • For RQ2, the study examines whether model-model agreement correlates with human-model agreement as a cheap, automated indicator of replacement feasibility.The method compares these two agreement measures across tasks.Source: D. Evaluation Methodology · p. 4
  • For RQ3, the output probability of a model answer serves as a confidence proxy for selecting samples, and the study measures agreement after replacing one human’s ratings on a fraction of them.The selected ratings are the model’s highest-confidence responses.Source: D. Evaluation Methodology · pp. 4–5
  • For RQ4, human effort saved is measured as the work to label one sample, while testing whether agreement declines significantly when one rating per sample is delegated.Saving 100% of one-rating effort means obtaining one LLM annotation for every sample; this can replace work otherwise done by one or more humans.Source: D. Evaluation Methodology · p. 5

A. RQ1: Level of Agreement between Humans and Models

Agreement between humans and models varies by task: it is generally similar for code summarization and name-value consistency, but lower for causality and static-analysis warnings; semantic similarity shows high agreement across pairings.

Code Summarization

  • Human-human and human-model agreement are generally similar across code-summarization criteria, at 0.21–0.48 and 0.24–0.40, respectively; model-model agreement averages 0.68–0.76.These table values use Claude, Gemini, and GPT-4; agreement is higher for human-human than human-model on similarity, and smaller models show answer biases such as Llama-3 favoring extreme response options.Source: A. RQ1: Level of Agreement between… · p. 5, Table I · p. 5

Name-Value Inconsistencies

  • For name-value consistency, mean human-model agreement is 0.49 versus 0.52 for human-human agreement; the top three models have mean model-model agreement of 0.66.The reported α values thereafter focus on those three models because they did not show answer preferences.Source: A. RQ1: Level of Agreement between… · p. 5, Table I · p. 5

Causality

Semantic Similarity

  • For functional semantic similarity, human-human agreement is 0.71–0.83, human-model 0.64–0.77, and model-model 0.69–0.83, the highest levels observed across all three pairings.The passage reports these ranges across goals, operations, and effects categories.Source: A. RQ1: Level of Agreement between… · p. 5, Table I · p. 5

Static Analysis Warning

  • For static-analysis warnings, human-human agreement is 0.80, but human-model agreement is 0.15 and model-model agreement is 0.12, suggesting LLMs cannot safely substitute for human ratings on this task.The task has only two human raters.Source: A. RQ1: Level of Agreement between… · p. 5, Table I · p. 5

B. RQ2: LLMs’ Applicability for a Specific Task

Model–model agreement helps identify annotation tasks where LLMs may align with humans, while confidence-based GPT-4 selection can preserve agreement for portions of several datasets. The results vary by task, with low agreement limiting substitution in some settings.

B. RQ2: LLMs’ Applicability for a Specific Task

  • Across tasks, mean model–model agreement among the top three models positively correlated with human–model agreement (Spearman ρ = 0.65, p < 0.05), making it a potential suitability signal.The code-summarization conciseness outlier had low human–human agreement (0.24).Source: B. RQ2: LLMs’ Applicability for a… · pp. 5–6

C. RQ3: LLMs’ Applicability for a Specific Sample

  • Where humans and models reasonably agreed, model–model agreement was 0.66–0.82; it was 0.39 for causality and 0.12 for static-analysis warnings, motivating GPT-4 sample-level analysis.GPT-4 was selected for that analysis because it was the best-performing model overall and exposed output probabilities.Source: C. RQ3: LLMs’ Applicability for a… · p. 6
  • For code-summarization accuracy, replacing one rating in every sample left inter-rater agreement unchanged; for similarity, probability-selected GPT-4 replacements maintained agreement through 50% of samples, potentially saving 16.5% of human effort.Source: C. RQ3: LLMs’ Applicability for a… · p. 7
  • GPT-4 could replace one rating in every sample for name–value inconsistencies and functional similarity while maintaining the same inter-rater agreement.The name–value dataset had few samples, so its human–human agreement confidence interval was wider.Source: C. RQ3: LLMs’ Applicability for a… · p. 7
  • Confidence-based GPT-4 selection maintained similar agreement through 60% of causality samples and 50% of static-analysis-warning samples, despite declines as replacement increased overall.The confidence intervals through those thresholds overlapped with human–human agreement.Source: C. RQ3: LLMs’ Applicability for a… · p. 7

D. RQ4: Human Effort to Potentially Save

Replacing one human rating with an LLM could save effort on seven of ten tasks. The estimated savings depend on ratings per sample and the share of samples that can be replaced without a statistically significant agreement change.

D. RQ4: Human Effort to Potentially Save

  • The replaceable sample fraction is the maximum that keeps inter-rater agreement statistically indistinguishable from a human-only study; required ratings per sample also constrain total savings.Replacing one rating per sample offers less overall savings when each sample requires more ratings.Source: D. RQ4: Human Effort to Potentially Save · pp. 7–8
  • For seven of ten tasks, one human rating could be replaced; potential overall savings were 33% for code-summary accuracy, 9% for name–value inconsistencies, and up to 33% for semantic similarity.The table estimates savings for one rating and for the overall annotation process.Source: D. RQ4: Human Effort to Potentially Save · p. 8
  • For code-summary accuracy, replacing the one rating for every sample could save 100% of that rating effort, or 33% of total annotation effort.The estimate excludes the few-shot labeling effort of three to four samples.Source: D. RQ4: Human Effort to Potentially Save · p. 8

V. DISCUSSION & LIMITATIONS

The discussion recommends selective replacement of one human rating and using model confidence to choose samples, while cautioning against replacing all human raters and noting limits to the evidence.

V. DISCUSSION & LIMITATIONS

  • The authors suggest replacing one human rating for 50%-100% of samples, but not replacing all human raters for any sample with current models.Selective replacement still carries some risk and requires further study.Source: §V · pp. 8–10
  • Replacing more than one human can inflate inter-rater agreement because model-model agreement is much higher, potentially misrepresenting the annotations’ original goals.The authors also note that automatically labeling additional samples could extend a dataset.Source: §V · p. 9
  • For causality and static-analysis-warning tasks, GPT-4 maintains output probability around 0.80 for the first 50%-60% of samples as confidence declines across the sorted samples.The discussion proposes computing a probability cutoff when model-model agreement is low.Source: §V · p. 9, Figure 12 · p. 9
  • Higher GPT-4 output probability makes majority-vote agreement more likely, but no probability split works consistently across datasets apart from semantic similarity.Even among the 10% highest-probability samples, some errors remain.Source: §V · p. 9, Figure 13 · p. 9
  • The study covers only 10 tasks and six models, and findings may vary on other datasets or models.Public datasets retaining individual annotations are difficult to find, and only Krippendorff’s α applied across all studied datasets.Source: §V · p. 10
  • Zero-shot human-model agreement was unsatisfactory even where few-shot learning performed well, while repeated sampling mostly generated the same samples.The authors recommend collecting human reasoning to support few-shot prompting and possible future chain-of-thought or self-consistency approaches.Source: §V · p. 10

VI. RELATED WORK

Related work places this study among efforts to automate software engineering and human annotation, while distinguishing its task-level delegation guidance from other annotation and data-synthesis approaches.

ML and LLMs in Software Engineering

  • Learning-based approaches and LLMs address software tasks such as code completion, test generation, fuzzing, automated repair, coding agents, and type prediction –.This work instead studies LLMs for automating manual human-subject evaluations.Source: §VI · p. 10

LLMs to Complement or Replace Human Annotators

  • NLP studies of LLM annotation report mixed results, including improved aggregated labels, performance over crowd-workers, and failures to represent demographic differences.Bavaresco et al. [13] propose an NLP dataset benchmark that this work envisions extending to software engineering.Source: §VI · p. 10

Collaborative Human-LLM Annotation

  • Prior human-LLM labeling workflows query an LLM and then have humans refine labels,, whereas this work uses inter-model agreement and output probabilities to guide delegation.The paper positions its study as the first careful evaluation of LLM annotation for software engineering artifacts.Source: §VI · p. 10

LLMs for Data Synthesis and Assessment

  • LLM data synthesis targets large training datasets that can tolerate noise, whereas human-subject studies use smaller datasets whose annotations should have high confidence [64]–.LLM-as-judge work, mainly addresses human preferences, while many software engineering annotation tasks have objectively correct answers humans can eventually agree upon.Source: §VI · p. 10

VII. CONCLUSION

The conclusion reports broadly consistent human-model agreement and identifies model agreement and sample-level confidence as useful indicators. It also limits the study’s scope to discrete responses and says bias and demographic effects were not studied.

VII. CONCLUSION

  • Human-model agreement is fairly consistent with human-human agreement in the majority of studied settings.The conclusion also reports model-model agreement as a good indication of human-model agreement.Source: §VII · p. 10
  • An LLM’s output probability for a sample is a good indication of whether its response agrees with the majority human rating for that sample.Source: §VII · p. 10
  • The study considers discrete, multiple-choice LLM responses rather than free-form annotations, and does not study model-output bias or demographic issues.Source: §VII · p. 11
Loading 2408.05534v1…