PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

Thomas Reiter, Christoph Kern, Fedor Miasnikov, Sofiia Nikolenko, Rob Chew, Stephanie Eckman, Frauke Kreuter

arXiv:2609.35824v1cs.CLstat.ME

TL;DR

LLM labels may reproduce under one setup yet change under other defensible task designs, a source of uncertainty not captured by repeating a single setup. We test multiple models and designs, finding that agreement falls across designs and that model and design choices add substantial prevalence variance.

  • Problem

    A single model–design comparison with a human benchmark measures performance under that setup but does not test whether labels persist across reasonable task-design changes.

    Source: §1 · p. 1

  • Method

    We test seven LLMs across 12 task designs and three runs on 3,000 tweets, comparing reliability, design sensitivity, prevalence variation, human instrument effects, and confidence.

    Source: §5 · p. 7, §6 · p. 9

  • Results

    Task design and model choice increase prevalence variance by 76.7 times for offensive language and 110.6 times for hate speech relative to nominal sampling variance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Researchers must compare defensible task designs and report their resulting spread; repeating one setup or using confidence scores cannot estimate instrument uncertainty.

    Source: §6 · p. 9

    The evaluation focuses on offensive-language and hate-speech detection, so its estimates may not transfer to objective NLP tasks with tight ground-truth constraints.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' $\kappa = 0.91$). Agreement fell when we changed the task design for the same tweets (median Cohen's $\kappa = 0.76$). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.

1 Introduction

A single model–design comparison with a human benchmark cannot establish whether labels persist under other defensible task designs. This study tests reliability and sensitivity across models and designs, finding that agreement drops across designs and that instrument choices add substantial prevalence variance.

  • Standard evaluations compare one model–design combination with a human benchmark, but do not test whether labels survive reasonable changes in task design.Source: §1 · p. 1
  • The study tests seven LLMs across three runs and 12 task designs on 3,000 tweets labeled for offensive language and hate speech.Source: §1 · p. 1
  • Median agreement falls from Fleiss’ κ = 0.91 within designs to Cohen’s κ = 0.76 across designs.Source: §1 · p. 2
  • Task design and model choice increase prevalence variance by 76.7 times for offensive language and 110.6 times for hate speech relative to nominal sampling variance.Source: §1 · p. 2
  • LLM task-design variation reaches 560–572 basis points, compared with 270–331 basis points across five human instrument versions.Source: §1 · p. 2
  • Confidence tracks repeated model outputs more closely than agreement with humans and changes when identical tweets are presented in batches.Source: §1 · p. 2

2 Related Work

Prior work studies prompt sensitivity and flexibility in AI-generated measures, but often seeks a preferred prompt or holds the model and prompt fixed. This paper extends that work by testing multiple task-design choices and connecting their variation to prevalence uncertainty, human instrument effects, and confidence.

  • The paper extends a human measurement perspective by estimating variation introduced by model and task-design choices, treating design decisions as part of the measurement instrument.Source: §2 · p. 2
  • Prior prompt-optimization and human-validation approaches generally hold the model and prompt fixed, so they do not measure variation across reasonable task designs.Source: §2 · p. 2
  • Prior studies document analytic flexibility across AI-generated measures, with correlations between human and synthetic association structures ranging from r = .23 to r = .84 across 66 configurations.Source: §2 · p. 2
  • The experiment crosses task structure, item batching, confidence elicitation, and model selection, linking sensitivity to prevalence uncertainty and comparing LLM and human instrument effects.Source: §2 · p. 3

3 Methods

The methods use a crossed annotation experiment to separate run-to-run reliability from sensitivity to task design, then estimate how design and model choices affect prevalence and whether confidence detects instability. They also compare LLM sensitivity with human instrument effects without treating human labels as ground truth.

3.1 Dataset and Models

  • Seven LLMs annotate 3,000 tweets for offensive language and hate speech, with the models treated as a sample of deployed annotators.Source: §3.1 · p. 3, §3 · p. 3

3.2 Factorial Experimental Design

  • The 3 × 2 × 2 factorial design yields 12 task designs varying joint or separate labeling, individual or six-tweet batches, and labels alone or labels with confidence.Base definitions and instructions are held constant.Source: §3.2 · p. 3

3.3 Analytic Strategy

  • Three runs per model–design condition measure fixed-design reproducibility with Fleiss’ κ and cross-design label persistence with Cohen’s κ.Source: §3.2 · p. 3, §3.3 · p. 3
  • A pooled model estimates average design effects; a varying-slope model captures model-specific effects, and a crossed prevalence model separates run, design, model, and interaction variation.Task-design variance combines average design and Model × Design components because an application is exposed to both.Source: §3.3 · pp. 3–4
  • A design effect compares cumulative variance with nominal binomial sampling variance, adding run, task-design, and model components sequentially.A design effect of 1 indicates no added variance; values above 1 show how many times larger cumulative variance is than sampling variance alone.Source: §3.3 · p. 4
  • Human benchmarking compares LLM task-design variation with 44,900 human ratings on the same tweets and constructs, without treating human labels as ground truth.Source: §3.3 · p. 4
  • The confidence audit compares stated scores with agreement across designs, repeated runs, individual human judgments, and the human-majority label.Expected Calibration Error compares mean confidence with observed agreement in confidence bins, weighted by each bin’s share of evaluated labels.Source: §3.3 · pp. 4–5

4 Results

Across the tested designs, repeated runs were highly consistent, but agreement and prevalence shifted when task designs or models changed. Those choices contributed substantially to prevalence uncertainty; LLM task-design variation also exceeded human variation, while stated confidence tracked repeated outputs more closely than human agreement.

4.1 Reliable Within Designs, Sensitive Across Designs

  • Median Fleiss’ κ was 0.91 within designs, versus median Cohen’s κ = 0.76 across designs, while prevalence varied more across designs and models than across repeated runs.The prevalence comparisons used the same 3,000 tweets in every combination.Source: §4.1 · p. 5, Figure 1 · p. 5

4.2 Task Design and Model Choice Create Instrument Variance

  • 2,120 basis points separated hate-speech prevalence for GPT-5.4 and GPT-4o-mini on identical tweets; task-design effects also differed in direction and magnitude across models.Batching raised hate-speech prevalence in three models and lowered it in four.Source: §4.2 · p. 6, Figure 2 · p. 6
  • deff reached 76.7 for offensive language and 110.6 for hate speech when task-design, Model × Design, and model-choice variation were included.For offensive language, the 95% interval half-width rose from 150 basis points under nominal sampling alone to 1,280 basis points after model choice was added.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 LLM Task-Design Variation Exceeds Human Task-Design Variation

  • 463 basis points for offensive language and 513 for hate speech were the standard deviations across six LLM designs without confidence requests, versus 331 and 270 basis points across human versions.The corresponding LLM-to-human ratios were 1.4 for offensive language and 1.9 for hate speech, though the versions differed between labeler groups.Source: §4.3 · p. 6
  • Presenting offensive language and hate speech jointly lowered human offensive-language labeling by 598 basis points but raised LLM labeling by 267 basis points.Human instrument versions had design effects of 13–15, compared with 76.7 for offensive language and 110.6 for hate speech when LLM task design and model choice were included.Source: §4.3 · p. 6

4.4 Stated Confidence Does Not Measure Stability Across Task Designs

  • 98.1% of labels in confidence-eliciting conditions had usable scores, but calibration estimates condition on valid scores and give more weight to individual-presentation conditions.Missing scores were concentrated in batched presentation and Llama 3.1 8B.Source: §4.4 · pp. 6–7
  • Expected Calibration Error was 0.074–0.096 against another run of the same design and 0.218 for offensive language and 0.163 for hate speech against an individual human annotator.Against human majority labels, ECE was 0.160 for offensive language and 0.111 for hate speech; all seven models reported lower confidence for batches of six than individual tweets.Source: §4.4 · p. 7

5 Discussion

Repeating one fixed model-design combination measures reliability, not whether labels survive other defensible designs. The discussion argues that researchers should measure and report instrument uncertainty by comparing designs, since prevalence estimates and confidence can shift with the instrument.

Reliability and Sensitivity Answer Different Questions

  • Median agreement falls from Fleiss’ κ = 0.91 within designs to Cohen’s κ = 0.76 across designs, separating reliability from sensitivity.Repeating a fixed design tests whether it reproduces itself; changing the design while holding items and model fixed tests sensitivity.Source: §5 · p. 7

Instrument Uncertainty Belongs in the Reported Error Budget

  • For 3,000 tweets, adding run, task-design, and model variation raises the nominal OL interval half-width from 150 to 1,280 basis points and yields a 1,770-basis-point HS half-width.This additional variation is called instrument uncertainty because it arises from the instrument used to elicit labels.Source: §5 · p. 7
  • More items under one fixed design reduce sampling error but do not reveal variation across untested designs, whose effects differ across seven models.Instrument uncertainty concerns the selected model and task design, unlike item ambiguity or parameter uncertainty.Source: §5 · p. 7

Human Task-Design Effects Do Not Predict LLM Effects

  • LLM task-design variation reaches 1.4 times the human value for OL and 1.9 times the human value for HS, while joint presentation moves human and LLM OL labeling in opposite directions.Joint presentation lowers human OL labeling by 598 basis points but raises LLM OL labeling by 267 basis points.Source: §5 · p. 7

Confidence Is Part of the Instrument

  • Stated confidence tracks agreement with another model output more closely than agreement with a human, and requesting confidence raises OL prevalence by 230 basis points.Presenting tweets in batches of six lowers mean OL confidence by 660 basis points across the seven models.Source: §5 · p. 8, Figure 3 · p. 8

Implications for Applied Annotation

  • An absolute-prevalence audit should compare at least two defensible designs and candidate models, but two designs cannot estimate the full range of instrument uncertainty.If design effects vary across groups or time, comparative differences may reflect the instrument; test that assumption before interpreting group differences, trends, or treatment effects.Source: §5 · p. 8

6 Conclusion

Across the tested models, designs, runs, and tweets, labels were more consistent within a design than across designs. The conclusion names this variation instrument uncertainty and calls for comparing defensible designs and reporting their spread.

6 Conclusion

  • Across seven models, twelve designs, three runs, and 3,000 tweets, median agreement falls from Fleiss’ κ = 0.91 within designs to Cohen’s κ = 0.76 across designs.Task design and model choice produce design effects of 76.7 for offensive language and 110.6 for hate speech.Source: §6 · p. 9
  • Researchers should compare defensible task designs, report the resulting spread, and include task-design and model settings in the measurement description.Repeating one model-design combination, collecting more items under it, or filtering by stated confidence cannot estimate instrument uncertainty.Source: §6 · p. 9

7 Limitations

The study’s limitations bound its findings to particular constructs, model and task-design choices, collection conditions, and labeling outcomes; the appendix illustrates two prompt conditions.

7 Limitations

  • Findings are bounded by subjective safety constructs, untested design choices, a nonrandom seven-model sample, fixed wording and temperature, one collection window, and labels not downstream training.The study does not estimate transfer to objective NLP tasks, surface-wording sensitivity, or temporal reliability; it also does not test whether label differences carry over to trained models.Source: §7 · p. 9

A Example Prompts and Model Outputs

A.1 Base Condition

  • In the Base condition, the model labels one tweet for OL and HS jointly, with OL requested first.Source: §A.1 · p. 11

System Prompt

  • The system prompt defines HS as group-directed hatred or intended derogation, humiliation, or insult, and requests OL then HS labels.The specified labels are OL/HS when present and NO/NH otherwise.Source: System Prompt · p. 11

User Prompt

A.2 Batch-Confidence Condition

B Additional Results

The appendix identifies pooled regression tables underlying the main-text instrument-variance analysis.

B Additional Results

  • Table 3 reports pooled mixed-effects linear probability models underlying the main-text instrument-variance analysis.Source: §B.1 · p. 11
  • Table 4 repeats the analysis using mixed-effects logistic regressions.Source: §B.1 · p. 11
  • The appendix presents both linear-probability and logistic specifications for the instrument-variance analysis.Source: §B.1 · p. 11

B.2 Model-Specific Task-Design Coefficients

The appendix reports model-specific task-design coefficients and their cross-model dispersion.

B.2 Model-Specific Task-Design Coefficients

  • Table 5 fits mixed-effects linear probability models separately to each LLM.Source: §B.2 · p. 11
  • The coefficients estimate how each task-design choice changes the probability of an OL or HS label within each model.Source: §B.2 · pp. 11–12
  • The final column reports the cross-model standard deviation of each estimate.Source: §B.2 · p. 12

B.3 Agreement Between Task Designs

The appendix describes agreement comparisons across task designs and confidence-score summaries by model.

B.3 Agreement Between Task Designs

  • The agreement analysis compares design agreement across three runs with modal-label agreement across designs and against human reference ratings.Source: §B.3 · p. 12

B.4 Confidence Scores by Model

  • Table 7 reports calibration error against three referents, mean stated confidence, label-change share, and confidence AUC for detecting label changes.Source: §B.4 · p. 12
  • Figure 4 shows the confidence calibration curves.Source: §B.4 · p. 12

B.5 Annotation Costs

Batch presentation reduces annotation costs, while GPT-5.4 is substantially more expensive than GPT-4o-mini for the same tweet set.

B.5 Annotation Costs

  • GPT-5.4 costs approximately 17 times more than GPT-4o-mini for the same 3,000 tweets.Source: §B.5 · p. 12, Table 8 · p. 16
  • Batch presentation reduces API calls sixfold and cost by approximately 76% within each model.Table 8 reports annotation costs for three runs of 3,000 tweets.Source: §B.5 · p. 12, Table 8 · p. 16
  • Table 8 compares annotation costs for three runs of 3,000 tweets.Source: Table 8 · p. 16

Related papers

Loading 2609.35824v1…