PaperlayerView exact revision on arXiv

Source-linked AI summary

High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank

arXiv:2610.35179v1cs.CLcs.CY

TL;DR

LLM codebook annotation can be inconsistent, unstable, and uneven across labels, while optimization may affect the constructs being measured. The paper evaluates failures in fixed-codebook inference and codebook optimization using controlled experiments and complementary measures. High aggregate scores coexist with multiple reliability failures, which optimization can compound, motivating validation beyond standard metrics.

  • Problem

    Codebook-based annotation may be inconsistent, unstable across runs, and uneven across labels, while aggregate metrics can mask category-level failures.

    Source: §1 · p. 1, §4.1 · p. 3

  • Method

    The study evaluates predictable failures in fixed-codebook inference and codebook optimization through controlled experiments and complementary metrics.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    High aggregate scores coexist with label disparities, label-name reliance, run instability, formatting sensitivity, and fragile domain transfer; optimization can compound these failures.

    Source: §7 · p. 9

  • Takeaways & Limitations

    A lightweight validation protocol turns the identified failure modes into routine checks beyond standard metrics.

    Source: §7 · p. 9

    The evaluation covers a small set of datasets, codebooks, and model configurations, and the human evaluation relies on only two experts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.

1 Introduction

LLM codebook annotation can be inconsistent and unstable, and prompt optimization can alter label performance or conceptual meaning. This study evaluates failure modes at both inference and optimization stages using controlled experiments and complementary metrics.

1 Introduction

  • Inconsistent instruction-following, run instability, and label-level performance disparities can undermine reliability, while optimization may alter label definitions or measured constructs.Annotations feed statistical analyses and qualitative interpretation, requiring balanced label performance, comparability, and construct preservation.Source: §1 · pp. 1–2
  • The study organizes predictable failures into inference, where an LLM applies a fixed codebook, and optimization, where an LLM refines it before inference.It operationalizes faithful-coding risks with controlled experiments and complementary metrics.Source: §1 · p. 2
  • The paper proposes a systematic failure-mode framework for breakdowns at both inference and optimization stages.Source: §1 · p. 2
  • Evidence across four datasets and five models shows that standard metrics can mask coding-reliability failures, motivating a minimal validation protocol.The paper also introduces two expert-annotated datasets sharing one codebook for controlled domain-transfer evaluation.Source: §1 · p. 2

2 Related Work

Prior work studies codebook-based LLM annotation and prompt optimization, but questions remain about robustness, label-level disparities, stability, and conceptual validity. This paper extends evaluation to controlled tests of these dimensions across annotation and optimization.

2.1 Reliability of LLM-Based Text Annotation

  • Prior work notes that prompt, domain, task, or model choices can change LLM annotations, with possible downstream effects on statistical estimates and inference.Source: §2.1 · p. 2
  • Prior content-analysis studies report that codebook definitions and examples can improve classification, while prompt design and repeated runs can affect predictions.Existing evaluations have primarily emphasized aggregate performance, leaving robustness and label-level failure modes open.Source: §2.1 · p. 2
  • The study tests published codebook-design recommendations, run-to-run stability, label-name reliance, and domain transfer under controlled conditions.It decomposes effects at label level and distinguishes stylistic robustness from adherence to coding concepts.Source: §2.1 · pp. 2–3

2.2 Codebook Optimization with LLMs

  • The study applies APO and SCALE to examine how codebook optimization affects robustness, comparability, and codebook quality, dimensions left underexplored by performance-focused optimization work.It reports that performance improvements can coincide with amplified failure modes.Source: §2.2 · p. 3

3 Data and Evaluation Setup

The evaluation spans four datasets that vary in domain, size, and label count, including two newly introduced Value Creation datasets sharing a codebook. It uses a primary dataset and a news-wire dataset for domain-shift evaluation, with multiple performance metrics and expert codebook ratings.

3 Data and Evaluation Setup

  • The evaluation covers four datasets varying in domain, size, and label count, including the CCC and MINI-MUM WAGE datasets from prior work.Source: §3 · p. 3
  • Two introduced Value Creation datasets use distinct sources: scientific literature for VALUE CREATION SCIENCE and news-wire corpora for VALUE CREATION NEWS.The latter was developed to evaluate robustness under domain shift; both share a codebook developed by social scientists.Source: §3 · p. 3
  • The shared codebook was refined through an iterative annotation process guided by Fleiss’ κ, and two domain experts provided qualitative validation.Source: §3 · p. 3
  • Inference experiments generally use Mistral-7B-Instruct-v0.3 at temperature 0.3, while all three optimization methods use GPT-4o for comparability with SCALE.Reported metrics include accuracy, micro-F1, macro-F1, macro-Precision, and Worst-Label-F1; optimized codebooks also receive expert Likert-scale evaluation.Source: §3 · p. 3

4 Failure Modes in Inference

Inference-stage evaluation finds uneven label performance, overprediction, reliance on label names, and sensitivity to prompts, datasets, and repeated runs. Results vary across models and domains, so aggregate scores alone do not capture these reliability problems.

4.1 Reliability of Label Application

  • Worst-label-F1 trails micro-F1 by 30 points for MINIMUM WAGE, 26 for VALUE CREATION SCIENCE, and 28 for CCC.For MINIMUM WAGE, the lowest-support label has F1 28.90, while SOCIAL-JUSTICE/INJUSTICE reaches 72.97.Source: §4.1 · p. 4
  • Precision falls below recall in 17 of 24 labels, indicating systematic overprediction that can inflate concept prevalence in frequency analyses.In two of the three datasets, macro-Precision is below macro-F1.Source: §4.1 · p. 4
  • Label-Only scores are surprisingly high, while Generic Label and Swapped Label performance drops, indicating reliance on label names over definitions.The pattern is especially pronounced for VALUE CREATION SCIENCE.Source: §4.1 · p. 4

4.2 Robustness

  • Prompt-format recommendations have dataset-dependent effects: structured markers improve nearly all CCC metrics but decrease all metrics for MINIMUM WAGE and VALUE CREATION SCIENCE.Majority-label ordering changes little for two datasets but degrades CCC; inverted ordering gives CCC its best micro/macro-F1 while harming the other datasets.Source: §4.2 · p. 5
  • Run-to-run variation is dataset- and label-dependent: CCC is stable, while MINIMUM WAGE varies more, especially for low-support TURNOVER.Averaging three runs reduces variation for VALUE CREATION SCIENCE; averaging across 15 runs is shown for each label.Source: §4.2 · p. 5, Figure 2 · p. 5
  • Across datasets, no model dominates: Gemma leads on MINIMUM WAGE but falls up to 25 percentage points below other models on CCC.For VALUE CREATION SCIENCE, Gemma has the highest precision while DeepSeek leads in micro/macro-F1.Source: §4.2 · pp. 5–6
  • Domain transfer raises VALUE CREATION NEWS micro-F1 from 44.89 to 62.22 and macro-F1 from 39.57 to 59.55, yet accuracy is 0.0.The codebook was developed on academic publications and applied to news articles within the same theoretical domain.Source: §4.2 · p. 6

5 Failure Modes in Optimization

Codebook optimization can amplify label disparities, produce dataset-contingent outcomes, and alter constructs, while its computational cost and accessibility vary substantially. Predictive performance and expert assessments of codebook quality do not always align.

5.1 Amplified Bias and Sensitivity

  • Adding examples improves micro/macro-F1, accuracy, and macro-Precision on CCC and VALUE CREATION SCIENCE while lowering worst-label-F1 to 5.67 and 11.32, respectively.Because APO optimizes micro-F1 by design, it has no mechanism to prevent minority-label degradation.Source: §5.1 · p. 7
  • Minimal codebook changes have inconsistent effects across datasets; FULL's macro-F1 falls by at most 5 points, while APO and SCALE show greater volatility.Repeated SCALE runs on VALUE CREATION SCIENCE ranged from leaving labels unchanged to completely rewriting the codebook.Source: §5.1 · p. 7

5.2 Conceptual Validity of Optimized Codebooks

  • For relevance, SCALE scores 3.56, FULL 3.44, and APO 3.19; FRAMEWORK was removed during APO optimization.Two domain experts rated label descriptions on a 1–4 Likert scale.Source: §5.2 · p. 8
  • SCALE's mean clarity score of 3.44 exceeds FULL's 3.25 and APO's 3.06, while FULL often achieves higher F1 despite lower expert-rated relevance and clarity than SCALE.The authors describe this as a disconnect between predictive gains and conceptual interpretability.Source: §5.2 · p. 8

5.3 Efficiency and Accessibility

  • APO requires 113,200 API calls, over two orders of magnitude more than SCALE and five orders more than FULL, without consistently outperforming either.FULL requires one call per dataset and SCALE 688–1,044; APO and SCALE also require non-trivial engineering skills.Source: §5.3 · pp. 8–9

6 Minimal Validation Protocol

The proposed protocol is a practical screening procedure to catch reliability problems that aggregate metrics may miss. It pairs label-sensitive evaluation with checks of meaning adherence, stability, sensitivity, transfer, and optimization safeguards.

Minimal Validation Protocol

  • Report macro-F1, worst-label-F1, per-label scores and supports, and macro-Precision alongside micro-F1 and accuracy to expose tail failures and overcoding.The protocol recommends repeating inference at least three times and reporting run counts and variability, with more runs when variance is high or the API is stochastic or opaque.Source: §6 · p. 9
  • Test Label-Only, Generic Label, and Swapped Label conditions, and perturb semantics-preserving formatting or label order to probe meaning adherence and prompt sensitivity.If possible, evaluate on another dataset from the same domain to detect shifts under domain or style changes.Source: §6 · p. 9
  • For codebook optimization, include worst-label-F1, verify constructs with experts, and report computational cost.Source: §6 · p. 9

7 Conclusion

The conclusion describes LLM content analysis as fragile and context-dependent, with optimization compounding rather than resolving its problems. It presents a lightweight validation protocol as a practical response to failures standard metrics often miss.

7 Conclusion

  • High aggregate scores coexist with label disparities, label-name reliance, run instability, formatting sensitivity, and fragile domain transfer.With codebook optimization, these problems compound through minority-label bias, dataset-contingent effects, and potential changes to theoretical constructs.Source: §7 · p. 9

Limitations

The evaluation covers a small set of datasets, codebooks, and model configurations; results may not generalize to other domains, languages, or annotation frameworks.

Limitations

  • The study is limited to a small set of datasets, codebooks, and model configurations, so results may not generalize across domains, languages, or annotation frameworks.Human evaluation of optimized codebooks also relies on only two experts, which may limit intersubjective validity.Source: Limitations · p. 9

Ethical Considerations

The paper reports that it introduces no new personal or sensitive data and focuses on methodological evaluation. Its prompts use codebooks to assign labels to text units across several annotation tasks.

Ethical Considerations

  • The study uses publicly available or author-released, documented datasets, introduces no new personal or sensitive data, and focuses on methodological evaluation rather than judgments about people or groups.The authors state that they verified the technical content, analyses, and conclusions; LLMs assisted with manuscript language refinement.Source: Ethical Considerations · p. 9

VALUE CREATION Codebook Prompt

  • For value-creation text units, the prompt asks models to identify all applicable aspects, defined as elements of the value creation system being discussed.The prompt presents aspect categories with label names and explanations.Source: VALUE CREATION Codebook Prompt · p. 11

MINIMUM WAGE Codebook Prompt

  • For minimum-wage argumentative text units, the prompt defines an aspect as a sub-topic of the broader debate and asks annotators to mark all applicable aspects.Source: MINIMUM WAGE Codebook Prompt · p. 12

CCC Protest Dataset Prompt

B Details on Dataset Creation

The appendix describes the value-creation annotation scheme and dataset construction, then details procedures for assessing run stability, instance-level uncertainty, and domain-shift comparisons.

B.1.1 Annotation Scheme

  • The annotation scheme defines eight interdependent value-creation categories relationally, and annotators labeled noun phrases in English-language scientific abstracts.The scheme includes negative instances, such as financial losses labeled REVENUE and material costs labeled RESOURCE.Source: §B.1.1 · pp. 13–14

B.1.2 Annotation Process

  • Pilot annotation revealed fair but insufficient agreement (Fleiss’ κ = 0.31) and disagreements involving category boundaries, coding rules, and attention to value-creation context.Guidelines were revised with sharper definitions, a stepwise procedure, explicit noun-phrase rules, and requirements to consider relationships among labeled entities.Source: §B.1.2 · p. 14

B.2 VALUE CREATION SCIENCE

  • The main campaign produced 921 annotated abstracts, which were proofed by a domain expert; selected labeled sentences were converted into a multi-label dataset with sufficient context.Source: §B.1.2 · p. 14, §B.2 · p. 14

B.3 VALUE CREATION NEWS

  • VALUE CREATION NEWS comprises 60 business-related news sentences labeled by the three most reliable pilot annotators.Source: §B.3 · p. 14

C Qualitative Examples of Domain Differences

D.1 Setup

  • The evaluation compares prompt prototypes with a fixed baseline using 10 independent runs per dataset-system pair and computes accuracy, macro-F1, micro-F1, macro-precision, and micro-precision.For multi-label tasks, accuracy is exact-match subset accuracy.Source: §D.1 · p. 15

D.2 Run-Level Stability

  • Run-level stability is tested with paired comparisons across run indices, including a t-test of mean deltas against zero and a Wilcoxon signed-rank sensitivity check.Source: §D.2 · p. 15

D.3 Instance-Level Uncertainty

  • Instance-level uncertainty uses a paired bootstrap that preserves baseline–prototype pairing while resampling instances (B = 20,000; seed = 42).For domain shift, the test sets are unpaired at the instance level, so the analysis compares per-run metrics and bootstraps over runs.Source: §D.3 · p. 15, §D.4 · p. 15

E Encoder Baseline: Fine-Tuned RoBERTa

The fine-tuned RoBERTa baseline shows that high aggregate performance can coexist with failure on individual labels. The accompanying figure reports per-label Likert scores by dimension.

E Encoder Baseline: Fine-Tuned RoBERTa

  • RoBERTa’s relatively high Micro-F1 coexists with substantially lower Macro-F1 and Worst-Label-F1 of 0.0 across the reported datasets.The authors attribute this label-level collapse more likely to label imbalance and sparse supervision than to the prompt and semantic failure modes discussed for LLM coding pipelines.Source: E Encoder Baseline: Fine-Tuned RoBERTa · p. 15

F Detailed Likert Scores

G Expert Evaluation Criteria

Expert evaluation rates codebook labels on a four-point scale across relevance, clarity, and distinctiveness.

G Expert Evaluation Criteria

  • The evaluation uses four ratings, from 1 (not relevant, clear, or distinctive) to 4 (highly relevant, clear, or distinctive).Ratings 2 and 3 correspond to “somewhat” and “quite” relevant, clear, or distinctive.Source: Likert Scale · pp. 15–16
  • Figure 5 presents relevance, clarity, and distinctiveness across optimization methods.Source: Figure 5 · p. 16
  • Relevance asks whether a label captures the original codebook concept, while clarity asks whether researchers can understand the label and definition.Source: Likert Scale · p. 16
  • Distinctiveness asks whether a label is separate from other codes or could overlap or be confused with them.Source: Likert Scale · p. 16
Loading 2610.35179v1…