PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation

Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza-del-Arco, Johannes B. Gruber, Dirk Hovy

arXiv:2509.08825v2cs.CLcs.AIcs.LG

TL;DR

LLM configuration choices can distort downstream scientific conclusions, but the factors governing this risk remain poorly understood. The paper formalizes and measures LLM hacking across annotation tasks, finding substantial risks from both deliberate manipulation and accidental configuration variation, and evaluates safeguards.

  • Problem

    Researchers lack principled empirical guidance on when LLM-based annotation compromises scientific validity or how configuration choices distort downstream statistical conclusions.

  • Method

    The paper defines LLM hacking and evaluates intentional and accidental risks across replicated social-science annotation tasks, models, prompts, and hypotheses.

  • Results

    LLM hacking is feasible for 94.4% of null hypotheses, while SOTA models still show a 31% average risk of incorrect conclusions.

  • Takeaways & Limitations

    Human annotations and methodological safeguards are needed to distinguish valid LLM-based findings from configuration artifacts, although available guidelines cannot eliminate the risk.

  • Takeaways & Limitations

    Risk estimates assume noise-free ground-truth annotations and depend on the configuration space examined, so real-world risk may be over- or underestimated.

Abstract

from arXiv · show

Large language models are rapidly transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis. However, LLM outputs vary significantly depending on the implementation choices made by researchers (e.g., model selection or prompting strategy). Such variation can introduce systematic biases and random errors, which propagate to downstream analyses and cause Type I (false positive), Type II (false negative), Type S (wrong sign), or Type M (exaggerated effect) errors. We call this phenomenon where configuration choices lead to incorrect conclusions LLM hacking. We find that intentional LLM hacking is strikingly simple. By replicating 37 data annotation tasks from 21 published social science studies, we show that, with just a handful of prompt paraphrases, virtually anything can be presented as statistically significant. Beyond intentional manipulation, our analysis of 13 million labels from 18 different LLMs across 2361 realistic hypotheses shows that there is also a high risk of accidental LLM hacking, even when following standard research practices. We find incorrect conclusions in approximately 31% of hypotheses for state-of-the-art LLMs, and in half the hypotheses for smaller language models. While higher task performance and stronger general model capabilities reduce LLM hacking risk, even highly accurate models remain susceptible. The risk of LLM hacking decreases as effect sizes increase, indicating the need for more rigorous verification of LLM-based findings near significance thresholds. We analyze 21 mitigation techniques and find that human annotations provide crucial protection against false positives. Common regression estimator correction techniques can restore valid inference but trade off Type I vs. Type II errors. We publish a list of practical recommendations to prevent LLM hacking.

1 Introduction

LLM adoption creates a scientific-validity risk because configuration choices can alter annotations and downstream conclusions. The paper quantifies intentional and accidental LLM hacking and evaluates mitigation strategies.

  • Key findings: 94.4% of null hypotheses could be turned into false positives, while true effects could be hidden in 98.1% of cases and reversed in 68.3%.These outcomes were feasible using tested models and a handful of prompt paraphrases.
  • Key findings: 31% of hypotheses showed accidental LLM hacking with state-of-the-art models, compared with 50% for smaller 1B-parameter models.Risk ranged from 5% for humor detection to over 65% for some ideology and frame-classification configurations.
  • Key findings: Error rates approached 70% near p = 0.05, while task characteristics explained 21% of variance and model performance explained only 8%.Prompt engineering choices explained less than 1% of variance, and high annotation performance did not prevent hacking.
  • Mitigation: 100 human annotations produced around 10% false-positive rates, outperforming uncorrected hybrid approaches using 100K LLM annotations at 30–40%.Human annotations therefore provided stronger protection against Type I errors in the tested approaches.
  • Mitigation: Regression correction methods restored nominal Type I error levels but increased Type II errors by up to 60 percentage points.The methods therefore trade false positives for false negatives rather than eliminating hacking risk.

2 Large Language Model (LLM) Hacking

LLM hacking is defined as incorrect scientific conclusions caused by researchers’ choices among plausible LLM configurations. The concept covers accidental and intentional manipulation of LLM-generated annotations before downstream statistical analysis.

  • Definition: LLM hacking occurs when an LLM configuration produces an incorrect downstream scientific conclusion.The study focuses on whether regression coefficients reach statistical significance, while defining the concept more broadly.
  • Definition: The configuration space includes model choice, prompts, decoding parameters, and output-to-category mapping, provided configurations remain plausible.Defaults and undocumented settings can also contribute to researchers’ degrees of freedom.
  • Intentional hacking: Intentional hacking is feasible when at least one plausible configuration produces a specified incorrect outcome.Minor prompt or temperature changes can propagate into false positives, false negatives, wrong directions, or distorted magnitudes.
  • Relation to p-hacking: LLM hacking differs from p-hacking because it manipulates data generation through annotation configurations rather than analytical choices.The two practices can be cumulative, exposing LLM-annotated studies to both configuration-induced and traditional analytical flexibility risks.

3 Literature review: Using LLMs for automated data annotation

LLM annotation was widely endorsed in the reviewed CSS literature, but validation of its reliability for downstream scientific inference remained limited. The review identifies a gap in principled guidance about when LLM errors distort statistical conclusions.

  • Adoption: 85 of 93 reviewed papers recommended LLMs for annotation, while only 8 advised against their use.Endorsement covered tasks including sentiment, stance, hate speech, and political discourse analysis.
  • Adoption: 77 papers benchmarked annotation quality using empirical datasets, and GPT-family models appeared in over 80% of reviewed studies.Open-weight alternatives such as Llama and Qwen were tested less frequently.
  • Validation: 56.5% of papers recommending LLMs disregarded model validation, while included validations mostly used simple performance metrics.Only four papers explicitly discussed false downstream statistical conclusions from LLM-generated annotations.
  • Research gap: The literature offered no empirical evidence identifying which factors determine whether LLM errors distort hypothesis tests, regressions, or related statistical procedures.This leaves unclear how well-intentioned researchers can avoid erroneous conclusions or how configuration choices could be exploited.

4 Experimental setup

The experiments evaluate LLM annotation reliability across diverse social-science tasks by comparing ground-truth and LLM-informed regression conclusions under many model and prompt configurations. They quantify discrepancies in significance, direction, and effect magnitude, while testing intentional hacking feasibility and mitigation using human annotations.

  • 4.1 Data: The benchmark covers 37 annotation tasks from 21 datasets, spanning diverse data types, domains, and classification problems.Tasks include stance detection, topic classification, sentiment analysis, misinformation, humor, and politeness classification.
  • 4.1 Data: The analysis uses objectively correct or expert- and crowd-annotated labels as ground truth, removing ambiguous cases to achieve average Krippendorff’s alpha of 0.91.This enables comparison of LLM-generated annotations with high-quality reference labels.
  • 4.2 Downstream statistical analysis: The study compares ground-truth and LLM-informed logistic regressions to test whether group differences in positive-class proportions reach significance at α = 0.05.Ground-truth and LLM annotations are analyzed separately for each hypothesis and LLM configuration.
  • 4.5 Metrics: The study measures Type I, Type II, Type S, and Type M discrepancies between LLM-based and ground-truth conclusions while assessing whether configuration choices enable deliberate manipulation.Type S errors concern opposite-sign significant effects, and feasibility is evaluated over methodologically defensible configurations.
  • 4.2 Downstream statistical analysis: Researchers generate 2,361 hypotheses from keyword-based or metadata-based dataset splits, with 48.1% based on original metadata.Each hypothesis compares two groups using binary logistic regression.

5 Results

LLM hacking is feasible both intentionally and accidentally: configuration choices can manufacture or hide effects, while even strong models produce unreliable downstream conclusions.

  • 5.1 Intentional LLM hacking: 94.4% of null hypotheses permit manufactured significant effects, 98.1% of true effects can be hidden, and 68.3% can be reversed in sign.These rates show that different tested configurations can support contradictory conclusions.
  • 5.1 Intentional LLM hacking: Even the seven best-performing models retain 69.7% Type I and 68.1% Type II error feasibility.The vulnerability therefore persists beyond poor-quality models and methodologically indefensible choices.
  • 5.2 Empirical LLM hacking risk: Empirical hacking risk ranges from 31% for 70B models to 50% for 1B models, varying from 5% to over 65% across tasks.Random-label baselines show a different failure pattern, with 7.8% Type I risk but 92.9% Type II risk.
  • 5.2 Empirical LLM hacking risk: Type S errors reverse effects in 4–16% of cases, while Type M errors distort estimated effect sizes by 41–77% on average.For GPT-4o, only 2.7% of correctly identified significant effects fall within 10% of the true magnitude.
  • 5.2 Empirical LLM hacking risk: Scaling reduces risk with diminishing returns: Llama risk falls from 50.3% at 1B to 36.6% at 8B and 31.6% at 70B parameters.Type II errors dominate across model families, and additional scaling may not overcome limitations beyond 30–70B parameters.
  • 5.3 Predictors of LLM hacking: Performance does not guarantee valid inference: tasks with weighted F1 above 0.93 still show hacking risk above 50%, and one reaches 50% risk at 93% F1.Risk is strongest near p = 0.05, where error probabilities approach 70%, but substantial risk remains farther from the threshold.

6 Limitations

The study’s risk estimates rely on noise-free ground truth and a fixed significance threshold, while real human annotations contain biases and errors. Estimates also depend on the configuration space, although robustness analyses indicate stability to ground-truth uncertainty.

  • Risk estimates assume noise-free ground truth annotations and effective p = 0.05 significance thresholds, although real human annotations contain biases and errors.Unavailable annotator-bias analyses make the resulting over- or underestimation difficult to quantify.
  • Risk estimates remain largely stable after accounting for ground-truth uncertainty, although comparisons with true population effects may still over- or underestimate risk.The authors report extensive robustness analyses addressing this uncertainty.
  • Risk estimates depend on the considered configuration space, so misspecified prompts could inflate estimated LLM hacking risk.The authors use prior-research models and prompts and exclude configurations producing invalid outputs.

7 Discussion

The discussion argues that LLM-generated annotations can produce incorrect downstream conclusions unpredictably across disciplines, motivating human validation, transparent multiverse reporting, and explicit safeguards. It recommends practical standards while emphasizing that human expertise remains important and current guidelines cannot eliminate risk.

  • 31-50% of cases produce incorrect downstream statistical conclusions, and researchers cannot predict when, how, or which LLMs will fail.The authors characterize this unpredictability as a threat to the validity of LLM-based hypothesis testing without rigorous safeguards.
  • LLM hacking concerns extend beyond computational social science to any discipline relying on AI-generated annotations, including medicine and law.The paper empirically demonstrates at scale how ignoring validation safeguards can produce false discoveries and offers actionable standards.
  • Human-LLM annotation collaboration: Human annotations and task-specific validation remain indispensable because pure LLM approaches underperform hybrid strategies for controlling Type I errors.Even 100 human annotations outperform sophisticated LLM-based corrections for Type I error control.
  • Multiverse LLM annotation approaches: Multiverse annotation reports coefficient distributions across reasonable LLM configurations instead of single estimates, exposing fragility in findings.The authors identify practical feasibility and p-curve analysis as areas for future work.
  • What distinguishes LLMs from traditional supervised learning: LLMs create more opportunities for manipulation than traditional supervised models because prompt paraphrases can generate different annotator models without retraining or ground truth data.The paper argues that this expanded researcher freedom changes the potential for intentional manipulation.
  • Guidelines require reporting tested models, prompts, parameters, selection criteria, annotations, code, and preregistered analysis decisions.These recommendations are synthesized as practical standards for methodological rigor.

A Literature review methodology

The literature review searched five databases, screened 1,592 records, and selected 21 publicly available computational social science datasets covering 37 annotation tasks. Selection emphasized task diversity, metadata suitable for hypothesis testing, and field usage.

  • The review searched five databases using combined annotation, LLM, performance, and social-science domain keywords.The databases included Scopus, Web of Science, Semantic Scholar, ACL Anthology, and Google Scholar.
  • 1,592 papers were identified, 102 duplicates and 1,030 low-citation papers were removed, leaving 453 for screening.
  • From screened literature, the study retained papers benchmarking LLM performance or recommending LLM use in social science research.Non-social-science domains were excluded during title and abstract screening.
  • For each reviewed paper, the extraction recorded task characteristics, models, prompting, validation, recommendations, ground-truth construction, and reported performance metrics.
  • The final selection comprised 21 publicly available computational social science datasets covering 37 annotation tasks.Tasks were selected for metadata availability, diversity, and broad field usage or citation.

B Structured results of literature review for in-scope articles

The review found that most in-scope papers recommend automated LLM annotation without rigorous validation, while only a small minority warn about downstream statistical errors. The study then operationalizes its review through diverse models, hypotheses, prompts, and confidence measures.

  • The structured review covers 93 papers and sorts them by LLM recommendation type, validation practice, dataset, model, and prompt characteristics.Table 7 defines recommendation and validation symbols used in the overview.
  • Only 8 reviewed papers opposed automated LLM annotation, while only four warned that erroneous annotations could affect downstream statistical conclusions.About 75% of studies focused exclusively on English text.
  • Datasets generate binary hypotheses from metadata, keyword presence, text length, and random splits to create diverse plausible group contrasts.
  • Each annotation task uses at least five prompt variations, with paraphrases manually reviewed and released alongside prompts, mappings, and code for reproducibility.
  • The study elicits verbalized confidence scores by asking models to output a number between 0 and 1 for each annotation.Confidence histograms and calibration plots are reported in Appendix E.7.

C.5 Active sampling implementation details

The study implements active-sampling and regression-correction methods with specified defaults, adaptive fallbacks, and human-annotation budgets. Experiments also report substantial computational and API resources.

  • Active sampling: Human-annotation budgets range from 25 to 1,000 samples, with 25 burn-in samples used to initialize and retraining after each batch.The prediction model is retrained after every batch of human annotations.
  • Regression correction: DSL and CDI use adaptive procedures that alter cross-fitting, sample-splitting, or trust-parameter initialization when sparse labels or numerical instability prevent successful estimation.CDI falls back to human annotations only when optimization produces instability or negative variance.
  • Active sampling: Low-confidence sampling assigns higher selection probabilities to instances with lower verbalized LLM confidence, while retaining 10% uniform sampling for stability.The sampling probability combines a baseline rate with the complement of the LLM confidence score.
  • Computational resources: The experiments required approximately $520 in API costs and 11,000 GPU hours across annotation, confidence elicitation, inference, and correction analyses.The reported API costs cover GPT-4o and GPT-4o-mini annotations and confidence elicitation.

D Additional Metrics

Additional metrics assess the reliability of LLM-based discoveries from complementary conditioning perspectives. They quantify false discoveries, missed effects, and incorrect signs among relevant discovery outcomes.

  • Discovery reliability: The False Discovery Rate measures the fraction of incorrect discoveries among cases where LLM annotations detect a significant effect.It conditions on discoveries made by the LLM testing procedure.
  • Discovery reliability: The False Nondiscovery Rate measures the fraction of missed true effects among cases where LLM annotations fail to detect a significant effect.It conditions on non-discoveries rather than on ground-truth outcomes.
  • Discovery reliability: The Type S Error Rate measures the fraction of significant LLM discoveries with an incorrect sign when ground truth also indicates a significant effect.Together, these metrics complement Type I and Type II risk by conditioning on LLM decisions.

E.1 Complete LLM hacking risk breakdown by task and model

LLM hacking risk varies substantially across annotation tasks and models, while instruction-following failures are more common in smaller models. Excluding combinations with excessive failures may make reported risks conservative.

  • Risk breakdown: Risk ranges from 5% for humor detection with larger models to over 75% for some relevance-classification tasks across the evaluated task-model combinations.The complete breakdown covers 37 annotation tasks and 18 models.
  • Instruction following: Instruction-following failures occur when models produce outputs outside the required format or explanatory text instead of valid categorical labels.Higher failure rates occur for smaller models.
  • Instruction following: Combinations with failure rates above 1% are excluded, so reported hacking risks may underestimate the true extent for models with poor instruction-following capabilities.The threshold is described as conservative.

E.3 LLM hacking predictors

LLM hacking risk is associated with prompt sensitivity, model capability, task and model performance, and proximity to statistical significance. Larger or stronger models generally reduce risk, but substantial susceptibility remains.

  • Regression predictors: LLM hacking is more likely when hypotheses involve significant differences and less likely as normalized distance from the significance threshold increases.The regression coefficients are 0.324 for significant_difference_found and −0.220 for normalized_distance_from_significance_treshold.
  • Regression predictors: Task and model performance measures have heterogeneous associations with hacking risk, including negative coefficients for several model weighted-F1 interactions.The table reports task and model interaction terms rather than a single uniform performance effect.
  • Regression predictors: The regression analyzes 1,429,925 observations and achieves 83.0% classification accuracy while jointly finding significant explanatory power for the predictors.The reported model has R2 = 0.154 and an F-statistic of 2,352.722 with p < 0.001.
  • Prompt and capability predictors: Prompt choice changes average annotation performance by more than 10 percentage points, with smaller models showing wider performance differences across prompts.The result indicates less stable annotation behavior for smaller models.
  • Prompt and capability predictors: Higher general model capability is negatively correlated with LLM hacking risk, although even the highest-scoring models retain risk levels up to 52%.The capability analysis covers eight Llama and Qwen models with available MMLU-PRO scores.

E.6 LLM hacking feasibility by annotation task

This section examines how annotation task, model, prompting, and validation choices relate to LLM hacking and downstream statistical reliability. It finds that prompt sensitivity and annotation quality metrics do not fully guarantee valid inference.

  • Passing the alt-test does not guarantee reliable downstream inference, because individual annotator alignment can coexist with biased regression coefficients and incorrect hypothesis conclusions.The analysis asks whether annotations preserve downstream statistical validity, whereas the alt-test evaluates replacement of an individual human annotator.
  • Sampling strategy has minimal effect when human-annotation counts are fixed, so sophisticated active or low-confidence sampling provides limited benefit over random sampling.The reported differences are negligible for both ground-truth-only and ground-truth-plus-LLM mitigation variants.
  • Prompt sensitivity can produce best-prompt versus baseline performance gaps of up to 5 percentage points or more, making it a task characteristic researchers should consider.Performance spread is measured as absolute weighted-F1 differences across prompt variations.
  • Higher-performing models generally have lower LLM hacking risk, but GPT-4o is selected as the best model in only about 49% of cases.The negative performance-risk relationship remains significant even when performance is estimated from a small human-annotation sample.
  • 35-42% of hypotheses show statistically distinguishable LLM and ground-truth regression coefficients, rising to 58-66% in LLM hacking cases.Correctly classified cases show distinguishable coefficients in only 21-28% of hypotheses, indicating that many errors reflect systematic differences rather than small threshold perturbations.
  • Accounting for ground-truth uncertainty raises GPT-4o’s estimated Type I risk from 19.7% to 31.3%, while Type II and Type S risks decrease.The empirical-Bayes adjustment changes risk estimates because local FDR assigns more conservative effect-existence probabilities near p = 0.05.

F.11.1 humor task

This appendix section documents the annotation datasets and task constructions used in the experiments, including humor, political, policy, misinformation, and content-moderation tasks. It also specifies the corresponding ground-truth sources and preprocessing choices.

  • humor task: The humor task predicts whether a post is funny to many people, operationalized as receiving more than 200 upvotes.Its ground truth comes from Reddit users, and the prompts are taken from prior work.
  • The experiments cover diverse text domains, including congressional bills, political tweets and advertisements, news headlines, party manifestos, and content-moderation tweets.Tasks include topic, party-affiliation, misinformation, tone, policy-dimension, manifesto-category, relevance, and frame classification.
  • Ground truth is supplied by trained annotators, experts, crowd workers, research assistants, or known author attributes, depending on the dataset.Several manifesto tasks retain only examples with more than 80% crowd-worker agreement, while some content-moderation tasks require unanimous annotator agreement.
Loading 2509.08825v2…