On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Etienne Casanova; Rafal Kocielnik; R. Michael Alvarez

Source-linked AI summary

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez

arXiv:2606.00467v1May 30, 2026cs.CLcs.AIcs.LGstat.ML

TL;DR

LLM annotation reliability depends on how internalized task concepts interact with user definitions, but it is unclear how much prompting can override those priors. The paper evaluates familiarity, error correction, and misaligned definitions across toxicity datasets and models, finding that definition alignment predicts performance while prompting often cannot repair errors and confidence does not reveal misalignment.

Problem
It is unclear how well users can override LLMs’ internalized task understanding through prompting or measure its alignment with task definitions.
Method
The study evaluates 9 LLMs on 5 toxicity datasets, measuring DSF, text memorization, prompt-based rescue, confidence, and responses to misaligned definitions.
Results
DSF is positively associated with accuracy (partial r = +0.41), while ROUGE-L, BERTScore, and embedding similarity show no positive association; overall rescue rate is 34.8%.
Takeaways & Limitations
Definition alignment is more informative than text-level memorization, while confidence thresholds are poorly suited to detecting definition errors.
Takeaways & Limitations
The conclusions are based on toxicity-related binary classification with concept-substitution misalignment, and all evaluated models are instruction-tuned.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = +0.41), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.

1. Introduction

The paper examines whether user instructions can override LLMs’ internalized task concepts in zero-shot annotation. It studies familiarity, correction of zero-shot errors, and responses to misaligned definitions across toxicity datasets.

Motivation: LLMs develop internalized concepts through training that may conflict with user-provided task definitions.Toxicity can be operationalized differently across applications, creating potential definition mismatch.
Research questions: The study asks whether task and data familiarity affect performance, whether prompting rescues zero-shot errors, and how models respond to incorrect definitions.These correspond to familiarity, decision stickiness, and misalignment susceptibility.
Study design: 9 LLMs are evaluated on 5 toxicity datasets spanning social media, gaming, news, and forums, using Definition-Specific Familiarity alongside memorization metrics.DSF measures alignment between the model’s internal task concept and the dataset’s definition.
Key findings: 34.8% is the overall rescue rate for initial zero-shot errors, leaving nearly two-thirds resistant to prompting.High-confidence errors are especially resistant to correction.
Key findings: DSF has a positive association with performance, whereas text memorization metrics show no positive association after dataset-level controls.The reported partial correlations are +0.41 for DSF and −0.19 for text memorization.

2. Background and Related Work

Related work positions LLM annotation as useful but vulnerable to configuration effects, internalized priors, and unreliable confidence. The paper connects these concerns to steerability and definition alignment.

Steerability and controllability: Research on steerability identifies limited coverage, miscalibration, and side effects as recurring intervention failures.This paper studies analogous failures for discrete annotation labels.
LLMs as annotators and judges: LLMs are used as annotators and judges, but configuration choices can produce incorrect conclusions across replicated annotation tasks.Specialized alignment and curated human definitions have been proposed for more reliable assessment.
Model-internalized priors: Training and post-training alignment can create strong internal priors that conflict with user-provided definitions.Prior work also suggests in-context learning may rely heavily on format and label space rather than input-label mappings.
Confidence and calibration: Confidence estimation remains unreliable because models may overstate confidence and vary when prompted to reconsider.This motivates examining confidence alongside annotation accuracy and steerability.

3. Methods

The methods distinguish text memorization from definition alignment, measure prompt-based steerability and confidence, and evaluate multiple models, datasets, and prompting conditions with regression analyses.

Familiarity measures: The framework separates text familiarity, based on memorized instances, from definition familiarity, based on conceptual alignment with the task definition.These represent lexical-content and conceptual-task mechanisms respectively.
Familiarity measures: Text familiarity uses continuation generation scored with ROUGE-L, BERTScore, and embedding cosine similarity.The metrics provide lexical, contextual, and semantic memorization proxies.
Familiarity measures: DSF measures semantic similarity between a model’s explanation of the target concept and the dataset’s full operational definition.Consensus DSF averages results from six sentence encoders.
Steerability metrics: Rescue Rate = P(Correct | Prompted, Zero-Shot Wrong) measures how often prompting corrects an initial error.A low rescue rate indicates limited reachability of correct behavior through prompt-based steering.
Experimental conditions: Concept substitution creates six misaligned conditions by applying related but semantically distinct dataset definitions to the target task.For example, hate-speech definitions may replace gaming-toxicity definitions.
Steerability metrics: The study also measures change, rescue, corruption, prediction bias, and verbalized confidence to characterize beneficial and harmful prompt effects.Confidence scores range from 0–100 and use a consistent suffix across models.
Experimental setup: Experiments cover five toxicity datasets and nine instruction-tuned models, including dense and mixture-of-experts architectures.The core analysis uses six models, with three additional models extending the evaluation.
Experimental setup: Prompting conditions include zero-shot, aligned definitions, few-shot examples, combined definition-plus-examples, misaligned definitions, and automated DSPy variants.Prompts use a consistent classification template ending with an “Answer:” suffix.

4. Results

Across toxicity datasets and prompting conditions, semantic definition alignment is associated with performance, while memorization metrics are not. Prompting rarely rescues confident errors, and misaligned definitions alter decisions without reliably changing confidence.

Familiarity and performance: DSF had a positive association with performance after controlling for dataset, while ROUGE-L, BERTScore, and embedding similarity had no positive association.Consensus DSF remained positively associated after controlling for text complexity.
Knowledge injection and decision stickiness: Correct zero-shot responses had 6.4× higher odds of remaining correct after aligned definitions or examples, indicating prompting consolidated correct answers more often than it rescued errors.Zero-shot correctness was associated with prompted correctness at OR = 6.43.
Knowledge injection and decision stickiness: 34.8% was the overall rescue rate, so nearly two-thirds of zero-shot errors resisted correction.Rescue probability fell to 20.8% for errors above 0.9 confidence, and each standard-deviation increase in confidence reduced rescue odds by 16%.
Definition effects: Misaligned definitions shifted prediction thresholds, with narrow definitions causing −7% to −12% bias and broad definitions causing +9% to +13% bias.Models adjusted decisions to match provided criteria rather than ignoring misaligned instructions.
Definition effects: 17% accuracy variation came from definition choice, compared with 5% from model choice on average.Fox News accuracy ranged from 61.6% with the Gaming Toxicity definition to 78.4% with the Twitter Hate Speech definition.

5. Discussion

The discussion argues that annotation reliability depends more on definition alignment than text memorization, while prompting has limited power to correct errors. Misaligned definitions can shift behavior without reliably reducing confidence, motivating alignment checks and sensitivity testing.

Definition Alignment: DSF remains positively associated with accuracy after dataset-level controls, whereas ROUGE-L, BERTScore, and embedding similarity show no positive association.The reported DSF association is partial r = +0.41; text memorization has little explanatory power.
Decision Stickiness: Most zero-shot errors persist under aligned definitions and few-shot examples, with high-confidence errors especially resistant to correction.A three-turn rescue sequence increases rescue from 7.5% at Turn 1 to only 18.7% at Turn 3, while high-confidence rescue reaches just 8.5%.
Definition Misalignment: Misaligned definitions systematically change prediction rates, consistent with models adjusting decision thresholds to match provided criteria.The models do not simply ignore conflicting instructions, but their responsiveness can make incorrect definitions operationally consequential.
Calibration: Models remain highly confident under misalignment, and calibration curves show little separation between aligned and misaligned conditions.Confidence therefore reflects certainty conditional on the instructions rather than uncertainty about definition validity.
Practical Implications: The paper recommends measuring definition alignment, testing multiple plausible formulations, and avoiding confidence as a proxy for definition appropriateness.These safeguards are proposed for large-scale LLM annotation pipelines.
Limitations: The conclusions are limited to instruction-tuned models and toxicity-related binary classification with concept-substitution misalignment.The study cannot distinguish capability limits from intentional steerability constraints or identify which training stage produces concept anchoring.

6. Conclusion

The paper examines how internalized priors affect prompt-based steering in LLM annotation. Across models and toxicity datasets, definition-specific familiarity predicts accuracy more consistently than text memorization, while prompting often fails to correct mistakes.

Conclusion: Across 9 models and 5 toxicity datasets, DSF is positively associated with accuracy after controlling for dataset difficulty, unlike three memorization metrics.The reported DSF association is partial r = +0.41 with N = 54.

Impact Statement

The paper’s definitions span different scopes and toxicity-related criteria across social media, gaming, news, and forum settings. They range from identity-targeted hate speech to broad rudeness, discouragement, and contextual gaming toxicity.

Task Coverage: The study uses definitions across social media, gaming, news, and forum-oriented toxicity tasks, including hate speech, offensive language, and general toxicity.The supplied definitions include Twitter, Fox News, GameTox, OLID, CONDA, and Perspective API/Jigsaw formulations.
Boundary Differences: Some task definitions explicitly exclude general insults without identity targeting, while others include insults or threats regardless of identity targeting.This difference creates substantially different operational boundaries for toxicity classification.
Gaming Definition: GameTox defines toxicity through multiple categories, including hate and harassment, threats, extremism, insults and flaming, and other offensive texts harming game enjoyment.Its non-toxic category excludes utterances without toxicity from those categories.
Definition Scope: Narrow definitions require identity-based targeting, whereas medium definitions include insults, threats, and profanity regardless of identity targeting.The scope classification contrasts these with broader criteria centered on rudeness, disrespect, discouragement, and implicit toxicity.

A.4. Non-Safety Task Definitions

The appendix defines non-safety tasks and details the experimental prompts, conditions, familiarity measures, and regression models. It covers irony, subjectivity, stance, and automated or manually specified prompting analyses.

Task Definitions: Irony detection labels statements whose intended meaning is opposite to or incongruent with their literal meaning, versus statements matching literal meaning.The definition includes humor, criticism, sarcasm, and discrepancies between expression and intent.
Task Definitions: Subjectivity detection distinguishes opinions, evaluations, emotions, and speculations from independently verifiable factual information.The task contrasts author perspective with objective facts.
Task Definitions: Stance detection classifies five targets as either in favor or against, rather than against a single concept definition.Targets include atheism, climate change, feminism, Hillary Clinton, and abortion legalization.
Prompting Conditions: All classification prompts present the task and text before an Answer suffix, with confidence elicitation appended to every prompt.The experiments compare 10 manually specified prompting conditions with 2 DSPy automated optimization conditions.
Familiarity Measures: Text familiarity uses 40% text prefixes for continuation generation, while embedding similarity and BERTScore compare generated continuations with ground-truth suffixes.Embedding Similarity uses cosine similarity; BERTScore uses token-level F1 from contextual-token similarities.
Statistical Analysis: Mixed-effects logistic regressions analyze domain alignment, model-alignment interactions, zero-shot prediction, and rescue among zero-shot errors.The models include message-level random intercepts to account for repeated measures across experimental conditions.

D.1. Multi-Turn Rescue Experiment

The multi-turn rescue experiment tests whether history-aware prompting can overcome decision stickiness. Iterative correction helps modestly, but high-confidence errors remain largely resistant.

Multi-Turn Rescue Experiment: 100 zero-shot errors per model–dataset pair underwent three rescue turns: few-shot examples, an aligned definition, then reconsideration.Each later turn included the model’s previous answers and confidence scores.
Multi-Turn Rescue Experiment: 8.5% of high-confidence errors were rescued after three turns.High-confidence errors remained almost entirely uncorrected despite iterative prompting.
Multi-Turn Rescue Experiment: Iterative prompting improved rescue modestly, especially after adding the aligned definition at Turn 2, but plateaued below the 34.8% one-shot rescue rate.The comparison is with the one-shot rate reported in Table 6.
Multi-Turn Rescue Experiment: 78.8% of sampled items were dropped because of classification refusals.This refusal-related exclusion constrains the analyzed sample.

E.1. DSF Robustness Across Embedding Models

The paper tests whether DSF and memorization findings depend on embedding choice or textual-overlap metric. DSF remains positively associated with accuracy across encoders, whereas text-memorization metrics remain non-positive.

DSF Robustness Across Embedding Models: Six diverse sentence encoders were used to recompute DSF, with consensus DSF defined as their unweighted mean over 54 model–dataset pairs.The panel covers nine models across six datasets, and consensus DSF is the primary metric.
DSF Robustness Across Embedding Models: +0.30 to +0.49: all six embeddings yielded positive DSF–accuracy partial correlations, while consensus DSF was +0.41.Correlations control for dataset; larger encoders tended to produce slightly stronger associations.
DSF Robustness Across Embedding Models: Above 0.87: pairwise correlations between DSF score vectors exceeded this value across all embedding choices, with most above 0.95.The embeddings closely agreed on the relative ordering of model–dataset cells.
Memorization Metrics: BERTScore uses contextual token-embedding alignment and credits paraphrases or near-synonyms that ROUGE-L may penalize.Its primary reported score is contextual F1.
Memorization Metrics: ROUGE-L, BERTScore, and embedding cosine similarity all had negative partial correlations with accuracy after controlling for dataset.The negative direction persisted from lexical to semantic notions of text reproduction.
Metric Definitions: DSF measures cosine similarity between a dataset definition and the model’s own concept description, whereas EmbSim compares whole-sentence representations of generated and reference text.The comparison isolates definition–self-description alignment from continuation–reference alignment.

F. Generalization Beyond Toxicity

The paper replicates its two core findings beyond toxicity using irony, subjectivity, and a preliminary stance-detection experiment. DSF remains positively associated with accuracy, while most zero-shot errors continue to resist correction.

Non-Safety Replications: +0.343: DSF’s partial correlation with zero-shot accuracy across irony and subjectivity was positive over 18 model–dataset pairs.The effect directionally replicates the toxicity finding and is reported as not safety-specific.
Non-Safety Replications: 45.0% of 3,052 zero-shot errors were rescued by one-shot few-shot prompting across irony and subjectivity.More than half of the errors remained uncorrected; the 95% confidence interval was [43.2%, 46.8%].
Stance Detection: +0.295: DSF’s partial correlation with accuracy remained positive in preliminary stance detection across 40 target–model pairs.Stance detection differs structurally because it chooses between explicit alternatives rather than applying a single concept definition.
Conclusion: Both core findings replicated outside toxicity: DSF was positively associated with zero-shot accuracy, and most zero-shot errors resisted correction despite aligned definitions or few-shot examples.The authors characterize these effects as properties of definition-driven annotation under internalized priors.

G. Model and Dataset Use Across Experiments

The experiments use distinct coverage tiers for models and datasets. Main descriptive results span all nine models and five primary toxicity datasets, while regression and supplementary analyses use narrower, specified subsets.

Model Coverage: Nine evaluated models provide the broad coverage for main descriptive RQ2/RQ3 tables and figures.The set includes dense, mixture-of-experts, open-weights, and proprietary systems.
Model Coverage: Six core open-weights models are used for mixed-effects regressions and the multi-turn rescue analysis.This restriction keeps statistical inference consistent across those analyses.
Dataset Coverage: Five primary toxicity datasets form the analytic core and appear in every result table.They cover Twitter Hate, OLID, GameTox, Fox News, and Jigsaw Toxic Comments.
Dataset Coverage: Jigsaw Unintended Bias serves as a label-bias robustness check, while irony and subjectivity are reserved for out-of-domain RQ1 and RQ2 replications.These supplementary datasets are used only in their specified analyses.