Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels

1 Introduction

• ICL is contested as either flexible task learning or refinement of existing classifiers.Task-learning accounts include Bayesian inference and gradient-descent simulation; prior-refinement evidence includes robustness to label permutations and bootstrapping from zero-shot predictions.Source: §1 · p. 1
• Label tokens carry semantic commitments: changing positive to great can shift accuracy by tens of points, and models favor common tokens.Prior work found that overriding such anchors can require massive scale: GPT-3 can eventually flip labels, whereas smaller models cannot.Source: §1 · pp. 1–2
• Across eight tasks and eight open-source models, natural ICL improves accuracy while retaining prior alignment, but inverted ICL yields exactly zero semantic override.The study covers models from 1–12B parameters across the LLaMA, Mistral, Qwen, and Gemma families.Source: §1 · p. 2
• The paper interprets ICL as adjusting input projections onto a pre-trained semantic space without redefining label meanings.Source: §1 · p. 2

Semantic manipulation in ICL

• Prior flipped-label studies found scale-dependent semantic override, while many-shot prompting also struggles at smaller scales.The paper extends this inquiry to 1–12B models using alignment metrics to distinguish failure modes beyond accuracy alone.Source: §2 · p. 2

Theoretical frameworks

• Bayesian and meta-optimization accounts often treat labels as arbitrary symbols, but prior evidence indicates lasting pre-trained label relationships.Findings of Bayesian inconsistency and persistent label relationships support prior-constrained learning over flexible remapping.Source: §2 · p. 2

Label semantics as constraints

• Label semantics and token biases can constrain ICL, while symbol tuning and contrastive decoding are described as ways to bypass those constraints.Prior work reports accuracy shifts from changing semantically loaded labels and probability competition between equivalent forms.Source: §2 · p. 2

3 Problem Setup

• ICL is formalized as a choice between a zero-shot prior and an in-context classifier induced by demonstrations.Source: §3 · p. 2

3.1 Zero-shot and In-Context Classifiers

• The zero-shot prior uses task instructions, while the in-context classifier conditions on a demonstration set; metrics use inputs with non-UNK predictions.Predictions are obtained by greedy decoding.Source: §3.1 · p. 3

3.2 Natural vs. Inverted Demonstrations

• Natural demonstrations use correct labels, whereas inverted demonstrations permute them; sentiment swaps POS and NEG, and NLI cycles its three labels.Source: §3.2 · p. 3

3.3 Alignment Metrics

• Three alignments measure accuracy, agreement with zero-shot predictions, and agreement with demonstrated mappings; the semantic override rate requires both correctness and inverted-mapping agreement.Prompt alignment equals accuracy under natural demonstrations, but the metrics diverge under inverted demonstrations.Source: §3.3 · p. 3

4.1 Models

• Table 1 identifies the evaluated model set.Source: Table 1 · p. 3

4.2 Tasks and Datasets

• The eight tasks cover sentiment, natural language inference, paraphrase detection, hate speech detection, and topic classification.Datasets include SST-2, IMDB, SNLI, MNLI, MRPC, QQP, ETHOS, and AG News.Source: §4.2 · p. 4

4.3 Prompting Conditions

• Table 2 illustrates sentiment prompts: natural k-shot uses correct labels, while inverted k-shot flips them; other tasks follow the same pattern.For each test example, demonstrations are sampled from the same dataset using fixed random seeds for consistent comparisons.Source: Table 2 · p. 4, §4.3 · p. 4

4.4 Implementation Details

• Predictions use greedy decoding for up to three generated tokens, with UNK outputs excluded from metrics and identical demonstration sets across models.The setup tests k ∈ {1, 2, 4, 8}, caps prompt lengths at 512 zero-shot and 2,048 few-shot tokens, and uses five sampling seeds.Source: §4.4 · p. 4

5.1 Natural ICL improves accuracy, inverted ICL degrades it

• Average accuracy rises from 69.5% (zero-shot) to 79.3% at k = 8 with natural ICL, but drops to 52.8% under inverted ICL.Inverted ICL still reaches 71.6% on QQP, above its weak zero-shot baseline but 7 points below natural ICL; weak priors can provide task-relevant signal without overriding semantic anchors.Source: §5.1 · pp. 4–5

5.2 ICL operates through prior refinement, not replacement

• At k = 8, SST-2 reaches 92.5% accuracy with 89.8% zero-shot alignment, while natural ICL raises accuracy and joint correctness and leaves raw prior alignment stable.On QQP, accuracy rises by 37 points even as the model agrees with its zero-shot prior on 34.9% of examples.Source: Figure 1 · p. 5, §5.2 · pp. 5–6

5.4 Effects strengthen with demonstration count

• As k rises from 1 to 8, natural ICL improves or holds across tasks, while inverted ICL sentiment losses grow from 5–10 points to over 40.Natural gains are larger with weak priors such as QQP than with strong priors such as SST-2.Source: Figure 2 · p. 5, §5.4 · p. 6

5.3 The semantic override paradox

• On IMDB and SST-2, 51.6% and 52.6% of predictions match inverted labels, yet the semantic override rate remains exactly zero.Under inversion, prompt-following rises as accuracy and prior alignment decay, but joint alignment stays at zero.Source: §5.3 · p. 6

6 Conclusion

• Across eight tasks and eight 1–12B models, the semantic override rate remains exactly zero under inverted demonstrations.Models cannot be both accurate and follow the inverted mapping; increasing prompt-following sacrifices accuracy.Source: §6 · p. 7
• Natural demonstrations refine zero-shot behavior: most correct outputs coincide with the prior, while inputs project onto pre-existing semantic directions rather than redefining label meanings.The paper characterizes semantic anchors such as POS, NEG, and ENTAILMENT as rigid constraints on few-shot prompting.Source: §6 · p. 7
• For tasks requiring non-standard label semantics, the authors recommend interventions such as symbol tuning, contrastive decoding, or fine-tuning rather than prompt engineering.Natural label semantics, by contrast, can benefit from ICL because demonstrations refine rather than fight the prior.Source: §6 · p. 7
• Whether the proposed semantic constraint extends to arbitrary symbol-concept mappings, and how model scale affects it, remain questions for future work.The paper presents stable semantic regions in the representation manifold as a suggested geometric interpretation.Source: §6 · p. 7

• The semantic override rate was exactly zero across all experimental conditions; Tables 5–12 report results from k = 1 to k = 8 over five seeds.Reported values are mean accuracy ± standard deviation.Source: A Complete Experimental Results · p. 10
• Tables 5 and 6 present sentiment classification results for SST-2 and IMDB.Source: A Complete Experimental Results · p. 10, Table 5 · p. 10, Table 6 · p. 10
• Tables 7–9 present natural language inference and paraphrase detection results for SNLI, MNLI, and MRPC.Source: A Complete Experimental Results · p. 10, Table 7 · p. 10, Table 8 · p. 11
• Tables 10–12 present results for QQP paraphrase detection, ETHOS hate speech detection, and AG News topic classification.AG News uses a four-way cyclic label permutation rather than binary inversion.Source: A Complete Experimental Results · p. 10, Table 10 · p. 11, Table 11 · p. 12

A.1 Key Observations

• The semantic override rate remained 0.0% across 320 conditions, and inverted-demonstration accuracy declined monotonically as k increased from 1 to 8.Source: §A.1 · p. 12
• Sentiment tasks showed the strongest degradation, often losing more than 40 points at k = 8.Source: §A.1 · p. 12
• Despite weak zero-shot QQP performance across models, models still did not override label semantics on that task.Source: §A.1 · p. 12
• The no-override pattern held across the 1B–12B parameter range, with no evidence that larger models in that range overcame semantic anchors.Source: §A.1 · p. 12

B.2 Complete Prompt Templates

• Table 13 provides the prompt templates and states that instructions are identical for natural and inverted conditions.Source: Table 13 · p. 13

B.3 Demonstration Example

• Table 14 gives a k = 2 example of natural versus inverted sentiment demonstrations.Source: Table 14 · p. 13

• Predictions used greedy decoding at temperature = 0, with a maximum of three new tokens.Source: §B.1 · p. 12
• Zero-shot inputs were truncated to 512 tokens and k-shot inputs to 2,048 tokens, using left-sided padding and truncation.Source: §B.1 · p. 12
• Experiments used five seeds, and demonstrations were sampled without replacement from training data while excluding the query example.Source: §B.1 · p. 12