Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

Motivation

• ICL’s underlying working mechanism remains an open question despite enabling LLMs to perform diverse tasks from few-shot demonstrations.The paper motivates studying how models learn from their supplied context without parameter updates.Source: §1 · p. 1

Method

• Saliency visualization and metrics are used to characterize information flow among demonstration text, label words, and the target position.The approach analyzes attention interaction patterns and distinguishes shallow aggregation from deep prediction-related flows.Source: Figure 1 · p. 1, §1 · p. 2

Information Flow with Labels as Anchors

• Label words aggregate demonstration information in shallow layers and provide information for final prediction in deep layers.The hypothesis describes shallow-layer semantic representation formation followed by deep-layer extraction for prediction.Source: §1 · pp. 1–2, Figure 2 · p. 2

Experiments

• Experiments with GPT2-XL and GPT-J across text-classification benchmarks validate the proposed information-flow hypothesis.Blocking shallow-layer aggregation paths significantly impairs performance, indicating that label words collect useful information during forward propagation.Source: §1 · p. 2

Applications

• Anchor re-weighting improves average accuracy by 16.7% over standard ICL baselines, while demonstration compression achieves a 1.8 × inference speedup with minimal performance trade-off.The paper also presents an error-analysis framework connecting label confusion with distances between anchor key vectors.Source: §1 · p. 2

2.1 Hypothesis Motivated by Saliency Scores

• The saliency matrix I_l measures the significance of information flow between input words, revealing increasing label-word dominance as layers deepen.It is computed by averaging attention-head saliency matrices derived from attention and loss gradients.Source: §2.1 · pp. 2–3
• The metrics Swp, Spq, and Sww separately quantify text-to-label, label-to-target, and other word-to-word information flow.Swp measures aggregation onto label words, while Spq measures extraction from label words toward the target position.Source: §2.1 · p. 3

2.2 Shallow Layers: Information Aggregation

• Across layers, Swp is initially substantial and decays, whereas Spq becomes dominant; label-word interactions generally exceed other word interactions.This pattern is reported on SST-2 and AGNews, with comparable findings described across the analyzed settings.Source: §2.1 · p. 3, Figure 3 · p. 4

2.1 Hypothesis Motivated by Saliency Scores

• Label words gather information from demonstrations in shallow layers and provide semantic representations for later computations and final prediction.The hypothesis distinguishes shallow-layer aggregation from deep-layer extraction.Source: §2 · p. 2, §2.1 · p. 3

2.2 Shallow Layers: Information Aggregation

• Isolating label words in the first five layers substantially affects behavior, unlike isolation in the last five layers or of random non-label words.The intervention supports a specific role for shallow-layer aggregation through label words.Source: §2.2 · p. 4, Figure 4 · p. 4

2.3 Deep Layers: Information Extraction

• In deep layers, target-position attention over label words strongly correlates with final predictions, while cumulative contribution rises mainly in middle and deep layers.The reported AUCROC for deep layers approaches 0.8, and shallow layers show negligible cumulative contributions.Source: §2.3.2 · p. 5, §2.3 · p. 5

3.1 Anchor Re-weighting

• Anchor re-weighting adjusts label-word contributions with a learnable vector, achieving a 16.7% average accuracy boost over standard ICL baselines.The method uses an auxiliary training set and is evaluated with GPT2-XL; its parameters are trained by gradient descent.Source: §3.1.2 · p. 6
• The method improves performance across tasks while avoiding the extra inference cost of adding auxiliary examples as demonstrations.The authors report particularly strong gains on SST-2 and EmoC and attribute the shorter context to nearly unchanged inference speed.Source: §3.1.3 · p. 6, §3.1.2 · p. 6

3.2 Context Compression

• Hiddenanchor compresses demonstrations into formatting and label-word hidden states, preserving aggregated information while reducing the input context.Label-word states can be cached because their aggregation depends only on preceding tokens, but formatting states are also needed to determine the output space.Source: §3.2 · p. 6, §3.2.1 · p. 7, §3.2.2 · p. 7
• Hiddenanchor performs best among the compression methods on all metrics for both GPT2-XL and GPT-J.The comparison includes Textanchor, Hiddenrandom, and Hiddenrandom-top, evaluated using accuracy, label loyalty, and word loyalty.Source: §3.2.3 · p. 7, §3.2.2 · p. 7
• Hiddenanchor yields speed-up ratios from 1.1× to 2.9×, with larger acceleration reported for GPT-J than GPT2-XL.For GPT-J, anchor-state compression produces a 1.5 accuracy-point drop relative to the uncompressed setting.Source: §3.2.3 · pp. 7–8

3.3 Error Analysis

• Anchor distances support an error-analysis framework because confusing categories tend to have similar label-anchor representations.Confusionpred identifies Description-Entity as the most confusing case and performs reasonably on Entity-Abbreviation and Description-Abbreviation.Source: §3.3.3 · p. 8, §3.3 · p. 8

Input and Demonstration Factors

• One research stream studies input factors such as demonstration order, formatting, and example selection, along with construction and calibration strategies.These approaches target performance changes caused by perturbing or designing the demonstrations.Source: §4 · pp. 8–9

Mechanistic Perspectives

• A second stream explains ICL through conceptual analogies, including gradient descent and Bayesian inference.The paper positions its information-flow perspective as distinct from these existing lenses.Source: §4 · p. 9

This Paper’s Perspective

• The paper investigates information flow in language models as a new perspective for understanding ICL and improving its effectiveness, efficiency, and interpretability.This framing connects mechanism analysis with practical ICL enhancements.Source: §4 · p. 9

• Label words serve as anchors that aggregate and distribute task-relevant information in ICL.Source: §5 · p. 9
• The paper proposes anchor re-weighting, demonstration compression, and an analysis framework for diagnosing ICL errors.Source: §5 · p. 9
• The study covers classification tasks and conventional ICL, leaving generative tasks and paradigms such as chain-of-thought prompting unexplored.Source: Limitations · p. 9
• The experiments mainly investigate models up to 6 billion parameters because of hardware constraints.Source: Limitations · p. 9

• The study uses GPT2-XL with 1.5B parameters and GPT-J with 6B parameters.Source: A Experimental Settings · p. 11
• The datasets are SST-2 for sentiment analysis, TREC for question type classification, AGNews for topic classification, and EmoContext for emotion classification.Source: A Experimental Settings · p. 11

• In shallow layers, Swp, the text-to-label information flow, is prominent while Spq, the label-to-target flow, is less significant.Source: B Results of Swp, Spq, and Sww on TREC… · p. 11
• In deeper layers, Spq dominates the information flow.Source: B Results of Swp, Spq, and Sww on TREC… · p. 11
• Swp and Spq generally exceed Sww, indicating that interactions involving label words predominate.Source: B Results of Swp, Spq, and Sww on TREC… · p. 11
• Word loyalty is used alongside label loyalty because label loyalty alone may miss changes in non-label-word probabilities and label-word probabilities relative to the vocabulary.Source: C Reason for Using Word Loyalty… · p. 11

D Isolating Different Numbers of Layers

• Isolating shallow layers causes a significant impact, whereas isolating deep layers has negligible impact even as the number of isolated layers increases.Source: D Isolating Different Numbers of Layers · p. 11
• Figure 8 compares label and word loyalty when label or non-label words are isolated in the first several versus last layers.Source: Figure 8 · p. 12
• These results further illustrate the important role of information aggregation via label words in shallow layers.Source: D Isolating Different Numbers of Layers · p. 12
• The analysis uses attention distributions on label words at the target position as a classifier prediction and quantifies its correlation with model outputs using AUC-ROC.Source: E Details for the Calculation of AUCROCl · p. 12
• With two demonstrations per class, the results for SST-2, TREC, AGNews, and EmoC are highly similar to results using one demonstration per class.Source: §F.1 · p. 12

• Using LLaMA-33B on SST-2, the experiment replaces labels with “A”/“B” and uses eight shots per class, achieving 83.0% accuracy.Source: §F.2 · p. 12
• Isolating label words in the first five layers has a stronger effect than isolating them in the last five layers or isolating non-label words.Source: §F.2 · p. 12
• The model leverages information from anchors in deeper layers to perform classification.Source: §F.2 · p. 13

• Anchor re-weighting modifies each attention head’s attention matrix before subsequent computations.Source: G Implementation of Anchor Re-weighting · p. 13
• The modification applies a learned weighting to attention from the target query position to selected demonstration label-word positions, while leaving other entries unchanged.Source: G Implementation of Anchor Re-weighting · p. 13
• Isolating label words within the first five layers has a more pronounced effect than isolating them within the last five layers.Source: Figure 12 · p. 14

• Anchor re-weighting optimization uses Adam with learning rate 0.01, β1 = 0.9, β2 = 0.999, batch size 1, and 10 epochs.Source: H Training Settings of Anchor… · p. 14
• Because of computational-resource limitations, the anchor re-weighting evaluation is restricted to GPT2-XL and excludes GPT-J.Source: H Training Settings of Anchor… · p. 14
• Acceleration ratios correlate with the ratio of total demonstration length Ldemo to predicted-text length Lx.Source: I The Factor of Ldemo and Lx · p. 14
• Datasets with longer demonstrations tend to have higher acceleration ratios; AGNews has the longest Ldemo and the highest acceleration ratio among the analyzed datasets.Source: I The Factor of Ldemo and Lx · p. 14
• For predicted confusion, query vectors are centered, their largest variation directions are selected, and key vectors are projected onto those directions.Source: J Calculation of ˆk · p. 15
• The analysis uses M = 10 because predicted confusion matrices show similar outcomes across tested M values.Source: J Calculation of ˆk · p. 15, Figure 14 · p. 16
• Confusionij is computed with AUC-ROC and approaches 1 when confusion between categories is less pronounced.Source: K Calculation of Confusionij · p. 15
• Five random seeds—42, 43, 44, 45, and 46—are fixed to support reproducibility across devices.Source: L Reproducibility · p. 15