How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models

Dun Li Chan; Emily Liu; Niyathi Allu; Christian Hoang

Source-linked AI summary

How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models

Dun Li Chan, Emily Liu, Niyathi Allu, Christian Hoang

arXiv:2609.03322v1Sep 3, 2026cs.CLstat.ML

TL;DR

Language-model robustness is often judged from outputs alone, leaving unclear how perturbations alter internal computation. This paper compares six perturbation types across behavior, hidden-state geometry, and attention-head function, finding that metric profiles and cross-model effects vary, while convergent metrics provide stronger evidence of genuine disruption.

Problem
Output degradation alone does not explain how perturbations change model computation, and similar behavioral effects may reflect different internal disruptions.
Method
The study compares six perturbation types using behavioral measures, representation geometry, and attention-head responses across decoder-only language models.
Results
Perturbation signatures are distinguishable but only partly conserved across model scales and families, while adversarial perturbations exceed rate-matched random controls across multiple behavioral and internal metrics.
Takeaways & Limitations
Robustness claims based on a single metric should be corroborated across behavioral and internal measures, with metric convergence offering stronger evidence of genuine disruption.
Takeaways & Limitations
CKA is restricted when retokenization prevents position-wise alignment, attention-head findings establish association rather than causality, and architectural explanations for cross-family differences remain unidentified.

Abstract

from arXiv · show

Language models encounter typos, corrupted text, altered words, and disrupted token order, yet robustness is usually evaluated only through output behavior. We study how six naturalistic and synthetic input perturbations propagate through decoder-only language models at three levels: output behavior, hidden-state geometry, and attention-head function. We evaluate behavioral effects across four GPT-2 and two Qwen2.5 checkpoints by analyzing layerwise geometry using centered kernel alignment and intrinsic dimension, and examine attention-head responses in GPT-2. Perturbation types produce distinguishable metric profiles that are not fully captured by output measures and are only partly consistent across the tested checkpoints. Copying scores are especially associated with activation-patching recovery under token substitution and shuffling. Gradient-guided HotFlip perturbations also cause stronger behavioral and representational disruption than rate-matched random token substitutions in GPT-2; their behavioral effects are consistent across all six tested checkpoints. Our results show that robustness claims based on a single behavioral or representational metric can be misleading, and motivate multi-level evaluation of how perturbations alter language-model computation.

1 Introduction

This study asks how input perturbations alter language-model computation beyond output behavior. It compares six perturbation types across decoder-only models using behavioral, representational, and attention-head analyses, including matched adversarial controls.

Output degradation alone cannot reveal how perturbations change computation inside language models.
The analysis compares six naturalistic and synthetic perturbation types across GPT-2 scales and the Qwen2.5 family.
The study jointly measures output behavior, hidden-state geometry, and attention-head responses.
Attention-based measurements and activation patching test whether perturbation effects are associated with functionally characterized attention heads.
HotFlip attacks are compared with rate-matched random token substitutions to separate adversarial optimization effects from edit rate.

2 Background and Related Work

Prior work documents substantial robustness failures from naturalistic and adversarial input changes, while representation metrics and attention-head functions provide complementary ways to study internal effects. This paper combines these perspectives and constrains activation-patching analyses to token-count-preserving perturbations.

Naturalistic text corruptions and adversarial edits can substantially degrade model performance despite preserving much of the meaning for human readers.
CKA is used as one diagnostic among several because it can respond to outliers or transformations without meaningful functional differences.
Intrinsic-dimension analysis estimates the lower-dimensional structure underlying high-dimensional learned representations.
The study tests whether established attention-head functions relate to responses across naturalistic and synthetic input corruptions.
Activation patching conclusions depend on corruption, metric, and alignment, so recovery is interpreted as functional association within this experimental setting.

3 Methods

The study evaluates how varied input perturbations affect decoder-only language models across output behavior, hidden-state geometry, and attention-head function. It combines behavioral metrics, representation analyses, head-function measures, and activation patching, including a rate-matched HotFlip comparison.

Evaluation setup: 300 WikiText-2 sequences of length at least 128 are evaluated with a fixed seed across six checkpoints.The checkpoints include four GPT-2 models and two Qwen2.5 models; behavioral comparisons use all six, while attention-head analysis and full adversarial representation comparison use GPT-2.
Inputs and perturbations: Six perturbation types span character substitution, keyboard typos, random token substitution, token shuffling, word substitution, and synonym substitution.Character, typo, word, synonym, and token perturbations may alter tokenization, whereas shuffling preserves token identity while disrupting local order and syntax.
Behavioral measurements: Output effects are measured with negative log-likelihood and normalized Levenshtein output divergence.Output divergence compares generated clean and perturbed outputs.
Representational measurements: Layerwise representation change is measured with CKA and intrinsic-dimension shifts between clean and perturbed hidden states.Layers 1 through n_layer are evaluated; layer 0 is excluded because it directly reflects the changed input embedding. For retokenizing perturbations, CKA combines retokenization and representation-change effects.
Head-level measurements: Attention-head analysis uses previous-token, duplicate, induction, and copying scores, plus entropy changes and clean activation-patching recovery.Patching replaces a perturbed head activation with its clean counterpart and is restricted to token substitution and shuffling because positional correspondence is required.
Adversarial comparison: A HotFlip attack selects positions randomly, ranks 50 vocabulary candidates by first-order gradient loss increase, and commits replacements only when they increase NLL.Each accepted adversarial substitution is paired with a random-token control at the same position, and both are compared using NLL, output divergence, CKA, and intrinsic-dimension change.

4 Experiments

Across GPT-2 and Qwen2.5, perturbations produce distinct behavioral, representational, and attention-related profiles, with some patterns varying across model families. HotFlip produces stronger behavioral and representational disruption than matched random substitutions, while copying scores align with patching recovery for selected perturbations.

Output behavior: Character substitution produces the largest output divergence, while keyboard typos are less disruptive; token and word substitutions have similar divergence but different NLL.These results show that generated-text change and predictive confidence need not agree.
Representation geometry: CKA generally decreases as perturbation strength increases, with character and typo corruption showing the lowest similarity and shuffling retaining comparatively high similarity.TwoNN further distinguishes perturbations: character and typo corruption lower early-layer intrinsic dimension at higher rates, whereas token substitution produces the largest positive change.
Attention-head responses: Induction score shows the largest-magnitude negative correlation with entropy change for most perturbations, ranging from −0.27 to −0.609.Under shuffling, previous-token score has the strongest positive correlation at 0.670, while duplicate and induction scores are strongly negative.
Attention-head responses: Copying score is most strongly associated with activation-patching recovery for token substitution and shuffling, with patched NLL-recovery correlations of 0.726 and 0.622.The corresponding output-divergence correlations are 0.652 and 0.348, while other head-function scores are smaller or inconsistent.
Cross-model generalization: Cross-model intrinsic-dimension patterns differ: Qwen models show larger early-layer increases for character and typo substitution, whereas GPT models show larger increases for shuffling and word-level perturbations.For token substitution, GPT models also show larger intrinsic-dimension changes than Qwen models, although raw family NLL differences are descriptive because clean baselines differ by 0.592.
Adversarial perturbations: HotFlip yields higher NLL and output divergence, lower CKA, and larger intrinsic-dimension increases than matched random substitutions in GPT-2 at 30%.Its higher mean NLL and output divergence relative to the matched control persist across all six tested checkpoints at 5% and 30%, while internal comparisons remain GPT-2-specific.

5 Discussion

The discussion finds that perturbation effects can dissociate across behavioral and internal metrics, vary by model family, and require convergent evidence for robust interpretation. Adversarial perturbations provide a contrasting case where metrics agree more consistently.

Behavioral and internal metrics can disagree about representational disruption, making single-metric robustness assessments incomplete.
Token substitution can match word substitution in output divergence while producing substantially higher NLL.
Copying-head correlations with attention-entropy shifts are weak, whereas copying heads dominate activation-patching recovery.
Token substitution appears architecture-agnostic in output divergence but differs across GPT-2 and Qwen2.5 in NLL and intrinsic dimension.
HotFlip exceeds rate-matched random substitution across NLL, output divergence, CKA, and intrinsic dimension in GPT-2 at 30% perturbation.
CKA alignment constraints, non-causal attention-head evidence, untested architectural explanations, and reliance on WikiText-2 and two model families limit generality.
Metric convergence may better distinguish robust perturbation effects from artifacts than any single metric’s value.

6 Conclusion

The paper shows that perturbation types leave distinguishable signatures across behavior, internal geometry, and circuit-level function, but these signatures are only partly conserved across model scales and families. It therefore calls for corroborating evidence and further mechanistic testing before treating single-metric robustness claims as general findings.

Perturbation types produce distinguishable signatures in behavior, internal geometry, and circuit-level function.
These signatures are only partially conserved across model scale and family, making the measurement metric a potential confounding factor.
GPT-2 and Qwen2.5 diverge under some perturbations, but the study does not identify a mechanistic basis for those differences.
Robustness claims grounded in a single metric warrant corroborating evidence before being treated as general findings.

A.1 Output Metrics

The output metrics quantify predictive confidence and generated-text change under clean and perturbed inputs. NLL measures continuation likelihood, while output divergence measures normalized edit distance between clean and perturbed generations.

Sequence-level NLL measures how well the model predicts the next token under a clean or perturbed input sequence.
Lower NLL indicates that the model assigns higher probability to the observed continuation.
Output divergence is computed as normalized Levenshtein distance between clean and perturbed generated sequences.
Output divergence measures how much the generated continuation changes under perturbation.

A.3.2 Duplicate Score

The duplicate score isolates duplicate-detection attention using repeated randomized-token probes, then compares attention between repeated positions and related earlier-sequence positions.

A.3.2 Duplicate Score: Repeated randomized sequences isolate duplicate detection from previous-token effects and general semantic associations.The probe repeats a randomized sequence of L tokens twice.
A.3.2 Duplicate Score: A related probe examines attention from repeated-sequence positions to positions immediately following their earlier counterparts.This reuses the repeated random-token probe while changing the compared positions.

A.4 Attention Entropy Norm Delta

Attention entropy norm delta compares clean and perturbed attention distributions under 30% perturbation, using normalized entropy across positions except the first.

A.4 Attention Entropy Norm Delta: Clean and perturbed inputs produce attention matrices α = h(x) and α′ = h(x′) under 30% perturbation for every perturbation type.The comparison is defined between corresponding clean and perturbed attention matrices.
A.4 Attention Entropy Norm Delta: Normalized entropy excludes the first position because it has no target attention distribution.Entropy is evaluated over the remaining positions.
A.4 Attention Entropy Norm Delta: Normalization divides each position’s total entropy by the maximum possible entropy to place values on a comparable scale.The resulting normalized values support the entropy-norm delta calculation across inputs.

B Adversarial Shortlist Selection

The adversarial shortlist uses 50 candidates as a practical balance between HotFlip attack strength and computational cost, rather than maximizing loss.

B Adversarial Shortlist Selection: Mean NLL rises from 8.91 at |S|=1 to 11.65 at |S|=100 in the GPT-2, p = 30% ablation.The ablation evaluates shortlist sizes |S| ∈ {1, 5, 10, 20, 50, 100}.
B Adversarial Shortlist Selection: The marginal gain per additional candidate declines from 0.22 between |S|=1–5 to 0.01 between |S|=50–100.This diminishing return motivates avoiding the largest shortlist.
B Adversarial Shortlist Selection: The study fixes |S| = 50 as a practical balance between attack strength and compute cost, not as a loss-maximizing choice.The selected shortlist size reflects an efficiency trade-off.