PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering Uncontrolled Repetition through Residual Stream Dynamics

Yuanhe Zhang, Xinyao Zhou, Haoran Gao, Yuyao Zhang, Zhenhong Zhou, Fanyu Meng, Li Sun, Sen Su

arXiv:2609.38802v1cs.CLcs.AI

TL;DR

Uncontrolled repetition can raise inference cost, yet how it develops before becoming prominent in later layers is insufficiently understood. TRC compares residual contributions across generated positions to localize shallow repetition signals; it reduces repetition by 57% on average and generalizes to LLMs and LRMs.

  • Problem

    How uncontrolled repetition emerges across layers during generation is insufficiently understood, despite its potential to increase inference cost and latency.

    Source: §1 · p. 1

  • Method

    Tokenwise Residual Comparison (TRC) compares residual-stream contributions across generated positions, identifies candidate layers using repetition signals and benign references, and selectively suppresses anomalous contributions.

    Source: §1 · p. 2

  • Results

    TRC reduces repetition by 57% on average and also mitigates repetitive generation while identifying corresponding early-layer changes in LLMs and LRMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Shallow residual changes provide targets for reducing repetitive generation while largely preserving benign task performance across LVLMs, LLMs, and LRMs.

    Source: §5 · p. 10

    The defense assumes backbone access and calibration on attack-induced repetitions plus separate benign references; evaluation uses existing models and benchmark data.

    Source: §3.1 · p. 3, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Uncontrolled repetition can prolong autoregressive generation in large language models (LLMs) and enable resource consumption attacks. Prior analyses of repetitive generation have identified strongly activated features in intermediate and late layers. However, how uncontrolled repetition activity emerges and develops before becoming prominent in these layers remains insufficiently understood. In this paper, we investigate this question primarily in large vision-language models (LVLMs), which support a richer set of uncontrolled repetitions through both visual and textual inputs. We propose Tokenwise Residual Comparison (TRC), a method that identifies and localizes anomalies associated with repetition from residual dynamics during generation. TRC compares attention and multilayer perceptron writes to the residual stream across generated tokens to identify patterns associated with repetition. It then selectively suppresses coordinates in the residual stream at the identified layer. Experiments show that TRC effectively mitigates uncontrolled repetition, reducing loop rates by 57\% on average. Our analysis further shows that repetition semantics emerge in shallow layers and propagate through the residual stream, disrupting normal representations. TRC also generalizes to large language models (LLMs) and large reasoning models (LRMs), where it consistently captures analogous repetition dynamics and achieves effective mitigation. Our work broadens the study of repetitive generation from its prominent internal representations to earlier opportunities for intervention, providing insights for mitigating resource consumption attacks.

1 INTRODUCTION

The paper examines how uncontrolled repetition emerges before becoming prominent in deeper layers, focusing on LVLMs that can be triggered through visual and textual inputs. It introduces TRC to localize shallow residual signals and suppress repetition, reporting reduced repetition and generalization across model families.

1 INTRODUCTION

  • Prior analyses locate strongly activated repetition features mainly in intermediate and later layers, leaving their emergence across layers during generation less understood.Existing activation interventions target representations after repetitive semantics become strongly activated.Source: §1 · pp. 1–2
  • TRC compares residual stream contributions across generated tokens to identify repetition signals and selectively suppresses anomalous contributions at candidate layers and coordinates.Candidate layers are selected using signal magnitude relative to benign references and consistency across layers.Source: §1 · p. 2
  • The study focuses on naturally emerging LVLM repetition under visual and textual triggers rather than artificially constructed repetitive outputs.Visual inputs provide a way to study a broader range of model-generated failure cases and activity before it becomes prominent in deeper layers.Source: §1 · pp. 1–2
  • Suppressing residual contributions at the layer with the minimum TRC score reduces repetition by 57% on average.TRC also identifies shallow-layer changes and generalizes to LLMs and LRMs.Source: §1 · p. 2

2 RELATED WORK

Prior adversarial-example research spans discrete textual perturbations, continuous visual and speech inputs, and coordinated multimodal attacks. Work on resource consumption describes ways to prolong autoregressive computation, existing defenses, and the limited insight prior internal analyses provide into when repetition first emerges.

2.1 INPUT MODALITIES AND ADVERSARIAL EXAMPLE CONSTRUCTION

  • Visual optimization and typographic prompts can bypass textual safety alignment, and visual optimization can also prolong generation in LVLMs.These perturbation spaces support diverse visual and textual triggers for studying model-generated repetition.Source: §2.1 · p. 2

2.2 RESOURCE CONSUMPTION ATTACKS AND DEFENSES

  • Resource-consumption attacks use varied mechanisms, while existing defenses include token budgets, context filtering, decoding strategies, and training objectives.Examples include energy- and latency-increasing sponge inputs and attacks that undermine early-exit efficiency.Source: §2.2 · pp. 2–3
  • Figure 1 depicts repetition across model families, tokenwise residual comparison and selective intervention, and mitigation outcomes.The overview places the comparison and intervention between the model-family examples and outcomes.Source: Figure 1 · p. 3
  • Prior internal analyses capture prominent repetition activations but provide limited insight into when repetition first emerges during generation.TRC instead tracks evolving semantic changes in residual contributions across the generated sequence.Source: §2.2 · p. 3

3 TOKENWISE RESIDUAL COMPARISON

TRC compares residual-stream contributions across generated tokens to identify repetition signals, localize a layer for intervention, and selectively suppress associated coordinates. Its scoring compares attack-induced changes with benign references and favors layers with low relative magnitude and stable cross-layer variation.

3.1 PRELIMINARIES AND PROBLEM SETUP

  • TRC records attention or MLP residual contributions during attack-induced repetition and benign generation to identify changes associated with repetition while preserving benign generation.The formulation analyzes attention and MLP modules independently and also applies to text-only LLMs and LRMs.Source: §3.1 · pp. 3–4

3.2 TRC SCORES AND LAYER LOCALIZATION

  • TRC finds the smallest repeated n-gram whose occurrences cover more than half of the output, then uses its positions to set token comparisons.If no such n-gram is defined, the comparison begins at position 1 with spacing 1.Source: §3.2 · p. 4
  • At each layer, TRC aligns contributions for paired positions, computes element-wise absolute differences, and averages across token positions and comparison pairs.Sample-level vectors are aggregated across attack calibration samples to produce a score for each hidden coordinate.Source: §3.2 · p. 4
  • Layer localization combines attack-change magnitude relative to benign generation with cross-layer variation normalized by a benign reference curve.Attention and MLP modules are calibrated separately within each model.Source: §3.2 · pp. 4–5
  • The intervention layer minimizes the localization score, with ties resolved in favor of the shallowest layer.The selected localization window is specified by its starting layer and controllable span.Source: §3.2 · p. 5

3.3 SELECTIVE RESIDUAL-STREAM INTERVENTION

  • At the selected layer, TRC selects the coordinates with the largest attack scores and applies a fixed soft mask during residual addition.The mask scales selected rows; after calibration, the selected layer, coordinates, and mask are fixed for evaluation, and other layers remain unchanged.Source: §3.3 · p. 5

4 EXPERIMENTS

Across LVLM, LLM, and LRM evaluations, TRC suppresses uncontrolled repetition while largely preserving benign performance. Mechanistic analyses localize repetition-related changes to shallow layers and show that targeted suppression and restoration of early residual coordinates can affect repetitive generation.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans LVLM, LLM, and LRM families, visual and textual attacks, and benign tasks, using generation length, task accuracy, and loop rate as metrics.The attack suite includes visual perturbations and textual induction; benign tasks include ScienceQA, TextVQA, and MMLU.Source: §4.1 · pp. 5–6

4.2 MITIGATION EFFECTIVENESS AND GENERALIZATION

  • TRC reduces generation length and loop rate in most attack settings, though its numerical efficiency is weaker than no-repeat; benign task accuracy changes little.Table 1 reports defense results on three LVLMs, and Table 2 reports benign task accuracy.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 6
  • Across LLMs and LRMs, TRC reduces generation length and loop rate when attacks succeed, especially on LRMs, with little change in benign MMLU accuracy.The results also show more stable intervention effects on newer and larger models.Source: §4.2 · p. 6, Figure 2 · p. 7, Table 3 · p. 7

4.3 MECHANISTIC ANALYSIS

  • Across three model families, TRC localizes attack signals to layers 0–3, while benign references localize to layers 9–18; Attention localization is at layer 1 in nine of ten conditions.The signal’s early profile is distinct from the cross-layer rankings for benign repetition and ordinary requests.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Benign repetition follows the localization profile of normal requests, unlike uncontrolled repetition, whose candidate locations rank higher early and lower at greater depths.This supports specificity to uncontrolled repetition rather than repeated tokens or semantics alone.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Full Attention-write suppression offers a better mitigation–utility tradeoff than suppressing individual heads, while MLP-only or joint suppression degrades benign tasks more.Layer 1 suppresses all repetitive failures while retaining 81.5% ScienceQA accuracy; deeper suppression causes larger accuracy degradation.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • Attention-update gaps remain within normal fluctuation bounds in the first two blocks, consistent with early repetition features being retained through the residual stream rather than repeatedly amplified.For L2/RMS, the gaps reach 23.4%/17.2% of the threshold in Block 0 and 73.0%/69.2% in Block 1.Source: Table 4 · p. 8, §4.3 · p. 9
  • Restoring first-layer TRC-localized coordinates converts 80% of repetitive attack trajectories to non-repetitive outputs and reduces average generation length from 4096 to 828 tokens.Random-coordinate restoration performs substantially worse; injecting the attack-associated difference disrupts normal generation but does not reproduce repetition.Source: §4.3 · p. 9, Table 5 · p. 9

5 CONCLUSION

The conclusion presents TRC as a way to track repetition signals in residual contributions across generation positions. It reports shallow-layer signals and targeted suppression across LVLMs, LLMs, and LRMs, with analyses supporting residual propagation as a plausible mechanism.

5 CONCLUSION

  • TRC identifies and mitigates uncontrolled repetition by comparing residual stream contributions across generation positions.It tracks fine-grained repetition signals at specific layers and coordinates.Source: §5 · pp. 9–10
  • Across LVLMs, LLMs, and LRMs, TRC identifies shallow repetition signals and enables targeted suppression that reduces repetitive generation while largely preserving benign task performance.Source: §5 · p. 10
  • Mechanistic analyses find that repetition signals emerge early, remain distinguishable from benign and legitimate repetition, and can be partially restored through localized activation replacement.These findings support residual propagation as a plausible mechanism for repetition-related features persisting through the network.Source: §5 · p. 10
  • The paper shifts analysis from where strong repetition representations appear to how repetition emerges early and becomes actionable.Source: §5 · p. 10

AI USE STATEMENT

The authors used generative AI tools for literature retrieval, manuscript formatting, and caption and editorial revisions. They reviewed the assisted outputs and retain responsibility for the final text, data, results, and claims.

AI USE STATEMENT

ETHICS STATEMENT

The ethics statement describes a study of repetition-inducing attacks and a defense using existing model and benchmark data. It notes possible misuse of attack procedures and reports defensive results alongside benign-task performance to show the trade-off.

ETHICS STATEMENT

  • The study evaluates repetition-inducing attacks and a defense using existing model and benchmark data.Source: ETHICS STATEMENT · p. 10
  • Reported attack procedures may be misused to increase inference costs or disrupt model services.Source: ETHICS STATEMENT · p. 10
  • The authors report defensive results alongside benign-task performance to make the trade-off visible.Source: ETHICS STATEMENT · p. 10

A MODEL AND ATTACK COVERAGE

The appendix describes model-specific attack pools, evaluation coverage, and benign tasks. Attack examples are screened on target models, while each model's TRC configuration is localized once and applied to held-out attack pools.

A MODEL AND ATTACK COVERAGE

  • Attack candidates are retained only when greedy decoding produces an uncontrolled one- or two-token cycle for at least ten consecutive repetitions.A dash in Table 6 marks a condition outside the evaluated scope, not a failed attack.Source: A MODEL AND ATTACK COVERAGE · p. 15, Table 6 · p. 15
  • TRC localization uses disjoint training attacks and is performed once per target model before applying that configuration to held-out attack pools.RECITE is used for LVLMs, while GCG is used for LLMs and LRMs.Source: A MODEL AND ATTACK COVERAGE · p. 15
  • The main experiments use suppression strength of approximately α_l⋆ = 0.73, direction-selection ratio ρ = 5%, and layer-window span h = 3.Source: A MODEL AND ATTACK COVERAGE · p. 15
  • RECITE constructs multimodal attacks by perturbing images while keeping each associated text request fixed, then retaining repetitions that meet the cycle criterion.Source: A MODEL AND ATTACK COVERAGE · p. 15
  • GCG and LoopLLM build model-specific request-suffix pairs, optimize suffixes toward repeated continuations, and screen them on the target checkpoint.LoopLLM optimizes for probability mass on a short cyclic token pattern.Source: A MODEL AND ATTACK COVERAGE · p. 15
  • Direct attacks prepend a repeated token sequence to the assistant-side generation prefix, while benign utility tests use ScienceQA and TextVQA for LVLMs and MMLU for LLMs and LRMs.Each benign dataset uses 200 samples; benign inputs receive no attack suffix or repetition prefix.Source: A MODEL AND ATTACK COVERAGE · p. 16

B DISTINCTION FROM EXISTING INTERPRETABILITY METHODS

TRC distinguishes itself by tracking cycle-aligned changes in native residual contributions, localizing shallow intervention sites, and applying the same analysis across model modalities. The legitimate-repetition controls pair an image-grounded description with a bounded, specified repetition.

B DISTINCTION FROM EXISTING INTERPRETABILITY METHODS

  • TRC measures cycle-aligned changes in native attention and MLP contributions, unlike approaches focused on activation magnitude, attribution, recoverable content, or learned features.The comparison is methodological, not a shared numerical benchmark; the listed approaches address complementary questions.Source: B DISTINCTION FROM EXISTING… · pp. 16–17, Table 7 · p. 16
  • TRC localizes layers and residual coordinates at module outputs before residual addition, with experiments locating repetition-associated low-change regimes in shallow layers.This location is supported for uncontrolled repetition in the evaluated settings, not asserted as universally optimal for other behaviors.Source: B DISTINCTION FROM EXISTING… · pp. 16–17
  • The same TRC score, layer-selection rule, and soft-mask construction apply to visually induced LVLM repetition and textual repetition in LLMs and LRMs.The shared interface operates on the Transformer backbone’s residual pathway after modality-specific inputs enter the autoregressive decoder.Source: B DISTINCTION FROM EXISTING… · p. 17

C CONSTRUCTION OF LEGITIMATE-REPETITION CONTROLS

  • All five constructed controls require an image-grounded description followed by ten contiguous copies of a designated token, preserving bounded, task-compliant repetition.Items were accepted only when the semantic prefix met the stated image-grounding and separation criteria and the output ended with ten copies of the same token ID.Source: C CONSTRUCTION OF… · p. 17

D LAYER-WISE TRC-L PROFILES UNDER REPETITION

TRC-l profiles place repetitive failures’ low-change minima in shallow layers, unlike the middle-layer minima for normal requests. Failure curves generally fall below normal curves, but that scale difference alone does not establish semantic similarity as its unique cause.

D LAYER-WISE TRC-L PROFILES UNDER REPETITION

  • Failure curves generally lie below normal curves, but the score gap does not establish semantic similarity as its unique cause.TRC-l includes cross-layer variation, and comparison positions depend on the detected repetition unit; Figure 5 shows the layer-wise profiles for attention and MLP across the tested spans.Source: D LAYER-WISE TRC-L PROFILES UNDER… · pp. 17–19, Figure 5 · p. 18
  • Normal-request minima fall in middle layers: Attention selects layers 19, 11, and 10, while MLP selects layers 12, 11, and 14 for h = 2, 3, and 4, respectively.The passage cautions that this does not establish a universal reasoning layer or unchanged normal representations between tokens.Source: D LAYER-WISE TRC-L PROFILES UNDER… · p. 19
  • All three attacks select Attention layer 1 at every tested span; in MLP, Direct and RECITE select layer 1 while GCG selects layer 0.The attack minima remain separated from normal requests for each tested window span.Source: D LAYER-WISE TRC-L PROFILES UNDER… · p. 19

E QUALITATIVE CROSS-INTERVENTION EXAMPLES

The paired examples contrast restoring localized coordinates on an attack trajectory with injecting attack differences into a normal trajectory. The outcomes distinguish recovery of a short normal response from disruption that does not recreate repetition.

E QUALITATIVE CROSS-INTERVENTION EXAMPLES

  • Restoring localized coordinates changed the attack from a repetitive 2,048-token output to an 11-token non-repetitive response, while equal-size random replacement left the loop intact.The normal residual replacement was applied to the localized coordinates.Source: E QUALITATIVE CROSS-INTERVENTION… · p. 19
  • Adding attack-minus-normal differences at localized coordinates terminated a normal response after two tokens without recreating repetition; matched random coordinates left the 11-token response unchanged.The localized perturbation disrupted the normal response, whereas the random-coordinate perturbation matched the normal baseline.Source: E QUALITATIVE CROSS-INTERVENTION… · p. 19

F.1 SENSITIVITY TO THE LAYER-WINDOW SPAN

The sensitivity and efficiency results show how window span, suppression ratio, repetition distance, and deployment relate to localization, attack outcomes, benign utility, and generation cost.

F.1 SENSITIVITY TO THE LAYER-WINDOW SPAN

  • Selected locations were invariant across h = 2, 3, and 4, preserving GCG’s attention-layer-1 versus MLP-layer-0 difference and layer-1 selections for RECITE and LoopLLM.Table 9 reports the Top-1 start layer separately for attention and MLP; layers are zero-indexed.Source: §F.1 · p. 20, Table 9 · p. 20

F.2 SENSITIVITY TO THE SUPPRESSION RATIO ON ATTACKS

  • Every tested suppression ratio lowered macro-average generation length and repetition rate versus no defense, but gains were non-monotonic; ρ = 20% gave the shortest output at 620.56 tokens, while ρ = 1% and 20% tied at 21.3% repetition.RECITE was strongly mitigated at every evaluated ratio, while GCG benefited most at ρ = 20%.Source: §F.2 · p. 20, Table 10 · p. 21

F.3 EFFECT OF THE SUPPRESSION RATIO ON BENIGN UTILITY

  • Across the tested suppression-ratio range, task accuracy dropped by at most 1.5 percentage points and average response length varied by no more than 0.1 tokens.The accuracy comparison is relative to the unsuppressed model.Source: §F.3 · p. 20, Table 11 · p. 21

F.4 SENSITIVITY TO THE MAXIMUM REPETITION DISTANCE

  • RECITE remained at layer 1 across tested qmax values, while GCG shifted from layer 12 at qmax = 1 to layer 1 for every qmax ≥2.The analysis attributes the one-token limitation to possible multi-subword repetition units and finds a short multi-token range sufficient in this experiment.Source: §F.4 · p. 20, Table 12 · p. 21

G ONLINE EFFICIENCY

  • TRC reduced attack-generation time by 90.9%–99.6% across listed LLM conditions, while throughput changed by −3.00 tokens/s for Qwen2.5-3B and +1.05 tokens/s for Llama-3.2-3B.The paper describes the throughput variations as small and generation-time reductions as resulting from earlier termination of attack-induced loops.Source: G ONLINE EFFICIENCY · p. 21, Table 13 · p. 22, Table 14 · p. 22

Related papers

Loading 2609.38802v1…