Source-linked AI summary
Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective
Qiyan Zhao, Xiaofeng Zhang, Shuochen Chang, Qianyu Chen, Xiaosong Yuan, Xuhang Chen, Luoqi Liu, Jiajun Zhang, Xu-Yao Zhang, Da-Han Wang
TL;DR
Cache mechanisms used to reduce dMLLM inference latency can cause repetitive generation, termed the Repeat Curse. The paper analyzes this failure through information flow and introduces CoTA, which preserves context-token patterns and penalizes uncertain decoding. CoTA reduces adjacent repetition by up to 92% and consistently improves performance across multimodal benchmarks.
Problem
Caching accelerates dMLLM inference but often introduces repetitive text that reduces output performance and readability.
Method
The paper analyzes context-token information flow and proposes CoTA, combining attention enhancement with entropy-guided confidence penalties in a training-free plug-and-play method.
Results
CoTA reduces the adjacent repetition rate by up to 92% and consistently surpasses baseline models across several multimodal benchmarks.
Takeaways & Limitations
Context tokens act as anchors that aggregate information and guide predictions, while disrupted information flow and nonconvergent deep-layer entropy are linked to repetition.
Takeaways & Limitations
CoTA has not yet been validated across more open-source dMLLMs or base models of different scales.
Abstract
from arXiv · showhide
Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA
1 INTRODUCTION
Caching accelerates dMLLM inference but can trigger the Repeat Curse. The paper analyzes this failure through information flow and proposes CoTA to preserve context-token behavior and reduce repetition.
- Motivation: Caching reduces dMLLM inference latency but can introduce substantial repetitive text, termed the Repeat Curse.The repetition reduces output performance and readability and consistently appears when cache techniques are employed.
- Information-flow findings: Information-flow analysis shows that context tokens adjacent to the query act as anchors that progressively aggregate semantic information across layers.The model’s bidirectional attention increasingly concentrates on these context tokens.
- Information-flow findings: Context-token entropy converges in deeper layers as information aggregation increases the model’s predictive certainty.When repetition occurs after caching, attention allocation becomes random and some context-token entropy fails to converge.
- CoTA: CoTA combines distance-aware context-token attention enhancement with entropy-guided confidence penalties to mitigate uncertain and repetitive outputs.It is designed as a plug-and-play, training-free method compatible with baseline dMLLMs and existing caching strategies.
- Evaluation: CoTA is validated through extensive experiments that show consistent performance improvements across multiple general multimodal tasks.The contribution is presented as empirical validation of CoTA’s effectiveness beyond repetition mitigation.
2 RELATED WORK
Related work frames information flow as an interpretable way to study black-box model mechanisms and abnormal outputs. Prior analyses use several internal signals and identify anchor-token behavior in autoregressive models.
- Information Flow: Information-flow studies analyze black-box model mechanisms using saliency scores, attention maps, Grad-CAM, and massive values.These approaches provide interpretable links between internal mechanisms and model outputs.
- Information Flow: Prior work identifies anchor tokens in autoregressive models that aggregate information despite carrying low semantic content.The paper relates this prior observation to its analysis of context-token behavior in dMLLMs.
3 MOTIVATION AND ANALYSIS
This section explains how caching accelerates dMLLM inference but can cause the Repeat Curse, then analyzes the information flow of context tokens to identify its mechanism. Context tokens act as anchors, normally achieve deeper-layer entropy convergence, and exhibit disrupted information flow and failed convergence during repetition.
- Repeat Curse: The analysis introduces ARR, SRR, MRL, ARL, and 95pRL to quantify the frequency and severity of repetitive outputs.ARR measures the proportion of repeated adjacent tokens in a response sequence.
- Repeat Curse: Caching reduces dMLLM inference latency but can introduce severe token repetition that harms output performance and readability.The paper names this side effect the Repeat Curse.
- Information Flow: In dMLLMs, context tokens aggregate information across layers, absorb attention, and guide final predictions.Attention progressively concentrates on context tokens from shallow to deeper layers.
- Information Flow: Under normal decoding, context-token entropy remains high in shallow layers but gradually converges in deeper layers as predictions become more stable and certain.The analysis defines context tokens as target tokens plus their two nearest relative-position neighbors.
- Repeat Curse: Repetition is accompanied by randomized attention, disrupted context-token information flow, and failed entropy convergence in deeper layers.These patterns contrast with the stable information flow observed during normal decoding.
4 METHOD
CoTA is a training-free, plug-and-play method that preserves context-token information flow and penalizes decoding choices associated with uncertain context tokens. Its two components enhance attention using relative distance and incorporate deep-layer context-token entropy into voting scores.
- CoTA overview: CoTA combines Context Tokens Attention Enhancement with Context Tokens Entropy-Guided Voting to mitigate the Repeat Curse.CTAE preserves context-token information flow, while CTEV prevents outputs driven by uncertain context tokens during decoding.
- CTAE: CTAE applies a distance-aware attention intervention that strengthens attention to context tokens and preserves their native information flow pattern.Attention is modified by multiplying Attn_i,j by G_i,j, with the decay determined by relative query-key distance.
- CTAE: The CTAE decay uses a Gaussian distance term with temperature τ fixed to 5 and a lower bound γ_min for stabilization.The decay depends on |i−j| between query and key tokens and is applied across layers and heads.
- Synergistic workflow: Together, CTAE restores context-token attention patterns while CTEV penalizes voting choices associated with uncertain context tokens.Their complementary roles jointly alleviate the Repeat Curse.
- CTEV: CTEV aggregates deep-layer context-token entropy and adds it as a penalty to confidence scores used for candidate-token voting.The method targets the persistent high entropy observed for repetitive context tokens and defines layers 26–30 as deep layers; target tokens and their two nearest relative-position tokens form the context tokens.
5 EXPERIMENTS
Experiments evaluate CoTA on multimodal benchmarks and repetition-focused captioning settings. The method consistently improves cache-degraded outputs while adding limited computational overhead.
- Experimental setup: CoTA is evaluated on eight multimodal benchmarks, including DocVQA, ChartQA, MMStar, MME, Seed, LLaVAW, MathVista, and MMBench.The setup uses the LLaDA-V baseline with a LLaDA-8B-Instruct language tower, SigLIP2 vision tower, and two-layer MLP projector.
- Repeat Curse evaluation: 13.1 ARR points are recovered under long responses and 6.1 under short responses when CoTA mitigates cache-induced repetition.The captioning evaluation uses 500 COCO2014 samples and reports ARR, MRL, ARL, and SRR.
- Repeat Curse evaluation: Caching sharply worsens repetition, with output length 512 increasing ARR by 14.1 and SRR by 75.4 relative to the minimally repetitive baseline.Longer responses are more susceptible to repetition, and each component alone also mitigates repetition in ablations.
- Generalization: CoTA consistently lowers response repetition across open-domain natural and mathematical tasks while alleviating cache-related performance degradation.On MathVerse, it gains +6.7 in Score and +5.9 in ARR; on MathVista, it gains +1.8 in ACC and +5.6 in ARR.
6 CONCLUSION AND LIMITATIONS
The paper concludes that cache-induced repetition is associated with disrupted context-token information flow and persistently high deep-layer entropy, motivating CoTA. Its evidence remains limited to a narrow set of available dMLLMs and cache methods.
- Conclusion: Context tokens aggregate information and guide predictions, while caching disrupts this pattern and repetitive context tokens retain high deep-layer entropy.CoTA is proposed as a plug-and-play mitigation based on these information-flow findings.
- Limitations: CoTA has not yet been validated across more open-source dMLLMs or base models with different scales.The limitation reflects limited research on baseline dMLLMs.
- Limitations: CoTA cannot be tested on a wider range of cache approaches because cache methods for dMLLMs remain scarce.The authors identify broader cache-method evaluation as future work.
A ANALYSIS OF CACHE MECHANISM AND REPETITION CURSE
Caching output-token states, rather than prompt-token states, is associated with the Repeat Curse: fewer recomputed outputs, lower similarity thresholds, and reuse policies that affect suffix tokens increase repetition. Across models, prefix-only caching avoids this issue, while CoTA suppresses repetition in MMaDA.
- Prefix KV cache does not trigger repetition because it reuses cached states only for prefix tokens.
- Prompt-token recomputation has negligible impact on repetition, whereas output-token recomputation intervals and similarity thresholds substantially affect it.Longer output-token recomputation intervals and lower similarity thresholds correspond to higher repetition rates.
- Fewer output tokens recomputed at each decoding step make repetition more likely.
- Caching dynamic output tokens can leave the model reliant on outdated states and disrupt deep-layer context-token entropy convergence.The paper presents this as a hypothesis explaining why output-token caching affects repetition more than prompt-token caching.
- MMaDA also exhibits the repetition curse under dLLM-Cache, with adjacent repetition increasing by +3.7% and sample repetition by +49%.
- CoTA and its CTAE and CTEV components suppress repetitive generation in MMaDA, indicating generalizability beyond LLaDA-V.
C MORE EXPERIMENTAL DETAILS OF CTEV
CTEV estimates context-token entropy from layers 26–30 because cache use most strongly disrupts entropy convergence there. Experiments show CoTA remains effective across block lengths, while this layer range gives the best performance and restores anchor-like attention patterns.
- Layers 26–30 are selected for CTEV because cache-induced entropy deviations are strongest in this deep-layer range.The choice is based on per-layer entropy computed across 100 samples and aggregated across samples.
- Across architectures, caching systematically disrupts entropy convergence in deep layers, supporting the use of layers 26–30.
- CoTA consistently reduces repetition across all tested decoding block lengths.
- Computing cumulative entropy over layers 26–30 yields the best performance among tested layer ranges.The paper attributes this to stronger separation between normal and abnormal entropy modes in that depth range.
- CoTA restores the context-tokens-as-anchors pattern by strengthening attention toward contextual tokens after cache produces more random attention.
E LINGUISTIC ATTRIBUTE ANALYSIS OF REPEATED TOKENS
Repeated tokens in cached LLaDA-V outputs are dominated by low-semantic-content function words, while CoTA is reported as more effective than n-gram penalties but adds modest latency and slightly reduces throughput.
- Repeated tokens are predominantly function words carrying low semantic content.The paper links this pattern to uncertain or repetitive text generation.
- CoTA reduces repetition more effectively than n-gram penalties, which can degrade accuracy through forced token substitution.
- Integrating CoTA adds a modest latency increase and slightly reduces throughput while improving over the baseline.Efficiency is evaluated on DocVQA and additionally reported for LLaDA-V and MMaDA.
- The paper frames its contribution as an information-flow analysis linking cache-disrupted context-token aggregation and failed deep-layer entropy convergence to repetition.
I EVALUATION CONFIGURATION DETAILS OF DIFFERENT DATASETS
The evaluation setup specifies maximum generation length, block length, decoding steps, and batch size for each benchmark.
- Evaluation configurations report maximum generation length, block length, decoding steps, and batch size for different benchmarks.
J MAXIMUM REPETITION LENGTH, AVERAGE REPETITION LENGTH, 95TH-PERCENTILE REPETITION LENGTH, AND SAMPLE REPETITION RATE
The paper evaluates repetition in generated outputs using repetition-segment statistics and sample-level repetition rate. It also reports that caching can produce excessive repeated words and punctuation, while the proposed mitigation reduces repetition and improves description quality.
- Repetition segments are identified by recording the lengths of consecutive identical token runs and retaining runs with length at least 2.These runs support maximum, average, and 95th-percentile repetition-length statistics.
- Sample Repetition Rate measures the fraction of generated results containing repetition among all generated results.It is defined using Ndup, the number of results containing repetition, and N, the total number of generated results.
- Caching can yield excessive repetition of words such as “the” and punctuation such as “,” in generated responses.The reported example contrasts dLLM-Cache outputs with the mitigation strategy.
- The mitigation strategy substantially reduces repetition and supports more detailed and coherent image descriptions.The passage attributes this improvement to more effective attention to information in surrounding tokens.