Source-linked AI summary
FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
Siyang He, Qiqi Wang, Xiaoran Liu, Hongnan Ma, Yiwei Shi, Yuerong Song, Ying Zhu, Tianyi Liang, Zengfeng Huang, Ziwei He, Xipeng Qiu
TL;DR
dLLM decoding retains positional bias despite its arbitrary-order potential, leaving the use of global bidirectional context incomplete. The paper analyzes hidden-state frequencies and introduces FourierSampler for structure-to-detail generation, reporting consistent gains across dLLM families and surpassing similarly sized autoregressive models.
Problem
Existing dLLM decoding strategies exhibit positional bias, limiting their ability to fully exploit arbitrary-order generation and global bidirectional context.
Method
FourierSampler uses frequency-domain analysis with a Translated Fourier Score and Adaptive Fourier Calibrator to guide structure-to-detail decoding.
Results
FourierSampler consistently improves math and code performance across LLaDA and SDAR, including relative gains up to 20.4% and 16.0% on LLaDA models, and surpasses similarly sized autoregressive models.
Takeaways & Limitations
The results provide an endogenous frequency-based approach for unlocking arbitrary-order generation potential in dLLMs.
Takeaways & Limitations
The frequency-to-semantics interpretation is supported by a hypothesis and analyses using selected code and math samples.
Abstract
from arXiv · showhide
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.
1 Introduction
dLLMs offer arbitrary-order generation and bidirectional context, but positional bias limits this flexibility. FourierSampler uses internal frequency characteristics to guide structure-to-detail decoding and reports improvements across LLaDA and SDAR.
- dLLMs support arbitrary-order generation and global bidirectional context, enabling text infilling and non-sequential planning beyond strict left-to-right decoding.
- Positional bias can make left-to-right constraints outperform confidence-based decoding on some math tasks, motivating optimization of the decoding schedule.
- FourierSampler analyzes internal representations and dynamically shifts frequency-band emphasis to decode global structure before local detail.Low-frequency components are associated with structure and long-range dependencies, while high-frequency components characterize local details.
- 20.4% and 16.0% relative improvements are reported for LLaDA1.5-8B and LLaDA-8B-Instruct, respectively, with further gains on SDAR-1.7B-Chat and SDAR-4B-Chat.The reported SDAR improvements are up to 45.1% and 26.5%, respectively.
2 Related Work
Related work frames dLLM decoding as a problem of positional bias and unmasking order, while prior enhancement methods use external signals or training. Frequency-domain analysis offers a complementary way to mine internal representations for decoding guidance.
- LLaDA and SDAR commonly use confidence-based unmasking, with variants based on maximum probability, entropy, or confidence gaps, alongside random sampling.
- Rule-based, reward-guided, and training-based methods improve generation but rely on external signals, costly training, or both.
- Transformer spectral studies report low-pass behavior and concentration of power spectra in low-frequency bins within deeper layers.
3 Method
FourierSampler analyzes frequency content in dLLM hidden states and uses a sliding spectral window to guide generation from structural content toward details. Its adaptive calibration combines frequency-based guidance with the model’s original confidence during decoding.
- 3.1 Spectral Semantic Analysis in dLLMs: The analysis extracts final-layer hidden states after one forward pass and separates the lowest-frequency half using a binary frequency mask.The low-frequency ratio measures retained spectral energy, while the high-frequency ratio is the remaining energy.
- 3.1 Spectral Semantic Analysis in dLLMs: Low-frequency hidden-state components correspond to global structure, while high-frequency components encode local details in dLLM outputs.This semantic correspondence motivates hierarchical structure-to-detail decoding.
- 3.2 Translated Filtering Score: FourierSampler applies a frequency-domain sliding window that shifts from low to high frequencies across decoding steps, enforcing structure-to-detail generation.The retained band favors low-frequency-dominated tokens early and high-frequency-dominated tokens later.
- 3.2 Translated Filtering Score: The Translated Filtering Score ranks token positions by their normalized energy within the frequency band selected at each decoding step.The score represents each token’s relative intensity under the step-specific filtered representation.
- 3.3 Adaptive Fourier Calibrator: Adaptive calibration adjusts frequency guidance according to the model’s confidence differences across masked positions.Guidance weakens when the model clearly distinguishes priorities and strengthens when its confidence is less differentiated.
4 Experiment
Experiments across LLaDA and SDAR show that FourierSampler consistently improves performance across math and code benchmarks, including models with full bidirectional and block-wise causal attention. Ablations further support adaptive weighting and selected frequency-window settings.
- LLaDA Results: FourierSampler consistently improves LLaDA performance across math and code tasks, achieving the highest average performance across benchmarks.It surpasses other competitive decoding strategies and similarly sized autoregressive models.
- LLaDA Results: FourierSampler bridges and exceeds LLaDA1.5-8B’s average-performance gap with similarly sized autoregressive models.The compared models include Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct.
- SDAR Results: FourierSampler consistently outperforms baselines across all evaluated benchmarks for SDAR models with block-wise causal attention.The method also achieves superior average performance to competing approaches.
- Cross-architecture Results: The results demonstrate generalization across full-bidirectional-attention and block-wise-causal-attention dLLM designs.Experiments cover both LLaDA and SDAR architectures.
- Ablation Studies: Fixed Fourier-calibration weights consistently underperform adaptive weighting, while different tasks on one model may prefer different weights.The ablation evaluates fixed maximum, minimum, and mean adaptive-weight values on GSM8K and MBPP.
- Ablation Studies: The selected frequency-window ratios are 0.2 for LLaDA-1.5B and 0.4 for LLaDA-8B-Instruct based on overall downstream-task performance.These values were chosen from experiments with different sliding-window settings.
5 Discussion
FourierSampler’s frequency-guided trajectory supports structure-to-detail generation: global program structure emerges early, while concrete details are filled later. Larger decoding blocks strengthen these gains by providing more continuous frequency signals.
- Block Size: Increasing decoding block size produces more pronounced FourierSampler gains on downstream tasks.Larger blocks provide more complete and continuous signals for frequency-domain analysis.
- Generation Order: Logical code keywords such as if and return receive high scores and are decoded during early steps.In Block 1, these structural tokens are determined during Steps 0–10.
- Generation Order: Concrete variables and numerical values, including fib, n, 0, and 1, generally emerge during later decoding stages.In Block 2, computation details involving fib and n appear around Step 20.
- Generation Order: The structure-to-detail trajectory maps low-frequency energy to structural text and enables global logic to be planned before local details are filled.This interpretation is supported by the observed ordering of structural keywords and concrete computation details.
- Part-of-Speech Analysis: Function words, connectives, prepositions, adverbs, and verbs show stronger low-frequency tendencies associated with sentence logic and structural scaffolding.Conjunctions such as but, if, and because exhibit especially high low-frequency ratios.
- Part-of-Speech Analysis: Nouns exhibit the strongest high-frequency characteristics and represent concrete entities, variables, or values that fill the syntactic skeleton.Examples include fib, n, and 0.
6 Conclusions
The paper analyzes dLLM decoding through signal processing, finding that frequency components distinguish structure from detail, and introduces FourierSampler for dynamic structure-to-detail guidance. Experiments across LLaDA and SDAR architectures show consistent improvements across math and code tasks.
- Conclusion: Fourier analysis shows that low-frequency components imply structure, whereas high-frequency components imply detail in dLLM representations.This provides the basis for interpreting dLLM decoding through signal processing.
- Conclusion: FourierSampler uses the Translated Fourier Score and Adaptive Fourier Calibrator to dynamically guide structure-to-detail generation.The method combines frequency-domain guidance with the model’s original decoding confidence.
- Conclusion: FourierSampler consistently improves performance across math and code tasks on both full-bidirectional-attention LLaDA and block-wise causal-attention SDAR models.The experiments also report gains over similarly sized autoregressive models.
A Details of Method
The method details identify FourierSampler’s adaptive weighting procedure and its documented hyperparameters, while the accompanying analysis figure concerns frequency-domain correspondence with textual information.
- Hyper-parameters: FourierSampler’s key hyperparameters for different dLLMs are documented in Table 5.The table is presented as the method’s hyper-parameter reference.
- Adaptive Weight: The adaptive weight β_s is based on the variance σ^2_s of maximum prediction probabilities across masked positions.The method records variance values over decoding steps to derive the adaptive guidance weight.
- Adaptive Weight: The procedure computes the current variance percentile over a 20-step history, maps it to [−3, 3], and converts it into a smooth weight through the normal CDF.The resulting value is w_s ∈ (0, 1).
- Adaptive Weight: Algorithm 1 presents the complete adaptive-weight computation procedure, including handling masked-position and variance-list conditions.The pseudocode includes checks for whether masked positions exist and whether the stored list exceeds 20 entries.
- Frequency-Text Correspondence: Figure 6 visualizes the correspondence between frequency-domain analysis and textual information.The supplied figure caption identifies the visualization but does not specify additional axes or comparison outcomes.
B Details of analysis
Analysis of a mathematical derivation extends the frequency-domain interpretation beyond code: narrative text appears low-frequency, while formulas and variables appear high-frequency.
- Natural-Language Analysis: Narrative text in a difference-of-squares derivation appears as low-frequency components, whereas formulas and variables emerge as high-frequency components.The observation corroborates the frequency-based structure-versus-detail distinction reported elsewhere in the paper.