Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation

1 Introduction

• Token merging reduces repeated diffusion computation, but merging content-relevant tokens can absorb local details and distort images.The central challenge is deciding which tokens to merge and which to retain as anchors for later computation.Source: §1 · p. 1
• CFG importance remains foreground-focused as denoising evolves, leaving actively changing tokens in non-prompt regions vulnerable to premature merging.The spatial distribution of evolving tokens expands beyond foreground objects at later timesteps.Source: §1 · p. 2
• CFG also assigns low importance to prompt-independent boundaries, background-object edges, and textures that contribute to visual coherence.These structures carry information despite receiving little text guidance.Source: §1 · p. 2
• DaTMe combines inter-step changes in Tweedie clean-image estimates with a high-pass structural signal to identify unresolved and structurally informative tokens.The structural signal measures the residual between the Tweedie estimate and a coarse-scale smoothed version.Source: §1 · pp. 2–3
• DaTMe preserves high-scoring tokens as constraint tokens and merges low-scoring tokens from stable, homogeneous regions, supplementing sources with grid sampling and nearby matching.The framework is training-free and evaluated on PixArt-α and FLUX, with strong image-wide detail preservation and state-of-the-art performance across most metrics.Source: §1 · p. 3

Token Reduction in Large Models

• Token reduction improves Transformer efficiency through pruning or merging, with ToMe using training-free bipartite matching and later methods refining selection with attention or frequency cues.Many such methods target single-forward-pass architectures with relatively static token importance.Source: §2 · p. 3

Token Merging in Diffusion Models

• Diffusion-specific acceleration includes samplers, quantization, pruning, caching, and merging; training-free token reduction avoids additional training or optimization overhead.Diffusion models repeatedly update tokens across many steps, motivating methods beyond those designed for static importance.Source: §2 · p. 3
• ToMeSD groups tokens by feature similarity, while later methods use cross-attention maps and CFG-based merging uses text-conditioning discrepancy to identify important tokens.The paper characterizes CFG importance as missing the full image's spatially heterogeneous and temporally evolving dynamics.Source: §2 · p. 3

3.1 Preliminaries

• The Tweedie denoised estimate expresses the current generation state in predicted clean-image space, providing a noise-normalized estimate of the clean sample.It is derived from Tweedie’s formula and the learned score estimator.Source: §3.1 · p. 4

3.2 Spatiotemporal Tweedie Importance Signal

• The Tweedie difference tracks token evolution more reliably than raw score slope or CFG, maintaining higher pseudo-GT correlation and lower temporal autocorrelation throughout denoising.The score slope becomes less reliable as its scale term grows dominant in later steps.Source: Figure 2 · p. 4, §3.2 · p. 5
• DaTMe combines inter-step changes in the Tweedie estimate with a spatial residual that responds to contours, boundaries, and fine textures.The temporal signal reflects ongoing change; the spatial signal distinguishes structurally salient regions from locally homogeneous ones.Source: §3.2 · p. 6

3.3 Dynamics-aware Token Merging (DaTMe)

• At each denoising step, DaTMe preserves tokens ranked highly on both axes and selects low-ranked tokens on both as merge sources under the target compression ratio.The preserved constraint tokens act as structural anchors.Source: §3.3 · p. 6
• DaTMe fills any remaining source budget with grid-sampled tokens, then merges sources with spatially nearest destinations to preserve local structure.The strict intersection defining safe sources may leave fewer tokens than the target budget.Source: §3.3 · pp. 6–7

4 Experiments

• Experiments use 10k COCO 2014 image-prompt pairs on PixArt-α at compression ratios 0.3, 0.5, and 0.7, and also evaluate DaTMe on FLUX.Metrics include CLIP-T, FID, HPSv2, SSIM, PSNR, LPIPS, and Aesthetic score.Source: §4 · p. 7

4.1 Main Experiments

• Across PixArt-α and FLUX, DaTMe reports favorable quality-efficiency trade-offs and stronger image fidelity under compression, with strongest FLUX alignment and fidelity at comparable efficiency.The supplied captions identify Tables 1–4 as text-alignment, efficiency, and image-fidelity evaluations; no cell values are provided.Source: Table 1 · p. 8, Table 2 · p. 8, Table 3 · p. 8
• On PixArt-α, ToMeSD causes spatial distortion, CFG-based merging weakens background preservation, and on FLUX DaTMe more faithfully preserves spatial configuration and object structure.On FLUX, ToMeSD and ToMA introduce structural drift; ToMA also adds visual degradation.Source: §4.1 · p. 8

4.2 Additional Analysis

• Using both temporal and spatial axes performs best overall; replacing the Tweedie difference with raw score slope degrades all metrics and introduces visible artifacts.Figure 7 illustrates the artifacts from score-slope substitution.Source: §4.2 · p. 9, Table 5 · p. 9, Figure 7 · p. 9
• Joint temporal and spatial signals preserve evolving background content and critical object boundaries, whereas temporal-only signals miss structurally important boundaries.Figure 8 illustrates the region-preservation comparison.Source: Figure 8 · p. 9, §4.2 · p. 10
• Constraint tokens selected for both high temporal and high spatial responses yield the best results; spatially low-response tokens fail to protect meaningful object structure.Table 6 compares constraint-set definitions.Source: Table 6 · p. 9, §4.2 · p. 10
• Replacing CFG importance with DaTMe’s signal improves CLIP-T, HPSv2, and fidelity metrics, while the full DaTMe strategy further improves all metrics.Table 7 evaluates the signal replacement at r = 0.3.Source: §4.2 · p. 10, Table 7 · p. 10
• At r = 0.3, human evaluators gave DaTMe the highest ratings for text alignment and visual quality and preferred it overall.The study used 20 PixArt-α cases and 32 evaluators.Source: §4.2 · p. 10, Figure 9 · p. 10

• DaTMe uses the Tweedie denoised estimate to represent token importance through temporal evolution and structural saliency.Source: §5 · p. 10
• The framework identifies safely mergeable tokens while preserving structurally critical regions.Source: §5 · p. 10
• The conclusion reports strong image fidelity and favorable quality-efficiency trade-offs across DiT-based diffusion models.Source: §5 · p. 10

Score-slope decomposition and late-step instability

• The score-slope decomposition separates actual changes in the model’s noise prediction from a second component determined by the noise schedule and current prediction.The first component reflects token-level evolution; the schedule component can dominate in later steps.Source: A Formal Decomposition of Score Slope… · p. 14
• Late in denoising, the score-slope scale term can dominate actual prediction changes, reducing the signal’s reliability for tracking token-wise evolution.The inverse noise-scale difference grows as noise decays, amplifying the raw noise prediction and aligning with high autocorrelation and reduced discriminative power.Source: A Formal Decomposition of Score Slope… · p. 14

Tweedie difference

• The derivation defines the Tweedie denoised estimate as a clean-data estimate and substitutes consecutive latent states to express its inter-step difference.The estimate uses the learned noise prediction, and the clean-space projection is embedded in deterministic updates such as DDIM.Source: A Formal Decomposition of Score Slope… · pp. 14–15
• The Tweedie difference scales prediction changes by σt−1/αt−1 rather than the rapidly growing inverse-noise factor, so its variation decays as predictions stabilize.The passage identifies σt−1/αt−1 as the scaling factor; it tends to zero as t → 0.Source: A Formal Decomposition of Score Slope… · p. 15

Extension to flow matching

• For FLUX rectified flow, the Tweedie inter-step difference isolates velocity-prediction changes and scales them by t − ∆t, which decays toward zero.The derivation uses the rectified-flow schedule and deterministic Euler update.Source: A Formal Decomposition of Score Slope… · p. 15

PixArt-α implementation

• For PixArt-α, token merging starts at the second denoising step and runs in layers 9–19 over 20 inference steps.The guidance scale is 4.5.Source: B DaTMe Details · p. 15

DaTMe Process

• DaTMe assigns tokens dynamic roles, completes source selection spatially, then matches source tokens to nearby destination tokens.The algorithm takes token features, a temporal map, compression ratio, and grid count; it outputs merged token features.Source: B DaTMe Details · p. 16

FLUX implementation

• For FLUX, merging is applied in dual-stream blocks 9–19 and single-stream blocks 7–22; its 35-step generation uses guidance scale 3.5.The temporal signal needs a previous Tweedie estimate, unavailable at the first step.Source: B DaTMe Details · p. 16

U-Net implementation

• For Stable Diffusion 1.5 and SDXL, DaTMe uses source-token selection without an explicit constraint set, relying on convolutional spatial locality.Both use 64 × 64 latent resolution, 50 inference steps, interpolation scale s = 2, and P = 4 grids; guidance scales are 7.5 and 5.0, respectively.Source: B DaTMe Details · p. 16

Extended experiments on U-Net architecture

• On Stable Diffusion 1.5 and SDXL, DaTMe better preserves structural fidelity while maintaining strong text alignment across compression ratios.The accompanying figures report that prior methods often lose structure in non-prompt regions, which DaTMe preserves more faithfully.Source: C Additional Experiments · pp. 16–17, Table 9 · p. 18, Table 8 · p. 18

Additional comparison results

• DaTMe more faithfully preserves composition and fine details across compression ratios on FLUX, while ToMeSD and ToMA show structural or visual degradation.On PixArt-α, it also preserves subject details and unconditioned content, including a woman’s hat and floor decoration beneath a cake.Source: C Additional Experiments · p. 16, Figure 11 · p. 17, Figure 10 · p. 17

Additional ablation studies on PixArt-α

• For PixArt-α, P = 4 grids gives the best overall quality while remaining faster than the unmerged baseline; finer grids improve structure but add latency.The interpolation scale s = 4 achieves the best overall trade-off across quality metrics.Source: C Additional Experiments · p. 17

Additional results

• Merging-map visualizations show DaTMe better preserves structural details such as bowl shape and leaves.Additional samples also report preserved structural detail alongside strong text alignment.Source: C Additional Experiments · pp. 17–18

Limitations

• In highly complex scenes with many objects or intricate backgrounds, excessive merging can remove necessary tokens and degrade generation quality.The limitations figure likewise describes loss of structural details and visual coherence under aggressive compression.Source: D Limitations · p. 18, Figure 18 · p. 18
• Diffusion-model layer-wise roles remain underexplored, particularly in DiT architectures, limiting understanding of which layers contribute to layout, detail, or background refinement.The paper identifies layer-aware compression as a possible improvement direction.Source: D Limitations · p. 18

Grid-size trade-off

• For PixArt-α, denser spatial grids better preserve structure by distributing source tokens more evenly, while the selected P = 4 balances quality and latency.The grid ablation caption identifies P = 4 as the best trade-off between latency and performance.Source: Figure 15 · p. 20, Table 12 · p. 20