PaperlayerView exact revision on arXiv

Source-linked AI summary

DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusion

Noam Issachar, Guy Yariv, Sagie Benaim, Yossi Adi, Dani Lischinski, Raanan Fattal

arXiv:2510.20766v2cs.CV

TL;DR

Training diffusion transformers at ultra-high resolutions is costly because self-attention scales quadratically with image-token count. DyPE dynamically adjusts positional encoding to diffusion’s spectral progression. It consistently improves ultra-high-resolution synthesis, with larger advantages over static extrapolation at higher resolutions.

  • Problem

    Training diffusion transformers at ultra-high resolutions remains costly because self-attention’s quadratic scaling makes memory and computation prohibitive.

    Source: §1 · p. 1

  • Method

    DyPE dynamically adjusts positional encoding through sampling to shift frequency emphasis in step with diffusion’s progression, without retraining.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    DyPE consistently outperforms static extrapolation techniques, with its advantage growing at higher resolutions.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Pre-trained FLUX can generate images exceeding 16M pixels without retraining or additional sampling costs.

    Source: §1 · p. 2

    The work focuses on methodological improvements and evaluation on standardized datasets; it states that efficiency improvements do not inherently increase established generative-model misuse risks.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Diffusion Transformer models can generate images with remarkable fidelity and detail, yet training them at ultra-high resolutions remains extremely costly due to the self-attention mechanism's quadratic scaling with the number of image tokens. In this paper, we introduce Dynamic Position Extrapolation (DyPE), a novel, training-free method that enables pre-trained diffusion transformers to synthesize images at resolutions far beyond their training data, with no additional sampling cost. DyPE takes advantage of the spectral progression inherent to the diffusion process, where low-frequency structures converge early, while high-frequencies take more steps to resolve. Specifically, DyPE dynamically adjusts the model's positional encoding at each diffusion step, matching their frequency spectrum with the current stage of the generative process. This approach allows us to generate images at resolutions that exceed the training resolution dramatically, e.g., 16 million pixels using FLUX. On multiple benchmarks, DyPE consistently improves performance and achieves state-of-the-art fidelity in ultra-high-resolution image generation, with gains becoming even more pronounced at higher resolutions. Project page is available at https://noamissachar.github.io/DyPE/.

1. Introduction

Training diffusion transformers at ultra-high resolutions is costly because self-attention scales quadratically with image-token count. DyPE dynamically adapts positional encoding to diffusion’s spectral progression, improving high-resolution synthesis without retraining or additional sampling costs.

1. Introduction

  • Training at ultra-high resolutions remains difficult because self-attention’s quadratic cost in image-token count makes memory and computation prohibitive.The paper notes this challenge at resolutions such as 4K and beyond.Source: §1 · p. 1
  • Static image adaptations of language-model positional extrapolation methods do not account for diffusion’s low-to-high-frequency progression.The paper identifies dynamic adjustment of positional embeddings to that progression as its guiding question.Source: §1 · pp. 1–2
  • Low-frequency Fourier components converge early while high-frequency components continue evolving, motivating DyPE to shift positional-encoding emphasis toward higher frequencies as sampling proceeds.DyPE is training-free and dynamically tailors positional encoding to the diffusion process.Source: §1 · p. 2
  • DyPE consistently improves ultra-high-resolution synthesis across benchmarks and resolutions, enabling pre-trained FLUX to generate images exceeding 16M pixels without retraining or added sampling costs.The paper reports that it evaluates image quality and prompt fidelity alongside qualitative and human evaluations.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe diffusion’s transition between data and noise, then introduce positional embeddings and RoPE. They review inference-time extrapolation methods and their frequency-allocation trade-offs.

2.1. Diffusion Models

  • Diffusion interpolates between pure data and Gaussian noise; the paper adopts Flow Matching’s linear schedule α_t = 1 − t and σ_t = t for its derivation.The schedule coefficients define the endpoints x_0 = x (pure data) and x_1 = ε (pure Gaussian noise).Source: §2.1 · p. 2

2.2. Rotary Positional Embeddings and Position Extrapolation

  • RoPE encodes positions as 2D rotations at multiple frequencies, applied axially to represent horizontal and vertical offsets independently.The number of frequencies is D = d_model/2, and each has wavelength λ_d = 2π/θ_d.Source: §2.2 · p. 2
  • Position interpolation uniformly rescales positions to the longer context, reproducing lower frequencies but missing higher ones, which can cause blur and loss of fine detail.The section frames inference-time positional adaptations as a response to degraded performance when models are applied beyond their training context.Source: §2.2 · pp. 2–3
  • NTK-aware interpolation scales low and high frequencies differently, preserving low frequencies while trading off representation of high frequencies in the extended context.Source: §2.2 · p. 3
  • YaRN extends NTK-aware interpolation with three frequency bands and a smooth ramp between position interpolation and no scaling; it also rescales attention logits to counter increased attention-weight entropy.The band boundaries control non-uniform frequency allocation.Source: §2.2 · p. 3

3. Method

DyPE analyzes how frequency bands evolve during reverse diffusion, then uses that progression to adjust positional encoding over sampling time. Its scaling gradually reduces spectrum compression as the process advances.

3. Method

  • The method section first analyzes frequency-mode evolution and then derives positional-encoding adjustments matched to that evolution..Source: §3 · p. 3

3.1. Evolution of Frequency Modes in the Diffusion Process

  • The analysis models the time-dependent mean power spectrum from the mixture of natural-image power-law structure and white noise.The empirical spectrum transitions between these components during denoising, with low-frequency structures growing and noise decaying as high-frequency detail emerges.Source: §3.1 · p. 3
  • Low-frequency modes converge early, while high-frequency modes evolve throughout sampling, motivating a shift in positional-encoding emphasis toward unresolved higher frequencies.The frequency progression map shows low-frequency modes ceasing to change early and high-frequency modes evolving across the reverse process.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Dynamic Position Extrapolation (DYPE)

  • DyPE gradually lowers the use of spectrum-compressing positional extrapolation as diffusion proceeds, aligning positional encoding with the evolving frequency spectrum.The method adds time dependence to existing positional-extrapolation strategies rather than retraining the model.Source: §3.2 · p. 4
  • DyPE introduces time dependence into PI, NTK-aware, and YaRN extrapolation, whose scalings collapse to no scaling when the scale factor reaches 1.For PI, NTK-aware, and YaRN, the unscaled setting returns their positional mapping to its original form.Source: §3.2 · p. 4
  • The scheduler moves from near-maximal scaling early in sampling toward no scaling late, with its attenuation governed by tunable parameters.λ_s sets maximal scaling, while λ_t controls how scaling attenuates over time.Source: §3.2 · pp. 4–5

4. Experiments

The experiments evaluate DyPE on text-to-image benchmarks, at multiple resolutions, against positional-extrapolation and broader high-resolution baselines, and on class-conditional ImageNet generation. Results report consistent gains across settings, with performance patterns varying by scale and metric.

4. Experiments

  • The evaluation spans global structure and fine detail, using DrawBench, Aesthetic-4K, automatic metrics, human judgments, and resolution scaling on FLUX.The study also tests transfer to class-conditional ImageNet generation with FiTv2 and includes zoom-in comparisons and further evaluations in the appendices.Source: §4 · p. 5

4.1.1. COMPARISON WITH POSITION-EXTRAPOLATION BASELINES

  • Position-extrapolation baselines are compared on DrawBench and Aesthetic-4K using text alignment, human-feedback reward, aesthetics, and FID measures.Aesthetic-4K uses 195 curated image–prompt pairs downsampled to the target test resolutions.Source: §4.1.1 · pp. 5–6
  • DyPE improves high-resolution generation across DrawBench and Aesthetic-4K, with its advantage becoming more pronounced as resolution grows.Table 1 covers CLIPScore, ImageReward, and Aesthetics for DrawBench and adds FID for Aesthetic-4K; the comparisons use FLUX-based methods.Source: Table 1 · p. 6, §4.1.1 · p. 7
  • Human raters preferred DYPE in 70.5%–94.2% of pairwise comparisons on Aesthetic-4K at 4096².The study used 20 prompts and 50 raters, comparing DYPE variants with NTK-aware, TASR, and YaRN baselines.Source: §4.1.1 · p. 7, Table 2 · p. 7
  • On ImageReward from 1024² to 6144², FLUX degrades sharply at 3072² and YaRN at 4096², while our method remains stable across scales until experiencing degradation at 6144².Source: §4.1.1 · p. 7

4.1.2. COMPARISON WITH OTHER APPROACHES — GENERAL BASELINES

  • At 2048², DY-YaRN leads all four metrics; at 4096², DY-YaRN leads patch-FID and patch-IS, while DY-YaRN+HiFlow leads global FID and IS.At both resolutions, at least one DYPE-based variant outperforms the baselines on every metric.Source: §4.1.2 · p. 7, Table 3 · p. 8

4.2. Higher-Resolution Class-to-Image Generation

  • On FiTv2-XL/2, DYPE improves over vanilla extrapolation baselines at 320² and 384², with DY-YaRN achieving the best overall performance.The evaluation uses the 50,000-image ImageNet validation set; PI underperforms the base model, with loss of high-frequency details cited as the reason.Source: §4.2 · p. 8, Table 4 · p. 8

5. Related Work

Related work frames ultra-high-resolution generation as difficult for DiTs because self-attention costs grow quadratically with image tokens. Existing high-resolution and positional-extrapolation approaches have limits that motivate a diffusion-aware alternative.

Diffusion Transformers

  • Training DiTs at 4K and beyond remains challenging because quadratic self-attention costs make memory and computation prohibitive.Source: §5 · p. 8

Ultra-High Resolution Image Synthesis

  • Prior ultra-high-resolution methods include fine-tuning, patch stitching, input-level changes, and tuning-free U-Net pipelines, each with stated scaling or fidelity limitations.Fine-tuning is costly; patch-based methods can duplicate or repeat regions, and U-Net tuning-free methods do not directly transfer to DiTs.Source: §5 · pp. 8–9

Position Extrapolation Schemes

  • Vision-adapted PI, NTK-aware, and YaRN support aspect-ratio changes and modest resolution gains, but do not account for diffusion's low-to-high frequency progression.Lumina-Next incorporates timestep dynamics through interpolation, while the paper describes that approach as suppressing high frequencies and producing blurry outputs.Source: §5 · p. 9

6. Conclusion

DyPE is a training-free approach for ultra-high-resolution DiT synthesis, based on Fourier-space analysis of how frequencies evolve during diffusion. Evaluations on pre-trained FLUX report improved generalization over static extrapolation methods, with larger advantages at higher resolutions.

6. Conclusion

  • Fourier-space analysis finds that low-frequency content converges faster than higher-frequency bands, motivating dynamic positional encoding that tracks the evolving spectrum.The conclusion states that this lets the denoiser operate more effectively within its training conditions.Source: §6 · p. 9
  • DyPE is training-free and adds no retraining or sampling overhead, while evaluations report stronger generalization than static extrapolation methods, especially at higher resolutions.Source: §6 · p. 9
  • Future work will explore higher resolutions through inference-time scaling and time-dependent positional extrapolation during a light tuning phase.Source: §6 · p. 9

Impact Statement

The statement frames the work as methodological research evaluated on public benchmarks, while acknowledging potential misuse of high-fidelity image generation. The accompanying illustration compares RoPE frequency behavior across scaling strategies.

Impact Statement

  • The authors acknowledge that high-fidelity image generation may be misused to create deceptive or harmful content, while focusing their contribution on methodology and standardized-dataset evaluation.They state that efficiency improvements do not inherently increase these risks beyond those already established in generative modeling.Source: Impact Statement · p. 9

A. Illustration of DYPE

  • Figure 6 compares frequency behavior across RoPE scaling strategies, with low and high frequencies distinguished and pretrained versus unseen positions marked.Its panels include Dy-NTK-aware settings at two diffusion timesteps.Source: Impact Statement · p. 9, A. Illustration of DYPE · p. 14

B. Comparison Between DYPE and Lumina-Next

The comparison evaluates DY-NTK-aware and Time-Aware Scaled RoPE on FLUX using Aesthetic-4K metrics. DY-NTK-aware performs best across the reported metrics, against a backdrop of different frequency-allocation strategies over sampling.

B. Comparison Between DYPE and Lumina-Next

  • DY-NTK-aware uses the low-to-high progression of diffusion: low-frequency modes converge early, leaving more opportunity to represent higher bands and reduce extrapolation compression.The passage contrasts this with fixed-capacity extrapolation strategies that trade off coverage across the spectrum.Source: B. Comparison Between DYPE and… · p. 14
  • The cited comparison describes Time-Aware Scaled RoPE as starting with PI and later switching to NTK-aware scaling, unlike DYPE’s denoiser operating under its trained positional encoding.The passage attributes the contrasting strategy to Zhuo et al. (2024).Source: B. Comparison Between DYPE and… · p. 14
  • Figure 7 contrasts the wavelength coverage of Time-Aware and DY-NTK-aware dynamic variants across sampling.The figure caption distinguishes interpolation with PI from interpolation with extrapolation.Source: B. Comparison Between DYPE and… · p. 14
  • DY-NTK-aware achieves the best reported performance across all Aesthetic-4K metrics in the comparison with Time-Aware Scaled RoPE and NTK-aware.The comparison uses CLIPScore, ImageReward, Aesthetic-Score, and FID; Figure 8 provides a qualitative comparison with Time-Aware Scaled RoPE.Source: B. Comparison Between DYPE and… · p. 14

C. Implementation Details

The implementation details specify a single-GPU setup and the sampling and method parameters used unless otherwise stated.

C. Implementation Details

  • Experiments use one L40S GPU, effective resolution L = 1024, and 28 diffusion sampling steps unless otherwise stated.The method uses α = 1, β = 32, and λs = λt = 2.Source: C. Implementation Details · p. 14

D. Ablation Study

The ablations test timestep-scheduler choices for DY-NTK-aware and DY-YaRN. Their reported findings favor an exponential DY-NTK-aware scheduler for balancing structural fidelity and aesthetics, and timestep scheduling alone among the tested DY-YaRN options.

DY-NTK-aware scheduler designs

  • The DY-NTK-aware ablation varies scaling magnitude and timestep progression, comparing multiplicative and exponential modulation of NTK-aware compression.It fixes λt = 1 while varying λs, then fixes λs = 2 while testing λt values for sublinear, linear, and exponential progression.Source: D. Ablation Study · p. 14
  • The exponential DY-NTK-aware scheduler with λs = 2 and λt = 2 achieves the reported best balance between structural fidelity and aesthetic outputs.The ablation reports that increasing initial scaling improves CLIP structural fidelity, while faster attenuation with t yields more aesthetic outputs.Source: D. Ablation Study · pp. 14–15, Table 6 · p. 16
  • Figure 6 illustrates RoPE frequency behavior across scaling strategies, including Dy-NTK-aware settings at t = 0.2 and t = 0.8.The figure distinguishes low and high frequencies and pretrained versus unseen position ranges.Source: Figure 6 · p. 15
  • Figure 7 depicts wavelengths for baseline and dynamic scaling variants, including DY-NTK-aware and Time-Aware RoPE.The caption distinguishes how the dynamic variants interpolate between baseline strategies.Source: Figure 7 · p. 15

DY-YaRN scheduler designs

  • DY-YaRN tests timestep dependence on mid-band NTK-aware components, ramp thresholds, or both together.The ramp assigns PI-like scaling to low frequencies, no scaling to high frequencies, and interpolation to middle bands.Source: D. Ablation Study · pp. 15–16
  • The DY-YaRN threshold scheduler uses κ(t) = t^2 in the reported configuration.The passage says the best scheduler for the thresholds is κ(t) = t^2.Source: D. Ablation Study · p. 16
  • Considering only κ(t) performs best among the tested DY-YaRN timestep-integration strategies.The passage relates the finding to low-frequency coverage early in denoising and sharper high-frequency detail later.Source: D. Ablation Study · p. 16

E. Additional Results

Additional evaluations extend DyPE to Qwen-Image, high-resolution video, image editing, and panoramic generation. The reported results include metric gains and qualitative improvements in these settings.

Additional Results

  • On Aesthetic-4K, DY-YaRN leads Qwen-Image and YaRN across all four metrics and produces fewer visual artifacts.The qualitative comparison is at 4096 × 4096 resolution.Source: E. Additional Results · p. 16, Table 8 · p. 17, Figure 9 · p. 17
  • DY-YaRN outperforms vanilla Wan2.1 across VBench categories; unlike YaRN, it improves Imaging Quality rather than degrading it.The video evaluation uses 1280 × 720 resolution and a fixed sequence length of 33 frames.Source: E. Additional Results · p. 16, Table 9 · p. 17
  • At 2656 × 2656, DyPE mitigates object duplication in multi-concept editing compared with vanilla Qwen-Image-Edit-2509.The examples include duplicated cats and baskets in the baseline output.Source: E. Additional Results · p. 17, Figure 11 · p. 18
  • For 4096 × 1365 panoramas, DY-YaRN outperforms YaRN on the reported metrics and preserves coherent spatial structure where YaRN distorts object placement.The comparison uses 20 Aesthetic-4K prompts and includes CLIP-Score, ImageReward, and Aesthetics-Score.Source: E. Additional Results · p. 17, Table 10 · p. 18

F. Attention Entropy Analysis

The analysis compares attention-entropy profiles at native and extrapolated resolutions across diffusion timesteps. DY-YaRN most closely preserves the baseline attention structure, with the lowest reported deviation.

Attention Entropy Analysis

  • The analysis averages normalized attention entropy across layers and heads over 20 Aesthetic-4K prompts, comparing native 1024 × 1024 FLUX with 4096 × 4096 extrapolations.The extrapolated conditions are FLUX, YaRN, and DY-YaRN.Source: F. Attention Entropy Analysis · p. 18
  • 0.0455 MAE for DY-YaRN is below YaRN (0.0476) and vanilla (0.0529), indicating the smallest deviation from the baseline attention profile.The figure plots normalized attention-entropy deviation across diffusion timesteps; lower values indicate better preservation.Source: F. Attention Entropy Analysis · p. 18, Figure 18 · p. 22

Related papers

Loading 2510.20766v2…