PaperlayerView exact revision on arXiv

Source-linked AI summary

SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers

Javad Rajabi, Kimia Shaban, Koorosh Roohi, David B. Lindell, Babak Taati

arXiv:2605.22668v1cs.CV

TL;DR

DiTs degrade when extrapolating beyond their training resolutions, while existing uniform RoPE scaling can trade global structure against fine detail. SEGA dynamically scales RoPE components using the latent’s spectral structure and consistently improves high-resolution synthesis across resolutions and model architectures.

  • Problem

    DiTs often suffer quality degradation and structural breakdown when generating beyond their training resolutions, while uniform RoPE scaling is suboptimal across spatial-frequency bands.

  • Method

    SEGA dynamically derives per-component RoPE attention scaling from the latent’s spectral structure at each denoising step.

  • Results

    SEGA consistently improves structural coherence and fine-detail fidelity across baselines, resolutions, and model architectures, including resolutions exceeding 36 million pixels.

  • Takeaways & Limitations

    Frequency-aware attention scaling provides a training-free, parameter-free modification for more stable high-resolution synthesis without retraining or architectural changes.

  • Takeaways & Limitations

    At extreme extrapolation factors, perceptual quality still degrades because the underlying model’s capacity, rather than positional encoding alone, becomes limiting.

Abstract

from arXiv · show

Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. However, these strategies apply a uniform and content-agnostic scaling across RoPE components with distinct frequency characteristics, inducing a trade-off between preserving global structure and recovering fine detail. We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent's spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity. Experiments show that SEGA consistently improves high-resolution synthesis across multiple target resolutions, outperforming state-of-the-art training-free baselines.

1 Introduction

SEGA is a training-free, content-aware method for resolution extrapolation in diffusion transformers that adapts attention scaling to latent spectral structure. It addresses RoPE-induced attention dilution beyond training resolutions, improving structural coherence and fine-detail fidelity without model or architectural changes.

  • DiTs remain constrained by training resolutions, typically 1024^2 to 2048^2, limiting high-resolution synthesis beyond that range.
  • At extrapolated resolutions, RoPE positional offsets deviate from training conditions, diluting attention across the expanded token grid and weakening spatial discrimination.
  • SEGA derives per-component attention-scaling magnitudes from the latent’s spatial-frequency energy at each denoising step.RoPE components correspond to spatial frequencies: low-frequency components support coarse structure, while high-frequency components support fine detail and texture.
  • SEGA consistently improves structural coherence and fine-detail fidelity across baselines and resolutions, including ultra-high resolutions exceeding 36 million pixels.It uses no learnable parameters, fine-tuning, or architectural changes and integrates directly into standard RoPE-based pipelines.
  • Static scaling cannot balance structural coherence against fine-detail fidelity, whereas SEGA dynamically allocates scaling according to spectral energy.

2 Related Work

Related work addresses high-resolution generation through either training-based adaptation or training-free inference-time modifications. In DiTs, this challenge parallels long-context extrapolation in LLMs and has motivated RoPE-based recalibration methods.

  • High-resolution generation: Training-based approaches use progressive upsampling, latent-space super-resolution, or high-resolution retraining and model-specific fine-tuning.Diffusion-4K is cited as an example of model-specific fine-tuning.
  • High-resolution generation: Training-free methods instead adapt pretrained models during inference to improve high-resolution generation.The passage contrasts these methods with training-based approaches and cites several training-free methods.
  • RoPE-based extrapolation: High-resolution generation in DiTs closely resembles long-context extrapolation in LLMs, with both challenges driven substantially by advances in RoPE.RoPE-based methods formulate extrapolation as recalibration of rotary frequencies.
  • RoPE-based extrapolation: Position Interpolation compresses position indices into the training range to limit phase drift, while NTK adjusts RoPE’s base frequency to redistribute positional variation.Both methods are presented as standard training-free approaches for extrapolation.

3 Preliminaries

RoPE supplies DiT models with relative spatial positional information by rotating paired embedding dimensions at frequency-specific rates. Although effective within training, RoPE-based models degrade at unseen resolutions, motivating inference-time extrapolation and attention-scaling methods that remain uniform across frequencies.

  • Rotary Position Embedding (RoPE): RoPE encodes spatial priors for transformer architectures by representing relative positions through rotations in embedding space.It addresses transformer permutation equivariance and is used in recent text-to-image models including Flux and Qwen.
  • Rotary Position Embedding (RoPE): RoPE partitions each D-dimensional vector into D/2 paired subspaces and rotates subspace d using an angular frequency determined by its dimension index.The frequency is θ_d = b^(-2d/D), with b = 10,000.
  • Rotary Position Embedding (RoPE): Applied to queries and keys before attention, RoPE makes their dot product depend only on relative distance and is typically applied axially to 2D images.Half the hidden dimensions encode horizontal positions and the other half encode vertical positions.
  • Resolution Extrapolation: At unseen resolutions, RoPE-based models encounter out-of-distribution positional offsets, prompting inference-time extrapolation methods such as PI and YaRN.PI compresses positions uniformly using n → n/s, while YaRN applies a constant logit scaling factor τ(s).
  • Resolution Extrapolation: Existing attention scaling remains uniform across RoPE frequencies, despite different RoPE dimensions exhibiting distinct characteristics and contributing unevenly to spatial structure.YaRN’s scaling factor was empirically determined for language-model length extrapolation and later adopted in image generation.

4 Method

SEGA is a training-free DiT method that replaces fixed, uniform RoPE scaling with content-aware, per-dimension scaling derived from the latent’s spectral structure at each denoising step. It combines axis-wise and radial spectral profiles to redistribute scaling across RoPE dimensions while controlling its overall strength.

  • Method overview: SEGA derives content-aware, per-dimension RoPE scaling from the current latent’s spectral structure at each denoising step.This replaces fixed and uniform scaling with dynamic modulation tied to latent content.
  • 4.1 Spectral structure extraction: SEGA transforms the latent into a zero-centered 2D spatial map, computes its power spectrum, and extracts axis-wise and rotation-invariant radial energy profiles.Axis-wise profiles preserve directional frequency information, whereas the radial profile summarizes energy across spatial scales.
  • 4.1 Spectral structure extraction: Axis-wise profiles independently determine scaling across height- and width-related RoPE dimensions, while the radial profile determines the scaling strength.This separates directional redistribution from global amplitude control.
  • 4.2 Spectral modulation: The modulator combines a resolution-ratio reference scale, a per-dimension correction based on RoPE wavelength and standardized log-energy, and a global amplitude gate.The reference scale depends on target-to-training resolution, while the correction redistributes scaling across dimensions with zero overall shift.
  • 4.2 Spectral modulation: Spectral flatness controls the gate: σ approaches 0 without clear spectral structure and approaches 1 as structural content resolves, enabling adaptive redistribution across denoising steps.The resulting scaling sharpens attention at under-resolved frequencies and softens it at over-emphasized ones.

5 Analysis of Spectral-Energy Guided Attention

SEGA adapts attention scaling to each image’s spectral structure during denoising, redistributing emphasis across RoPE dimensions. This content-aware modulation sharpens under-resolved frequencies while improving attention stability relative to fixed, uniform scaling.

  • Scaling Behavior: SEGA produces a customized scaling map for each image, with scaling factors distributed throughout the denoising process.The analysis compares scaling maps generated from two distinct prompts.
  • Spectral Redistribution: SEGA redistributes scaling across RoPE dimension d, sharpening under-resolved spatial-frequency bands and softening over-emphasized ones.This spectral redistribution is guided by the latent’s spatial-frequency structure.
  • Attention Stability: YaRN’s fixed, uniform scaling causes attention dilution, whereas SEGA shapes the attention grid earlier in denoising by dynamically modulating rotary-embedding magnitude.The analysis attributes YaRN’s failure to reduced discrimination between positional offsets.
  • Attention Stability: SEGA’s dynamic rotary-embedding modulation preserves semantic locality and entity consistency while stabilizing attention.These properties are presented as consequences of content-aware spectral redistribution.

6 Experiments

SEGA is evaluated across Flux and Qwen using high-resolution benchmarks, where it improves structural coherence, semantic fidelity, and image quality over training-free baselines. Its gains remain consistent across resolutions and aspect ratios, including 4096^2, while ablations support dynamic spectral guidance over fixed scaling.

  • Baselines: SEGA is compared with direct-inference methods and multi-stage guidance approaches across Flux and Qwen, with multi-stage methods evaluated only on Flux.Compared baselines include NTK, YaRN, DyPE, UltraImage, HiFlow, I-Max, and ScaleDiff.
  • Evaluation: Experiments use Aesthetic-4K prompts and reference images across four high-resolution configurations, evaluating FID, MUSIQ, CLIP-IQA, and semantic alignment.The configurations include 2048 × 4096, 4096 × 2048, 3072^2, and 4096^2.
  • Qualitative comparison: SEGA better preserves global structure, fine-grained semantic fidelity, and overall visual quality than direct-inference methods across Flux and Qwen, including complex prompts.YaRN, DyPE, and UltraImage often exhibit structural degradation, artifacts, and semantic omissions at high resolutions.
  • Quantitative comparison: At 4096^2 resolution, SEGA outperforms all baseline models across every evaluated metric, establishing a new benchmark for high-resolution generation.The broader quantitative comparison reports state-of-the-art performance across both Flux and Qwen architectures.
  • Robustness and ablation: SEGA maintains stable results across increasing resolutions and non-square aspect ratios, while ablation results show dynamic spectral guidance improves all metrics over fixed scaling.The ablation is conducted on Flux at 4096^2, where fixed scaling fails to maintain structural integrity at extreme resolutions.

7 Conclusion … A.1 High-Resolution Image Synthesis

SEGA is a training-free DiT method that adapts RoPE-component attention scaling to each latent’s spectral structure, addressing the trade-off between global coherence and fine-detail fidelity without retraining or architectural changes. The related work contrasts training-based and training-free approaches for high-resolution synthesis across latent diffusion models and DiTs.

  • 7 Conclusion: SEGA adapts RoPE-component scaling to the current latent’s spectral structure for training-free high-resolution extrapolation in DiTs.Its frequency-aware attention scaling targets limitations of uniform scaling strategies.
  • 7 Conclusion: Frequency-aware scaling addresses the trade-off between preserving global coherence and recovering fine-detail fidelity.The method modifies attention scaling across RoPE components according to their frequency characteristics.
  • 7 Conclusion: SEGA requires no retraining or architectural changes while consistently improving high-resolution synthesis.The supplied conclusion characterizes this as a simple modification to existing DiT inference behavior.
  • A.1 High-Resolution Image Synthesis: Training-based methods fine-tune models on curated high-resolution data, with Diffusion-4K using wavelet supervision for 4K synthesis.Diffusion-4K achieves strong perceptual quality but incurs retraining costs and reduced architectural generalizability.
  • A.1 High-Resolution Image Synthesis: Training-free U-Net methods include progressive upscaling, scale fusion, frequency-aware sampling, and inference-time dilated convolutions.The cited examples are DemoFusion, FreeScale, FreCaS, ScaleCrafter, and DiffuseHigh.
  • A.1 High-Resolution Image Synthesis: Training-free DiT methods generally use direct inference or multi-stage guidance approaches.Multi-stage methods condition high-resolution sampling on guidance extracted from base-resolution generation; I-Max uses projected flows for coarse-structure stabilization, while HiFlow constructs a virtual reference flow.

A.2 RoPE-Based Length Extrapolation Methods … C Additional Implementation Details

The paper reviews RoPE-based extrapolation methods for 2D image generation and motivates SEGA through their limitations: fixed or schedule-driven corrections fail to account for content-dependent spectral structure. Additional analyses show that latent spectra evolve across prompts and denoising, while SEGA is evaluated under specified attention and implementation settings.

  • A.2 RoPE-Based Length Extrapolation Methods: RoPE extrapolation methods modify the schedule θ_d, attention scaling, or both, with axial RoPE using separate height and width schedules and per-axis ratios s_H and s_W.This defines the 2D spatial setting used for image-generation extrapolation.
  • A.2.1 Position Interpolation (PI): Position Interpolation uniformly contracts all RoPE frequencies by each axis ratio, reducing phase drift but treating coarse structure and fine detail identically.PI maps extrapolated positions back into the training range through n^(a) → n^(a)/s_a.
  • A.2.2 NTK: NTK changes the RoPE base per axis, but the original correction is too weak for 2D extrapolation, causing inadequate positional discrimination and blurred or repetitive outputs.The stronger variant better preserves positional contrast, while remaining fixed with respect to the latent, sample, and denoising state.
  • A.2.3 YaRN: YaRN partitions RoPE dimensions into frequency bands and blends PI-style interpolation with extrapolation using a wavelength-dependent ramp, yet frequently causes spatial structure collapse and layout confusion in images.The paper attributes this failure to selectively blending frequency dimensions despite jointly encoded 2D spatial structure.
  • A.2.4 DyPE: DyPE adapts the extrapolation ratio to the denoising timestep, making it more adaptive than PI, NTK, and YaRN, but its schedule remains predefined rather than latent-driven.SEGA is complementary because it derives modulation from the current latent’s spectral structure.
  • A.2.5 UltraImage: UltraImage corrects dominant RoPE frequencies recursively to address repetition and uses entropy-guided attention concentration for quality degradation, whereas SEGA differs in diagnosis and mechanism.UltraImage’s corrections are discrete, while SEGA is motivated by spectral-energy allocation across RoPE dimensions.
  • B.1 Content-Dependent Spectral Structure of Latent Representations: Latent spectral energy varies across prompts and denoising: landscapes spread energy into mid- and high-frequency bands, portraits concentrate it more sharply at low frequencies, and low frequencies emerge first during denoising.These observations motivate content-aware scaling rather than fixed global correction.
  • B.2 Attention Entropy Analysis; B.3 Additional Attention Evolution Results; C Additional Implementation Details: Attention entropy measures whether high-resolution attention remains close to the native-resolution baseline, with deviations indicating diluted or collapsed focus; additional analyses compare YaRN and SEGA across spatial tokens at 4096^2 resolution.Experiments average entropy differences across seeds, prompts, layers, and heads, while implementation uses Flux and Qwen DiTs with bfloat16 weights and activations on NVIDIA H100 GPUs.

D Limitation and Discussion … F.1 Generalization to Alternative Backbones

SEGA generalizes across model architectures and high-resolution settings, but remains dependent on an underlying RoPE length-extrapolation method and degrades at extreme extrapolation factors. The paper also discusses societal risks, licensing, safeguards, and quantitative comparisons across several backbones and resolutions.

  • D Limitation and Discussion: SEGA modulates rotary-embedding magnitude without extending RoPE’s positional range, so it is composed with an underlying length-extrapolation method and inherits some structural priors.The experiments use NTK as the underlying length-extrapolation method.
  • D Limitation and Discussion: Perceptual quality continues to degrade at the most extreme extrapolation factors, despite SEGA scaling up to 8192^2.The passage identifies extreme extrapolation as a remaining limitation.
  • E Societal Impact and Safeguards: Higher-resolution generation can benefit creative workflows, design, visualization, and research by enabling more realistic and detailed synthesis without additional training.The passage frames these benefits alongside potential misuse risks.
  • E Societal Impact and Safeguards: Increased realism may heighten risks including disinformation, impersonation, non-consensual synthetic imagery, and amplification of existing social biases.These risks motivate safeguards for deployment.
  • F Additional Quantitative Results: Table 4 compares SEGA with state-of-the-art baselines on SDXL and Diffusion-4K across four high-resolution settings on Aesthetic-4K.The table marks best and second-best results with bold and underlining.
  • F Additional Quantitative Results: Table 5 reports a quantitative comparison at 4096^2 on a zero-shot benchmark, grouping methods by backbone model.Best and second-best results are bolded and underlined within each group, and † denotes a closed-source proprietary model.
  • F.1 Generalization to Alternative Backbones: On an alternative backbone, SEGA is evaluated across four high-resolution settings against fine-tuning and multi-stage-guidance baselines, including Diffusion-4K, DemoFusion, FreCas, FreeScale, and DiffuseHigh.The evaluation uses Aesthetic-4K and includes methods built on SDXL.
  • F.1 Generalization to Alternative Backbones: SEGA’s spectral-energy-guided scaling transfers effectively across different model architectures without architecture-specific tuning.Tables 6 and 7 provide comparisons at 5120^2 and 6144^2 resolution on Aesthetic-4K.

F.2 Zero-Shot Benchmark · F.3 Extreme Resolution: 51202 and 61442

The paper introduces a zero-shot benchmark to test generalization beyond possible Aesthetic-4K familiarity, where SEGA leads across metrics. It also evaluates extreme 51202 and 61442 resolutions, finding SEGA remains substantially more consistent as resolution increases.

  • F.2 Zero-Shot Benchmark: The zero-shot benchmark addresses whether Aesthetic-4K results reflect genuine generalization or dataset familiarity.The benchmark was motivated by concerns that some models may have encountered Aesthetic-4K during large-scale pretraining.
  • F.2 Zero-Shot Benchmark: An LLM generated 200 curated high-detail prompts spanning diverse scenes, lighting, subjects, styles, and compositions while minimizing Aesthetic-4K overlap.The benchmark additionally includes Nano Banana 2, a closed-source commercial baseline.
  • F.2 Zero-Shot Benchmark: At 40962 resolution, SEGA achieves the best performance across all metrics on both Flux and Qwen backbones.These results are reported in Table 5 for the zero-shot benchmark.
  • F.2 Zero-Shot Benchmark: On Qwen, SEGA reaches an ImageReward score of 1.58 and a PickScore of 23.86, approaching or matching Nano Banana 2.Nano Banana 2 reports IR: 1.37 and PS: 23.43.
  • F.3 Extreme Resolution: 51202 and 61442: Tables 6 and 7 evaluate SEGA at extreme resolutions of 51202 and 61442, approximately 26 and 38 million pixels.These correspond to 25× and 36× area extrapolation beyond Flux’s 10242 training resolution.
  • F.3 Extreme Resolution: 51202 and 61442: Because generation at these scales is costly, evaluation uses a randomly selected subset of 20 prompt–image pairs from Aesthetic-4K.The results show that SEGA remains substantially more consistent as resolution increases.

G Additional Qualitative Results

SEGA consistently produces sharp, coherent, and prompt-aligned images across varied aspect ratios, zero-shot prompts, guidance-based comparisons, and challenging 5K–6K extrapolation settings. Qualitative results further show that it preserves both global structure and local texture without common high-resolution artifacts.

  • Aspect-Ratio Robustness: SEGA preserves intended image geometry across vertical and horizontal aspect ratios without stretching or distorting objects.Comparisons with YaRN, DyPE, and UltraImage on Flux and Qwen also show sharpness, visual coherence, and prompt alignment.
  • Zero-Shot Generation: On zero-shot prompts, SEGA avoids repeated structures, distorted layouts, and semantic-clarity loss while retaining fine detail and global composition.The comparison includes YaRN, DyPE, and UltraImage on both Flux and Qwen.
  • Guidance-Based Comparisons: Against ScaleDiff, I-Max, and HiFlow, SEGA directly improves high-resolution denoising rather than relying on upsampled or guided low-resolution generation.The guidance-based alternatives may preserve coarse structure but can introduce artifacts, uneven detail, or foreground-background inconsistencies.
  • Ultra-High-Resolution Extrapolation: At 5K and 6K, SEGA maintains visual sharpness, structural consistency, and prompt alignment where direct-inference baselines may produce severe artifacts or incoherent images.Figures 12 and 13 compare SEGA with DyPE and UltraImage in challenging extrapolation regimes.
  • Fine-Detail Preservation: Zoomed-in ultra-high-resolution results show that SEGA preserves local texture and object-level detail while maintaining the broader image structure.The result supports benefits for fine-scale fidelity and global coherence simultaneously.

H Additional Ablation: Choice of Baseline Scaling mref

The reference scale mref anchors rotary scaling across RoPE dimensions and varies with the target-to-training resolution ratio. Power-law and logarithmic formulations behave similarly at moderate extrapolation, but the power-law form is more stable as resolution increases.

  • Baseline scaling choice: mref anchors the shared rotary scaling across all RoPE dimensions and is defined as a function of the resolution ratio s = Rtarget/Rtrain.Both formulations reduce to mref = 1 at s = 1 and grow monotonically with s.
  • Baseline scaling choice: The power-law and logarithmic formulations behave similarly for moderate extrapolation, approximately s ≈1–2, but diverge as s grows.The reported experiments use κ = 0.08 for both formulations.
  • High-resolution behavior: At larger extrapolation factors, a larger mref provides a stronger positional-discrimination anchor to counter attention dilution from the expanded token grid.The motivation is that increasing target resolution makes positional offsets harder to discriminate even with RoPE extrapolation.
  • Empirical comparison: At 4096^2, the two forms perform similarly, while the power-law variant has a small but consistent advantage; its gap widens at 5120^2 and remains clear at 6144^2.The comparison uses identical SEGA settings on FLUX, with lower FID and stronger alignment across most metrics for the power-law form at 6144^2.
  • Empirical comparison: The power-law baseline extrapolates more stably as resolution increases, consistent with its faster growth at large s.Table 8 attributes this faster growth to a stronger positional-discrimination anchor at extreme extrapolation factors.
Loading 2605.22668v1…