Source-linked AI summary
UltraGen: High-Resolution Video Generation with Hierarchical Attention
Teng Hu, Jiangning Zhang, Zihan Su, Ran Yi
TL;DR
Existing diffusion-transformer video models are constrained to lower resolutions by the quadratic cost of full attention. UltraGen uses hierarchical global-local attention with compressed global modeling and cross-window local attention for native high-resolution synthesis. The paper reports scaling to 1080P and 4K and outperforming existing methods and super-resolution pipelines in qualitative and quantitative evaluations.
Problem
Method
UltraGen combines local and global attention branches with spatially compressed global modeling and hierarchical cross-window local attention.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported results show that pre-trained low-resolution video models can be scaled to native high-resolution generation, up to 4K, for training and inference.
Source: §6 · p. 9Because its base model was designed for lower-resolution outputs, UltraGen may struggle with rapid or large-scale motion, producing minor artifacts or less natural motion.
Source: H. Limitations · p. 14
Abstract
from arXiv · showhide
Recent advances in video generation have made it possible to produce visually compelling videos, with wide-ranging applications in content creation, entertainment, and virtual reality. However, most existing diffusion transformer based video generation models are limited to low-resolution outputs (<=720P) due to the quadratic computational complexity of the attention mechanism with respect to the output width and height. This computational bottleneck makes native high-resolution video generation (1080P/2K/4K) impractical for both training and inference. To address this challenge, we present UltraGen, a novel video generation framework that enables i) efficient and ii) end-to-end native high-resolution video synthesis. Specifically, UltraGen features a hierarchical dual-branch attention architecture based on global-local attention decomposition, which decouples full attention into a local attention branch for high-fidelity regional content and a global attention branch for overall semantic consistency. We further propose a spatially compressed global modeling strategy to efficiently learn global dependencies, and a hierarchical cross-window local attention mechanism to reduce computational costs while enhancing information flow across different local windows. Extensive experiments demonstrate that UltraGen can effectively scale pre-trained low-resolution video models to 1080P and even 4K resolution for the first time, outperforming existing state-of-the-art methods and super-resolution based two-stage pipelines in both qualitative and quantitative evaluations.
1. Introduction
UltraGen addresses the cost and authenticity limits of high-resolution video generation with a hierarchical attention framework for native synthesis. Its experiments report native 1080P and 4K generation and stronger qualitative and quantitative results than existing methods.
1. Introduction
- Full attention scales quadratically with video spatiotemporal size, making native high-resolution generation prohibitively expensive for training and inference.Existing two-stage pipelines upscale low-resolution videos but may produce pseudo-high-resolution content with limited authenticity and richness.Source: §1 · p. 2
- UltraGen decomposes full attention into local and global branches for fine-grained regional content and overall semantic coherence.Its spatially compressed global modeling applies attention after frame-wise convolutional compression, then uses 3D convolutions to restore spatial fidelity and temporal continuity.Source: §1 · p. 2
- Hierarchical cross-window local attention uses intersecting windows across adjacent layers to improve information flow and consistency between local regions.The framework targets smooth content transitions and enhanced local detail.Source: §1 · p. 2
- UltraGen extends Wan-1.3B to native 1080P and 4K, reports the first native high-quality 4K synthesis, and outperforms existing methods qualitatively and quantitatively.The comparisons include state-of-the-art models and low-resolution-generation-plus-super-resolution pipelines.Source: §1 · p. 2
2. Related Work
Video generation has moved toward diffusion transformers that jointly model spatiotemporal relationships, but their attention costs limit resolution. Existing high-resolution approaches either blur beyond 2K or rely on super-resolution that may not add authentic detail.
2.1. Video Generation Foundation Models
- Early video-generation methods added temporal modules to image diffusion models, while DiT-based models treat videos as spatiotemporal volumes and use full attention.The related work describes Wan and HunyuanVideo as examples of models whose scale and data improve video quality and diversity.Source: §2.1 · pp. 2–3
- Wan and HunyuanVideo produce semantically rich, temporally consistent videos but remain limited to relatively low resolutions such as 720P because of quadratic self-attention costs.Source: §2.1 · p. 3
2.2. High-resolution Video Generation
- Beyond 2K, directly scaled models can produce blurry results, while two-stage super-resolution may sharpen video without adding enough authentic detail.The related work identifies limited spatial-resolution focus in some long-video methods and frames native HD generation as its target.Source: §2.2 · p. 3
3. Preliminaries
DiT-based video generation encodes video into latent tokens and applies full self-attention across the sequence. The resulting quadratic attention-map scaling makes high-resolution training and inference costly.
3. Preliminaries
- A 3D-VAE encodes video into a D × T × H × W latent representation, which patchification reshapes into a sequence of N = T × H × W tokens.Source: §3 · p. 3
- Full self-attention operates across the entire token sequence and forms an N × N attention map.Source: §3 · p. 3
- Attention complexity is O(N^2 · D); doubling video height and width quadruples token count and increases attention-map size sixteen-fold, limiting 1080P and 4K generation.Source: §3 · p. 3
4. UltraGen: Born for HD Video Generation
UltraGen decomposes full attention into complementary local and global branches, then combines them with spatial compression and hierarchical window interactions to model detail efficiently while retaining global coherence.
4.1. Time-Aware Global-Local Attention
- Local attention limits computation to spatial windows, while global attention connects regions to capture long-range dependencies and maintain semantic consistency.A timestep-conditioned fusion factor combines the fine-grained local latent with the semantically coherent global latent.Source: §4.1 · pp. 3–4
4.2. Spatially-Compressed Global Attention
- Spatially compressed global attention downsamples video latents before attention; depthwise convolution and domain-aware LoRA adapt global modeling using fewer spatial tokens and lightweight residuals.The compression convolution uses stride k, while LoRA adapts local attention parameters for global modeling.Source: §4.2 · p. 5
- After global modeling, bilinear upsampling and 3D convolution restore the latent resolution and jointly process spatial and temporal information for temporally consistent decompression.The convolution also mitigates interpolation's over-smoothing effect.Source: §4.2 · p. 5
4.3. Cross-window Hierarchical Local Attention
- Cross-window attention alternates window partitions across adjacent layers, using overlapping regions to propagate boundary information and improve continuity across local windows.The described alternation uses K × K windows and shifted (K+1) × (K+1) partitions.Source: §4.3 · p. 6
- Hierarchical Local Attention adds intermediate-scale attention over coarse windows to address fast-moving small objects that span multiple windows and need finer detail than global attention captures.Its shifted hierarchical windows and domain-specific adaptation support boundary information flow and fine-grained motion modeling.Source: §4.3 · p. 6
5. Experiments
UltraGen is evaluated for high-resolution video quality, inference efficiency, and the contribution of its proposed modules. Comparisons include direct generation and super-resolution pipelines, while ablations test attention and adaptation variants.
5.1. Implementation Details
- The evaluation compares Wan, HunyuanVideo, and CogVideo-X using direct target-resolution generation and low-resolution generation followed by super-resolution where supported.CogVideoX cannot directly generate HD video, so it is paired with super-resolution.Source: §5.1 · p. 6
5.2. Comparison Results
- UltraGen generates 1080P videos with finer details and prompt fidelity than the compared methods, whose outputs can be blurry, semantically inconsistent, or overly smooth.Figure 4 highlights zoomed-in regions; HunyuanVideo+SR has relatively good results but less detail than UltraGen.Source: §5.2 · p. 7, Figure 4 · p. 7
- UltraGen achieves the lowest HD-FVD on both 1080P and 4K, best HD-MSE, HD-LPIPS, and temporal consistency, and the best CLIP score among native-HD methods.Its CLIP score remains below Wan+SR; the reported comparison covers state-of-the-art methods and two-stage pipelines.Source: §5.2 · pp. 7–8, Table 1 · p. 7
- 4.78× speedup over Wan is reported for 4K inference, alongside a 2.7× speedup at 1080P.The inference-time comparison is shown in Table 2.Source: §5.2 · p. 8, Table 2 · p. 8
5.3. Ablation Studies
- The ablations associate removing global attention with disjoint content, while removing cross-window or hierarchical attention leaves coarse global relationships and window-boundary inconsistencies.Removing domain-aware LoRA yields blurrier results, and replacing the local module with Swin-Attention can produce semantically inconsistent content across windows.Source: §5.3 · p. 8, Figure 6 · p. 8
6. Conclusion
UltraGen combines hierarchical dual-branch attention with spatially compressed global modeling and cross-window local attention to support native high-resolution generation. The paper reports scaling pre-trained models to 1080P and 4K while outperforming existing methods and super-resolution pipelines in its evaluations.
6. Conclusion
- Its hierarchical dual-branch architecture separates local and global attention to produce high-fidelity regional details while maintaining overall semantic consistency.The conclusion also names spatially compressed global modeling and hierarchical cross-window local attention as components of the framework.Source: §6 · pp. 8–9
- UltraGen scales pre-trained low-resolution models to native 1080P and 4K generation and is reported to outperform state-of-the-art methods and super-resolution pipelines qualitatively and quantitatively.The conclusion attributes reduced computational complexity to spatially compressed global modeling and hierarchical cross-window local attention.Source: §6 · pp. 8–9
A. Overview
- The supplementary material covers efficiency, implementation, high-resolution metrics, VBench evaluation, quantitative ablations, visualizations, and limitations.Source: A. Overview · p. 11
B. Efficiency Analysis
The efficiency analysis decomposes full attention into local, global, and hierarchical components and estimates their computational cost. The accompanying material specifies training constraints and high-resolution evaluation metrics.
B. Efficiency Analysis
- Local attention partitions input into K × K windows, while global and hierarchical attention use compressed attention maps; together their cost is expressed as O(5(TWH)^2D/(4K^2) + (TWH)^2D/K^4).The analysis compares these components with full attention at O((TWH)^2D).Source: B. Efficiency Analysis · p. 11
- With K = 4, the analysis estimates an approximate 12-fold speedup over standard attention, though actual speedup is lower and approaches that value as resolution increases.The total complexity is O(5(TWH)^2D/(4K^2) + (TWH)^2D/K^4).Source: B. Efficiency Analysis · p. 11
C. More Implementation Details
- The 4K model is trained with 29 frames because of GPU memory constraints, whereas the 1080P model uses 81 frames.Training uses 42,000 4K videos over 50 epochs on 32 H20 GPUs.Source: C. More Implementation Details · p. 11
D. High-resolution Video Evaluation Metrics.
- HD-FVD compares generated and reference high-resolution video features extracted from high-resolution patches using pretrained I3D.Source: D. High-resolution Video Evaluation… · p. 11
- HD-MSE and HD-LPIPS assess fine detail by comparing videos with versions downsampled and upsampled to the original resolution; HD-LPIPS substitutes LPIPS for MSE.Higher HD-MSE indicates more fine details are lost during downsampling, reflecting high-resolution content.Source: D. High-resolution Video Evaluation… · pp. 11–12
E. Evaluation on Vbench
The VBench evaluation reports the highest overall average for UltraGen at both 1080P and 4K, with the best Imaging Quality score. Its comparison excludes super-resolution methods because VBench requires resizing videos to standard resolution.
E. Evaluation on Vbench
- VBench requires resizing videos to pretrained-model standard resolution, so super-resolution methods are excluded because downsampling would assess their base models rather than high-resolution generation.Source: E. Evaluation on Vbench · p. 12
- The method achieves the highest overall average score among compared methods at both 1080P and 4K, and the best Imaging Quality score, while not outperforming competitors on every individual metric.Source: E. Evaluation on Vbench · p. 12
F. Quantitative Ablation Studies.
UltraGen’s ablations report the strongest overall high-definition results, while omissions of global or cross-window attention expose boundary inconsistencies. Hierarchical attention and domain-aware LoRA also contribute to quality, and the cross-layer mechanism outperforms the Swin-Attention variant.
F. Quantitative Ablation Studies.
- UltraGen achieves the best HD-FVD and CLIP-L results, leads VBench in motion smoothness, aesthetic quality, and average score, and ranks second in imaging quality.The passage reports these outcomes in the quantitative ablation study; the supplied table caption does not provide cell values.Source: F. Quantitative Ablation Studies · p. 12, Table 4 · p. 13
- Removing global attention or cross-window local attention produces severe boundary inconsistencies and the lowest aesthetic and imaging quality.The comparison examines variants without global attention and without cross-window local attention.Source: F. Quantitative Ablation Studies · p. 12
- Omitting either hierarchical attention or domain-aware LoRA moderately decreases generation quality.The ablation study identifies both modules as contributors to improved generation quality.Source: F. Quantitative Ablation Studies · pp. 12–13
- UltraGen’s hierarchical cross-layer mechanism performs better in high-resolution video generation than the Swin-Attention variant.The compared variant replaces UltraGen’s local attention module with Swin-Attention [25].Source: F. Quantitative Ablation Studies · p. 12
G. More Visualization Results.
The additional visualizations compare UltraGen with native high-definition generators and super-resolution pipelines, reporting stronger prompt alignment and high-definition quality for UltraGen. Examples span 1080P and 4K, though 4K clips are limited to 29 frames by GPU memory constraints.
G. More Visualization Results.
- Native HunyuanVideo introduces noise and prompt-following failures at 4K, while native Wan outputs are overly smooth and lack detail.Super-resolution-based models follow prompts but their reliance on resolution leads to reduced detail and texture.Source: G. More Visualization Results · p. 13
- UltraGen aligns closely with prompts and achieves superior high-definition video quality compared with native HunyuanVideo and Wan and super-resolution-based methods.The comparisons include CogVideoX [37]+SR, HunyuanVideo [21]+SR, Wan [32]+SR, and the native HunyuanVideo and Wan models.Source: G. More Visualization Results · p. 13
- UltraGen produces high-quality, prompt-faithful examples at 1080P and 4K, but GPU memory limits 4K videos to 29 frames.The authors note that this frame limit may somewhat restrict temporal dynamics.Source: G. More Visualization Results · p. 13
H. Limitations
UltraGen inherits limitations from its low-resolution base model, particularly in challenging motion scenarios. Rapid or large-scale motion may be captured less accurately, with occasional minor artifacts or less natural motion.
H. Limitations
- UltraGen inherits limitations from its base model, which was originally designed for lower-resolution outputs.The paper identifies this inheritance as a limitation despite UltraGen’s high-resolution video capability.Source: H. Limitations · p. 14
- Rapid or large-scale motion can challenge accurate capture of complex dynamics, sometimes yielding minor artifacts or less natural motion.The passage frames these as occasional difficulties in particularly challenging scenarios.Source: H. Limitations · p. 14
- The authors identify improving robustness in demanding high-resolution settings as future work.Source: H. Limitations · p. 14