Source-linked AI summary
LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Experts
Chen Zhao, Jiawei Chen, Hongyu Li, Zhuoliang Kang, Shilin Lu, Xiaoming Wei, Kai Zhang, Jian Yang, Ying Tai
TL;DR
Ultra-high-resolution video generation remains difficult because motion modeling, semantic planning, and detail synthesis degrade at scale. LUVE addresses these issues with a three-stage latent-cascaded framework and dual-frequency refinement, achieving state-of-the-art UHR video performance while retaining computational efficiency as an open challenge.
Problem
UHR video generation remains challenging because high resolution amplifies difficulties in motion modeling, semantic planning, and fine-grained detail synthesis.
Method
LUVE combines low-resolution motion generation, direct video latent upsampling, and high-resolution content refinement with low- and high-frequency experts.
Results
LUVE achieves state-of-the-art performance in UHR video generation, with ablation studies confirming the effectiveness of each component.
Takeaways & Limitations
LUVE provides a three-stage framework for synthesizing high-quality, detail-rich UHR videos while improving semantic coherence and fine-grained detail generation.
Takeaways & Limitations
Computational efficiency remains a key challenge, motivating future work on efficient UHR video generation.
Abstract
from arXiv · showhide
Recent advances in video diffusion models have significantly improved visual quality, yet ultra-high-resolution (UHR) video generation remains a formidable challenge due to the compounded difficulties of motion modeling, semantic planning, and detail synthesis. To address these limitations, we propose \textbf{LUVE}, a \textbf{L}atent-cascaded \textbf{U}HR \textbf{V}ideo generation framework built upon dual frequency \textbf{E}xperts. LUVE employs a three-stage architecture comprising low-resolution motion generation for motion-consistent latent synthesis, video latent upsampling that performs resolution upsampling directly in the latent space to mitigate memory and computational overhead, and high-resolution content refinement that integrates low-frequency and high-frequency experts to jointly enhance semantic coherence and fine-grained detail generation. Extensive experiments demonstrate that our LUVE achieves superior photorealism and content fidelity in UHR video generation, and comprehensive ablation studies further validate the effectiveness of each component. The project is available at \href{https://unicornanrocinu.github.io/LUVE_web/}{https://github.io/LUVE/}.
1. Introduction
LUVE addresses the degradation of ultra-high-resolution video generation by combining low-resolution motion generation, latent-space upsampling, and high-resolution content refinement with dual-frequency experts.
- Challenges: UHR video generation is hindered by degraded quality, static or incoherent motion, semantic repetition, motion blur, and texture degradation.These limitations constrain applications requiring fine-grained detail and high visual fidelity.
- LUVE Framework: LUVE uses three collaborative stages: low-resolution motion generation, video latent upsampling, and high-resolution content refinement.The stages generate motion-consistent latents, upscale directly in latent space, and refine high-resolution content.
- Contributions: LUVE is presented as a three-stage cascaded architecture for producing high-quality, detail-rich UHR videos.The paper summarizes the framework as integrating LMG, VLU, and HCR.
- LUVE Framework: Video latent upsampling performs arbitrary-resolution upsampling directly on video latents.This design avoids the substantial memory and computation overhead associated with VAE codecs.
- Dual-Frequency Experts: Dual-frequency experts assign global semantic coherence to the low-frequency expert and fine-grained realistic texture synthesis to the high-frequency expert.The high-resolution refinement stage integrates both experts to improve semantic fidelity and detail generation.
2. Related Works
Related work develops video generation through spatiotemporal diffusion modeling and addresses UHR synthesis with training-free, fine-tuning, and super-resolution paradigms, each involving quality or scalability trade-offs.
- Video Generation: Early video diffusion methods added temporal modules to text-to-image models but often failed to model holistic spatiotemporal dependencies.DiT-based approaches subsequently became a dominant paradigm for jointly modeling spatial and temporal correlations.
- Video Generation: Modern text-to-video models commonly combine 3D VAE spatiotemporal compression with DiT latent-space denoising.Transformer-based models jointly model spatial and temporal correlations through full or interleaved attention mechanisms.
- UHR Generation: UHR visual generation is challenged by immense computational demands, limited high-quality data, and current models’ scalability constraints.Existing UHR research primarily follows training-free, fine-tuning, and super-resolution frameworks.
- UHR Generation: Training-free approaches improve computational efficiency by modifying denoising or attention, but often produce over-smoothed textures and unrealistic details.Fine-tuning on high-resolution datasets enhances fidelity while preserving generative priors.
- UHR Generation: Super-resolution pipelines upscale low-resolution generations to recover finer details but mainly improve perceptual sharpness without adding semantic or structural content.These methods typically use dedicated spatial or video super-resolution networks after low-resolution generation.
3. Methodology
LUVE uses a three-stage cascade to address motion degradation, latent upsampling costs, and semantic-detail limitations in ultra-high-resolution video generation. Its high-resolution refinement combines frequency-specialized experts, while VLUer performs direct latent-space upsampling and improves reconstruction quality.
- Framework Overview: LUVE integrates low-resolution motion generation, video latent upsampling, and high-resolution content refinement.LMG provides motion-consistent low-resolution latents, VLU upsamples within latent space, and HCR refines high-resolution content.
- Video Latent Upsampling: VLUer performs flexible arbitrary-resolution upsampling directly on video latents, avoiding the memory and computational burden of conventional VAE codec processing.The upsampler uses an encoder, video INR upsampler, and decoder; the decoder further models temporal dependencies in the high-resolution latent space.
- High-Resolution Content Refinement: LUVE's high-resolution stage is designed to enhance semantic fidelity and content richness rather than merely sharpen perceptual details.This distinguishes the framework from cascaded approaches whose high-resolution stages primarily function as perceptual enhancers.
- Framework Overview: High-resolution generation can produce nearly static motion, spatial inconsistencies, motion blur, texture degradation, and insufficient high-frequency information.These difficulties arise from the compounded challenges of motion modeling, semantic planning, and fine-grained detail synthesis.
- Video Latent Upsampling: The VLUer decoder alleviates blurriness, while pixel-level supervision mitigates blocky artifacts in decoded videos.The training objective combines latent reconstruction with pixel-space supervision, with an additional temporal loss introduced to improve coherence.
- High-Resolution Content Refinement: Dual-frequency experts assign global semantic coherence to the low-frequency expert and fine-grained realistic textures to the high-frequency expert.The low-frequency expert operates during high-noise denoising, while the high-frequency expert refines details during low-noise denoising.
4. Experiments
LUVE is evaluated against UHR text-to-video and video super-resolution methods, with additional human studies and ablations. The experiments report stronger generation quality, detail preservation, realism, and component effectiveness.
- Comparison with T2V Models: LUVE achieves the highest VBench score and substantial improvements across UHR generation metrics compared with UltraWan and CineScale.Visual comparisons also show coherent complex motion, fine-grained detail preservation, and strong semantic consistency.
- Comparison with VSR Models: LUVE outperforms RealBasicVSR, VEnhancer, STAR, and FlashVSR across all reported video super-resolution metrics.The comparisons attribute the advantage to better intricate-detail enhancement and overall video realism.
- Human Study: LUVE receives the highest human preference score across video quality, detail quality, temporal consistency, and text-video alignment.The study uses 60 VBench-generated videos and 20 participants making pairwise comparisons.
- Ablation with Different Upsampling: The proposed latent upsampler achieves the best quantitative results while improving computational efficiency over RGB interpolation.Latent interpolation causes color distortion, whereas RGB interpolation causes blurriness; the proposed method preserves color fidelity and fine structure.
- Ablation with Dual Experts: The low-frequency expert primarily improves content fidelity, while the high-frequency expert focuses on fine-grained detail generation.Removing either expert degrades its associated function, with low-frequency removal weakening semantic planning and consistency and high-frequency removal reducing details.
5. Conclusion
The paper concludes that LUVE combines a three-stage cascade with dual-frequency experts for high-quality UHR video synthesis. It reports state-of-the-art performance, while identifying computational efficiency as an ongoing challenge.
- Conclusion: LUVE integrates low-resolution motion generation, video latent upsampling, and high-resolution content refinement in a three-stage UHR video framework.The conclusion presents this architecture as the basis for synthesizing high-quality UHR videos.
- Conclusion: LUVE achieves state-of-the-art performance, and ablation studies confirm the effectiveness of each component.
- Limitations and Future Works: Computational efficiency remains a key challenge despite LUVE's reported performance.Future work is directed toward more efficient UHR video generation.
Impact Statement
LUVE is presented as a latent-cascaded UHR video generation framework that combines dual-frequency experts. The supplied analysis examines how refinement-step choices affect motion and semantic correction.
- Impact Statement: LUVE combines a latent-cascaded architecture with dual-frequency experts for native UHR video synthesis and active content completion.The framework is positioned against training-free and VSR approaches that can produce pseudo-high-resolution artifacts and limited realism.
- Impact Statement: Using S = 5 achieves the best overall performance, while smaller or larger skipped-step settings impair motion coherence or semantic correction.S = 2 struggles with coherent motion, whereas S = 10 or S = 15 fails to correct semantic inconsistencies.
B. Discussion on Efficiency
LUVE is evaluated for inference efficiency against UHR video generation baselines under a shared Wan2.1 1.3B foundation. The results report lower latency and memory use, but substantial UHR computational demands remain a commercialization boundary.
- Efficiency Comparison: LUVE consistently outperforms UltraWan and CineScale in inference latency and memory under the efficiency benchmark.The reported advantage is attributed primarily to the lightweight VLUer and Dual Frequency Expert designs.
- Evaluation Setting: The efficiency evaluation uses 49-frame videos for temporal scalability, while main-text comparisons retain the 29-frame 4K setting supported by UltraWan.
- Component Efficiency: Removing the Dual Frequency Experts produces only a marginal inference-time reduction, indicating substantial quality gains with negligible computational-budget impact.
- Limitations and Future Works: Ultra-high-resolution video generation still requires substantial computational expenditure, leaving a significant gap before large-scale commercial applications.Future work targets more efficient UHR generation paradigms.
C.1. Training Data
VLUer training uses curated UHR videos, latent-space LR/HR pairs, and arbitrary-scale upsampling to support efficient temporally coherent reconstruction. Its lightweight encoder–INR–decoder design is validated through reconstruction experiments and ablations.
- Training Data: Approximately 20,000 videos with native resolution at least 1440×1440 are retained for VLUer training.The data are resized and center-cropped to 1440×1440 before latent encoding.
- Training Data: HR videos are downsampled by 1.5×, 2.0×, and 3.0× to create LR input latents for simulated super-resolution training.Both HR and LR videos are encoded with the frozen Wan2.1 VAE.
- VLUer Architecture: The VLUer performs arbitrary-scale latent upsampling with temporal coherence using an encoder, INR upsampler, and lightweight decoder.The model contains approximately 22M parameters and operates directly on video latents.
- VLUer Architecture: The INR takes queried 3D coordinates and encoded features to produce a 16-channel coarse high-resolution latent representation.Its MLP uses four hidden layers with dimensions [512, 512, 256, 256].
- Reconstruction Evaluation: On 60 generated clips, VLUer provides comparable RGB reconstruction metrics while outperforming RGB interpolation in latent-space evaluation.Ablations also identify the decoder and pixel-level loss as important for balancing perceptual RGB quality and latent fidelity.
E. Human Study Settings
The human study compares LUVE with three alternative methods through anonymized, randomized side-by-side viewing. Participants assess four quality dimensions, and LUVE receives the highest preference scores with confidence intervals above 50%.
- Evaluation Setup: Sixty prompts generate comparison videos from LUVE, UltraWan, CineScale, and STAR.STAR serves as the VSR baseline by upsampling Wan2.1-1.3B outputs.
- Evaluation Setup: Videos are shown simultaneously in a randomized, anonymized 2 × 2 grid with the corresponding text prompt.Participants can play, pause, and replay each video in the web-based interface.
- Evaluation Setup: Twenty participants rate Overall Video Quality, Detail Quality, Temporal Consistency, and Text-Video Alignment.Full-screen viewing on high-definition displays is required for the UHR samples.
- Results: LUVE achieves the highest human preference scores, with confidence-interval lower bounds above 50% across all four metrics.The intervals are evaluated at confidence levels ranging from 95% to 97.5%.
F. FIDpatch Evaluation Detail
FIDpatch evaluates localized texture and high-frequency fidelity in UHR videos, complementing global metrics that can miss fine detail. The section positions this evaluation against training-free and super-resolution alternatives and their known artifact risks.
- Metric: FIDpatch measures the statistical distribution of localized patches to assess UHR textural fidelity and high-frequency detail.It is used because standard global FID can overlook localized nuances after downsampling.
- Metric: All baseline methods use the same experimental configuration with a fixed 256 × 256 patch size.This standardizes the patch-based comparison.
- Comparison Context: Training-free methods can produce over-smoothed textures and unrealistic high-frequency details because their pretrained T2V models lack UHR exposure.These approaches modify architectures or inference strategies rather than training on UHR data.
- Comparison Context: VSR-based frameworks can sharpen perception but may hallucinate details and lose structural integrity at extreme scaling factors such as 8× or 16×.The cited limitation concerns outputs that appear sharp while lacking authentic realism and structural richness.
H. MLLM-Based UHR Video Evaluation
The MLLM-based evaluation uses multidimensional expert-style rubrics to assess UHR video realism, detail, and prompt alignment. The protocols emphasize meaningful high-frequency content, physical plausibility, temporal stability, and semantic fidelity rather than sharpness alone.
- Evaluation Dimensions: Doubao-1.5 Pro evaluates UHR videos across Realism, Detail, and Alignment.These pillars cover physical authenticity and content fidelity, textural richness, and semantic consistency with the prompt.
- Realism: The realism rubric checks semantic fidelity, object permanence, biomechanically plausible motion, physically consistent rendering, and temporal coherence.It specifically warns that sharp resolution should not compensate for physically impossible content.
- Scoring Protocol: Each dimension uses a 1–10 scoring scale with conservative criteria and requires reasons tied to specific video timestamps.The scoring instructions reserve top scores for near-perfect realism, detail stability, or semantic alignment.
- Detail: The detail rubric distinguishes meaningful texture and structural high-frequency information from upscaled noise, over-sharpening, and repetitive artifacts.Fine structures should remain stable rather than vibrate, disappear, or blur during motion.
- Alignment: The alignment rubric checks entity presence, action accuracy, attribute and style consistency, and spatial-temporal logic against the target prompt.High visual quality does not receive credit when the generated video fails to follow the prompt.
I. Addition Results
LUVE produces high-fidelity UHR videos across diverse scenes, improving not only sharpness but also semantic fidelity, motion coherence, and scene understanding at extreme resolutions.
- LUVE consistently produces high-fidelity UHR videos across diverse scenes in additional qualitative results.
- The method improves semantic fidelity through more accurate object details, structurally coherent motion patterns, and better scene understanding at extreme resolutions.
- Additional results demonstrate LUVE's performance at both 2K and 4K video resolutions.