Source-linked AI summary
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, Nan Duan
TL;DR
Generating arbitrarily sized high-resolution images and long-duration videos remains challenging, especially when patch dependencies and generation order matter. NUWA-Infinity combines patch- and token-level autoregression with nearby-context caching and direction-aware ordering. It supports arbitrary-size images and long-duration video generation, with improved quality over NUWA reported by the authors.
Problem
Generating arbitrarily sized high-quality images or videos is difficult because prior patch-based methods may not model patch dependencies, while visual content has multiple possible generation directions.
Source: §1 · p. 5Method
NUWA-Infinity combines global patch-level and local token-level autoregression with Nearby Context Pool caching and an Arbitrary Direction Controller for generation order and positional embeddings.
Source: §1 · p. 5Results
NUWA-Infinity generates arbitrary-size high-resolution images and additionally supports long-duration videos, with significantly improved generation quality over NUWA.
Source: §1 · p. 5Takeaways & Limitations
The authors hope NUWA-Infinity can help visual content creators save time and costs while improving productivity and creativity.
Source: §6 · p. 21Autoregressive generation's training and inference efficiency remains a blocking issue for practical deployment.
Source: §5 · p. 21
Abstract
from arXiv · showhide
In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos. An autoregressive over autoregressive generation mechanism is proposed to deal with this variable-size generation task, where a global patch-level autoregressive model considers the dependencies between patches, and a local token-level autoregressive model considers dependencies between visual tokens within each patch. A Nearby Context Pool (NCP) is introduced to cache-related patches already generated as the context for the current patch being generated, which can significantly save computation costs without sacrificing patch-level dependency modeling. An Arbitrary Direction Controller (ADC) is used to decide suitable generation orders for different visual synthesis tasks and learn order-aware positional embeddings. Compared to DALL-E, Imagen and Parti, NUWA-Infinity can generate high-resolution images with arbitrary sizes and support long-duration video generation additionally. Compared to NUWA, which also covers images and videos, NUWA-Infinity has superior visual synthesis capabilities in terms of resolution and variable-size generation. The GitHub link is https://github.com/microsoft/NUWA. The homepage link is https://nuwa-infinity.microsoft.com.
1 Introduction
NUWA-Infinity targets arbitrarily sized high-quality images and videos, a task complicated by weak patch-dependency modeling in prior approaches and the need for task-specific generation orders. It combines patch- and token-level autoregression with context caching and direction-aware ordering, and supports arbitrary-size images and long-duration videos.
1 Introduction
- Infinite Visual Synthesis means generating arbitrarily sized high-quality images or videos, which can improve visual effects in practical applications such as design, advertising, and entertainment.The paper notes growing interest in higher-resolution images and longer-duration videos.Source: §1 · p. 5
- Prior patch-based approaches often generate patches independently, leaving their dependencies unmodeled and making consistency difficult for high-resolution images and long-duration videos.Visual synthesis also needs to account for different generation orders across image and video dimensions.Source: §1 · p. 5
- NUWA-Infinity uses global patch-level and local token-level autoregression, caches nearby generated patches to save computation, and uses ADC to select generation orders and learn order-aware positional embeddings.The paper identifies ADC as especially useful for image outpainting.Source: §1 · p. 5
- Compared with fixed-resolution DALL·E, Imagen and Parti, NUWA-Infinity generates arbitrary-size high-resolution images and additionally supports long-duration videos.The paper also reports significantly improved generation quality over NUWA, which supports both image and video synthesis.Source: §1 · p. 5
2 Related Work
Related work includes autoregressive and diffusion-based synthesis, as well as methods for generating large images or videos from patches. NUWA-Infinity distinguishes itself by modeling dependencies across patches while supporting variable-size generation.
Autoregressive Methods
- DALL·E [19] and Parti [30] use autoregressive visual-token generation, while NUWA [28] supports both image and video synthesis; NUWA-Infinity adds variable-size generation through autoregression over autoregression.Parti follows DALL·E’s architecture but uses ViT-VQGAN [29] for image discretization and reconstruction.Source: §2 · p. 5
Diffusion Methods
Infinite Visual Synthesis
- Independent patch generation can yield inconsistent results; sliding-window autoregressive and Mask-Predict approaches model patch dependencies at inference, despite training patches independently.NUWA-Infinity combines patch-level autoregression with Nearby Context Pool and Arbitrary Direction Controller for variable-size image and video generation and lower computation cost without losing dependency and consistency modeling.Source: §2 · p. 6
3 Model
NUWA-Infinity generates user-sized images and videos through global patch-level and local token-level autoregression. ADC controls patch order and positional embeddings, while NCP manages context from previously generated patches.
3 Model
- NUWA-Infinity generates user-sized images or videos through global patch autoregression and local token autoregression, then composes patches at the requested dimensions and duration.Source: §3 · pp. 6–7
3.1 Arbitrary Direction Controller (ADC)
- ADC selects patch orders, including four basic traversals and an outpainting order, and dynamically assigns relative positional embeddings to the current patch and its context.Source: §3.1 · pp. 7–8
3.2 Nearby Context Pool (NCP)
- NCP selects nearby cached patches as context, stores generated patches’ multilayer hidden states, and removes caches only after they can no longer support future patches.Source: §3.2 · pp. 8–9
3.3.1 Training Strategy
- Training samples a patch order, predicts each patch from selected context and positional embeddings, updates caches, and optimizes cross-entropy against ground-truth patches.Source: §3.3.1 · pp. 9–10
3.3.2 Inference Strategy
- For image outpainting and animation, input-image patches initialize NCP as a prefix for generating spatially or temporally extended content.Source: §3.3.2 · p. 11
- PG-VQGAN fuses prior-frame encoder features into decoder layers; image animation starts from a ground-truth prior frame, while text-to-video uses standard VQGAN for frame one.Source: §3.3.2 · p. 11
4.1 Datasets
The evaluation uses newly built high-resolution datasets because most existing datasets do not fit the paper’s focus on high resolutions and long durations. Four datasets support the five visual synthesis tasks.
4.1 Datasets
- Four high-resolution datasets support evaluation across the paper’s five visual synthesis tasks.The section introduces RQF, LHQC, LHQ-V, and PeppaPig.Source: §4.1 · p. 12
- RQF provides 128 images from overlapping 2048 × 2048 patches and supports qualitative evaluation of 2048 × 38912 unconditional generation.Training uses the 2048 × 2048 images; evaluation uses the much wider resolution.Source: §4.1 · p. 12
- LHQC contains 90K captioned landscape images, split into 85K training and 5K test pairs for text-to-image and outpainting evaluation.Captions are generated with an image-captioning model and manually corrected.Source: §4.1 · p. 12
- LHQ-V contains 40K landscape videos, split into 38K training and 2K test videos for image-animation evaluation.The videos are filtered using detected objects to remove clips containing objects considered unrelated to landscape.Source: §4.1 · p. 12
- PeppaPig contains 10K annotated text-video pairs, split into 9K training and 1K test pairs for text-to-video evaluation.Annotators captioned clips, and clips labeled “N/A” were removed.Source: §4.1 · p. 12
4.2 Implementation Details
Training has two stages: encode cropped image and video samples into discrete visual tokens, then train the model with Adam.
4.2 Implementation Details
- Images are cropped to 1024 × 1024 and videos to 1024 × 1024 × 5 at 5 fps before VQGAN encoding at compression ratio 16.This is the first of the two training stages.Source: §4.2 · p. 13
- The second stage uses Adam with learning rate 1 × 10^-4, batch size 256, and warm-up over 5% of 50 total epochs.Four models are trained on four datasets and used for inference on five tasks.Source: §4.2 · p. 13
- Task-specific implementation settings are provided in Table 1.The passage does not enumerate those settings.Source: §4.2 · p. 13
4.3 Metrics
The evaluation metrics cover image quality, diversity, video quality, and text-image semantic consistency; Block-FID adapts FID to large images.
4.3 Metrics
- Block-FID splits a large image into blocks and averages their Fréchet Inception Distance scores to assess generated-image quality.FID is also used to calculate generated-image quality.Source: §4.3 · p. 13
- IS measures the diversity of generated results, with a higher score indicating more diversity.Source: §4.3 · p. 13
- FVD measures distance between ground-truth and generated videos, with a lower score denoting higher similarity.Source: §4.3 · p. 13
- CLIP-SIM uses CLIP to measure semantic consistency between a generated image and its input text.Source: §4.3 · p. 13
4.4 Main Results
Across the reported tasks, NUWA-Infinity generates very large images and produces strong results in text-to-image, image-animation, and outpainting evaluations. The experiments also show long-range visual continuity and comparisons across generation strategies and extension directions.
4.4.1 Unconditional Image GenerationHD
- NUWA-Infinity generates a 38912 × 2048 unconditional image, 19 times longer than each 2048 × 2048 training instance, with NCP computation growing linearly with output size.The complete image is split into six display sections; generated content also includes scenes absent from the source painting.Source: §4.4.1 · p. 14, Figure 11 · p. 14, Figure 12 · p. 15
- Local autoregression produces detailed visual tokens, while patch-level autoregression yields smooth global transitions in the extremely long image.Reported details include faces, gestures, leaves, and roof tiles; figures gradually transition from people to mountains.Source: §4.4.1 · p. 16
4.4.2 Text-to-ImageHD
- At 4096 × 1024, NUWA-Infinity maintains Block-FID 15.65 while MaskGIT performance decreases rapidly; at 1024 × 1024, NUWA-Infinity outperforms Taming Transformer and MaskGIT on Block-FID, CLIP-SIM, and IS.The models are trained from scratch on LHQC; the reported comparison with these baselines is supported by Table 2.Source: §4.4.2 · p. 16, Table 2 · p. 16, Figure 13 · p. 16
4.4.3 Image AnimationHD
- NUWA-Infinity (AR) achieves FVD 62.57 and more realistic generated images than NAR and P-NAR, with slower generation than those alternatives.The compared models all generate patches globally autoregressively but differ in their local token-generation methods.Source: §4.4.3 · p. 17, Table 3 · p. 17, Figure 14 · p. 17
4.4.4 Image OutpaintingHD
- NUWA-Infinity outperforms Taming and MaskGIT in all four outpainting directions; adding text improves results for vertical extensions and gives similar results for horizontal extensions.The reported comparison measures Block-FID on the extended area; Taming and MaskGIT use text prompts by default for the fair comparison.Source: §4.4.4 · pp. 17–18, Table 4 · p. 18, Figure 15 · p. 18
4.5 Ablation Studies
The ablations examine ADC, NCP, and vision-decoder choices, including how they affect generation quality, computation, and convergence.
4.5.1 Ablation Study on ADC
- For Text-to-ImageHD, smaller ADC patch sizes harm FID, whereas larger sizes require more GPU memory; M = 256 balances performance and memory.Figure 16 plots FID at 1024 × 1024 and Block-FID at 1024 × 4096 against patch size.Source: §4.5.1 · p. 19, Figure 16 · p. 19
- ADC pre-feeding outperforms post-feeding by using positional information to control which contexts self-attention attends to.Pre-feeding adds relative positional embeddings to the attention key before attention computation; post-feeding adjusts attention distributions afterward.Source: §4.5.1 · p. 19, Table 5 · p. 19
4.5.2 Ablation Study on NCP
- NCP caches transmit information from previously generated patches, and the cache-enabled design significantly outperforms the version without that transmission.Hidden states of nearby patches can retain information from distant patches removed from NCP.Source: §4.5.2 · p. 19, Table 6 · p. 19
- For NCP, larger spatial extents improve Block-FID with diminishing returns beyond eh, ew > 2, while temporal FVD improvements become limited at ef = 3.The selected default is (ew, eh, ef) = (2, 2, 3), balancing performance and computation.Source: §4.5.2 · p. 20
4.5.3 Ablation Study on Vision Decoder
- The AR vision decoder achieves the best performance, while the NAR decoder achieves the fastest speed.Global autoregression is maintained in all three decoder variants; local generation is AR, NAR, or P-NAR.Source: §4.5.3 · p. 20, Table 7 · p. 20
- The Base model has acceptable performance relative to Large, and the authors attribute this to limited training samples while prioritizing architecture over large-scale pretraining.They plan to pre-train with more data in the future.Source: §4.5.3 · p. 20, Table 7 · p. 20
- Patch loss converges in 50 epochs versus 70 for accumulated loss and improves Block-FID by sharing optimized parameters across patches.The authors say this sharing helps the model learn large images or videos.Source: §4.5.3 · p. 20, Table 7 · p. 20
5 Discussions
The discussion identifies data availability, evaluation, efficiency, and pretraining scope as constraints and future-work areas.
Training Data
- High-resolution images and videos are required for training, but quality and licensing issues make such data harder to collect.The authors plan to collect larger datasets that meet those criteria.Source: §5 · p. 20
Evaluation Metric
- Visual synthesis is difficult to evaluate because a model's output may have unlimited ground truths; the paper uses FID, FVD, IS, and CLIP-SIM and plans to explore better metrics.Source: §5 · p. 20
Inference Speed
- Autoregressive models handle dependencies well, but their training and inference efficiency remains a barrier to practical deployment.The authors propose exploring combinations of autoregressive and non-autoregressive methods to pursue quality and efficiency.Source: §5 · p. 21
Pre-trained Version
- Because large-scale high-quality visual data are lacking, this paper trains NUWA-Infinity directly for different downstream tasks rather than pre-training it.The authors plan a future version with more collected visual data and evaluation of open-domain generalization.Source: §5 · p. 21
6 Conclusion
NUWA-Infinity is a visual synthesis framework for generating high-quality images and videos from text or image inputs. Its autoregressive-over-autoregressive mechanism supports variable-size tasks, and the authors hope it will help content creators.
6 Conclusion
- NUWA-Infinity uses autoregressive-over-autoregressive generation for variable-size tasks including image outpainting, image animation, text-to-image, and text-to-video.The authors hope such models help creators save time, cut costs, and increase productivity and creativity.Source: §6 · p. 21