PaperlayerView exact revision on arXiv

Source-linked AI summary

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan

arXiv:2111.12417v1cs.CVcs.AI

TL;DR

Visual synthesis models have typically handled images and videos separately, limiting their ability to use both modalities together. NÜWA introduces a unified 3D transformer encoder-decoder with 3D Nearby Attention for language, images, and videos, achieving state-of-the-art results across several synthesis tasks and strong zero-shot manipulation capabilities.

  • Problem

    Existing visual synthesis solutions treat images and videos separately, limiting their ability to benefit from both types of visual data.

  • Method

    NÜWA uses a unified 3D transformer encoder-decoder with an adaptive encoder for text or visual sketches, a shared decoder, and 3D Nearby Attention across spatial and temporal axes.

  • Results

    NÜWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, and video prediction, with strong zero-shot text-guided image and video manipulation capabilities.

  • Takeaways & Limitations

    NÜWA supports generating or manipulating images and videos across 8 visual synthesis tasks within one unified pre-trained model.

Abstract

from arXiv · show

This paper presents a unified multimodal pre-trained model called NÜWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate NÜWA on 8 downstream tasks. Compared to several strong baselines, NÜWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.

1. Introduction

NÜWA addresses the limitations of separate image and video synthesis models with a unified multimodal framework for visual synthesis. Its 3D Nearby Attention reduces computational complexity, while the model achieves strong benchmark and zero-shot manipulation results.

  • Visual synthesis models aim to generate or manipulate images and videos for diverse visual scenarios.
  • Pixel-by-pixel autoregressive methods offer explicit density modeling and stable training but are computationally expensive and limited to low-resolution visual data.
  • VQ-VAE-based approaches enable efficient large-scale visual pre-training but generally treat images and videos separately, preventing shared use of both data types.
  • NÜWA uses a general 3D transformer encoder-decoder that jointly covers language, images, and videos, with an adaptive encoder and a decoder shared across 8 synthesis tasks.
  • 3D Nearby Attention models local spatial and temporal structure, reducing computational complexity and improving the visual quality of generated results.
  • NÜWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, and video prediction, alongside zero-shot text-guided image and video manipulation.

2. Related Works

Prior visual autoregressive models improved scalability through discrete tokens but did not jointly train on image and video data. NÜWA unifies both modalities and introduces nearby attention to address self-attention complexity across visual dimensions.

  • Earlier pixel-by-pixel visual autoregressive models were difficult to scale beyond low-resolution images and videos because of high computational cost.
  • VQ-VAE-based models enabled efficient large-scale pre-training and higher-resolution generation, but existing systems were not trained jointly on images and videos.
  • NÜWA is a unified autoregressive model pre-trained on visual data covering both images and videos for varied downstream tasks.
  • Self-attention introduces quadratic complexity, creating a challenge for high-resolution image synthesis and video synthesis.
  • Block-wise sparse attention separates visual blocks without modeling their relationships, while axial-wise sparse attention operates along representation axes.
  • NÜWA extends local sparse attention into 3D nearby attention for images and videos, and reports local-wise attention as superior to axial-wise attention for visual generation.

3. Method

NÜWA represents text, images, videos, and sketches in a unified 3D token space, then uses 3DNA-based encoder-decoder layers for multimodal visual synthesis. Its training combines text-to-image, video prediction, and text-to-video objectives.

  • 3.1. 3D Data Representation: NÜWA represents texts, images, videos, and their sketches as tokens in a unified 3D tensor X ∈ R^{h×w×s×d}.The spatial dimensions are h and w, the temporal dimension is s, and each token has dimension d.
  • 3.1. 3D Data Representation: Text uses BPE embeddings with representation R^{1×1×s×d}, while images use VQ-based discrete visual tokens with a placeholder temporal dimension.Images are encoded through a learnable codebook and reconstructed by a decoder.
  • 3.1. 3D Data Representation: Videos are encoded by applying a 2D VQ-GAN to each frame, producing R^{h×w×s×d} while sharing the image representation scheme.The paper reports that this approach can generate temporally consistent videos and benefit from both image and video data.
  • 3.2. 3D Nearby Self-Attention: 3DNA defines nearby self- and cross-attention over spatial and temporal neighborhoods rather than attending to all positions.For position (i,j,k), the module gathers a local sub-tensor from condition C; when C = X, it performs nearby self-attention.
  • 3.3. 3D Encoder-Decoder: The 3D encoder-decoder uses stacked 3DNA layers, with the encoder modeling condition interactions and the decoder combining generated-token self-attention with condition cross-attention.The decoder generates target tokens autoregressively from a learned <bos> token.
  • 3.4. Training Objective: NÜWA is trained jointly on text-to-image, video prediction, and text-to-video tasks using three cross-entropy components.Text conditions are used for T2I and T2V, while video prediction uses a constant representation of “None.”

4. Experiments

NÜWA is evaluated across multiple visual synthesis tasks, including generation, prediction, sketch conditioning, completion, and zero-shot manipulation. Ablations show benefits from multi-task pre-training, VQ-GAN tokenization, and 3D nearby attention.

  • State-of-the-art comparisons: NÜWA achieves the best performance on all evaluated Text-to-Video metrics on Kinetics, including visual quality and semantic consistency.Evaluation uses FID-img, FID-vid, and generated-video label accuracy.
  • State-of-the-art comparisons: 86.9 FVD improves on the prior 94±2 state-of-the-art score for video prediction when conditioning on only one frame.The comparison uses 64×64 videos on the BAIR Robot Pushing dataset.
  • State-of-the-art comparisons: NÜWA generates varied, realistic outputs for sketch-to-image and richer completions for images, including diverse lower halves of towers.The sketch-to-image comparison uses MSCOCO stuff, while image completion is evaluated zero-shot against Taming Transformers.
  • Zero-shot manipulation: Zero-shot text-guided image manipulation preserves unrelated image regions while producing text-consistent edits, and inference is reported as practically 5The example changes a firetruck while leaving the background buildings unchanged.
  • Ablation study: Multi-task pre-training improves Text-to-Video generation by raising CLIPSIM from 0.2314 to 0.2379 with T2I, and lowering FVD from 52.98 to 51.81 with V2V.The paper attributes these gains to text-image semantic connections from T2I and common unconditional video patterns from V2V; all three tasks perform best together.
  • Ablation study: 3D nearby attention outperforms full and axial-sparse alternatives for Sketch-to-Video, while nearby-sparse attention is more efficient for longer videos.Nearby attention models interactions across spatial and temporal axes rather than treating axes separately.

5. Conclusion

NÜWA is presented as a unified pre-trained model for generating and manipulating images and videos across eight visual synthesis tasks, combining a 3D encoder-decoder, nearby-sparse attention, and comprehensive experiments.

  • NÜWA generates new or manipulates existing images and videos for 8 visual synthesis tasks.
  • Its contributions include a general 3D encoder-decoder framework covering text, images, and videos simultaneously.
  • NÜWA introduces nearby-sparse attention that models nearby information across spatial and temporal axes.
  • The paper reports comprehensive experiments on 8 synthesis tasks and frames NÜWA as an initial step toward visual world creation.

3D block-sparse 3D axial-sparse (row) 3D nearby-sparse (ours)

Figure 10 compares different 3D sparse-attention patterns on a 4 × 4 × 2 input, showing which tokens are attended and how masks encode sparsity.

  • All attention comparisons use 3D input containing 4 × 4 × 2 = 32 tokens.
  • The upper illustrations identify attended blue tokens used to generate each target orange token.
  • The lower 32 × 32 matrices visualize sparse-attention masks, with black entries denoting masked tokens.

A. Comparisons between 3D Sparse Attentions

The paper contrasts block- and axis-sparse attention with 3D nearby-sparse attention, which captures fuller local spatial-temporal context and can be more efficient in large settings.

  • 3D block-sparse: Block-sparse attention divides 3D data into fixed blocks, potentially hiding nearby information outside each block.
  • 3D axial-sparse: Axial-sparse attention follows visual-data axes, limiting nearby context to information aligned with those axes.
  • 3D nearby-sparse (ours): 3D nearby-sparse attention is proposed to consider full nearby information across spatial and temporal dimensions.
  • 3D nearby-sparse (ours): For long videos or high-resolution frames, nearby-sparse attention is more efficient than axial-sparse attention when h, w, s < (h + w + s).
  • 3D block-sparse: Block-sparse attention remains suitable when the 3D data can be divided into independent parts without dependencies.

B. Details of Multi-task Pre-training

NÜWA uses two implementation settings that trade image quality against video length, while human evaluations assess its text-to-image and image-completion performance on MSCOCO.

  • Implementation settings: NÜWA-256 uses F8 compression and 32 × 32 discrete tokens, prioritizing high-quality image generation.
  • Implementation settings: NÜWA-336 uses F8 compression and 21 × 21 discrete tokens to support relatively longer videos.
  • Implementation settings: NÜWA-256 can generate only 4-frame videos because of transformer efficiency constraints.
  • Implementation settings: Both settings use the same learning rate of 10^-3 and 50M training steps without over-adjusting parameters.
  • Human evaluation: Human evaluations on MSCOCO cover Text-to-Image and Image Completion tasks.

C. Human Evaluation

Human evaluations favor NÜWA over CogView for text-to-image visual quality and over VQGAN for image completion, supporting its reported zero-shot capability.

  • 62% of visual-quality votes favored NÜWA over CogView, compared with 23% for CogView and 15% undetermined.The comparison used 2,000 MSCOCO texts and asked annotators to judge visual quality and semantic consistency.
  • 89% of image-completion votes favored NÜWA over VQGAN.This evaluation removed semantic consistency because the task has no text input.
  • The paper illustrates NÜWA outputs across text-to-image, sketch-to-image, image completion, and text-guided image manipulation tasks.
  • The paper also presents examples of text-to-video, sketch-to-video, video prediction, and text-guided video manipulation.For video prediction, only one frame is used as the condition.
Loading 2111.12417v1…