PaperlayerView exact revision on arXiv

Source-linked AI summary

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, Yao Lu

arXiv:2409.04429v3cs.CVcs.LG

TL;DR

VILA-U addresses the complexity of combining visual understanding and generation with separate model components. It uses one autoregressive next-token framework and achieves competitive understanding and generation performance across image and video tasks.

  • Problem

    Existing visual tokenizers often require external generation models, adding infrastructure complexity to unified visual-language systems.

  • Method

    VILA-U unifies visual and textual inputs under autoregressive next-token prediction, using text-aligned vision-tower pretraining.

  • Results

    VILA-U narrows the visual-understanding gap with leading VLMs and provides competitive image and video generation without external diffusion components.

  • Takeaways & Limitations

    A single token-based autoregressive framework can support diverse visual-language understanding and generation tasks with performance comparable to more complex systems.

  • Takeaways & Limitations

    Balancing vision-tower text alignment and reconstruction remains challenging, and the paper observes no significant synergy between understanding and generation.

Abstract

from arXiv · show

VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

1 INTRODUCTION

VILA-U unifies visual understanding and generation in an end-to-end autoregressive framework using one next-token prediction objective. It targets competitive performance without external components such as diffusion models while narrowing understanding gaps and providing native visual generation.

  • 1 INTRODUCTION: The work is motivated by the complementary progress of visual understanding and generation and the emerging effort to unify them in a single multimodal framework.Earlier visual-language efforts achieved strong visual language understanding, while generation research pursued diffusion-based and discrete-token autoregressive approaches.
  • 1 INTRODUCTION: Prior unified approaches often use VQGAN-based tokenizers and autoregressive models, but their tokenizers lack decoding capability and require an external visual generation model.This dependency increases infrastructure complexity despite supporting unified modeling of understanding and generation.
  • 1 INTRODUCTION: VILA-U uses an end-to-end autoregressive framework with a unified next-token objective for visual and text inputs, eliminating external components such as diffusion models.The framework is designed to support both visual language understanding and generation within one system.
  • 1 INTRODUCTION: VILA-U evaluates image-language understanding, video-language understanding, image generation, and video generation, narrowing the understanding gap while providing competitive native visual generation.The evaluation spans both understanding and generation across image and video modalities.

2 RELATED WORK

The related work progresses from large language models and vision-language models toward unified systems that generate both text and visual content. Existing unified approaches commonly augment VLMs with diffusion models for visual generation.

  • Large Language Models and Visual Language Models: LLMs provide the transformer-based foundation for modern language tasks, while VLMs combine vision foundation models with LLMs for cross-modal understanding.The passage highlights CLIP, BLIP, and CoCa as visual feature extractors aligned with text before being fed into an LLM.
  • Unified Visual Language Models: Unified visual language models aim to generate both textual and visual content, including images and videos.The related work identifies unified generation of multiple modalities as a distinct research direction.
  • Unified Visual Language Models: A mainstream unified-model strategy combines VLMs with diffusion models such as Stable Diffusion for visual generation.The passage lists multiple works following this VLM-plus-diffusion design.

3 METHODS

VILA-U unifies visual and language processing through a discrete-token vision tower and a single multimodal generative training procedure. Residual quantization, staged tower training, and next-token prediction support both visual understanding and generation.

  • Framework Overview: The framework combines a unified foundation vision tower that converts visual inputs into text-aligned discrete tokens with unified multimodal generative training.The overall multimodal training and inference process is illustrated in Figure 1.
  • Unified Foundation Vision Tower: The vision tower jointly develops text alignment and discrete image tokenization using text-image contrastive loss and VQ-based image reconstruction loss.Image features are discretized through residual quantization, then used for reconstruction and text-image alignment.
  • Unified Training Recipe: Because simultaneous alignment and reconstruction training from scratch causes conflicting objectives and yields only 5% Top-1 accuracy, VILA-U first learns alignment and then reconstruction while preserving alignment.The staged recipe initializes the vision and text encoders with pretrained CLIP weights.
  • Residual Vector Quantization: Residual vector quantization represents each visual vector with D sequential codes, increasing approximation capacity while keeping visual generation tokenized.A depth transformer sequentially produces codes at different depths during multimodal training and inference.
  • Multimodal Pretraining: Multimodal pretraining concatenates visual and text tokens using image-text, text-image, text-video, and interleaved text-image forms, supervising only the latter modality in paired sequences.Special start and end tokens distinguish image and video modalities and enable visual content generation.
  • Training Objective: Since text and visual tokens are discrete, the LLM uses next-token prediction, while a depth transformer autoregressively predicts each visual token’s D residual codes.The depth transformer is trained jointly with the LLM from random initialization.

4 EXPERIMENTS

Experiments evaluate VILA-U across visual understanding and generation tasks, including image and video benchmarks. The unified vision tower and autoregressive framework achieve strong understanding performance and competitive image and video generation results.

  • Experimental Setup: The evaluation covers zero-shot image-language benchmarks, video-language benchmarks, image-generation benchmarks, and VBench for video generation.The listed benchmarks include VQAv2, GQA, TextVQA, POPE, MME, SEED, MM-Vet, ActivityNet, MSVD, MSRVTT, TGIF, MJHQ-30K, GenAI-Bench, and VBench.
  • Unified Foundation Vision Tower: The unified foundation vision tower achieves significantly better reconstruction than VQ-GAN, while its reconstruction FID is slightly worse than RQ-VAE with the same code shape.Reconstruction FID and ImageNet zero-shot Top-1 accuracy are used to measure reconstruction and text-alignment capabilities.
  • Visual Understanding Tasks: VILA-U performs close to leading VLMs on image- and video-based visual-language benchmarks, surpassing many methods with the same language-model size despite using discrete visual tokens.Tables 2 and 3 summarize these comparisons and attribute the results to the unified foundation vision tower.
  • Visual Generation Tasks: VILA-U achieves better FID than other autoregressive methods and comparable performance to some diffusion-based methods on image generation, while remaining competitive on advanced prompts with much less training data.The GenAI-Bench results show a relatively small gap to diffusion-based methods for advanced prompts, and the MJHQ-30K comparison is reported in Table 4.
  • Qualitative Results: Qualitative examples show that VILA-U generates good-looking images and continuous videos that adhere to user inputs, even with a relatively small data corpus.Additional visualizations cover multi-image understanding, in-context learning, image generation, and video generation.

5 ABLATION STUDY

The ablations show that contrastive text alignment in the vision tower is crucial for visual-language understanding, especially with larger-scale training. Generation remains competitive with the unified tower, while classifier-free guidance performs best at CFG 3.0.

  • Qualitative capability demonstrations: Qualitative examples show that VILA-U captions videos in detail, answers visual questions, performs in-context learning, and reasons over multiple images.The figures illustrate these capabilities across video captioning, VQAv2 question answering, image-text prompting, and multi-image reasoning.
  • Vision-tower text alignment: Contrastive text alignment is crucial for strong visual-language understanding, and scaling vision-tower training from 25M to 700M data further improves performance.The study trains with and without contrastive loss and reports that larger-scale text-alignment learning enhances downstream visual-language results.
  • Vision-tower generation ablation: The unified vision tower yields slightly worse FID results than the RQ-VAE-based tower in the text-to-image generation ablation.The comparison uses RQ-VAE with rFID 1.30 and a reduced Sheared-LLaMA-1.3B setup.
  • Classifier-free guidance: Classifier-free guidance achieves the best FID score at a CFG value of 3.0 for the 256-resolution model.The ablation varies CFG during visual-content generation.

6 CONCLUSION AND LIMITATION · APPENDIX · A DIFFERENCE WITH RELATED WORKS

VILA-U unifies visual understanding and generation in a single autoregressive framework, while its unified vision tower addresses key limitations of prior tokenizer-based approaches. The model remains limited by a reconstruction–contrastive-loss trade-off and the lack of clear synergy between understanding and generation.

  • 6 CONCLUSION AND LIMITATION: VILA-U integrates video, image, and language understanding and generation through one autoregressive next-token prediction framework.This removes the need for additional components such as diffusion models while achieving performance comparable to state-of-the-art VLMs.
  • 6 CONCLUSION AND LIMITATION: The unified vision tower’s reconstruction–contrastive-loss trade-off remains a central limitation requiring further exploration.Contrastive loss improves one capability while affecting reconstruction ability, making the balance between the two challenging.
  • 6 CONCLUSION AND LIMITATION: VILA-U currently shows no significant synergy or mutual enhancement between visual understanding and generation tasks.The authors identify enabling stronger interaction between these tasks as a direction for future work.
  • A DIFFERENCE WITH RELATED WORKS: VQGAN-based tokenizers support autoregressive visual generation but produce tokens with insufficient semantic information for strong multimodal understanding.Their reconstruction-only training makes visual generation straightforward because tokens can be decoded into outputs using lightweight decoders.
  • A DIFFERENCE WITH RELATED WORKS: CLIP-feature tokenizers improve understanding performance through richer semantics but lack decoding capability and therefore require external visual generation models.Diffusion models can be used to process the generated visual-token conditions.
  • A DIFFERENCE WITH RELATED WORKS: VILA-U addresses both prior limitations with a vision tower that combines CLIP-like semantic features with VQGAN-like image reconstruction.The design uses reconstruction and contrastive losses during autoencoder training together with residual quantization.

B QUALITATIVE RESULTS · B.1 RECONSTRUCTION · B.2 VISUAL UNDERSTANDING

The qualitative results show that VILA-U reconstructs images in detail from text-aligned discrete visual tokens and answers image- and video-understanding questions accurately. These demonstrations cover 256/384-resolution vision towers and examples from VQAv2 and TGIF test splits.

  • B.1 RECONSTRUCTION: VILA-U effectively reconstructs images in detail using text-aligned discrete visual tokens at 256/384 resolution.Figure 8 visualizes these reconstruction results.
  • B.1 RECONSTRUCTION: The reconstruction demonstrations are presented as qualitative results for the 256/384-resolution vision tower.The results are shown in Figure 8.
  • B.2 VISUAL UNDERSTANDING: Figure 9 presents additional image-understanding examples from the VQAv2 test split.These examples accompany the reported accurate answers.
  • B.2 VISUAL UNDERSTANDING: Figure 10 presents additional video-understanding examples from the TGIF test split.The image- and video-understanding examples are provided together as further qualitative demonstrations.
  • B.2 VISUAL UNDERSTANDING: VILA-U successfully answers image-understanding questions accurately on examples from the VQAv2 test split.Figure 9 provides the image-understanding examples.
  • B.2 VISUAL UNDERSTANDING: VILA-U successfully answers video-understanding questions accurately on examples from the TGIF test split.Figure 10 provides the video-understanding examples.

B.3 IN-CONTEXT LEARNING EXAMPLES

VILA-U demonstrates good in-context learning capabilities across the examples evaluated from Lin et al. (2023).

  • B.3 IN-CONTEXT LEARNING EXAMPLES: VILA-U exhibits good in-context learning capabilities across all tested examples from Lin et al. (2023).The qualitative examples are presented in Figure 11.

B.4 VISUAL GENERATION

VILA-U generates high-quality images and videos from text input, as illustrated by the provided image and video generation examples.

  • B.4 VISUAL GENERATION: VILA-U can generate high-quality images and videos given text input.Examples are provided in Figures 12 and 13.

C FAILED TRAINING RECIPES.

Alternative training recipes underperform because they either lack feasible text-aligned vision initialization or prevent the vision tower from learning reconstruction-relevant features. The final recipe uses pretrained CLIP vision weights while keeping the vision encoder trainable, enabling small-batch training and low-level feature learning.

  • C FAILED TRAINING RECIPES.: None of four alternative training recipes matches the effectiveness of the final approach.The alternatives include text-only CLIP initialization, RQ-VAE initialization, a frozen vision encoder, and a trainable text encoder.
  • C FAILED TRAINING RECIPES.: Recipes 1 and 2 are impractical because CLIP training from scratch needs large batches and many GPU days, whereas VQ reconstruction requires small batches.The passage contrasts global batch sizes of 32k for CLIP training and 512 for stable VQ-based reconstruction training.
  • C FAILED TRAINING RECIPES.: Making the text encoder trainable fails because chaotic initial quantized features and contrastive loss disrupt its weights, slowing training.This is the stated failure mode of recipe 4.
  • C FAILED TRAINING RECIPES.: The final recipe initializes the vision encoder with pretrained CLIP weights but keeps it trainable, supporting small-batch training and low-level reconstruction learning.Pretrained initialization preserves semantic features instead of relearning them from scratch.
Loading 2409.04429v3…