PaperlayerView exact revision on arXiv

Source-linked AI summary

X-VILA: Cross-Modality Alignment for Large Language Model

Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, Hongxu Yin

arXiv:2405.19335v1cs.CVcs.CLcs.LG

TL;DR

Existing multimodal LLMs largely link one input modality to text output, while cross-modality instruction data is limited. X-VILA combines modality-specific encoders and diffusion decoders with textual and visual alignment, and its reported results show stronger cross-modality abilities and visual correspondence than prior methods.

  • Problem

    Existing multimodal LLMs largely center on one input modality with text-only output, while available instruction data is mostly X-to-text or text-to-X.

    Source: §1 · p. 2

  • Method

    X-VILA aligns modality-specific encoders to LLM inputs and diffusion decoders to outputs, using textual alignment and a Visual Embedding Highway for visual features.

    Source: §1 · p. 2

  • Results

    X-VILA shows stronger cross-modality understanding, reasoning, and generation across conversation data, with improved visual correspondence over previous methods.

    Source: §3.2 · p. 7, §3.3 · p. 8

  • Takeaways & Limitations

    The authors present the curated X-to-X instruction dataset as a resource for future multimodality foundation-model research.

    Source: §1 · p. 3

Abstract

from arXiv · show

We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities. By aligning modality-specific encoders with LLM inputs and diffusion decoders with LLM outputs, X-VILA achieves cross-modality understanding, reasoning, and generation. To facilitate this cross-modality alignment, we curate an effective interleaved any-to-any modality instruction-following dataset. Furthermore, we identify a significant problem with the current cross-modality alignment method, which results in visual information loss. To address the issue, we propose a visual alignment mechanism with a visual embedding highway module. We then introduce a resource-efficient recipe for training X-VILA, that exhibits proficiency in any-to-any modality conversation, surpassing previous approaches by large margins. X-VILA also showcases emergent properties across modalities even in the absence of similar training data. The project will be made open-source.

1 Introduction

X-VILA integrates video, image, language, and audio inputs and outputs in an any-to-any LLM, addressing the limited scope of systems centered on a single input modality and text output. It combines modality alignment, a cross-modality instruction dataset, and staged training to support understanding, reasoning, and aligned multimodal generation.

1 Introduction

  • Existing multimodal LLMs largely focus on a single input modality with text-only output, leaving broader modality integration relatively unexplored.Examples include image-input systems such as Flamingo, LLaVA, and VILA, and video-input systems such as Video-LLaMA and LITA.Source: §1 · p. 2
  • X-VILA aligns modality encoders with LLM inputs and diffusion decoders with LLM outputs, then adds visual alignment to preserve details lost in text-space projection.The Visual Embedding Highway lets visual features guide visual decoders while bypassing the LLM.Source: §1 · p. 2, Figure 2 · p. 3
  • The curated X-to-X dataset contains more than 1.5M multimodal conversations, each with at least one cross-modality question-and-answer pair.It draws on multimodality data from WebVid and ActivityNet Captions and spans text, audio, image, and video interactions.Source: §1 · p. 2
  • X-VILA is trained through data-effective alignment, interleaved multimodal pre-training, and two-step textual and visual instruction tuning.The resulting model is intended to comprehend and generate multimodal content.Source: §1 · p. 2
  • X-VILA recognizes image subjects, reasons about their content, and generates multimodal outputs aligned with the given context.The model is presented as supporting cross-modality understanding, reasoning, and generation across video, image, language, and audio.Source: §1 · p. 3

2 Methodology

X-VILA connects modality-specific encoders and decoders to an LLM for multimodal input and output. Its visual embedding highway addresses visual information loss in text-space alignment, and training proceeds in three phases.

2.1 X-VILA Architecture

  • Modality-specific encoders project text, image, video, and audio inputs into the LLM's textual embedding space, while the LLM predicts output embeddings for modality-specific decoders.The system uses a text tokenizer and ImageBind transformers for non-text modalities; generated modality tokens are mapped to image, video, or audio synthesis.Source: §2.1 · pp. 3–4
  • The visual embedding highway uses layer-wise ImageBind feature maps and a visual controller to send control signals into visual-decoder U-Net layers.Its four stages each contain two residual convolutional blocks, and zero-initialized blocks generate the stage control signals.Source: §2.1 · pp. 4–5, Figure 3 · p. 4
  • A conditioning rate α regulates the proportion of reverse diffusion steps conditioned on visual features, and both the U-Net and controller are fine-tuned during instruction tuning.The control signals are initially zero; “Null” denotes a timestep when no visual highway feature is passed to the U-Net.Source: §2.1 · p. 5
  • The visual embedding highway is reported to significantly increase consistency between generated content and visual context.It bridges visual encoders and decoders to alleviate information loss when projecting high-dimensional visual content into the textual embedding space.Source: §2.1 · pp. 4–5

2.2 X-VILA Training

  • X-VILA training has three phases: encoder–LLM–decoder alignment, interleaved-data pre-training, and X-to-X cross-modality instruction fine-tuning.Training details are provided in Appendix A.Source: §2.2 · p. 5

3 Experiments

The experiments evaluate X-VILA on cross-modality conversations and benchmarks, finding stronger alignment with visual embedding highway (VEH) and emergent abilities beyond the trained task types.

3.1 Datasets and Evaluation

  • The evaluation uses X-to-X conversations from WebVid and ActivityNet Captions, with X2A scores measuring cosine similarity between ImageBind embeddings of predictions and ground truths.The training dataset includes six input-output modality combinations and multi-pair conversations.Source: §3.1 · pp. 5–6

3.2 Quantitative Analysis and Ablation Study

  • X-VILA significantly outperforms Next-GPT across conversation types, with consistent gains as VEH is added to image and video decoders.Both models use ImageBind scores, which the authors describe as a fair comparison.Source: §3.2 · pp. 6–7, Table 2 · p. 7
  • Increasing the VEH conditioning rate, which exposes more reverse sampling steps to VEH features, improves multi-modality alignment.The paper also reports that text-aligned decoding struggles with visual style, object appearance, and precise actions, while VEH improves visual consistency.Source: §3.2 · p. 7, §3.3 · pp. 8–9, Figure 7 · p. 8
  • On zero-shot VQAv2, VisWiz, and MMMU-val, X-VILA is competitive with leading domain-expert VLMs while retaining X-to-X capability.The cited table caption describes these results as comparable performance to domain experts.Source: §3.2 · p. 7, Table 3 · p. 7

3.3 Qualitative Analysis and Ablation Study

  • In image-to-video comparisons, X-VILA shows better visual correspondence than Next-GPT and CoDi; GPT-4o generates images but not videos.The comparison prompts models to generate content similar to an input image.Source: Figure 6 · p. 7, §3.3 · p. 8
  • Without explicit training on similar tasks, X-VILA handles long-context cross-modality generation and image-to-audio and audio-to-image generation.The paper attributes these emergent abilities to exposure to its X-to-X dataset.Source: Figure 5 · p. 6, §3.3 · p. 8

4 Related Work

Related work traces LLM and vision-language model development, including multimodal input-output systems. It positions Next-GPT as a textual-alignment approach that struggles with visual details.

LLM development

  • LLMs developed from transformer and scaling work into chat systems, with cited advances in bidirectional processing, text-to-text tasks, and longer context.The passage highlights GPT, BERT, T5, Transformer-XL, and ChatGPT.Source: §4 · p. 9

Vision-language models

  • Vision-language research includes textually prompted, visually prompted, and multimodal input-output models, with alignment commonly learned between pretrained encoders and LLMs.The passage cites linear projection, cross-attention, and instruction tuning among approaches.Source: §4 · p. 9

Related multimodal LLMs

  • Next-GPT can understand multimodal inputs and generate multimodal outputs through textual alignment, but it does not effectively handle visual details in the input.Source: §4 · pp. 9–10

5 Conclusion

X-VILA combines any-to-any modality alignment and an instruction-tuning dataset with a visual embedding highway to preserve input details; experiments indicate the data and methods are effective.

5 Conclusion

  • X-VILA is an any-to-any modality LLM for understanding, inference, and generation of multimodal content.Source: §5 · p. 10
  • The paper curates an any-to-any modality instruction-tuning dataset and identifies visual-detail loss in previous textual alignment methods.Source: §5 · p. 10
  • A visual alignment mechanism with a visual feature highway helps preserve essential visual details from inputs.Source: §5 · p. 10
  • Quantitative and qualitative experiments indicate the effectiveness of the data and methods, with further gains across VLM benchmarks.Source: §5 · p. 10

A X-VILA Training

X-VILA training has three phases: encoder-LLM-decoder alignment, interleaved data pre-training, and X-to-X cross-modality instruction fine-tuning.

A X-VILA Training

A.1 Encoder-LLM-decoder alignment training phase.

The first alignment phase connects modality-specific encoders and decoders to the LLM’s textual embedding space. It trains projection layers on modality-to-text and text-to-modality tasks while keeping most LLM parameters frozen.

A.1 Encoder-LLM-decoder alignment training phase.

  • The phase trains input and output projection layers, plus the LLM vocabulary embedding layer, while keeping its other parameters frozen.Training uses video-text, image-text, or audio-text pairs.Source: §A.1 · p. 15
  • X-to-text captioning aligns modality-encoder outputs with the LLM’s textual embedding space.The inputs can be video, image, or audio.Source: §A.1 · p. 15
  • Text-to-X generation aligns LLM text embeddings with the input of modality-specific video, image, and audio decoders.Only the output projection layers are optimized for this task.Source: §A.1 · p. 15
  • The projected textual controller embedding is trained to match the pretrained diffusion text encoder’s feature distribution, then replaces that feature to control decoder U-Nets via cross-attention.The feature-distance objective is described in the continuation passages.Source: §A.1 · p. 15

A.2 Interleaved data pre-training phase.

Interleaved pretraining uses sequential multimodal content from video clips to train across modalities. The subsequent X-to-X instruction-tuning phase uses six conversation types and separates textual alignment from visual alignment.

A.2 Interleaved data pre-training phase.

  • The interleaved corpus combines image-text pairs with ActivityNet-derived video, audio, image, and text sequences to support multimodal pretraining.The figure caption identifies six types of cross-modality understanding and generation tasks in the dataset examples.Source: §A.2 · p. 15, Figure 9 · p. 16
  • Interleaved pretraining constructs sequences of image, audio, video, and text samples from video chunks, with evenly sampled beginning, middle, and ending chunks and n = 3.Input and output projection layers are jointly trained, with LoRA fine-tuning on the LLM.Source: §A.2 · p. 15

A.3 X-to-X cross-modality instruction tuning phase.

  • X-to-X instruction tuning uses six conversation types: video-to-image, video-to-video, image-to-video, video-to-audio, audio-to-video, and image+audio-to-video.The dataset is based on WebVid and ActivityNet Captions, and conversations may contain multiple rounds.Source: §A.3 · pp. 15–16
  • Textual alignment projects multimodal inputs into the LLM’s text space and trains it to generate embeddings for corresponding modalities, blending X-to-X data with common VLM SFT datasets.Decoder weights remain frozen while projection layers, the vocabulary embedding layer, and LLM LoRA parameters are trained.Source: §A.3 · p. 16
  • Visual alignment updates the visual decoders and visual controller while freezing the other network parameters, including the LLM and projection layers.The visual embedding highway is used for image and video decoders when the input includes a visual modality; the authors state this preserves performance on other modalities.Source: §A.3 · p. 16

B.1 Examples of our X-to-X dataset.

The examples illustrate X-to-X conversations across modalities, while a visual comparison reports stronger visually aligned generation from X-VILA than CoDi. Additional demonstrations cover varied human-model interaction tasks.

B.1 Examples of our X-to-X dataset.

  • The X-to-X dataset examples cover varied conversation templates designed to connect modalities in any-to-any interactions.Figure 9 visualizes the six conversation types.Source: §B.1 · p. 17

B.2 Visual comparison with CoDi on cross-modality alignment.

  • X-VILA produces superior visually aligned generation results to CoDi, which the authors report fails to capture input semantics and details and cannot perform X-to-X chatting.The comparison is shown in Figure 10.Source: §B.2 · p. 17, Figure 10 · p. 17

B.3 Human-model interaction demonstration.

  • Human-model examples across diverse use cases demonstrate X-VILA’s comprehensive multimodal understanding and generation capabilities.The examples appear in Figures 11–14.Source: §B.3 · p. 17

C More Implementation Details

The implementation uses three training phases, with separate textual and visual alignment stages in the final phase. Qualitative examples show cross-modality conversation and visual reasoning.

C More Implementation Details

  • The three phases use 20,000, 10,000, and 30,000 iterations for initial alignment, interleaved pretraining, and textual alignment, respectively.Visual alignment trains the Visual Embedding Highway and modality-specific decoders for 20,000 iterations.Source: C More Implementation Details · p. 17
  • The initial alignment, interleaved pretraining, textual alignment, and visual alignment phases use batch sizes of 200, 192, 192, and 64, respectively.The reported learning rates are 4 × 10−4 for initial alignment, 1 × 10−4 for pretraining and textual alignment, and 1 × 10−6 for visual alignment.Source: C More Implementation Details · p. 17
  • The training pipeline uses four NVIDIA A100 80GB server nodes, and the authors describe it as highly efficient compared with several prior vision-language models.The cited prior work markers are [15] [20] [17].Source: C More Implementation Details · p. 17
  • Figures 11–14 provide cross-modality conversation examples, including visual-input comprehension and reasoning.Figure 14’s caption says the visual alignment technique plays a crucial role in visual consistency between input and output.Source: Figure 11 · p. 18, Figure 12 · p. 19, Figure 13 · p. 20
Loading 2405.19335v1…