PaperlayerView exact revision on arXiv

Source-linked AI summary

NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua

arXiv:2309.05519v3cs.AIcs.CLcs.LG

TL;DR

Existing MM-LLMs largely understand multimodal inputs without generating arbitrary modalities, motivating an end-to-end any-to-any system. NExT-GPT connects an LLM with multimodal adaptors and diffusion decoders, achieving multimodal perception and generation while updating only 1% of parameters. The paper presents this system and MosIT as evidence for the possibility of MM-LLM agents modeling universal modalities.

  • Problem

    Most MM-LLMs focus on input-side multimodal understanding, while external-tool systems lack end-to-end training and can struggle with intricate instructions.

  • Method

    NExT-GPT connects multimodal encoders and diffusion decoders to an LLM, using projection-layer alignment and modality-switching instruction tuning across text, image, video, and audio.

  • Results

    NExT-GPT supports perceiving and generating arbitrary combinations of text, image, video, and audio, while fine-tuning only 1% of parameters.

  • Takeaways & Limitations

    The work showcases the possibility of an any-to-any MM-LLM agent capable of modeling universal modalities.

  • Takeaways & Limitations

    The system is constrained by fine-tuning data quantity and base-model quality and may produce low-quality or hallucinated content.

Abstract

from arXiv · show

While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities. As we humans always perceive the world and communicate with people through various modalities, developing any-to-any MM-LLMs capable of accepting and delivering content in any modality becomes essential to human-level AI. To fill the gap, we present an end-to-end general-purpose any-to-any MM-LLM system, NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion decoders, enabling NExT-GPT to perceive inputs and generate outputs in arbitrary combinations of text, images, videos, and audio. By leveraging the existing well-trained highly-performing encoders and decoders, NExT-GPT is tuned with only a small amount of parameter (1%) of certain projection layers, which not only benefits low-cost training and also facilitates convenient expansion to more potential modalities. Moreover, we introduce a modality-switching instruction tuning (MosIT) and manually curate a high-quality dataset for MosIT, based on which NExT-GPT is empowered with complex cross-modal semantic understanding and content generation. Overall, our research showcases the promising possibility of building an AI agent capable of modeling universal modalities, paving the way for more human-like AI research in the community. Project page: https://next-gpt.github.io/

1. Introduction

NExT-GPT addresses the limitation of MM-LLMs that mainly understand multimodal inputs but cannot generate arbitrary modalities. It connects an LLM with multimodal adaptors and diffusion decoders, using lightweight alignment and modality-switching instruction tuning for any-to-any generation.

  • Most MM-LLMs focus on multimodal input understanding, while existing multimodal generators are largely confined to interleaved text and image outputs.
  • External-tool systems approximate any-to-any interaction but pass information through discrete text and lack end-to-end training, causing noise propagation and limited handling of intricate instructions.
  • NExT-GPT combines multimodal encoders, an LLM for semantic reasoning, and diffusion decoders to process and generate arbitrary combinations of text, images, videos, and audio.
  • The system uses existing pretrained encoders and decoders while locally fine-tuning projection layers through encoding-side and decoding-side alignment.
  • The work presents NExT-GPT as an end-to-end any-to-any MM-LLM and a promising path toward agents modeling universal modalities.
  • NExT-GPT introduces a modality-switching instruction-tuning dataset to support cross-modal understanding, reasoning, and generation with only 1% parameter adjustment.

2. Related Work

Related work has expanded MM-LLMs from multimodal perception toward multimodal generation and tool-based orchestration. However, prior systems remain limited in arbitrary-modal generation, end-to-end integration, or LLM-centered reasoning.

  • Multimodal learning research covers cross-modal understanding and high-quality content generation across diverse tasks and modalities.
  • Most MM-LLMs align pretrained modality encoders with textual LLM features but remain limited to perceiving multimodal data.
  • Some systems use LLMs as decision-makers with pretrained encoders and decoders as tools, enabling multimodal input and output but relying on external pipeline components.
  • The related approaches motivate an MM-LLM that combines multimodal perception and generation more directly.

3. Overall Architecture

NExT-GPT uses a three-stage architecture: multimodal encoding, LLM understanding and reasoning, and multimodal decoding. Existing encoders, an LLM, and diffusion models are connected through projection layers, with only 1% of parameters updated during fine-tuning.

  • NExT-GPT consists of encoding, LLM understanding and reasoning, and decoding stages.
  • Multimodal Encoding Stage: Existing encoders produce representations that a projection layer maps into language-like features comprehensible to the LLM.
  • 1% of parameters need updating during fine-tuning.
  • LLM Understanding and Reasoning Stage: The LLM processes multimodal representations for semantic understanding and reasoning, then emits textual responses or modality-specific signal tokens for decoding.
  • Multimodal Generation Stage: Transformer-based output projections map LLM signal tokens into representations consumed by Stable Diffusion, Zeroscope, and AudioLDM for image, video, and audio synthesis.

4. Lightweight Multimodal Alignment Learning

NExT-GPT performs lightweight alignment at the encoding and decoding interfaces rather than retraining its multimodal backbones. The approach aligns modality representations with LLM semantics and teaches the LLM to emit instructions usable by diffusion decoders.

  • NExT-GPT updates only the encoding-side and decoding-side projection layers to bridge feature spaces across its loosely coupled tiers.
  • Encoding-side LLM-centric Alignment: Encoding-side alignment addresses the mismatch between patch-based multimodal features and the separate concepts represented by language tokens.
  • Encoding-side LLM-centric Alignment: An X-to-text generation task trains a frozen LLM to produce captions from image, audio, or video representations.
  • Decoding-side Instruction-following Alignment: Decoding-side alignment uses special modality signal tokens to encode flexible instructions for downstream diffusion models.
  • Decoding-side Instruction-following Alignment: The diffusion backbones remain frozen, keeping training lightweight while projected signal tokens are aligned with their conditional text representations.
  • Alignment training combines signal-token negative log-likelihood, caption alignment, and conditional latent denoising losses.

5. Modality-switching Instruction Tuning

NExT-GPT uses instruction tuning to improve controllable multimodal interaction, while MosIT supplies diverse, complex dialogues for arbitrary modality switching. The section combines alignment procedures, dataset construction, and benchmark settings.

  • 5.1. Instruction Tuning: Instruction tuning improves the capabilities and controllability of the overall MM-LLM through user-instruction and desired-output pairs.LoRA updates a small parameter subset alongside two projection layers during tuning.
  • 5.2. Instruction Dataset: MosIT extends existing text-to-text-plus-modality datasets to target outputs containing both textual and multimodal content.Earlier datasets primarily provide textual responses, whereas the any-to-any setting requires multimodal generations.
  • 5.2. Instruction Dataset: Existing instruction datasets lack variable modalities and sufficiently long dialogues to represent complex real-world interactions.The limitation motivates more diverse and dynamically changing user-machine exchanges.
  • 5.2. Instruction Dataset: MosIT supports complex cross-modal understanding, reasoning, and sophisticated multimodal content generation.Its dataset includes varied inputs and outputs designed to support accurate responses across diverse interactions.
  • 5.2. Instruction Dataset: The MosIT dataset uses template dialogues expanded with GPT-4 across more than 100 topics, including explicit and implicit requirements, perception, reasoning, suggestion, and planning.The interactions are designed to remain logically connected, semantically coherent, and reasoning-intensive.
  • 5.2. Instruction Dataset: After human inspection and filtering, the MosIT collection contains 5K high-quality dialogues.Its statistics are compared with existing multimodal instruction-tuning datasets in Appendix Table 6.
  • 5.2. Instruction Dataset: The experiments evaluate zero-shot image captioning and image question answering across multiple datasets, including NoCaps, Flickr 30K, COCO, VQAv2, VizWiz, OKVQA, MMB, and SEED.CIDEr is used for image captioning, while MMB and SEED are evaluation-only benchmarks.
  • 5.2. Instruction Dataset: Additional evaluations cover video reasoning on MSRVTT, MSVD-QA, MSRVTT-QA, and NExTQA, audio captioning on AudioCaps, and text-to-image, audio, and video generation.Table 3 marks fine-tuned results with an asterisk, while Table 4 marks zero-shot results with a dagger.

6. Experiments

Experiments assess NExT-GPT’s multimodal perception, generation, architectural choices, signal-token allocation, human-evaluated instruction following, and qualitative cross-modal behavior. The reported results show strong performance across these settings.

  • 6. Experiments: The experiments quantify NExT-GPT on downstream tasks requiring perception and generation across modalities.Implementation details and additional analyses are provided in the appendices.
  • 6.1. Main Results: NExT-GPT shows strong image comprehension, comparable performance on MMB and SEED, and effective video and audio comprehension.It improves over Codi in comparisons attributed to direct text generation from the LLM.
  • 6.1. Main Results: NExT-GPT achieves superior text-conditioned image, audio, and video generation while supporting a broader modality range than LLM-centric models.It maintains optimal individual-modality performance even in zero-shot scenarios and improves over non-LLM-centered models.
  • 6.2. In-depth Analysis: Video generation requires the highest number of modality signal tokens, whereas images and audio achieve satisfactory generation with 4 and 8 tokens, respectively.The suitable token count depends on training-data size and diffusion-model selection.
  • 6.2. In-depth Analysis: Qualitative examples show cross-modal comprehension, reasoning, implicit-intent inference, and modality-specific responses such as cheerful puppy videos or presentation visualizations.The examples span arbitrary combinations of modalities and include empathetic responses to user mood.
  • 6.2. In-depth Analysis: Removing the grouping mechanism with a simple linear projection significantly decreases perceptual capabilities across image, video, and audio tasks.A Q-former provides a modest improvement, but NExT-GPT’s grouping mechanism performs best.
  • 6.2. In-depth Analysis: NExT-GPT outperforms the compared systems in following complex instructions and generating high-quality images.The qualitative and human-evaluation results complement the benchmark-based perception and generation experiments.

7. Conclusion

NExT-GPT is presented as an end-to-end any-to-any multimodal language model. It connects an LLM with multimodal adaptors and diffusion decoders, using lightweight tuning and modality-switching instruction data.

  • NExT-GPT is an end-to-end general-purpose any-to-any multimodal large language model.
  • By connecting an LLM with multimodal adaptors and diffusion decoders, NExT-GPT handles arbitrary combinations of text, images, videos, and audio.
  • Training updates only 1% of certain projection-layer parameters while using existing encoders and decoders.This supports lower-cost training and expansion to additional modalities.
  • MosIT and its manually curated dataset support complex cross-modal semantic understanding and content generation.

Impact Statement

The paper identifies quality, safety, licensing, privacy, and fairness considerations for NExT-GPT. It cautions that limited fine-tuning data and base-model quality may produce harmful low-quality or hallucinated content.

  • Limited fine-tuning data and base-model quality may cause low-quality or hallucinated content that could be harmful.
  • Users are cautioned to interpret results carefully and comply with licensing rules, including a prohibition on commercial use.
  • The paper addresses privacy by following platform terms, obtaining consent when necessary, and anonymizing or obfuscating personal information.
  • Dataset collection seeks to minimize bias by pursuing representative and fair data that does not favor or disfavor particular groups or perspectives.

A. Potential Limitation and Future work

The paper outlines future work to broaden NExT-GPT’s modality and task coverage, support more LLM variants, improve generation quality, and expand MosIT data.

  • Modalities & Tasks Expansion: Resource limitations currently restrict NExT-GPT to language, images, videos, and audio; future work targets additional modalities and tasks.Examples include web pages, 3D vision, heat maps, tables, object detection, segmentation, grounding, and tracking.
  • LLM Variants: Future plans include incorporating LLM types and sizes beyond the implemented 7B Vicuna version.This would let practitioners choose models suited to their requirements.
  • Multimodal Generation Strategies: Generative output quality can be limited by diffusion-model capabilities, motivating retrieval-based approaches as a possible complement.
  • MosIT Dataset Expansion: The MosIT instruction-tuning dataset has room for expansion, with plans to increase annotated data.

C.2. Model Training

NExT-GPT training uses staged alignment and instruction tuning with multimodal datasets. The process combines projection-layer learning, modality-switching data, and evaluation resources spanning text, image, video, and audio tasks.

  • Training Stages: NExT-GPT uses a three-stage learning process beginning with encoding-side alignment learning.Stage 1 trains only the input projection layer while the remaining model is frozen, using cross-entropy and Adam optimization.
  • Training Data: The training collection includes text-image, text-video, and audio-text datasets alongside textual and multimodal instruction datasets.Examples include CC3M, COCO-caption, WebVid-2M, AudioCaps, Cleaned-Alpaca, LLaVA-150K, and VideoChat.
  • MosIT Dataset: MosIT dialogues contain 3–7 turns with multimodal inputs or outputs and alternating modality switches.
  • MosIT Dataset: The instruction-tuning resources include 15k text-to-multimodal instances and a 5k-instance modality-switching dataset named MosIT.
  • Inference Procedure: During inference, encoders and projectors pass user modalities to the LLM, which emits text or modality signal tokens that control decoding.

D.1. Additional Multimodal Comprehension and Generation Results

NExT-GPT performs competitively across text-conditioned synthesis, cross-modal captioning, and modal editing tasks. Its strongest results appear in text-to-video generation and X-to-text semantic understanding, while editing remains competitive rather than superior.

  • Text-conditioned generation: NExT-GPT matches the best-performing baselines on text-to-image and text-to-audio generation tasks.These comparisons evaluate synthesis quality against non-LLM-based methods.
  • Text-conditioned generation: NExT-GPT shows significant superiority in zero-shot text-to-video generation.The result is presented as evidence of strong generalization under zero-shot conditions.
  • Cross-modal comprehension: NExT-GPT mostly outperforms CoDi on X-to-text generation tasks.The paper attributes this advantage to direct text generation by the LLM.
  • Modal editing: NExT-GPT remains competitive, but not superior, on text-conditioned image, video, and audio editing tasks.These tasks compare text-conditioned modal editing across the reported datasets and tables.
  • Inference process: The inference process selectively deactivates modules for different multimodal generation settings.Figure 7 depicts the NExT-GPT inference process, with grey indicating deactivated modules.

D.2. Human Evaluation on Complex Any-to-any QA

Human evaluation examines NExT-GPT on complex cross-modal conversions, while case studies compare it with pipeline-style systems. The examples emphasize that intermediate captions can lose modality-specific visual information during complex instruction following.

  • Human evaluation: Human evaluators score NExT-GPT from 1 to 10 across complex input-output modality conversions.Because no standard benchmark is available, the study uses human evaluation and reports comparisons in Figure 8.
  • Case studies: Pipeline-style systems can generate incorrect images for complex instructions involving visual numeration and spatial relations.The simple-instruction case appears correct, whereas the complex cases in Figures 10 and 11 show biased or incorrect content.
  • Case studies: Intermediate captions risk omitting modality-specific visual features when non-linguistic content is expressed solely through language.The paper identifies caption-based transfer as a source of information loss for complex visual attributes.
  • End-to-end unification: NExT-GPT passes implicit modality-signal token embeddings carrying non-linguistic features to downstream modules such as Stable Diffusion.The end-to-end design is contrasted with pipeline systems that pass intermediate captions to downstream tools.
Loading 2309.05519v3…