Source-linked AI summary
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu
TL;DR
AnyGPT addresses the limitation of text-only LLMs and multimodal systems that primarily generate text. It uses discrete multimodal representations, tokenizers, de-tokenizers, and multimodal datasets to support unified any-to-any processing; experiments report zero-shot performance comparable to specialized models across modalities.
Problem
Existing LLMs are confined to text, while multimodal systems commonly support multimodal perception but remain limited to text generation.
Method
AnyGPT converts non-text modalities into discrete tokens for autoregressive processing, decodes them back through de-tokenizers, and trains on multimodal alignment and instruction datasets.
Results
AnyGPT achieves zero-shot performance comparable to specialized models across various modalities and facilitates any-to-any multimodal dialogue.
Takeaways & Limitations
Discrete multimodal representations can unify speech, text, images, and music within a language model while supporting arbitrary combinations of multimodal inputs and outputs.
Takeaways & Limitations
Higher loss than unimodal training prevents optimal performance in each modality.
Abstract
from arXiv · showhide
We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/
1 Introduction
AnyGPT addresses the limitation of text-only LLMs by using discrete representations to unify multimodal understanding and generation across speech, text, images, and music. It combines this architecture with multimodal alignment and instruction data, enabling arbitrary combinations of multimodal inputs and outputs.
- LLMs remain largely confined to text despite real-world information arriving through vision, language, sound, and touch.
- Existing multimodal approaches often support text generation but not multimodal output, while integrating three or more modalities creates alignment and training challenges.
- AnyGPT converts modalities into discrete semantic tokens so one LLM can perform perception, understanding, reasoning, and generation autoregressively without changing its architecture or training paradigms.
- AnyGPT achieves zero-shot performance comparable to specialized models across various modalities and supports any-to-any multimodal dialogue.
- AnyGPT supports speech, text, images, and music through a token-based any-to-any multimodal language model.
- 108k multi-turn multimodal dialogues in AnyInstruct-108k provide interleaved instruction-following data for arbitrary combinations of multimodal inputs and outputs.
2 Related Work
Related work has expanded LLMs toward multimodal understanding and generation, but existing systems typically support limited modalities or use non-unified representations. Discretization offers a route toward a more unified multimodal language model.
- Adaptor-based systems connect pre-trained modality encoders to LLMs but are often limited to text generation.
- NExT-GPT uses separately trained encoders and decoders without a unified input-output representation, complicating unified training and inference.
- Discretization methods such as VQ-VAE compress representations into tokens, with residual quantization added in some work to improve fidelity.
- Semantic tokenizers separate high-level semantic information from residual acoustic or perceptual information across modalities.
3 AnyGPT
AnyGPT uses modality-specific tokenizers, a shared LLM backbone, and de-tokenizers to process multimodal data as interleaved discrete sequences. Semantic tokens support efficient autoregressive modeling, while non-autoregressive decoders restore high-fidelity outputs.
- Architecture: AnyGPT comprises multimodal tokenizers, a multimodal language-model backbone, and de-tokenizers that transform between continuous modalities and discrete sequences.
- Music Tokenization: AnyGPT models music by flattening four-layer codes frame by frame, with 5 seconds encoded as 250 latent frames.
- Language Model Backbone: The expanded vocabulary combines modality-specific token sets, while the rest of the pre-trained LLM remains unaltered apart from its embedding and prediction layers.
- Architecture: The language model trains interleaved multimodal sequences with next-token prediction, enabling autoregressive perception, understanding, reasoning, and generation.
- Generation: High-fidelity generation uses two stages: semantic modeling by the LLM followed by non-autoregressive conversion into perceptual content.
- Generation: Images use SEED tokens and diffusion decoding, while speech uses SoundStorm to generate acoustic tokens from semantic tokens.
4 Multimodal Data
The multimodal data pipeline aligns non-text modalities with text and constructs large-scale interleaved conversations. Pre-training uses token-based balancing, while AnyInstruct converts text-based dialogues into multimodal interactions.
- Alignment data: Text serves as an intermediary for aligning different modalities and enabling mutual multimodal alignment.
- Pre-training data: Pre-training data are quantified by token counts, with lower-volume modalities oversampled for balanced batches.
- Pre-training data: The pre-training corpus includes image-text, image-text interleaved, speech-text, and music-text data.
- Instruction data: AnyInstruct construction has two stages: generating text-based conversations with multimodal descriptions, then converting descriptions into multimodal elements.
- Instruction data: 108k filtered multimodal dialogues combine text, speech, images, and music, including around 205k images, 503k voice recordings, and 113k music tracks.
- Instruction data: The two-stage synthesis approach efficiently collects diverse, high-quality multimodal conversations at scale.
5 Experiment
AnyGPT is evaluated zero-shot on cross-modal understanding and generation tasks spanning images, speech, and music. Case studies further examine any-to-any dialogue after instruction fine-tuning.
- Evaluation setup: Zero-shot evaluation tests both text-to-X and X-to-text tasks for image, music, and speech modalities.
- Image: Image captioning is evaluated on MS-COCO 2014 using the Karpathy split, while text-to-image generation uses 30k MS-COCO validation images and CLIPscore.
- Speech: ASR is evaluated on LibriSpeech test-clean with Word Error Rate, using Wav2vec 2.0 and Whisper Large V2 as baselines.
- Speech: TTS is evaluated zero-shot on VCTK using speaker similarity and Word Error Rate.
- Music: Music understanding and generation are evaluated on MusicCaps using CLAPscore for alignment between music and textual captions.
- Music: Existing objective metrics may inadequately express music-captioning performance because music diversity and subjectivity produce varying human judgments.
- Any-to-any dialogue: After fine-tuning on AnyInstruct-108k, AnyGPT handles instructions interwoven with text, voice, images, and music in arbitrary combinations.
6 Conclusion
AnyGPT unifies speech, text, images, and music through discrete representations without changing the existing LLM architecture or training paradigms. Its AnyInstruct-108k dataset supports arbitrary multimodal input-output combinations, and experiments report promising cross-modal performance.
- AnyGPT uses discrete representations to process speech, text, images, and music within one multimodal language model.
- The approach integrates modalities without altering the existing LLM architecture or training paradigms.
- AnyInstruct-108k consists of multi-turn conversations interweaving modalities to support arbitrary multimodal inputs and outputs.
- Experiments report promising results across cross-modal tasks and support the feasibility of unifying modalities in an LLM.
Limitations and Future Work
The paper identifies several boundaries for AnyGPT, including the lack of a comprehensive benchmark, higher multimodal loss, tokenizer quality constraints, and limited context for audio and dialogue.
- Benchmark: Any-to-any multimodal LLMs lack a dedicated benchmark covering multiple capabilities and potential risks.The paper therefore calls for a comprehensive benchmark.
- Enhancing LLMs: Higher loss than unimodal training prevents discrete-representation multimodal LLMs from achieving optimal performance in each modality.The paper suggests scaling LLMs and tokenizers or using a Mixture-Of-Experts architecture as potential remedies.
- Better Tokenizer: Tokenizer quality sets a ceiling for multimodal LLM comprehension and generation.Potential improvements include better codebook training, more cohesive multimodal representations, and information disentanglement.
- Longer Context: AnyGPT limits music modeling to 5 seconds, restricting the practical usefulness of its audio output.Longer context could also support more conversational exchanges and richer interaction.
A.1 Data
Table 7 presents the data details for AnyGPT’s pre-training stage.
- Data: Table 7 documents the data used during pre-training.
- Data: The table is organized around pre-training data details.
- Data: Pre-training data composition is reported in Table 7.
A.2 pre-training
AnyGPT pre-training converts multimodal content into token sequences and uses templates, instructions, interleaved data, and continued training on selected high-quality datasets.
- A.2 pre-training: Non-text modalities are marked with special beginning and ending tokens and paired with captions or transcriptions.The paired modalities include images, speech, and music.
- A.2 pre-training: Training hyperparameters are reported in Table 8.
- A.2 pre-training: Bidirectional X-to-text and text-to-X instructions are generated and inserted with token sequences and related text.The resulting triplet is formed from an instruction, token sequence, and text.
- A.2 pre-training: Interleaved multimodal documents replace non-text content with corresponding token sequences.
- A.2 pre-training: After initial pre-training, selected high-quality image, captioning, and music datasets are used for an additional 4000 steps.The remaining data are kept unchanged during this continuation.
- A.2 pre-training: The paper reports that web-sourced image and music data contain noise that can affect multimodal generation quality.
- A.2 pre-training: Figure 4 illustrates autoregressive processing of discrete tokens for multimodal dialogue.An image and voice command produce music together with text and voice responses.
C Evaluation
The evaluation describes zero-shot TTS testing on VCTK, metrics for speaker similarity and transcription accuracy, comparisons with VALL-E and USLM, and decoding strategies reported in Table 9.
- C Evaluation: Table 9 reports the generation decoding strategies used in evaluation.
- C Evaluation: Zero-shot TTS evaluation uses VCTK speakers absent from training data, with a 3-second vocal prompt and separate text input.
- C Evaluation: Speaker similarity is measured with WavLM-TDNN2 cosine similarity between generated-speech and prompt-speech embeddings.
- C Evaluation: WER is computed by transcribing generated speech with Whisper medium, with lower WER indicating higher synthesis quality.
- C Evaluation: AnyGPT is compared with VALL-E and USLM, which use two autoregressive speech models and different speech tokenizers.VALL-E uses Encodec, while USLM uses SpeechTokenizer.
D Prompts for Constructing Multimodal Interleaved Instruction Data
The instruction-data pipeline uses GPT4 prompts to brainstorm topics, construct multimodal chat scenarios, and generate detailed conversations. Demonstration figures show conversations combining speech, text, images, and music across varied input-output configurations.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: GPT4 prompts support three pipeline stages: brainstorming a topic pool, constructing chatting scenarios, and detailing chat contents.Multimodal content is represented as text descriptions during prompt-based construction.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: The brainstorming prompt requests non-academic conversation topics involving image or music understanding and generation.Topics are requested for conversations between an ordinary person and a helpful chatbot.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: 100 metatopics are expanded through four brainstorming rounds, producing 20,000 topics with 200 topics per metatopic.Figure 5 describes the resulting topic-pool construction process.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: Scenario prompts sample demonstrations, topics, and user-action requirements to guide GPT4 toward diverse multimodal chat situations.The requirements include users providing or requesting images and sharing or requesting music.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: The constructed conversations allow users to provide images or music and allow AnyGPT to respond with images or music in its utterances.This establishes multimodal content on both sides of the dialogue.
- D Prompts for Constructing Multimodal Interleaved Instruction Data: The content-writing prompt supplies scenario descriptions and corresponding chats as demonstrations for generating conversations involving AnyGPT.The prompt frames the chatbot as an assistant speaking with a user.
- E Examples Demonstration: Demonstration figures include speech instructions with image inputs and text, music, and speech responses, as well as speech conversations using voice cloning.Figures 9–11 illustrate speech-centered multimodal interactions.
- E Examples Demonstration: Other demonstrations show text, image, and music inputs producing image, music, text, or mixed multimodal outputs.Figures 10 and 12–16 cover several any-to-any combinations, including text plus music producing text plus image.