Source-linked AI summary
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, Bo Xu
TL;DR
X-LLM addresses the unknown architecture and training strategy behind GPT-4’s multimodal abilities by converting images, speech, and videos into language-like inputs for ChatGLM. It aligns frozen modality encoders and the LLM through X2L interfaces in three stages, achieving broad multimodal behaviors and an 84.5% relative score against GPT-4 on a synthetic instruction-following dataset. The paper also reports limitations from ChatGLM’s size and limited multimodal training data.
Problem
GPT-4 demonstrates advanced multimodal abilities, but its architecture and training methods are unknown, leaving the basis of those capabilities unclear.
Method
X-LLM converts images, speech, and videos into language-like representations through X2L interfaces and aligns frozen single-modal encoders with frozen ChatGLM in three training stages.
Results
84.5% relative score compared with GPT-4 is reported on a synthetic multimodal instruction-following dataset, alongside broad image, video, speech, and multimodal chat abilities.
Takeaways & Limitations
X-LLM supports multimodal interaction by extending an LLM’s language-based instruction abilities across independently aligned image, video, and speech modalities.
Takeaways & Limitations
X-LLM is limited by ChatGLM’s 6B parameters and unreliable reasoning or fabricated facts, while its multimodal connections use relatively little training data.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable language abilities. GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities beyond previous visual language models. We attribute this to the use of more advanced LLMs compared with previous multimodal models. Unfortunately, the model architecture and training strategies of GPT-4 are unknown. To endow LLMs with multimodal capabilities, we propose X-LLM, which converts Multi-modalities (images, speech, videos) into foreign languages using X2L interfaces and inputs them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X'' denotes multi-modalities such as image, speech, and videos, and ``L'' denotes languages. X-LLM's training consists of three stages: (1) Converting Multimodal Information: The first stage trains each X2L interface to align with its respective single-modal encoder separately to convert multimodal information into languages. (2) Aligning X2L representations with the LLM: single-modal encoders are aligned with the LLM through X2L interfaces independently. (3) Integrating multiple modalities: all single-modal encoders are aligned with the LLM through X2L interfaces to integrate multimodal capabilities into the LLM. Our experiments show that X-LLM demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 84.5\% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. And we also conduct quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote the era of LLM-based speech recognition.
1 Introduction
X-LLM addresses the unknown design of GPT-4 by converting images, speech, and videos into language-like representations for ChatGLM. It reports broad multimodal abilities, including GPT-4-like behaviors and an 84.5% relative score against GPT-4 on a synthetic instruction-following dataset.
- Motivation: GPT-4’s multimodal architecture and training methods are unknown, motivating an approach that leverages advanced language-model capabilities.The paper attributes GPT-4’s multimodal abilities to its more advanced and larger language model.
- Method: X-LLM converts images, speech, and videos into foreign-language representations through X2L interfaces before feeding them into frozen ChatGLM.The framework aligns frozen single-modal encoders and a frozen language model through image, video, and speech interfaces.
- Results: X-LLM demonstrates multimodal behaviors such as complex image description, visual reasoning, video understanding, and spoken questions combined with images or videos.The reported abilities include recognizing locations, describing food and recipes, creating stories, retrieving movie names, and answering multimodal spoken questions.
2 Related Work
Prior work developed specialized vision-language and visual-dialogue models through multimodal pretraining and task-specific finetuning. X-LLM instead presents a universal framework that treats modalities as foreign languages and supports images, videos, and speech.
- Vision-Language Models: Vision-language models use visual and textual encoders with objectives including masked language modeling, image-text matching, and contrastive learning.The related work traces this progression from region-based features to vision transformers and CLIP-style separate encoders.
- Visual Dialogue: Visual-dialogue systems commonly finetune pretrained vision-language models on dialogue data using masked language modeling and image-text matching.Examples include VisDial-BERT, VD-BERT, and AlignVD.
- X-LLM: X-LLM differs as a universal multimodal framework that injects images, videos, and speech into an LLM as foreign languages through X2L interfaces.The framework is described as extensible to modalities such as robot states, terminal information, and audio.
3 Approach
X-LLM aligns pretrained modality encoders with ChatGLM through X2L interfaces that convert multimodal signals into language-like representations. Its staged training supports independent modality alignment and later multimodal integration, including joint-instruction finetuning.
- Architecture: X-LLM uses ChatGLM with pretrained image, video, and speech encoders connected through X2L interfaces.The interfaces include image, video, and speech components that align multimodal information with the LLM.
- X2L Interfaces: The image interface converts image features into quasi-linguistic embeddings with a Q-Former and aligns their dimensions to the LLM with an I-Adapter.The video interface uses the same structure, samples T frames, and concatenates frame-level sequences of length Li into T × Li embeddings.
- X2L Interfaces: The speech interface uses CIF-based variable-length down-sampling, a 12-layer transformer, and an S-Adaptor to map speech into the LLM input space.The CIF output is intended to have the same length as the transcription token sequence, while the S-Adaptor narrows the speech-language semantic gap.
- Training Strategy: Training proceeds by converting multimodal information, aligning each X2L representation with the LLM, and then integrating multiple modalities.The first two stages align modality-specific components independently, while later training supports multimodal integration.
- Multimodal Integration: After the first two stages, X-LLM performs multimodal tasks such as visual spoken dialogue without joint multimodal instruction training.The paper attributes this to integrating the LLM’s instruction generalization capability with multimodal adaptation.
- Multimodal Integration: The third stage finetunes X-LLM with compact multimodal instruction data and prompts containing image, video, and speech inputs.The dataset combines translated and manually selected image-text, speech, video-text, and visual-dialogue examples.
4 Experiments
The experiments evaluate X-LLM on multimodal chat, visual instruction following, and speech or multimodal speech recognition. X-LLM performs strongly relative to GPT-4 and competing visual models, while speech recognition remains weaker and sensitive to training and instruction choices.
- Multimodal Chat: X-LLM recognizes Chinese visual content more specifically than LLaVA and MiniGPT-4 in example prompts, including the Forbidden City and Honor of Kings.LLaVA and MiniGPT-4 generally provide generic or incorrect descriptions in these examples.
- Multimodal Chat: 84.5% relative score to GPT-4 is reported for X-LLM on the multimodal instruction-following evaluation.The evaluation uses 30 COCO validation images, three question types per image, and ChatGPT-based scoring; comparisons are described as rough because evaluation models differ.
- Multimodal Chat: X-LLM outperforms LLaVA on conversation and detailed description but is inferior on complex reasoning.The authors attribute the complex-reasoning difference to LLaVA’s larger instruction dataset, larger language model, and language-model finetuning.
- Multimodal Chat: Using BLIP2 pretrained Q-Former parameters significantly improves performance, supporting transferability from English image-text alignment.Increasing image-text training pairs also improves performance, while the Wukong-containing variant performs worse on complex reasoning than the 4M variant.
- ASR and Multimodal ASR: X-LLM’s speech recognition is weaker than the fully trained CIF-based model, while adding images increases error rates before targeted finetuning.On VSDial-CN, multimodal recognition improves from 4.7 to 4.3 on test-orig and from 6.8 to 6.3 on test-art after finetuning with 2k speech instructions.
- ASR and Multimodal ASR: X-LLM can answer spoken questions and incorporate images without finetuning, with this ability further improved after small-scale third-stage finetuning.The addition of images causes a slight decrease in speech-recognition performance in the reported evaluation.
5 Discussions
X-LLM uses a three-stage strategy to inject multiple modalities into an LLM through X2L interfaces. The resulting system supports multimodal interaction without joint instruction fine-tuning, while additional multimodal instruction data further improves performance.
- X-LLM injects images, speech, and videos into a large language model as foreign languages through X2L interfaces.The approach aligns multiple frozen single-modal encoders and a frozen LLM.
- The first two training stages keep modality interfaces highly independent, facilitating simultaneous training and interaction with each modality through language.
- X-LLM completes multimodal tasks such as visual spoken question answering without further fine-tuning on joint instruction datasets.This is attributed to combining LLM instruction generalization with multimodal adaptation.
- Fine-tuning on a streamlined, high-quality multimodal instruction dataset further improves X-LLM’s performance.
- The system’s current limitations include ChatGLM’s unreliable reasoning and fabricated facts, plus limited multimodal perception from small-scale connection data.
A.1 Training Data
The appendix describes datasets and training resources spanning vision-language, video-language, ASR, and visual spoken dialogue data across the three training stages. It also records test-question and training-detail tables, along with the construction of multimodal ASR data.
- Training datasets combine vision-language, video-language, automatic ASR, and visual spoken dialogue data.Their statistics are listed in Table 5.
- The first two image-interface stages use caption datasets including CC3M, COCO, Visual Genome Captions, Flickr30k, SBU, AI-Caps, and Wukong.
- The CIF-based ASR model uses AISHELL-1, AISHELL-2, and VSDial-CN, while the second speech-interface stage uses AISHELL-2 and VSDial-CN.
- AISHELL-2 contains approximately 1,000 hours of Mandarin speech across 12 domains, recorded by 1,991 speakers from different Chinese accent areas.
- VSDial-CN contains approximately 1.2 million multimodal ASR training samples derived from around 120,000 visual-dialogue images and ten dialogue rounds per image.After deduplication, approximately 370,000 unique question utterances remain.
- The appendix includes tables of test-question examples and three-stage training details; the final image and video interfaces omit stage-one training after initialization from BLIP2 Q-Formers.
- Third-stage data include translated image-text instructions, manually selected ASR and video-text examples, and rewritten VSDial-CN data.
A.2 Examples of Test Questions
The test set provided by LLaVA contains 30 randomly sampled COCO Val 2014 images, each paired with short, detailed, and complex reasoning questions.
- The test set contains 90 questions over 30 randomly sampled images from COCO Val 2014.Each image has one short, one detailed, and one complex reasoning question.
A.3 Training Details
The appendix specifies the architectures and optimization settings for X-LLM and its CIF-based ASR model. The ASR decoder incorporates visual and linguistic features through cross-attention for multimodal speech recognition.
- X-LLM has 7.6 billion parameters, including frozen single-modality encoders and the ChatGLM language model.Image and video encoders share ViT-g, while the speech encoder uses convolutional layers and a conformer structure.
- The CIF-based ASR model uses a warm-up, hold, and decay learning-rate schedule lasting 24,000, 36,000, and 120,000 steps.The hold-stage learning rate is 3e-4, with weight decay 0.01, trained on 8 A100-80G GPUs.
- The CIF-based ASR model combines a convolutional front-end, an 18-block conformer encoder, a CIF module, and a six-block transformer decoder.
- Visual features enter decoder blocks 3–4 and linguistic features enter blocks 5–6 through cross-attention for multimodal ASR.
B Qualitative Examples
X-LLM demonstrates multimodal understanding across video, speech, text, and images, including translation that depends on visual context and responses to legal and ethical scenarios.
- X-LLM answers spoken questions about images without requiring automatic speech recognition, integrating speech and visual information directly.
- X-LLM performs multimodal machine translation by selecting the visually grounded meaning of ambiguous words such as “bank.”
- X-LLM responds to video prompts based on video content, demonstrating video-input understanding.
- X-LLM refuses requests involving legally problematic images and provides reminders to comply with relevant laws.
- X-LLM declines ethical or moral judgments and recommendations while emphasizing traffic regulations, safety knowledge, and careful driving.