Source-linked AI summary
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, Zhaopeng Tu
TL;DR
Instruction-tuned LLMs have mainly been studied with text, leaving their effectiveness on other modalities insufficiently explored. MACAW-LLM integrates image, video, audio, and text through multimodal encoders, an alignment module, and a pretrained LLM, using one-stage instruction tuning. The paper demonstrates multimodal understanding through examples while acknowledging that its evaluation and training data do not yet establish robust multi-turn performance.
Problem
Instruction-tuned LLMs are primarily limited to text, so their effectiveness on other data modalities has not been fully studied.
Method
MACAW-LLM combines modality encoders, an alignment module that maps multimodal features to LLM embeddings, and a cognitive module based on a pretrained LLM, trained in one step.
Results
The paper demonstrates MACAW-LLM’s multimodal understanding through examples producing contextually relevant, fluent responses to image and video questions.
Takeaways & Limitations
MACAW-LLM provides a single model and instruction dataset for integrating multiple modalities into language-model interactions.
Takeaways & Limitations
The evaluation may not accurately reflect true capabilities, and the mainly single-turn training data leaves MACAW-LLM not optimized for multi-turn dialogue.
Abstract
from arXiv · showhide
Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal LLMs and expand the capabilities of LLMs to handle diverse data modalities and address complex real-world scenarios.
1 INTRODUCTION
MACAW-LLM extends instruction-tuned LLMs beyond text by integrating image, video, audio, and text, while simplifying multimodal adaptation through alignment and one-stage tuning. It also introduces a diverse multimodal instruction dataset to address limitations in existing task-focused datasets.
- Model and alignment: MACAW-LLM integrates image, video, audio, and text within a single multimodal instruction-tuned LLM.The model jointly handles four modalities rather than focusing primarily on text.
- Model and alignment: Its alignment approach maps multimodal features to LLM embedding representations so they can be injected naturally into the input sequence.This design is intended to streamline adaptation from modality modules to the language model.
- Training strategy: One-stage instruction fine-tuning replaces the two-stage training commonly used by previous multimodal systems.The paper describes this as promoting a simpler learning experience and avoiding a separate alignment-training stage.
- Instruction dataset: The MACAW-LLM instruction dataset addresses existing datasets’ limited task diversity and weak alignment with human-written instructions.The dataset is curated with the generative capability of GPT-3.5-TURBO.
- Contributions: The work contributes a multimodal architecture that jointly aligns multimodal and textual features while generating output sequences, alongside a large-scale instruction dataset.The stated dataset contribution covers diverse instructional tasks using image and video modalities.
2 RELATED WORK
Multimodal research has pursued shared representations and instruction-following capabilities, building on the strong generalization of instruction-tuned LLMs. MACAW-LLM contributes a model that integrates visual, audio, and textual information for following multimodal instructions.
- Instruction-tuned LLMs: Instruction-tuned LLMs achieve few-shot and zero-shot generalization across many NLP tasks.These capabilities motivate extending LLMs to additional modalities.
- Multimodality: Multimodal research commonly studies joint embedding spaces and combinations of pretrained vision-only and language-only models.These approaches aim to align or combine representations across modalities.
- Multimodal instruction following: Recent work has enabled multimodal pretrained models to follow instructions across diverse tasks and categories.The related work includes multimodal instruction-tuning datasets and machine-generated instruction data.
- Our Work: MACAW-LLM integrates visual, audio, and textual information to comprehend and execute instructions.This positions the model within the broader effort to combine modalities with instruction-following language models.
3 METHODOLOGY
MACAW-LLM combines modality encoders, an alignment module, and an instruction-tuned LLM to process image, video, audio, and text. Its alignment strategy maps modality representations into the LLM’s textual embedding space, enabling concatenation with textual instructions and one-step instruction fine-tuning.
- Architecture: MACAW-LLM uses modality, alignment, and cognitive modules to integrate visual, audio, and textual information.The modality module encodes non-text inputs, the alignment module unifies representations, and the cognitive module uses an instruction-tuned LLM.
- Modality Module: CLIP encodes images and video frames, WHISPER-BASE encodes audio, and LLAMA-7B provides the textual foundation.These pretrained encoders supply modality-specific representations for MACAW-LLM.
- Modality Alignment: A 1-D convolution compresses modality-feature sequences, followed by a linear layer that matches the LLM embedding dimension.The transformed features have fixed length L′, substantially smaller than the original modality sequence lengths, and embedding dimension de.
- Modality Alignment: Attention aligns transformed visual and audio features with the LLM’s textual embedding space by treating them as soft tokens.This establishes a common representation space so the cognitive module can process representations from different modalities.
- Integration: Aligned modality representations are concatenated with textual token embeddings to form the multi-modal instruction input.The textual instruction remains part of the input sequence, while aligned modality features are inserted through concatenation.
- Training: MACAW-LLM uses one-step instruction fine-tuning over all parameters, minimizing response negative log-likelihood.Unlike common two-step approaches, this strategy is described as simplifying adaptation and harmonizing the model’s modules.
4 MACAW-LLM INSTRUCTION DATASET
The MACAW-LLM instruction dataset addresses limited task diversity and weak human-text alignment in existing multimodal datasets by generating varied instruction-response pairs from image and video captions. It contains approximately 69K image-based and 50K video-based examples, while remaining focused on single-turn dialogue.
- Existing multimodal datasets often emphasize specific task types and may not align target text with human-written instruction styles.
- GPT-3.5-TURBO generates instruction-response pairs from image or video captions, producing 10 pairs per query for efficiency.
- The image data samples 10,000 MS COCO images, while video data comes from 9,848 captioned videos in Charades and AVSD.
- Approximately 69K examples are based on COCO image captions and about 50K on Charades and AVSD video captions.
- The current dataset focuses on single-turn dialogues and is being expanded toward multi-turn dialogues and broader multimodal content.
5 EXPERIMENTAL SETUP
The experimental setup combines text, image, and video instruction data for training MACAW-LLM, with audio inputs currently attached to video examples. Dataset construction uses GPT-4-based instruction generation, and training uses DeepSpeed on eight A100 GPUs.
- Training draws on text, image, and video instruction datasets, with approximately 52K text examples and around 69K image instruction-response pairs.
- The final training set randomly samples 50K examples from each instruction-data type to form 150K examples.
- Audio inputs are currently associated with video instruction data because a separate audio instruction dataset is still being created.
- Instruction-response examples are generated from image or video caption content using a GPT-4 prompt designed to avoid explicitly mentioning the caption.
- Training uses DeepSpeed on 8 Nvidia A100 GPUs for 5 epochs, with a learning rate of 3×10^-5 and maximum sequence length 512.
6 EXAMPLES
The examples illustrate MACAW-LLM’s ability to answer questions grounded in images and videos, combining object recognition with contextual, temporal, and cross-modal reasoning. The demonstrations also include audio-grounded interpretation.
- The examples are presented as producing contextually appropriate and logically consistent responses to diverse visual-content questions.
- Image-grounded question answering: The model identifies two giraffes and trees in an image and infers that the photograph was taken during daytime.
- Image understanding and reasoning: Image examples show recognition of basic objects, color, and light, alongside attempts to estimate human age and room location.
- Video-grounded question answering: The video-grounded example recognizes boats, counts them, and identifies their actions over time.
- Visual reasoning: A visual reasoning example infers snow from a small visible portion of white outside a door and attempts to estimate a woman’s age.
- Video- and audio-grounded question answering: A video-and-audio example describes a dog on a grassy field and uses the audio track to indicate that the dog remains silent.
7 LIMITATIONS
The paper identifies limitations in evaluation coverage, dialogue context, and unassessed safety-related properties. It also notes that the model still requires textual instructions and that encoder choices remain for future investigation.
- Evaluation: The evaluation relies on illustrative examples and may not accurately or comprehensively reflect MACAW-LLM’s capabilities.
- Single-turn dialogue: Training instructions are limited to single-turn interactions, so MACAW-LLM is not optimized for multi-turn dialogue or long-range context.
- Hallucination, toxicity and fairness: The paper does not evaluate hallucination, toxicity, or fairness because suitable evaluation resources are unavailable.
8 CONCLUSION AND FUTURE WORK
MACAW-LLM integrates image, video, audio, and text through shared representation alignment and single-step instruction tuning. The authors also identify current limitations and plan broader multi-turn and multilingual dialogue coverage.
- Conclusion: MACAW-LLM accommodates image, video, audio, and text in one multi-modal instruction-tuned LLM.Its architecture combines modality and cognitive modules with an approach that aligns representations from different encoders into a shared space.
- Conclusion: Single-step representation alignment and instruction tuning mitigates potential error propagation during multi-step tuning.
- Conclusion: The authors curate a large-scale multi-modal instruction dataset using GPT-3.5-TURBO and demonstrate MACAW-LLM’s multi-modal understanding ability.
- Future Work: Current multi-modal instruction-tuned LLMs may suffer from various limitations that the authors leave for future investigation.
- Future Work: The authors plan to broaden the corpus to include multi-turn and multilingual dialogues.They intend to use LLMs to generate and translate long-document texts for this expansion.