PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, Jing Liu

arXiv:2305.16103v1cs.CVcs.AIcs.CLcs.MM

TL;DR

Many multimodal approaches require paired data for every modality combination and are tuned to specific tasks. ChatBridge uses language-paired two-modality data and instruction-tunes with MULTIS; the authors report strong zero-shot results across text, image, video, and audio tasks, while noting weaknesses with long-range video and audio.

  • Problem

    Existing multimodal paradigms often require paired data across all modality combinations and remain limited to specific tuned tasks.

    Source: §1 · p. 1

  • Method

    ChatBridge aligns each modality with language using two-modality pairs, then instruction-finetunes on MULTIS, a dataset spanning 16 multimodal task categories.

    Source: §1 · p. 2

  • Results

    ChatBridge achieves strong zero-shot performance across text, image, video, and audio tasks, with a 21.8% accuracy boost in MSVDQA after MULTIS instruction tuning.

    Source: §1 · p. 2, §4.1 · p. 7

  • Takeaways & Limitations

    The authors conclude that language-paired two-modality data can support multimodal correlation and collaboration, including zero-shot perception without explicitly paired training examples.

    Source: §1 · p. 2

    The model has weaknesses in understanding and grounding long-range videos and audios, for which the authors say more precise temporal modeling is needed.

    Source: §5 · p. 9

Abstract

from arXiv · show

Building general-purpose models that can perceive diverse real-world modalities and solve various tasks is an appealing target in artificial intelligence. In this paper, we present ChatBridge, a novel multimodal language model that leverages the expressive capabilities of language as the catalyst to bridge the gap between various modalities. We show that only language-paired two-modality data is sufficient to connect all modalities. ChatBridge leverages recent large language models (LLM) and extends their zero-shot capabilities to incorporate diverse multimodal inputs. ChatBridge undergoes a two-stage training. The first stage aligns each modality with language, which brings emergent multimodal correlation and collaboration abilities. The second stage instruction-finetunes ChatBridge to align it with user intent with our newly proposed multimodal instruction tuning dataset, named MULTIS, which covers a wide range of 16 multimodal tasks of text, image, video, and audio modalities. We show strong quantitative and qualitative results on zero-shot multimodal tasks covering text, image, video, and audio modalities. All codes, data, and models of ChatBridge will be open-sourced.

1 Introduction

ChatBridge uses language-paired two-modality data to connect modalities through an LLM, then instruction-tunes the model with MULTIS. The paper reports strong zero-shot performance across text, image, video, and audio tasks.

1 Introduction

  • Existing multimodal approaches often require paired data for all modality combinations and remain tuned for specific tasks.The introduction cites vision-language, video understanding, and audio-visual speech recognition as areas of prior progress.Source: §1 · p. 1
  • ChatBridge is presented as a unified multimodal model that uses an LLM to interpret, correlate, and reason about modalities through natural-language interaction.The paper describes zero-shot tasks and multi-round dialogues following human instructions.Source: §1 · pp. 1–2
  • ChatBridge uses language as a catalyst, requiring language-paired two-modality data rather than data containing every modality combination.The paper says this supports multimodal correlation and collaboration, including zero-shot perception without explicitly paired training examples such as video-audio-text triples.Source: §1 · p. 2
  • The model aligns each modality with language, then instruction-finetunes on MULTIS, which covers 16 multimodal task categories.The paper describes these as two training stages for aligning modalities and user intent.Source: §1 · p. 2
  • ChatBridge achieves strong zero-shot performance across tasks covering text, image, video, and audio modalities.The authors report quantitative and qualitative evaluations.Source: §1 · p. 2
  • The authors say they will open-source the codebase, MULTIS data, model checkpoint, and a multimodal chat demo.Source: §1 · p. 2

2 Related works

Prior multimodal work either learns joint representations or aligns separate modalities in a shared space; ChatBridge projects modality embeddings into an LLM’s semantic space. Related LLM-based multimodal systems include tool-using agents and end-to-end models.

2.1 Multimodal Learning

  • Multimodal learning has used joint representations or separate modality representations aligned in a shared latent space, with CLIP and ImageBind as examples of alignment approaches.Later methods extended CLIP’s visual-textual alignment paradigm to additional modalities.Source: §2.1 · p. 2
  • ChatBridge projects embeddings from different modalities into the LLM’s semantic space to facilitate modality collaboration within the LLM.The authors characterize this integration as drawing on strengths of both multimodal learning approaches.Source: §2.1 · p. 3

2.2 LLM and Multimodal LLM

  • Instruction tuning has been applied to language models and vision-language models, following the success of ChatGPT as a general-purpose assistant.Source: §2.2 · p. 3
  • LLM-based multimodal systems include tool-using agents that select vision experts and end-to-end models that integrate inputs such as images or sensor features.The cited examples include Visual ChatGPT, MM-REACT, HuggingGPT, Flamingo, BLIP-2, and PaLM-E.Source: §2.2 · p. 3

3 Methods

ChatBridge combines modality-specific encoders, perceivers, and an LLM, and trains the model in two stages. MULTIS supplies task-specific and open-ended chat data spanning multiple modalities.

3 Methods

  • ChatBridge considers image, video, and audio inputs, while leaving modalities such as sketch and point cloud for future work.Source: §3 · p. 3

3.1 Architecture Overview

  • Perceiver modules transform variable-length encoder features into a shared latent space and fixed-shape outputs, while using fewer tokens to reduce LLM computation.The architecture includes modality-specific encoders and perceiver modules with a transformer-decoder-based LLM.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Two-stage Training

  • Training first aligns each modality with language using image-text, video-text, and audio-text pairs, then instruction-finetunes on multimodal tasks to align with user intent.The first stage trains the LLM to produce corresponding text from unimodal embeddings; the second uses multimodal inputs and text responses.Source: §3.2 · pp. 4–5

3.3 Multimodal Instruction Tuning Dataset — MULTIS

  • MULTIS combines standardized task-specific data with open-ended multimodal chat data, covering 16 task categories and 15 source datasets, with six datasets held out for evaluation.Source: §3.3 · p. 5

3.3.1 Task-Specific Data

  • Task-specific MULTIS data reformats public image-text, video-text, and audio-text datasets as instruction-tuning examples using 10–15 templates per task.Templates are manually filtered and refined, with modifiers specifying desired response styles.Source: §3.3.1 · p. 5

3.3.2 Multimodal Chat Data

  • MULTIS chat data adds open-ended image, video, and audio dialogues; its collection pipeline converts video and audio into textual descriptions for prompting ChatGPT or GPT-4.The resulting collection includes 24k conversation, 18K detailed-description, and 9k complex-reasoning samples; MULTIS overall contains 4.4M task-specific and 209k chat samples.Source: §3.3.2 · pp. 5–6

4 Experiments

ChatBridge performs strongly across zero-shot unimodal and multimodal tasks, with both video and audio improving multimodal task and chat performance over single-modality inputs. Instruction tuning with MULTIS improves several downstream metrics, while qualitative evaluations show multimodal reasoning and multi-round instruction following.

4.1 Zero-shot Task Evaluation

  • ChatBridge is comparable to advanced image-based methods on image-text tasks and achieves zero-shot state of the art on Flickr30k and VATEX captioning.The paper reports successful alignment of unimodal inputs with language.Source: §4.1 · p. 6, Table 1 · p. 7
  • Using both video and audio improves performance across all three multimodal tasks, while video-only input performs relatively better than audio-only input.Combining audio and video enhances performance across the tasks to varying extents.Source: §4.1 · pp. 6–7, Table 2 · p. 7
  • With MULTIS instruction tuning, MSVDQA accuracy rises 21.8%, OKVQA improves 3.8%, and GQA improves 3.6%.The paper also reports improvements on captioning tasks in CIDEr score.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Multimodal Chat Evaluation

  • On multimodal chat, GPT-4 scores the model with both audio and video above its unimodal-input versions across all three subtasks.Table 3 reports GPT-4 (text-only) assessment of response quality.Source: §4.2 · p. 8, Table 3 · p. 8
  • Qualitative cases show more comprehensive responses with video and audio together, and multi-round conversations demonstrate instruction following across unimodal and multimodal content.The paper also presents an image-based reasoning comparison with state-of-the-art models.Source: §4.2 · p. 8, Figure 5 · p. 8

5 Discussions

The discussion presents ChatBridge as a language-bridged multimodal model, reports zero-shot results and qualitative examples, and identifies limitations in long-range audio/video understanding and frozen-module performance.

5 Discussions

  • ChatBridge uses language-paired two-modality data and MULTIS to support zero-shot multimodal tasks across text, image, video, and audio.The discussion describes strong quantitative results and examples of multimodal reasoning, knowledge-grounded understanding, and multi-turn conversations.Source: §5 · pp. 8–9
  • Figures 6 and 8 show multi-round conversations with audio or video inputs, while Figure 7 compares responses across modalities for video-audio input.These examples accompany the reported instruction-following and multimodal-comprehension capabilities.Source: §5 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9
  • ChatBridge is weak at understanding and grounding long-range video and audio, and frozen modules may limit performance and introduce pretrained-model biases.The authors identify more precise temporal modeling as a needed improvement and note that the framework could add sketch and point-cloud modalities.Source: §5 · p. 9

A More Details about MULTIS Dataset

The appendix tables describe MULTIS task-specific data, multimodal chat data, and examples of task-specific instruction templates.

A More Details about MULTIS Dataset

  • Table 4 presents the compositions of task-specific data in MULTIS.Source: Table 4 · p. 15
  • Table 5 presents the compositions of multimodal chat data in MULTIS and identifies the LLaVA image-instruction dataset as its image chat data.Source: Table 5 · p. 16
  • Table 6 provides three examples per task of instruction templates, including prompts for image, audio, video, and video with background audio inputs.Source: Table 6 · p. 16

B Example Samples of MULTIS Dataset

The appendix provides example samples from MULTIS for audio, video, and combined video-audio inputs.

B Example Samples of MULTIS Dataset

  • Table 7 shows audio-input samples, with red text highlighting audio-related content in generated data.Source: Table 7 · p. 17
  • Table 8 shows video-input samples, with green text highlighting video-related content in generated data.Source: Table 8 · p. 18
  • Table 9 shows video-audio input samples, with green and red text highlighting video- and audio-related content, respectively.Source: Table 9 · p. 19

C More Case Studies of ChatBridge

The case studies depict ChatBridge handling varied image, audio, video, and cross-modal tasks, including multimodal analysis after first-stage alignment.

C More Case Studies of ChatBridge

  • Image examples cover advice and step-by-step instructions, reasoning and creation, background knowledge, and explaining humor.Another example generates HTML from a hand-drawn draft and revises it based on human opinions.Source: Figure 9 · p. 20, Figure 11 · p. 20, Figure 10 · p. 20
  • Audio examples depict inferring emotions from audio and writing lyrics based on audio.Source: Figure 14 · p. 22, Figure 15 · p. 22
  • Video examples depict informative multi-turn conversation, summarizing and extending video content, and understanding video alongside its background audio.Source: Figure 16 · p. 22, Figure 17 · p. 22, Figure 18 · p. 23
  • A further case depicts understanding and reasoning about independent image and audio inputs.Source: Figure 19 · p. 23
  • After first-stage multimodal alignment, ChatBridge is shown analyzing inputs containing image, video, and audio, and handling multiple images or videos.The examples include writing a story with images and identifying similarities and differences among multiple inputs.Source: Figure 20 · p. 24, Figure 21 · p. 25

Related papers

Loading 2305.16103v1…