PaperlayerView exact revision on arXiv

Source-linked AI summary

IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities

Bin Wang, Chunyu Xie, Dawei Leng, Yuhui Yin

arXiv:2408.12902v2cs.AIcs.CLcs.LG

TL;DR

Fine-tuning language models for multimodal learning can degrade NLP ability, while prior frozen-language-model approaches have struggled to achieve strong multimodal performance. The paper introduces IAA, which inserts multimodal adaptors into a frozen language model and supports separate multimodal and text-only workflows. IAA achieves strong vision-language benchmark performance, including with smaller datasets, while preserving NLP capabilities, at the cost of additional inference time.

  • Problem

    Multimodal fine-tuning can reduce language-model performance, while existing methods that freeze the language model have not achieved sufficiently strong multimodal capabilities.

    Source: Introduction · pp. 1–2

  • Method

    IAA inserts multimodal adaptors at multiple depths while keeping the language model frozen and provides separate multimodal and text-only workflows.

    Source: Methodology · p. 3, Training Paradigm · p. 4

  • Results

    IAA achieves state-of-the-art or superior performance across multiple vision-language benchmarks, including with smaller datasets, without impairing NLP ability.

    Source: Introduction · p. 2, Experimental Results · p. 6, Conclusion · p. 8

  • Takeaways & Limitations

    Frozen-language-model training can support strong multimodal and visual-grounding capabilities while preserving the original language workflow.

    Source: Conclusion · p. 8

    The additional adaptor parameters reduce inference speed; on MME, the 8-layer structure increases average inference time from 0.103s to 0.124s.

    Source: Limitations · p. 8

Abstract

from arXiv · show

In the field of multimodal large language models (MLLMs), common methods typically involve unfreezing the language model during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their natural language processing (NLP) capabilities. To avoid this performance degradation, a straightforward solution is to freeze the language model while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the language model, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen language model to acquire multimodal capabilities. Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. We conduct extensive experiments to improve the general multimodal capabilities and visual grounding abilities of the MLLM. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks. Code and models are available at https://github.com/360CVGroup/Inner-Adaptor-Architecture.

Introduction

MLLM training commonly improves visual capabilities by fine-tuning the language model, but this can reduce text-only language performance. The paper proposes IAA, which freezes the language model while adding multimodal capacity and reports strong benchmark performance with preserved NLP ability.

  • Fine-tuning the language model for multimodal learning significantly reduces performance on the MMLU and C-Eval text-only benchmarks.The decline is observed for LLaVA-1.5 architectures based on both Qwen2 and Llama3.Source: Figure 1 · p. 1, Introduction · p. 2
  • Training predominantly on image-text data can produce catastrophic forgetting, and adding some text-only data does not prevent reduced language comprehension.Source: Introduction · p. 2
  • IAA freezes the language model and inserts multimodal adaptors to enhance multimodal and visual-grounding capabilities without affecting original language modeling capabilities.Source: Introduction · p. 2
  • IAA achieves competitive or superior performance across MME, MMBench, MMMU, and RefCOCO while working with considerably less aligned data than prior frozen-language-model approaches.Source: Introduction · p. 2

Related Work

LLMs have expanded from text processing toward multimodal interaction, but maintaining language proficiency during visual instruction tuning remains a central challenge. Prior MLLMs use varied vision-language connectors and preservation strategies, motivating IAA.

Large Language Models.

  • Large language models have advanced language understanding and are increasingly being extended toward visual and multimodal interactions.Source: Related Work · p. 2

Multimodal Large Language Models.

  • Most MLLMs combine a visual encoder, vision-language connector, and language model, using components such as Q-Former, Resampler, linear layers, or adaptive connectors.Source: Related Work · p. 2
  • Fine-tuning language models with visual instruction data improves multimodal performance but commonly degrades language proficiency over long-term multimodal training.Source: Related Work · p. 2
  • Existing preservation strategies either continue training the language model or retain more language data at considerable training cost, motivating a different paradigm.Source: Related Work · p. 2

Methodology

IAA provides separate multimodal and text-only workflows that share a frozen language model while using multimodal-specific adaptors. Its final design emerged from structural exploration that removed adaptor-to-adaptor feature propagation.

Overview.

  • IAA supports multimodal and text-only workflows that share the frozen transformer layers, while only the multimodal workflow uses the image encoder, projector, adaptors, embedding layer, and language-model head.Source: Methodology · pp. 2–3, Figure 2 · p. 3

Image Encoder and Projector.

  • The multimodal pipeline uses a CLIP ViT-L/14 image encoder at 336px resolution and a two-layer MLP projector, with Llama3-8B as the base language model.Source: Methodology · p. 3

Inner-Adaptor Architecture.

  • The initial ControlNet-inspired design combines each frozen language-model layer with an insertion layer and zero-initialized gating, but it did not yield satisfactory performance.Insertion layers are transformer decoder layers initialized from corresponding language-model layers.Source: Methodology · p. 3
  • The refined design removes feature propagation between insertion layers so each insertion layer receives the output of its corresponding frozen language-model layer; this significantly improves performance.Source: Methodology · pp. 3–4, Figure 3 · p. 4
  • The final IAA scheme uses adjustable insertion-layer placement plus separately initialized multimodal embedding and language-model-head components while keeping the original text components frozen.Source: Methodology · p. 4
  • After comparing alternative structures and strategies in ablations, the authors select the Figure 3(c) design as IAA.Source: Methodology · p. 4, Figure 3 · p. 4

Experiments

The experiments evaluate the method’s visual understanding across general multimodal and visual-grounding benchmarks, followed by ablation studies.

  • The evaluation covers general multimodal benchmarks, visual-grounding benchmarks, and ablation experiments.Source: Experiments · p. 4

Training Paradigm

IAA trains the projector and inner-adaptor structures while keeping the image encoder and language model frozen. Its training paradigm uses staged pre-training, instruction fine-tuning, and grounding fine-tuning.

Training Paradigm

  • The image encoder and language model remain frozen throughout training, while the projector and inner-adaptor architecture are trained for multimodal capability.This design preserves foundational vision and language knowledge.Source: Training Paradigm · p. 4
  • A dual-stage pre-training procedure first trains a high-quality projector, then adds the inner-adaptor with a lower learning rate of 2e-5.The first stage uses a learning rate of 0.001, while the second stage initializes the projector from stage one.Source: Training Paradigm · pp. 4–5

Implementation details.

  • Across pre-training, the method uses 658K image-text pairs, including translated Chinese data, and the full four-stage process runs on A800×8 in 48 hours.The implementation uses zero technology and flash-attention v2.Source: Training Paradigm · p. 5

Training Paradigm

  • Instruction fine-tuning freezes the language model and image encoder while using 865K examples spanning multimodal, document, spatial, science, and Chinese data.The dataset combines 665K instruction samples with DocVQA, VSR, ScienceQA, an in-house dataset, and translated Chinese data.Source: Training Paradigm · p. 5
  • Grounding fine-tuning uses approximately 2M instances from RefCOCO, COCO, Flickr30k Entities, and Objects365 to improve fine-grained and multi-target localization.This stage follows instruction fine-tuning and specializes the model in visual grounding.Source: Training Paradigm · p. 5

Experimental Results

IAA is evaluated on general multimodal and visual grounding benchmarks under zero-shot comparison protocols, alongside deployment memory analysis. It delivers strong multimodal and grounding performance while preserving text-only language ability and using about 30GB of memory in its 8-layer configuration.

Main Results on General Multimodal Benchmarks.

  • IAA-8 and IAA-14 achieve competitive or superior multimodal results against frozen-language-model methods, while IAA-8 outperforms LLaVA-Llama3 at the same 1.2M-data scale.IAA-14 with 14 insertion layers performs better than IAA-8 with 8 layers.Source: Experimental Results · p. 6
  • IAA preserves NLP performance across MMLU and C-Eval, whereas LLaVA-Llama3 trained on the same data shows deterioration on both benchmarks.The text-only workflow also preserves the original language model’s conversational capabilities.Source: Experimental Results · pp. 6–7

Results on Visual Grounding Benchmarks.

  • IAA achieves competitive visual grounding performance on RefCOCO, RefCOCO+, and RefCOCOg despite training on relatively limited datasets.The comparison excludes methods using extremely large datasets or pretrained object detectors.Source: Experimental Results · p. 7, Table 4 · p. 7

Efficiency in Deployment.

  • The 8-layer IAA configuration combines general multimodal and visual grounding capabilities with a deployment memory footprint of about 30GB.This is presented as a reduced footprint relative to concurrently deploying separate high-quality language, multimodal, and grounding models.Source: Experimental Results · p. 7

Ablation Study

The ablations examine architecture, training stages, insertion depth, data scale, and inference cost. They show gains from architectural refinements and complete staged training, while more insertion layers improve metrics at an efficiency cost and freezing the language model adds latency.

Structure Analysis.

  • Architectural refinements consistently improve performance, and adding a specialized multimodal embedding layer and language-model head significantly boosts results.These comparisons are reported in the inner-adaptor structure ablation.Source: Ablation Study · p. 7, Table 5 · p. 8

Comparison of Training Stages.

  • The complete two-stage pre-training procedure outperforms variants that omit either alignment stage, while joint training requires a learning rate near 2e-5 to avoid loss overflow.Skipping the first stage harms performance, and skipping the second prevents replication of the best results.Source: Ablation Study · p. 7, Table 6 · p. 8

Impact of Insertion Layer Quantities.

  • Increasing insertion layers from 8 to 14 improves all performance metrics but reduces efficiency, making 8 layers adequate for foundational requirements.The paper presents 8 layers as the recommended balance for basic needs.Source: Ablation Study · p. 7, Table 7 · p. 8

Training Data Influence Assessment.

  • Adding 0.3M mostly Chinese training examples substantially improves all metrics, especially on the Chinese benchmark MMBench-CNT.The comparison uses a LLaVA-Llama3 baseline trained on 1.2M examples.Source: Ablation Study · p. 8

Limitations

  • Freezing the language model introduces additional parameters and raises average inference time from 0.103s to 0.124s for the 8-layer structure, despite key-value-cache extensions.The authors consider this latency increase relatively reasonable.Source: Limitations · p. 8

Conclusion

The Inner-Adaptor Architecture is introduced to improve multimodal and visual grounding capabilities while keeping the language model frozen. It achieves strong multimodal performance and preserves NLP proficiency through dual workflows.

  • IAA is designed to enhance general multimodal and visual grounding capabilities through architectural exploration with a frozen language model.Source: Conclusion · p. 8
  • IAA can surpass models with fine-tuned language models in multimodal performance.Source: Conclusion · p. 8
  • IAA achieves state-of-the-art performance across multiple public evaluation datasets while preserving the language model’s NLP proficiency through dual workflows.Source: Conclusion · p. 8

The Details of Dateset

IAA is trained through pre-training, instruction fine-tuning, and grounding fine-tuning stages using progressively specialized image-text and visual-grounding datasets. The stages combine translated Chinese data with publicly available and in-house resources.

The Details of Dateset

Pre-training

  • Pre-training uses 658K cumulative image-text pairs, including 558K from LLaVA, 100K translated ALLaVA pairs, and Chinese-task data.Source: Pre-training · p. 11

Instruction Fine-tuning

  • Instruction fine-tuning uses 865K samples while freezing both the language model and image encoder, combining LLaVA, DocVQA, VSR, ScienceQA, in-house, and translated ALLaVA data.Source: Instruction Fine-tuning · p. 11

Grounding Fine-tuning

  • Grounding fine-tuning uses approximately 2M instances from RefCOCO, COCO, Flickr30k Entities, and Objects365 to improve fine-grained and multi-target localization.Source: Grounding Fine-tuning · p. 11

Supplementary Display

The supplementary displays illustrate IAA’s multimodal understanding and visual grounding abilities. They cover image comprehension, knowledge answering, text recognition, writing, multi-object detection, and fine-grained perception.

Multimodal Capability.

Grounding Capability.

Related papers

Loading 2408.12902v2…