PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Instruction Tuning with Conditional Mixture of LoRA

Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang

arXiv:2402.15896v2cs.CV

TL;DR

Shared low-rank adaptation parameters can interfere across diverse multimodal instruction tasks, a concern for parameter-efficient tuning. MixLoRA dynamically selects low-rank factors for each input, and experiments report better performance than LoRA at equal ranks and competitive or superior performance against higher-rank LoRA.

  • Problem

    Conventional LoRA uses shared adaptation parameters across diverse multimodal instruction tasks, while task interference in parameter-efficient multimodal tuning remains insufficiently investigated.

    Source: §1 · p. 1

  • Method

    MixLoRA dynamically selects factors to construct input-specific low-rank adaptation matrices, with Conditional Factor Selection refining B based on the selected factors for A.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Across MME and seven other multimodal tasks, MixLoRA consistently beats equal-rank LoRA; rank-2 MixLoRA with 16 factors exceeds rank-32 LoRA by 1.7% in MME and 1.6% on average across other evaluations.

    Source: §6 · p. 6

  • Takeaways & Limitations

    The results support dynamic factor selection as an approach that performs effectively on unseen multimodal tasks and mitigates task interference within the evaluated setting.

    Source: §7 · p. 9

    The study covers image and text modalities, uses a scaled-down Vision-Flan dataset, and incurs extra training overhead versus same-rank LoRA.

    Source: §8 · p. 9

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pre-trained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameter-efficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks.

1 Introduction

The introduction frames task interference as an under-explored challenge in parameter-efficient multimodal instruction tuning and presents MixLoRA, which dynamically constructs adaptation matrices for each input. It reports that MixLoRA outperforms LoRA across unseen multimodal tasks at equal or higher ranks.

1 Introduction

  • Multimodal instruction tuning fine-tunes pretrained models on diverse instruction-based tasks and has demonstrated efficacy for zero-shot generalization to unseen multimodal problems.MLLMs process and integrate information across modalities, notably text and image.Source: §1 · p. 1
  • Shared LoRA parameters must accommodate diverse tasks, creating a task-interference concern that remains insufficiently investigated in parameter-efficient multimodal instruction tuning.The introduction gives OCR and domain-specific classification as an example of tasks whose updates may conflict.Source: §1 · p. 1
  • MixLoRA uses two Independent Factor Selection routers and a Conditional Factor Selection router to construct input-tailored A and B matrices with coordinated factor choices.The Conditional Factor Selection router refines the selection for B based on the factors chosen for A.Source: §1 · p. 2, Figure 1 · p. 2
  • MixLoRA outperforms LoRA across unseen multimodal tasks at equal or higher ranks, according to the introduction’s summary of the experiments.The paper reports evaluations on MME and seven additional multimodal evaluation datasets.Source: §1 · p. 2

2 Related Work

Related work describes multimodal instruction tuning and parameter-efficient adaptation, then identifies task interference in multimodal instruction tuning as under-explored. The paper investigates that issue in parameter-efficient multimodal tuning.

Multimodal Instruction tuning

  • Instruction tuning uses natural-language instructions to improve large language models’ generalization to unseen tasks, and multimodal instruction tuning extends this approach to multimodal and vision tasks.Vision-Flan is described as a human-annotated visual instruction-tuning dataset covering 187 tasks.Source: §2 · pp. 2–3

Parameter-efficient fine-tuning (PEFT)

  • PEFT adapts pretrained models with minimal parameter adjustments, commonly freezing most parameters; LoRA is described as offering competitive performance–efficiency trade-offs.PEFT approaches may share adaptation parameters across tasks or train task-specific adapters.Source: §2 · p. 3

Task Interference

  • Task interference can arise when simultaneous multi-task training causes performance decline through conflicting gradients, but its presence in multimodal instruction tuning remains under-explored.The paper focuses its investigation on parameter-efficient multimodal instruction tuning.Source: §2 · p. 3

3 Task Interference in Multimodal Instruction Tuning with LoRA

The section describes LoRA’s low-rank weight updates and investigates task interference in multimodal instruction tuning through gradient-direction analysis. The analysis reports both positive and negative interference across tasks and layers.

3.1 Background: Low-Rank Adaptation

  • LoRA updates a pretrained Transformer layer with trainable low-rank matrices A and B, where their product BA forms the weight adjustment.The rank r is much smaller than the input and output dimensions, and α controls the adjustment’s influence.Source: §3.1 · p. 3

3.2 Investigating Task Interference in Multimodal Instruction Tuning

  • The method estimates how task j’s loss gradient changes task i’s loss while updating shared parameters, then quantifies the interference between the task pair.The task-pair loss-change estimate follows Zhu et al. (2022).Source: §3.2 · p. 3
  • Positive interference scores indicate aligned gradient directions, whereas negative scores indicate divergent directions and an adverse effect of task j on task i.The heatmap places task j in columns and task i in rows, using red for positive and blue for negative impact.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • The analysis uses six Vision-Flan tasks spanning reasoning, perception, knowledge-intensive, OCR, and domain-specific capabilities.It computes average interference based on gradients for LoRA A and B across layers, with Figure 2 showing fifth- and twenty-fifth-layer MLP and self-attention results.Source: §3.2 · p. 4, Figure 2 · p. 4
  • PACS negatively affects COCO with a −7.3 interference score at the fifth layer for LoRA A, while PACS positively affects ST-VQA at that layer for LoRA B.The examples illustrate both negative and positive task effects in the reported interference analysis.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The observed positive and negative task-pair scores reveal notable task interference in parameter-efficient multimodal instruction tuning and motivate adaptation methods for diverse tasks.The paper emphasizes the need for robust and versatile performance across diverse multimodal tasks.Source: §3.2 · p. 4

4 Conditional Mixture-of-LoRA

MixLoRA treats low-rank factors as experts and dynamically selects them to build input-specific adaptation matrices. Its selection process uses independent routers for A and B, plus a conditional router that lets A’s selected factors inform B’s selection.

4 Conditional Mixture-of-LoRA

  • MixLoRA represents weight adjustments with low-rank factors and selects factors from an expanded pool to construct input-specific adaptation matrices.The selected factors assemble the LoRA A and B matrices.Source: §4 · pp. 4–5

4.1 Dynamic Factor Selection

  • The Dynamic Factor Selection module uses independent factor-selection routers for A and B, with an additional conditional router to refine B’s selection.The conditional selection uses factors chosen for A to inform the selection for B.Source: §4.1 · p. 5

4.1.1 Independent Factor Selection

  • The IFS routers select r factors for A and B separately using instance-based routing, which is more memory-efficient than input-token-based routing.They average preceding-layer hidden states across the sequence, then apply a dense layer, softmax, and top-k selection.Source: §4.1.1 · p. 5

4.1.2 Conditional Factor Selection

  • CFS conditions B’s factor selection on factors chosen for A, combining independent and conditional router outputs before selecting the final r factors.The conditional router maps each selected A factor into the expert dimension; the paper proposes this to leverage a hypothesized interdependence between A and B selections.Source: §4.1.2 · pp. 5–6

4.1.3 Reconstruction of Dynamic Adaptation Matrices

  • The selected factors assemble the final A and B matrices, and each forward pass computes the weight adjustment as ∆W = BA.The reconstruction uses r selected factors for each matrix.Source: §4.1.3 · p. 6

5 Experimental Methodology

The experiments tune on a scaled-down Vision-Flan dataset and evaluate across MME and seven additional multimodal datasets. MME scores aggregate perception and cognition metrics, while other dataset predictions are scored for accuracy against ground truth.

5.1 Datasets

  • Instruction tuning uses 182,167 Vision-Flan instances, capped at 1,000 instances per task across 187 diverse tasks.Vision-Flan is human-annotated and used in scaled-down form to minimize computational cost.Source: §5.1 · p. 6
  • Evaluation covers MME’s 14 perception and cognition subtasks plus seven datasets spanning OCR, reasoning, perception, visual entailment, and object hallucination.The additional datasets are Text-VQA, VSR, CIFAR-10/100, MNIST, SNLI-VE, and POPE.Source: §5.1 · p. 6

5.2 Evaluation Metrics

  • MME scores aggregate perception and cognition metrics; other multimodal datasets use Vicuna 1.5 13B to assess prediction accuracy against ground-truth outputs.Source: §5.2 · p. 6

6 Results and Discussion

MixLoRA generally outperforms LoRA across multimodal evaluations, including at higher LoRA ranks, while analyses report reduced task interference and adaptive factor selection for unseen tasks.

Increase the Number of Rank

  • Increasing rank from 2 to 4 improves performance, while further increasing it to 8 yields diminishing returns.With E=16, the reported gains are 1.8% in MME and 3.1% in MMAvg; with E=32, they are 3.5% in MME and 0.6% in MMAvg.Source: §6 · p. 6

Increasing the Number of Factors

  • Holding rank constant, increasing the number of factors generally improves MixLoRA performance.The authors attribute this trend to greater capacity to provide factors tailored to multimodal tasks.Source: §6 · p. 6

Comparison with LoRA

  • 1.7% in MME and 1.6% on average across other multimodal evaluations: MixLoRA (r=2, E=16) outperforms LoRA (r=32).Table 1 covers zero-shot evaluation; MMAvg averages seven multimodal datasets, excluding MME.Source: §6 · p. 6, Table 1 · p. 7

The Effect of Routing Strategies

  • Instance-based routing significantly outperforms task-specific routing on MME and MMAvg, and both routing strategies surpass random routing.Table 2 compares routing strategies and reports MMAvg across seven multimodal datasets.Source: §6 · p. 7, Table 2 · p. 7

Impact of Conditional Factor Selection

  • Conditional Factor Selection generally improves averaged performance across seven multimodal datasets and different factor and rank settings.The authors hypothesize that CFS strengthens interdependency between LoRA A and B factor selections.Source: §6 · p. 7, Figure 4 · p. 7

Factor Selection Pattern

  • Instances from identical tasks tend to cluster in the factor-selection visualization, consistent with instance-based routing assigning diverse factor sets across tasks.Figure 5 visualizes MixLoRA factor selection with E=32 and r=8.Source: §6 · p. 8, Figure 5 · p. 8

Factor Selection Pattern on Unseen Tasks

  • For five matched seen and unseen task pairs, MixLoRA activates analogous factors, suggesting it adapts selection on unseen tasks based on similar training tasks.The pairs cover visual entailment, OCR, reasoning, hallucination detection, and perception.Source: §6 · p. 8, Figure 6 · p. 8

Analysis of Task Interference

  • MixLoRA shows less performance degradation than conventional LoRA relative to task-specialized LoRA, and lower negative interference scores than LoRA (r=16).The interference-score comparison uses MixLoRA (E=16, r=4); scores are aggregated across adaptation layers.Source: §6 · pp. 8–9, Figure 7 · p. 8, Table 3 · p. 9

7 Conclusion

Conditional Mixture-of-LoRA (MixLoRA) dynamically constructs low-rank adaptation matrices for individual inputs to mitigate task interference. Experiments across multimodal datasets report enhanced performance on unseen tasks compared with conventional LoRA.

  • MixLoRA dynamically constructs low-rank adaptation matrices specific to individual inputs to mitigate task interference during parameter-efficient multimodal instruction tuning.Source: §7 · p. 9
  • Experiments across a variety of multimodal datasets demonstrate MixLoRA’s efficacy.Source: §7 · p. 9
  • MixLoRA demonstrates enhanced performance on unseen multimodal tasks compared with conventional LoRA.Source: §7 · p. 9

8 Limitations

The study focuses on image and text modalities, uses a scaled-down version of Vision-Flan, and reports extra training overhead compared with same-rank standard LoRA.

  • The study covers image and text modalities; integrating sound and 3D point clouds is left for future work.Source: §8 · p. 9
  • Experiments used a scaled-down version of Vision-Flan because training large models is costly.Source: §8 · p. 9
  • MixLoRA introduces extra training overhead compared with standard LoRA of the same rank.Source: §8 · p. 9

A Task-Specific Routing

Task-Specific Routing uses multimodal task characteristics to select decomposition factors, drawing on detailed task definitions. A pre-trained LLM encodes the task definition.

B Implementation Details

The experiments fine-tune stage-one LLaVA with Vicuna-7B v1.3 on scaled-down Vision-Flan for three epochs. MixLoRA is inserted into all linear layers in the Transformer, while the remaining LLaVA parameters stay frozen.

C Evaluation Metrics

The evaluation uses Vicuna 1.5 13B to judge predictions on unseen multimodal datasets against ground-truth outputs, considering task instructions. Because all tasks are classification tasks, accuracy is computed from Vicuna’s judgments.

  • Vicuna 1.5 13B judges prediction accuracy on unseen multimodal datasets using task instructions and ground-truth target outputs.Source: C Evaluation Metrics · p. 12
  • The prompt asks whether each prediction is correct given the question and answer, with instance-specific placeholders, and requires a Yes-or-No response.Source: C Evaluation Metrics · p. 12
  • Accuracy is computed from Vicuna’s judgments because all evaluated tasks are classification tasks.Source: C Evaluation Metrics · p. 12
Loading 2402.15896v2…