Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

1 Introduction

• The authors identify negative transfer during LVLM instruction tuning, where conflicting task losses can produce sub-optimal performance.Multi-task instruction tuning is needed for generalization to novel unseen instructions, but can fail to improve zero-shot tasks for certain model sizes.Source: §1 · pp. 1–2
• Task experts outperform full-data instruction tuning on five of seven downstream tasks, supporting positive transfer among similar tasks and harm from dissimilar ones.The cited tasks include Flickr30k, GQA, HM, SciQA, and IconQA.Source: Figure 1 · p. 1, §1 · p. 2
• Manually partitioned task experts require human expertise and make expert selection for unseen tasks difficult, limiting scalability and generalization.Some new tasks may benefit from multiple training tasks.Source: §1 · p. 2
• MoCLE automatically clusters training instructions, routes inputs to cluster-specific experts, and combines them with a universal expert trained on all tasks.The design aims to balance specialization and generalization while mitigating conflicts between less similar tasks.Source: §1 · p. 2
• MoCLE achieves reported performance gains over dense models and other MoE baselines on InstructBLIP and LLaVA-1.5.The paper also reports gains on held-in and held-out tasks.Source: §1 · p. 2
• The paper frames its contribution as identifying task conflicts and proposing a parameter-efficient MoCLE framework for vision-language instruction tuning.The framework is intended to retain benefits of large-scale data training while mitigating task conflicts.Source: §1 · p. 2

2.1 Multi-Task Instruction Tuning

• Instruction tuning across tasks and templates aims to teach task intentions and generalize to novel instructions, but can fail to outperform untuned models on unseen tasks for certain model sizes.The cited explanation is full capacity utilization for learning task mixtures.Source: §2.1 · p. 2

2.2 Mixture of Experts (MoE)

• MoE increases model capacity through parameter expansion, while adapter-based work explores combining pretrained adapters and parameter-efficient capacity expansion.The passage cites few-shot and zero-shot adapter applications.Source: §2.2 · p. 2
• Prior approaches do not explicitly incorporate task or domain priors in routing, while concurrent domain-informed routing work does not address zero-shot generalization to unseen tasks.The cited concurrent studies are Liu et al. (2024) and Li et al. (2024).Source: §2.2 · p. 3

3.1 Problem Formulation

• The formulation fine-tunes on held-in datasets and evaluates zero-shot on held-out datasets, with templates wrapping text inputs into instructions.Instructions pair template-wrapped text with image inputs.Source: §3.1 · p. 3

3.2 Clustering Data by Instructions

• A pretrained sentence encoder and k-means cluster instruction representations, grouping similar tasks to limit conflicts and enable automatic assignment of novel tasks.The clustering iteratively assigns instructions to the nearest centroid and updates centroids from assigned representations.Source: §3.2 · p. 3

3.3 Mixture of Cluster-Conditional LoRA Experts

• Cluster-conditioned gates route each input to LoRA experts, whose weighted outputs combine with the pretrained LVLM layer; cluster embeddings are initialized from centroids.The gate is learned at each layer where a MoE block is inserted.Source: §3.3 · pp. 3–4

3.4 Universal Expert

• Because task experts see fewer instructions and can hurt generalization, MoCLE adds a universal expert trained on all data and activated for every input alongside a task expert.Load balancing is omitted during training because it might distort task specialization.Source: §3.4 · p. 4

4.3 Evaluation Results

• MoCLE improves InstructBLIP over the original LVLM on held-out and held-in tasks, and reduces LLaVA-1.5’s medical-task gap while outperforming single-task models on MME, MMB, and GeoQA.InstructBLIP held-out absolute gains are 2.9% on IconQA, 3.9% on VSR, 3.2% on TextVQA, and 1.9% on ScienceQA; LLaVA-1.5 gains on natural-image and geometric tasks are reported relative to single-task models.Source: §4.3 · p. 5, Table 2 · p. 5, Table 3 · p. 5

4.4.1 Effects of Different Components

• Removing the universal expert sharply drops VSR and TextVQA performance, while omitting Cluster MoE drops SQA, IQA, and VQAT; raising LoRA rank from 8 to 64 improves average performance by only 0.7%.The universal expert is necessary to maintain generalization ability. Promoting task specialization via clustering achieves notable improvement with ×4 additional parameters, versus ×8 for increasing rank to 64.Source: §4.4.1 · p. 6

4.4.2 Universal Expert vs. Top-2 Experts

• Top-2 gating performs worse than MoCLE and even its variant without the universal expert, consistent with shared task experts needing to learn common features.The universal expert is shared across inputs, freeing task experts from that role and alleviating conflicts.Source: §4.4.2 · p. 6, Table 7 · p. 6, Table 6 · p. 6

4.4.3 Gating Strategies

• Token-MoLE and Sentence-MoLE perform worse on evaluation tasks, while dataset-level routing is less flexible than cluster routing because one dataset may contain tasks assigned to different clusters.For example, llava_150k includes reasoning, conversation, and captioning tasks assigned to different clusters; Figure 3 reports average performance across six evaluation tasks for the gating ablation.Source: §4.4.3 · pp. 6–7, Figure 3 · p. 7

4.4.4 Number of Clusters

• With K=4, MoCLE performs poorly; increasing K to 16 or 64 improves performance, while K=256 can route paraphrased instructions to different experts, so 64 is the default.The number of clusters controls the granularity of task specialization.Source: §4.4.4 · p. 7, Figure 3 · p. 7

4.4.5 Temperature; 4.4.6 Number of Task Experts

• Temperatures 0.05 and 0.1 balance specialization and generalization, with 0.05 used by default; increasing task experts from 2 to 4 improves performance, while 8 behaves similarly to 4.A temperature that is too small (0.01) or too large (0.2) leads to inferior results.Source: §4.4.5 · p. 7, §4.4.6 · p. 7, Figure 3 · p. 7

4.5.1 Clustering

• Instruction embeddings group samples from the same task and place similar tasks close together; k-means clusters also group similar task subsets, including heterogeneous tasks within llava_150k.The t-SNE visualization shows captioning datasets coco_cap and textcaps close together; the 64-cluster assignment groups llava_150k reasoning, conversation, and detail subsets with question-answering tasks.Source: §4.5.1 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8

4.5.2 Routing Results

• MoCLE routes VQA and VQG datasets to different experts, whereas Sentence-MoLE routing shows little correlation between datasets and experts.Figure 6 visualizes routing for one LoRA mixture and includes held-out tasks.Source: §4.5.2 · p. 8, Figure 6 · p. 8

5 Conclusions

• MoCLE activates task-customized model parameters based on instruction clusters to address task conflicts in vision-language instruction tuning.The paper describes it as a novel mixture-of-experts architecture.Source: §5 · p. 8
• A separate universal expert is used alongside task experts to achieve task specialization and generalization simultaneously.Source: §5 · p. 8
• Evaluations on held-out and held-in tasks show the effectiveness of MoCLE.Source: §5 · p. 8

6 Limitations

• The paper's focus is task conflicts among text-based conversation tasks.Source: §6 · p. 9
• Support for more complicated visual perception tasks remains outside the stated focus and is described as appealing.Source: §6 · p. 9
• The paper notes that visual perception tasks have shown more severe task conflicts with conversation tasks.Source: §6 · p. 9

A.1 InstructBLIP

• InstructBLIP models were trained for up to 60K steps with batch size 128 and AdamW using β1=0.9, β2=0.999, and weight decay 0.05.The learning rate warms up linearly for the first 1,000 steps, from 10−8 to 10−5, before cosine decay.Source: §A.1 · p. 13

A.2 LLaVA-1.5

• LLaVA-1.5 training used 1 epoch for LLaVA-150K, 2 for Geo170K, and 9 for Med. Mix.Source: §A.2 · p. 13
• For LLaVA-1.5 training on all datasets, each dataset was copied according to its independent-training epoch count before merging.Each job used batch size 128, weight decay 0, and learning rate 1e−4 with warmup over the initial 3% of steps followed by cosine decay.Source: §A.2 · p. 13

B Weights of the Universal Experts

• More task-specific training data tend to receive higher task-expert weights and lower universal-expert weights, while more general data receive larger universal-expert weights.The paper attributes good generalization in MoCLE to complementarity between the task experts and universal expert.Source: B Weights of the Universal Experts · p. 13
• ocr_vqa has the lowest average universal-expert activation weight, consistent with its book-title and author questions having little overlap with other datasets.Figure 7's colors distinguish datasets.Source: B Weights of the Universal Experts · p. 13, Figure 7 · p. 13
• ok_vqa, ok_vqg, aok_vqa, aok_vqg, llava_*, and coco_vqa receive much higher universal-expert weights, consistent with VQA abilities being fundamental in LVLM.Source: B Weights of the Universal Experts · p. 13

• Training and evaluation data are listed in Table 8.Source: C Data · p. 13

C.1 Dataset Abbreviation

• InstructBLIP uses 13 datasets, with abbreviations and manually defined task categories; LLaVA-150K is used without task templates.Source: §C.1 · p. 13, Table 9 · p. 14

D Task Templates

• InstructBLIP uses the same task templates as Dai et al. (2023) for instruction tuning and held-in/out evaluation.Training and evaluation templates are shown in Tables 10 and 11.Source: D Task Templates · p. 13

E Case Studies

• The case studies examine object counting, OCR, image introduction, and sample instructions assigned to different clusters.Source: E Case Studies · p. 13

E.1 Conversations

• MoCLE gives a more appropriate response than InstructBLIP on difficult object counting and OCR examples.The examples are shown in Tables 12 and 13.Source: §E.1 · p. 14, Table 12 · p. 16, Table 13 · p. 16
• MoCLE handles a celebrity-introduction request and a request for detailed image description better than InstructBLIP in the reported examples.The authors attribute the introduction example to routing the query to a conversation-specialized expert; examples appear in Tables 14 and 15.Source: §E.1 · p. 14, Table 14 · p. 16, Table 15 · p. 16

E.2 Sample Instructions in Clusters

• Four sampled clusters contain VQA-related instructions focused on food, pets, people, and counting, supporting the authors' use of many instruction clusters.Sampled instructions are shown in Table 16.Source: §E.2 · p. 14, Table 16 · p. 17

F Future Works

• The study focuses on conflicts among text-based conversation tasks; support for more complex visual perception tasks such as object detection remains a future direction.The authors also identify visual pre-training and diffusion models as potential applications.Source: F Future Works · p. 14