PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

Runze Liu, Naibin Gu, Mingxu Ai, Yuqing Li, Peng Fu, Zheng Lin, Weiping Wang

arXiv:2608.27867v1cs.AI

TL;DR

Continual multimodal instruction tuning must add new abilities while preserving prior knowledge, yet repeatedly expanding complete LoRA-MoE experts is costly. CoRe-MoE reuses SVD-derived direction bases and trains compact coordinates with task-specific routers, achieving up to 5.90 points higher final average performance while using less than 1% of sequential LoRA’s later-task trainable parameters.

  • Problem

    Continual multimodal instruction tuning must acquire heterogeneous new task abilities while preserving previously learned knowledge, but continual LoRA-MoE expansion introduces parameter overhead.

  • Method

    CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank and trains later tasks with compact coordinate experts and task-specific low-rank routers.

  • Results

    CoRe-MoE improves final average performance by up to 5.90 points and uses less than 1% of the trainable parameters required by sequential LoRA for later tasks.

  • Takeaways & Limitations

    Later tasks can exploit previously discovered LoRA direction bases instead of repeatedly learning complete LoRA experts.

  • Takeaways & Limitations

    CoRe-MoE depends on initial-task bases, and its scope is currently mainly UCIT experiments with two 7B-level backbones.

Abstract

from arXiv · show

Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based capacity, but repeatedly learning and maintaining full LoRA experts leads to substantial parameter overhead. This raises a natural question: is full expert expansion necessary for every new task? To answer it, we analyze the SVD of task-specific LoRA updates and observe substantial overlap in their input- and output-side LoRA direction subspaces, with task-specific adaptation largely captured by lightweight coordinates over these subspaces. Motivated by this observation, we propose CoRe-MoE, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning. CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank, and for subsequent tasks trains only compact coordinate experts together with task-specific low-rank routers. Experiments on two representative MLLMs show that CoRe-MoE improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks. The code is publicly available at https://github.com/runzezz/CoRe-MoE.

1 Introduction

Continual multimodal instruction tuning must add abilities sequentially while preserving prior knowledge, but expanding complete LoRA-MoE experts creates overhead. CoRe-MoE reuses initial direction subspaces and trains compact later-task coordinates, improving performance with far fewer parameters.

  • Continual multimodal instruction tuning requires learning heterogeneous new tasks while avoiding catastrophic forgetting of previously learned knowledge.
  • Full LoRA-MoE expert expansion adds parameter overhead as task sequences grow and may entangle task-specific knowledge in shared experts and routers.
  • SVD analysis finds substantial overlap between later-task and initial-task input- and output-side LoRA direction subspaces.
  • CoRe-MoE extracts reusable direction bases from an initial expert bank, then trains compact task-coordinate matrices and task-specific low-rank routers for later tasks.
  • 5.90 points: CoRe-MoE improves final average performance by up to 5.90 points while using less than 1% of sequential LoRA’s later-task trainable parameters.

2 Related Work

Related work covers continual multimodal instruction tuning, LoRA-MoE expert routing, and compact representations of LoRA updates. These studies motivate examining reusable structure inside task-specific adaptations.

  • Continual Instruction Tuning for Multimodal Large Language Models: Continual multimodal instruction tuning extends continual learning to multimodal instruction-following models that must acquire abilities sequentially while retaining prior knowledge.
  • LoRA-MoE and Compact LoRA Representations: LoRA injects low-rank trainable updates into frozen weights, while LoRA-MoE methods use multiple expert branches and routing to increase adaptation capacity.
  • LoRA-MoE and Compact LoRA Representations: Compact LoRA research decomposes or reallocates update structure through rank importance, subspaces, shared bases, and compact coordinate matrices.

3 Pilot Study

The pilot study compares LoRA updates through invariant SVD-derived direction subspaces rather than raw factor coordinates. It finds cross-task overlap and shows that frozen bases with target-task coordinates can support later adaptation.

  • Subspace Comparison: Raw LoRA factors are not uniquely identifiable because invertible transformations can change their internal bottleneck coordinates without changing the update.
  • Subspace Comparison: SVD provides invariant bases: U_B spans the output-side direction subspace, while V_A spans the input-side direction subspace.
  • Subspace Comparison: The factor-side view represents a LoRA update as a transformation from an input-side subspace to an output-side subspace through task-dependent coordinates.
  • Cross-Task Overlap: High SAR across layers and modules indicates that ArxivQA and ImageNet-R share both input-side V_A and output-side U_B direction subspaces.
  • Frozen-Basis Adaptation: Frozen U_B/V_A bases with target-task coordinates remain close to full LoRA on ArxivQA and CLEVR-Math, whereas mismatched Flickr30k coordinates lower performance.

4 Methodology

CoRe-MoE retains LoRA-MoE’s expert routing while replacing repeated full-expert learning with reusable direction bases and compact task-specific coordinates. Subsequent tasks train compact coordinate experts and low-rank routers, with prototype-guided task selection and compact token-level routing.

  • Compact Coordinate Conversion: CoRe-MoE warms up a vanilla LoRA-MoE expert bank on the initial task, then converts its experts into reusable direction bases and coordinate matrices.The conversion separates reusable basis structures from task-specific compact coordinates.
  • Compact Coordinate Training: Subsequent tasks freeze the learned direction bases and train only task-specific compact coordinate matrices instead of complete LoRA experts.This preserves expert-style adaptation while reducing repeated expert expansion.
  • Compact Coordinate Training: Compact coordinate matrices reduce trainable expert parameters from O(E(dout + din)r) for full LoRA experts to O((Er)2).The compact representation retains the expert-style structure of LoRA-MoE.
  • Low-Rank Routing: Later-task routers project hidden states into the reusable input-side space before computing expert logits, aligning routing with compact coordinate experts.Routing in the projected space further reduces trainable and stored parameter overhead.
  • Task-Aware Routing: Visual and textual prototypes select the task-related router and compact coordinate expert when task identity is unavailable during inference.Within the selected task, token-level routing operates in its compact coordinate space.

5 Experiments

Experiments on UCIT evaluate CoRe-MoE across two 7B multimodal backbones, comparing continual performance, forgetting, ablations, routing, efficiency, and task-order robustness. CoRe-MoE delivers strong results while substantially reducing later-task trainable parameters.

  • Main results: CoRe-MoE achieves 71.20% Avg. and 71.20% Last on LLaVA-1.5-7B, exceeding second-best results by 3.33 and 5.90 points, respectively.
  • Main results: CoRe-MoE records BWT of −0.02% on LLaVA-1.5-7B and −0.01% on Qwen2-VL-7B, the highest among compared continual methods.
  • Main results: CoRe-MoE obtains 81.26% Avg. and 81.27% Last on Qwen2-VL-7B, exceeding second-best results by 2.32 and 4.58 points.CLEVR-Math is excluded from this five-task sequence.
  • Expert parameterization: Under matched adaptation capacity, CoRe-MoE reaches 71.20 Avg. versus 70.61 for budget-matched LoRA-MoE, while probe-based selection reduces Avg. to 60.16.
  • Dual-modality routing: Combining visual and textual prototypes with α = 0.5 achieves 99.74% routing accuracy, versus 82.52% for text-only and 97.95% for visual-only routing.
  • Parameter efficiency: CoRe-MoE uses about 0.6M trainable parameters per later task, compared with about 60M for LoRA-FT and vanilla LoRA-MoE and over 260M for MoELoRA/CL-MoE.A full-rank router increases this overhead to about 8M.
  • Robustness to task order: Across task orders, CoRe-MoE achieves around 69.8%–69.9% Avg. in three alternative sequences versus 71.20% in the default order.

6 Conclusion

CoRe-MoE identifies cross-task redundancy in LoRA direction subspaces and reuses those bases with compact task-specific coordinates. Experiments show improved continual performance and forgetting reduction with much lower later-task parameter costs.

  • LoRA updates from different tasks exhibit high overlap in input- and output-side direction subspaces.
  • CoRe-MoE extracts reusable direction bases from the initial expert bank, then trains coordinate matrices and low-rank routers for later tasks.
  • CoRe-MoE improves continual learning performance and reduces forgetting while greatly lowering trainable parameters for subsequent-task adaptation.

Limitations

The method depends on direction bases extracted from the initial task and uses prototype-based routing, with current validation limited to UCIT and two 7B backbones.

  • CoRe-MoE may provide less suitable later-task bases when the initial task is not representative of subsequent tasks.
  • Prototype-based routing remains to be explored under ambiguous task boundaries or open-world task distributions.
  • Current experiments mainly use UCIT with two 7B-level backbones, leaving longer sequences, larger models, and more diverse scenarios for future evaluation.

Ethics Statement

The work uses public datasets and benchmarks without collecting or processing private or personally identifiable information. It reports evaluation metrics and backward transfer definitions for continual multimodal instruction tuning.

  • Ethics Statement: The study uses publicly available datasets and benchmarks for continual multimodal instruction tuning.
  • Ethics Statement: The authors acknowledge that multimodal models may inherit biases or unintended behaviors and require further evaluation before real-world deployment.
  • Evaluation Protocol: The evaluation uses Accuracy for four tasks and dataset-level Average scores for VizWiz-caption and Flickr30k.
  • Evaluation Protocol: Backward Transfer measures forgetting over historical tasks by comparing scores when tasks were first learned with their scores after the final step.
  • Evaluation Protocol: Higher BWT indicates better knowledge retention or positive backward transfer, whereas smaller values indicate stronger forgetting.

C Hyperparameter Study

The hyperparameter study fixes Task 1 and tunes the learning rate and router load-balancing coefficient for later tasks. The selected setting achieves the best reported continual-learning metrics.

  • C Hyperparameter Study: The study fixes Task 1 trained with vanilla LoRA-MoE, then varies later-task learning rate and auxiliary load-balancing coefficient λaux.λaux controls the strength of the router load-balancing objective.
  • C Hyperparameter Study: λaux = 10^-3 and learning rate 2 × 10^-4 achieve the best Avg. and Last scores and the highest BWT.This configuration is used for the main LLaVA experiments.

D Per-Task Parameter Overhead

CoRe-MoE adds only a small amount of stored parameter overhead as UCIT tasks accumulate, unlike vanilla LoRA-MoE’s rapid growth from complete task-specific experts.

  • D Per-Task Parameter Overhead: CoRe-MoE introduces only a small increase in stored parameters as new UCIT tasks are added.
  • D Per-Task Parameter Overhead: Vanilla LoRA-MoE rapidly accumulates parameters by storing complete task-specific LoRA experts throughout the task sequence.

E CLIP-Based Task Prototype Extraction

CoRe-MoE extracts dual-modality task prototypes and uses weighted similarity for task-aware routing. After six UCIT tasks, routing is nearly diagonal, with most samples assigned correctly.

  • E CLIP-Based Task Prototype Extraction: CoRe-MoE maintains image and text prototypes for each learned task using running-mean representations from training samples.
  • E CLIP-Based Task Prototype Extraction: The image branch projects pooled visual representations into the CLIP embedding space, while the text branch encodes the cleaned user query in the same space.
  • E CLIP-Based Task Prototype Extraction: During evaluation, task identity is predicted from weighted similarity between a test sample and the stored task prototypes.The final image and text prototypes are normalized and saved for inference.
  • E CLIP-Based Task Prototype Extraction: 17,953 of 18,000 samples are assigned to the correct task after all six UCIT tasks are learned.Remaining errors mainly occur among the visually or semantically closer ArxivQA–IconQA and VizWiz–Flickr30k pairs.
Loading 2608.27867v1…