PaperlayerView exact revision on arXiv

Source-linked AI summary

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Xi Zhang, Boyu Diao, Zesen Chen, Ruiqi Liu, Hangda Liu, Yongjun Xu

arXiv:2610.13215v1cs.LG

TL;DR

In multimodal continual instruction tuning, learning new tasks without replaying earlier examples risks forgetting previously acquired capabilities. Monarch-MoE replaces dense expert factors with permuted block-diagonal factors and combines them with execution optimizations. Across several continual-learning strategies, experiments report improved first-task retention with fewer adapter parameters.

  • Problem

    Continual instruction tuning must learn evolving task sequences without replaying earlier examples while retaining previously acquired capabilities.

    Source: §1 · p. 1

  • Method

    Monarch-MoE replaces dense low-rank expert factors with block-diagonal factors connected by a fixed permutation, and uses batched computation and optimized operators.

    Source: §1 · p. 2

  • Results

    Across several continual-learning strategies, Monarch improves first-task retention with fewer adapter parameters; on DCL, DISCO + Monarch reduces first-task forgetting from 2.51 to 0.52 pp.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The results support structured expert parameterization as a useful design choice for continual adaptation in the measured configurations, including on a single RTX 4090.

    Source: §1 · p. 2, §5 · p. 10

    Evaluation covers one backbone and fixed task orders, with multi-seed evidence limited to the RS→Med prefix; unmatched effective update scales limit causal attribution.

    Source: §3.4 · p. 5, §5 · p. 10

Abstract

from arXiv · show

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.

1 INTRODUCTION

Multimodal continual instruction tuning must learn evolving task sequences without replaying earlier examples or losing earlier capabilities. Monarch-MoE structures expert updates to limit their direct output spread, and the paper evaluates retention, adapter size, and execution efficiency.

1 INTRODUCTION

  • New-task training examples are unavailable in the stated continual-learning setting, making retention of previously acquired capabilities a central challenge.Updates driven by new data can alter parameters on which earlier tasks depend.Source: §1 · p. 1
  • Although multi-expert LoRA distributes adaptation across experts, maintaining experts and optimizer states can still strain a single GPU.When an expert is reused, changing one entry of a dense LoRA input factor can affect many output coordinates.Source: §1 · pp. 1–2
  • Monarch-MoE replaces dense low-rank expert factors with block-diagonal factors connected by a fixed permutation, localizing pair updates while preserving cross-block connections.Figure 1 illustrates the factorization and channel permutation.Source: §1 · p. 2, Figure 1 · p. 2
  • Across continual-learning strategies, experiments report improved first-task retention with fewer adapter parameters, alongside trade-offs in new-task acquisition.The evaluation also examines capacity controls, repeated runs, learned representations, and single-GPU runtime and memory.Source: §1 · p. 2
  • Batched computation, fused forward operators, and optimized backward and inference execution support fine-tuning and inference on a single 24 GB RTX 4090.The implementation combines compact expert factors with execution optimizations.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses multimodal continual instruction tuning, parameter-efficient adaptation, and efficient structured computation through complementary methods. Monarch-MoE studies structured experts within continual-learning strategies and develops an implementation for their execution.

Multimodal continual instruction tuning

  • MCITlib unifies continual-learning benchmarks and implementations, while the paper follows its no-replay setting and examines expert structure within its methods.CoIN evaluates learning and forgetting across visual instruction-task sequences.Source: §2 · p. 3

Parameter-efficient continual learning

  • Continual parameter-efficient methods use subspace constraints, expert routing, or knowledge organization; Monarch-MoE complements them by restricting block-pair updates within each expert.Its structure adds no cross-task orthogonality penalty, and near-orthogonal learned update subspaces are an empirical observation rather than a guarantee.Source: §2 · p. 3

Structured adaptation and efficient execution

  • Building on Monarch matrices and their rectangular adaptation use, the paper develops multi-expert batched computation, fused forward operators, and an optimized backward pass.Single-RTX-4090 measurements assess memory and runtime in tested configurations.Source: §2 · p. 3

3 MONARCH EXPERTS FOR CONTINUAL ADAPTATION

The method adapts a frozen backbone through expert factors integrated with a host continual-learning method. Monarch experts combine pair-specific low-rank channels with local output support, while retaining cross-block connectivity; the paper also specifies budget and execution choices.

3.1 PROBLEM SETUP AND ADAPTATION FRAMEWORK

  • The adaptation framework learns sequential task datasets without replaying earlier examples, using trainable expert factors and host-method components on a frozen backbone.Expert updates are combined using dispatch weights and output scales; routing follows the standalone or host method.Source: §3.1 · p. 3

3.2 MONARCH-PARAMETERIZED EXPERTS

  • Each input–output block pair receives a rank-at-most-r channel, and block-diagonal factors joined by a fixed permutation let every input block reach every output block.The partition concerns hidden feature coordinates, not task assignments; the permutation adds no trainable parameters.Source: §3.2 · p. 4

3.3 LOCAL UPDATE SUPPORT AND CROSS-TASK INTERFERENCE

  • A pair-specific Monarch update writes directly to one output block, and both factors receive gradients through that block; full steps may update all pairs.Task specialization is not assigned to disjoint blocks, and routing or later layers can introduce further interactions. Analyses include capacity and partition controls, representation CKA, principal angles, and expert utilization.Source: §3.3 · p. 4

3.4 PARAMETER BUDGET AND TRAINING

  • The parameter count is r(mwdo + mhdi), while matching Monarch’s rank bound requires m times as many factor parameters as dense LoRA; budget matching does not ensure matched update scale.The default initialization sets L to zero and R to random; O-LoRA replacements adapt the host’s orthogonality strategy to blockwise overlaps.Source: §3.4 · p. 5

3.5 MEMORY-EFFICIENT TRAINING AND INFERENCE

  • Batched expert computation and a recomputing custom backward reduce intermediate storage, while a fused R→P→L operator handles supported shapes and compilation reduces generation dispatch overhead.Other configurations use a batched path, and sparse top-2 gating need not skip unselected expert arithmetic.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments assess Monarch-MoE’s continual-learning performance, structural explanations, and single-GPU costs across MLLM-DCL and UCIT.

4.1 EXPERIMENTAL SETUP

  • LLaVA-1.5-7B is evaluated on five MLLM-DCL domains and six UCIT tasks without replay; MFT, MFN, MAA, BWT, and first-task forgetting assess learning and retention.MCITlib baselines use published results rather than reruns, and only the standalone DCL pair is parameter matched.Source: §4.1 · pp. 5–6

4.2 CONTINUAL LEARNING PERFORMANCE

  • DCL DISCO + Monarch raises MFN 60.33→63.56 and cuts first-task forgetting 2.51→0.52 pp; UCIT DISCO + Monarch reaches BWT −1.18 and forgetting 0.10, while MFN drops 69.66→67.58.Across the three host replacements on UCIT, BWT and first-task forgetting improve, but MFN decreases.Source: Table 1 · p. 6, §4.2 · p. 6
  • At equal DCL adapter budget, standalone Monarch reduces first-task forgetting from 25.98 to 3.01 pp and raises MFN from 48.03 to 51.21, but lowers MFT by 3.71 points.On UCIT, standalone Monarch improves MFN from 46.06 to 53.16 and BWT from −28.16 to −19.33 with fewer parameters, while first-task forgetting remains 35.50 pp.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 CAPACITY CONTROLS AND ROBUSTNESS

  • DCL controls retain Monarch’s forgetting advantage at the dense baseline’s rank bound, while raising dense LoRA to q = 160 does not recover it.The 44.5M Monarch variant has 3.07 pp first-task forgetting and MFN 51.23; dense LoRA q = 160 has 26.19 pp forgetting and MFN 47.27, though output scales and initialization dynamics are not fully matched.Source: §4.3 · p. 7, Table 2 · p. 7
  • Across three RS→Med seeds, first-task forgetting is 1.12 ± 0.13 pp for Monarch versus 32.47 ± 1.43 pp for LoRA; initialization and scale sweeps also show sensitivity.The seed study covers a two-task prefix, not repeated five-task sequences. For the 44.5M Monarch, SVD-Low improves MFN from 51.23 to 52.05 and forgetting from 3.07 to 1.58 pp, while SVD-High gives MFN 24.96 and forgetting 20.47 pp; UCIT DISCO + Monarch MFN is 67.34–67.81 over α ∈ {192, 256, 384, 512}, but deteriorates at 16 and 1024.Source: §4.3 · p. 7

4.4 STRUCTURAL AND INTERPRETABILITY ANALYSES

  • In the one-epoch block-count probe, finer partitions coincide with lower forgetting (3.29 to 1.74 pp) and lower immediate Med accuracy (52.91 to 50.48); directional probes are consistent with output-local updates but confound scale.The fixed factor budget does not fix the rank bound as block count changes.Source: §4.4 · p. 8, Table 3 · p. 8
  • Monarch has larger principal angles across adjacent-task update subspaces and higher layer-16 CKA after RS→Med (0.934 vs 0.338); expert-selection shares are near-uniform for both methods.These analyses do not measure raw task gradients or guarantee zero interference; the CKA comparison uses one layer and 256 RS examples, and aggregate routing does not establish identical token-level routing.Source: §4.4 · p. 8

4.5 SINGLE-GPU TRAINING AND INFERENCE

  • On one RTX 4090, equal-budget Monarch training reaches 1.606 versus 1.344 samples/s with lower peak memory; a 44.5M Monarch generation run reaches 5.42 versus 5.90 samples/s for 164.5M LoRA.These are short probes, not full-sequence timing, and the generation comparison does not isolate individual execution optimizations.Source: §4.5 · pp. 8–9

5 CONCLUSION AND LIMITATIONS

The conclusion presents Monarch-MoE as a structured expert design for limiting the direct spread of block-pair updates while connecting all input and output groups. Its scope remains bounded by trade-offs, one backbone, fixed task orders, and limited repeated-run evidence.

5 CONCLUSION AND LIMITATIONS

  • Block-diagonal factors and a fixed permutation combine local output support with connections between all input and output groups.The paper presents this structure as limiting the direct spread of individual block-pair updates.Source: §5 · p. 10
  • The gains involve method-dependent acquisition–retention trade-offs and do not guarantee higher final performance.Evaluation covers one backbone and fixed task orders, with multi-seed evidence limited to the RS→Med prefix.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

  • Reproducibility details distinguish configuration-based parameter estimates from checkpoint counts and do not imply every checkpoint or diagnostic log is included.Appendices B.1–G provide configuration distinctions, metric formulas, and stagewise results.Source: REPRODUCIBILITY STATEMENT · p. 11

AI USE STATEMENT

  • The authors report AI assistance across drafting, literature retrieval, analysis, derivations, and preparation of tables, figures, and LaTeX source.They state that revision generated no new training results and that authors are responsible for verifying claims, references, and artifacts.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The study evaluates established benchmarks and does not establish safety, fairness, or suitability for medical or autonomous-driving deployment.The authors note that the method inherits backbone and dataset limitations and that task scores do not resolve misuse risks.Source: ETHICS STATEMENT · p. 11

A.1 PARAMETER COUNT AND ATTAINABLE RANK

  • For a square symmetric partition, the attainable rank is ρ when dimensional conditions hold; those conditions reduce to m2r ≤ d.The rank is bounded by min(do, di, ρ); the stated full-rank construction attains ρ under its dimensional conditions.Source: §A.1 · p. 13
  • At factor budget 2dmr, square LoRA has rank at most mr while Monarch can attain m2r, but their matrix families impose different constraints.Each Monarch block pair still has rank at most r; the comparison is not a containment or universal-approximation claim.Source: §A.1 · p. 13

A.2 JACOBIAN SUPPORT AND THE PERMUTATION

  • Local support applies to one block pair, not a full expert or end-to-end network; gate derivatives and later dense layers can add coupling.A forgetting bound would require assumptions about task gradients, routing, feature distributions, and layer propagation not established here.Source: §A.2 · p. 13

A.3 BLOCKWISE SVD INITIALIZATION

This appendix describes blockwise SVD initialization and distinguishes configuration budgets, training records, host-specific regularization, and comparison limits. It cautions that initialization reconstruction does not ensure the routed network reproduces the pretrained function.

A.3 BLOCKWISE SVD INITIALIZATION

  • The SVD reconstruction identity does not guarantee exact reproduction of the pretrained function when sparse gate coefficients differ from its reconstruction coefficients.The experiment tests the resulting initialization together with its routing interaction.Source: §A.3 · p. 13
  • Table 4 uses standalone K = 4, m = 4, r = 1, α = 1 configurations at the same 44.5M adapter budget.The optional initialization assigns disjoint groups of r singular components to experts; SVD-High selects the largest Kr and SVD-Low the smallest.Source: §A.3 · p. 13, Table 4 · p. 14

B.1 CONFIGURATION DISTINCTIONS AND PARAMETER ACCOUNTING

  • Main configurations distinguish per-expert LoRA rank q from symmetric Monarch (m, r), and label parameter counts as checkpoint-derived C or configuration estimates E.Parameter counts exclude the frozen backbone and other non-adapter tensors, so they are not total saved-checkpoint sizes.Source: Table 5 · p. 14, §B.1 · p. 14
  • Native baseline budgets divide total rank across task experts; parameter estimates vary by method and are not equal implementation-level peak-memory measurements.The passage reports different factor estimates for O-LoRA, CL-MoE, MoELoRA, and DISCO, and omits a uniform-rank estimate for HiDe-LLaVA.Source: §B.1 · p. 14
  • Training records specify different DCL and UCIT schedules and batch configurations, but do not uniquely identify every reported run.Efficiency measurements use a separate single-GPU protocol rather than full-sequence timing.Source: §B.1 · pp. 14–15
  • The O-LoRA replacement adapts orthogonality regularization to blockwise R-factor overlaps; this host-specific penalty is distinct from local support, which follows from factorization.Available records do not establish run-specific regularization coefficients.Source: §B.1 · p. 15
  • Parameter- or rank-matching alone does not remove optimization differences, and the available records lack an independent validation-set protocol for the scale sweep.The DCL standalone configurations have different implemented scales, while UCIT LoRA and Monarch share nominal scale 2 but have different factor counts and rank bounds.Source: §B.1 · p. 15

B.2 METRIC DEFINITIONS AND ROUNDING

The section specifies aggregate-score interpretation and rounding practice, while distinguishing immediate learning from forward transfer. It also frames final scores as the sum of immediate performance and subsequent change.

B.2 METRIC DEFINITIONS AND ROUNDING

  • MFT measures immediate learning, not zero-shot forward transfer; negative F1 can reflect improvement over a poorly learned first checkpoint.The four aggregate metrics and F1 capture different aspects of the task sequence.Source: §B.2 · p. 15
  • Displayed aggregates are recomputed from archived two-decimal task scores and rounded half up to two decimals.For DCL O-LoRA, the archived matrix gives BWT −12.03 while the MCITlib v3 summary reports −12.53; the manuscript uses the matrix calculation and discloses the discrepancy.Source: §B.2 · p. 15
  • Each final score equals its immediate score plus its subsequent change.The section identifies this as an exact identity from the metric definitions.Source: §B.2 · p. 15

T BWT. (9)

Acquisition and subsequent change can pull final performance in opposite directions: a retention improvement may or may not offset lower immediate learning. The supplied tables identify the scope of seed and probe comparisons.

T BWT. (9)

  • For DCL DISCO, BWT improvement contributes about 4.13 points against a 0.89-point MFT decrease; for UCIT DISCO, 5.23 points does not offset a 7.30-point MFT decrease.Thus improved retention can coexist with lower final performance, and first-task forgetting alone does not describe the trade-off.Source: T BWT. (9) · p. 15
  • Table 6 reports first-task forgetting across three seeds on the RS→Med prefix, not repeated full DCL sequences.Both models use the 164.5M-parameter standalone configurations; the table reports mean ± sample standard deviation (n = 3).Source: Table 6 · p. 16
  • Table 7’s upper section compares output-only and input-only partitions; its lower section is a symmetric r = 1 sweep with increasing parameter count, not a same-budget experiment.The (8, 8, 1) row is the same run as Table 3, not an independent replicate.Source: Table 7 · p. 16

C REPEATABILITY AND STRUCTURAL ABLATIONS

The three-seed RS-to-Med check reports lower forgetting for Monarch in every seed, while the structural ablations indicate that partition comparisons do not isolate the effect of output localization.

C.1 THREE-SEED RS-TO-MED CHECK

  • Monarch has lower forgetting in all three seeds, with a mean gap of 31.35 pp; the check repeats only the two-task protocol, not the full sequence.Seed 42 is the prefix of the main run, not an independent repetition; seeds 1234 and 2026 repeat the protocol.Source: §C.1 · p. 16

C.2 BLOCK COUNT AND PARTITION DIRECTION

  • Lower forgetting in the output-only setting is compatible with localized output support, but scale and allocation changes prevent isolating its causal effect.A decisive control would match these quantities and repeat the runs across seeds.Source: §C.2 · p. 16
  • The reported block-count comparison is a fixed-budget sweep varying rank bound m2r, not a fixed-rank sweep.The asymmetric and symmetric sweeps likewise do not simultaneously hold rank and parameter count fixed.Source: §C.2 · p. 16

D REPRESENTATION AND ROUTING DIAGNOSTICS

The diagnostics compare representations, routing, and scale settings across continual-learning runs. They report descriptive differences alongside explicit limits on what those summaries establish.

D REPRESENTATION AND ROUTING DIAGNOSTICS

  • CKA values across five checkpoints are LoRA (1, .3382, .1851, .3559, .2472) and Monarch (1, .9339, .5675, .4062, .3530).The analysis uses 256 RS test examples selected with seed 0, at layer 16; no repeated-sample uncertainty is available.Source: D REPRESENTATION AND ROUTING DIAGNOSTICS · p. 16
  • The archived overall principal-angle summaries are 47.7° for LoRA and 82.8° for Monarch, but they are descriptive rather than independently reproduced statistics.Large angles describe learned update subspaces; they do not establish disjoint task information or zero interference.Source: D REPRESENTATION AND ROUTING DIAGNOSTICS · pp. 16–17
  • Taskwise router-selection shares are close to uniform, but some layer–expert slots remain unused; the analysis checks gross load imbalance only.No hypothesis test or token-level routing-equivalence comparison was retained.Source: D REPRESENTATION AND ROUTING DIAGNOSTICS · p. 17, Table 8 · p. 17

E.1 COMPLETE SCALE SWEEPS

  • For DCL O-LoRA α = 64, large positive BWT or negative first-task forgetting can coexist with weaker immediate learning; UCIT DISCO plateaus at α = 192–512 and deteriorates at 1024.The complete scale runs are reported without evidence of independent validation-based selection.Source: §E.1 · p. 17

E.2 SINGLE-GPU EXECUTION MEASUREMENTS

The execution measurements cover training probes, autoregressive generation, and memory-related calculations on a single GPU. Their results are specific to the tested configurations and do not isolate individual optimizations.

E.2 SINGLE-GPU EXECUTION MEASUREMENTS

  • Training throughput gain is 19.5% at one operating point, with no controlled ablation of batched execution, fusion, or custom backward.The probe has no retained repetitions or error bars; the table distinguishes forward probes from autoregressive generation.Source: §E.2 · p. 17, Table 11 · p. 18
  • The compact Monarch configuration completes on one GPU while the compared local dense path is recorded as OOM_or_crash; this does not establish that all native CL-MoE or O-LoRA configurations exceed 24 GB.The detailed stderr/CUDA traceback is unavailable.Source: §E.2 · pp. 17–18
  • The smaller Monarch has 8.1% lower throughput and 230 MiB lower peak memory in the archived generation benchmark.The benchmark uses model-level compilation for both models and about three generated tokens per example; it is distinct from the forward-probe timings.Source: §E.2 · p. 18, Table 13 · p. 18
  • The batched training path recomputes intermediate channels during backward, while the fused forward path is shape-dependent and trades arithmetic for reduced intermediate memory traffic.The m = 4, r = 1 configuration uses the batched/compiled path; the record does not establish a universal fused-kernel speedup.Source: §E.2 · p. 19
  • Reducing factors from roughly 400M to 100M yields an illustrative 2.4 decimal GB saving in two FP32 Adam moment arrays alone.This calculation excludes other memory components and is not measured end-to-end GPU memory.Source: §E.2 · p. 19

F EVIDENCE PROVENANCE AND SCOPE

  • The paper consolidates archived experiments without new training runs, and adapter-count estimates require checkpoint verification before being treated as measured counts.DCL counts come from checkpoint-inventory summaries; UCIT counts use six-expert configuration snapshots.Source: F EVIDENCE PROVENANCE AND SCOPE · p. 19

G STAGEWISE TASK SCORES

The stagewise task-score matrices organize results by training stage and task order. Their retained captions identify the layout, but the supplied passages do not provide score values for comparison.

DCL TASK-SCORE MATRICES

DCL TASK-SCORE MATRICES (CONTINUED)

UCIT TASK-SCORE MATRICES

DISCO 192

  • For the UCIT task-score matrices, columns follow the task order in Appendix B.1, while rows correspond to successive training stages.Source: DISCO 192 · p. 25
Loading 2610.13215v1…