PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

Xiwen Wei, Mustafa Munir, Radu Marculescu

arXiv:2512.03125v1cs.LGcs.AI

TL;DR

UMGMs can forget earlier capabilities while learning new tasks, and inter-modal forgetting remains less explored than within-modality forgetting. The paper introduces MoDE, which decouples modality-specific updates and uses knowledge distillation; experiments report mitigation of both kinds of forgetting and stronger results than prior baselines.

  • Problem

    Continual learning in UMGMs raises an underexplored challenge: forgetting can occur across modalities as well as within them, while prior multimodal continual-learning work focused on understanding tasks.

    Source: §1 · p. 2, §2.3 · p. 4

  • Method

    MoDE uses modality-decoupled LoRA experts and adapters, with knowledge distillation to preserve pretrained image-generation capabilities while addressing both forms of forgetting.

    Source: §1 · pp. 2–3

  • Results

    MoDE achieves the highest accuracy of 33.47% and the lowest FID of 53.74, while experiments report mitigation of both intra- and inter-modal forgetting.

    Source: §5.2 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The reported experiments establish MoDE as a strong solution for continual learning in unified multimodal generation.

    Source: §6 · p. 10

    The study focuses on visual-generation forgetting induced by tuning on understanding tasks; complementary results for understanding forgetting induced by image-generation tuning are in Appendix F.

    Source: §3 · p. 4

Abstract

from arXiv · show

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git

1 Introduction

UMGMs combine multimodal understanding and generation, but continual instruction tuning raises a largely unexplored risk of forgetting across modalities as well as within them. The paper identifies this problem and proposes MoDE to mitigate both forms of forgetting.

1 Introduction

  • Continual instruction tuning of UMGMs can cause forgetting both within a modality and across modalities, a challenge not addressed by prior work focused on understanding tasks.UMGMs can produce textual and visual outputs within a shared backbone, making cross-modal forgetting relevant to this setting.Source: §1 · p. 2
  • Existing methods can mitigate either intra-modal or inter-modal forgetting, but none effectively address both simultaneously across tasks including VQA, classification, visual reasoning, and image generation.Sequentially adding task data and retraining is described as computationally expensive and impractical as new multimodal tasks emerge.Source: §1 · p. 2
  • MoDE combines modality-aware sparse text LoRA experts with an image LoRA adapter, using expert routing and knowledge distillation to address intra- and inter-modal forgetting.The pretrained UMGM parameters remain frozen during continual tuning to preserve cross-modal alignment.Source: §1 · pp. 2–3
  • Experiments show that MoDE mitigates both intra-modal and inter-modal catastrophic forgetting, and outperforms state-of-the-art baselines on both dimensions.The paper identifies modality gradient conflict as an explanation for inter-modal forgetting and reports theoretical analysis of modality decoupling.Source: §1 · p. 3

2 Related work

Related work traces unified understanding and generation models, mixture-of-experts approaches, and continual-learning methods for catastrophic forgetting. It highlights that prior approaches generally do not address inter-modal forgetting in UMGMs.

2.1 Unified multimodal generative models (UMGM)

  • Earlier unified-model designs coupled diffusion with MLLMs but lacked tight integration of image generation and language modeling, while later models unified them through next-token prediction.The passage associates the earlier design with suboptimal instruction-based generation.Source: §2.1 · p. 3
  • Recent UMGMs frame understanding and generation as next-token prediction with an autoregressive transformer backbone, using varied image encoders and decoders.The paper's method is designed for transformer-based UMGMs regardless of encoding or decoding mechanism.Source: §2.1 · p. 3

2.2 Mixture of experts (MoE)

  • Prior MoE-based continual-learning methods commonly share experts across modalities, which can cause modality interference and exacerbate inter-modal forgetting.MoE combines specialized experts with a gating mechanism; prior approaches also use MoE with LoRA for scalable training.Source: §2.2 · p. 3

2.3 Catastrophic Forgetting

  • Existing forgetting methods mainly address same-output-modality tasks, leaving them insufficient for UMGMs that generate both text and images.The surveyed approaches include regularization-, architecture-, replay-, and prompt-based methods; Figure 2 illustrates degradation and misalignment in image generation during VQA-task tuning.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 Inter-modal Catastrophic Forgetting in UMGMs

The paper examines inter-modal forgetting in UMGMs, focusing on visual generation degradation after tuning on multimodal understanding tasks. It attributes this interference to conflicting modality gradients in shared parameters and proposes modality-decoupled adapters to mitigate it.

3 Inter-modal Catastrophic Forgetting in UMGMs

  • The analysis focuses on visual-generation forgetting induced by tuning on multimodal understanding tasks, with complementary reverse-direction results reported in Appendix F.The paper restricts its focus to multimodal tasks.Source: §3 · p. 4
  • In a shared-parameter architecture, a text-task update can interfere with image performance when text and image-generation gradients conflict.The paper defines modality gradient conflict as a negative inner product between the two gradients; under this condition, a sufficiently small-step SGD update on text generation increases visual loss.Source: §3 · p. 5
  • MoDE decouples modality-specific updates and is described as eliminating first-order modality conflict, thereby mitigating inter-modal forgetting.The paper reports a theoretical proof and experimental support for this design claim.Source: §3 · p. 5

4 Methodology

The method formalizes sequential instruction tuning and image generation, then assigns text and image tokens to separate adapters in a shared UMGM. Knowledge distillation anchors visual adaptation to the pretrained model to help preserve image-generation capability.

4.1 Problem formulation

  • The formulation treats instruction tuning as sequential adaptation across S tasks and image generation as autoregressive prediction from prompt tokens.Source: §4.1 · p. 5

4.2 Modality-Decoupled Experts (MoDE)

  • MoDE assigns image tokens to a V-Adapter and text tokens to routed T-MoE experts, separating modality-specific updates within the shared UMGM.The V-Adapter supports visual understanding and image generation, while T-MoE provides task-level flexibility for multimodal understanding.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • MoDE unties interleaved hidden states by modality, applies the corresponding adapter, then restores the outputs to their original sequence positions.Source: §4.2 · p. 7
  • A text-task gradient step updates only T-MoE parameters; MoDE bounds inter-modal interference by O(η²), versus O(η) under the Proposition 1 baseline.The paper links this bound to the minimal gradient conflict and visual-generation retention observed in its experiments.Source: §4.2 · p. 7

4.3 Knowledge Distillation

  • For visual adaptation, reference images from LAION-5B provide tokens for teacher and student next-image-token predictions, whose softened probabilities are KL-matched.The V-Adapter combines cross-entropy and distillation losses, with λ set to 0.3 in the experiments.Source: §4.3 · p. 7

5 Experiments

MoDE is evaluated for continual instruction tuning and image generation, and the reported results show it mitigates both intra- and inter-modal forgetting. Ablations examine its modality-specific adapters and knowledge distillation, with additional results reporting an accuracy–efficiency trade-off.

5.1 Experimental setup

  • The evaluation uses Chameleon and Janus-Pro backbones, five understanding datasets, and CustomConcept101 for image generation, comparing MoDE with continual-learning baselines and joint training.Continual instruction tuning reports average accuracy and average forgetting; image generation uses image alignment, text alignment, and FID.Source: §5.1 · p. 8

5.2 Main results

  • 33.47% accuracy and 53.74 FID are MoDE’s reported best accuracy and lowest FID, respectively, across the compared methods.MoDE’s FID is close to the pretrained model’s 52.13; MoE-LoRA’s FID is 65.16, compared with 56.12 for vanilla SeqLoRA.Source: §5.2 · pp. 8–9
  • MoDE consistently outperforms comparison methods across five continual instruction-tuning datasets and sustains effective learning across tasks.DualPrompt’s low average forgetting of 6.82% accompanies only 31.92% average accuracy and a best ImageNet accuracy of 24.55%.Source: §5.2 · pp. 8–9, Table 2 · p. 9
  • MoDE preserves image-generation capability in qualitative comparisons, including generating yellow leaves around a barn where Model Tailor and CL-MoE miss that detail.The reported evaluation uses Chameleon, with image quality assessed using image alignment, text alignment, and FID.Source: §5.1 · p. 8, §5.2 · p. 9

5.3 Ablation results

  • Ablations indicate that both modality-specific adapters and knowledge distillation are important; using only T-MoE preserves image generation but yields suboptimal visual understanding.The paper also reports favorable accuracy–efficiency trade-offs for MoDE, including training time, memory consumption, and parameter overhead.Source: Table 3 · p. 9, §5.3 · pp. 9–10

6 Conclusion

The paper presents MoDE as a solution to intra- and inter-modal catastrophic forgetting in continual instruction tuning for UMGMs, supported by theoretical analysis and experiments.

6 Conclusion

  • MoDE decouples modality-specific updates to mitigate gradient conflict, and experiments report that it significantly outperforms state-of-the-art baselines for continual learning in unified multimodal generation.The conclusion identifies inter-modal forgetting and attributes it to modality gradient conflict.Source: §6 · p. 10

A Theoretical Analysis

The theoretical analysis contrasts visual-loss drift after text updates in modality-coupled and modality-decoupled architectures. Gradient-distribution results are reported as empirical evidence of conflict in a coupled model and its absence in MoDE.

A Theoretical Analysis

  • MoDE limits worst-case image-generation loss drift to O(η^2), compared with O(η) for a modality-coupled architecture.The analysis considers a text-loss gradient step and its change to visual loss; disjoint text and visual parameter blocks make the first-order term vanish.Source: A Theoretical Analysis · p. 15

B Modality gradient conflict

  • Figure 5 is reported to show significant text–image gradient conflict in modality-coupled MoE LoRA on ScienceQA, while Figure 6 shows zero cosine disagreement in MoDE.The paper describes MoDE’s modality-specific update subspaces as completely isolated, consistent with the theoretical analysis.Source: B Modality gradient conflict · p. 16, Figure 5 · p. 16, Figure 6 · p. 16

C Additional Qualitative Results

The additional qualitative results compare MoDE with baseline methods on Chameleon image generation and report stronger prompt alignment and visual fidelity for MoDE.

C Additional Qualitative Results

  • For the barn-in-fall prompt, only Chameleon and MoDE generate yellow leaves around the barn; the comparison is part of the qualitative results against baseline methods.The table caption identifies Table 4 as the full qualitative comparison.Source: C Additional Qualitative Results · p. 17, Table 4 · p. 17

D Implementation Details

MoDE outperforms baselines under a different task ordering, and four T-MoE experts offer a reported balance between performance and computational cost. The implementation uses two UMGMs and selects λ = 0.3 to balance image generation and visual understanding.

D Implementation Details

  • Experiments use Chameleon and Janus-Pro backbones, with one training epoch per task and LoRA rank 8.Training uses eight RTX 6000 GPUs, bf16 mixed precision, and an effective batch size of 8.Source: D Implementation Details · p. 18
  • The selected λ = 0.3 balances image-generation fidelity and visual-understanding accuracy; larger λ favors generation but constrains beneficial understanding updates.The paper reports that λ weights the V-Adapter knowledge-distillation loss.Source: Table 5 · p. 18, D Implementation Details · p. 18

E.1 Different Tasks Order

  • MoDE consistently outperforms baseline methods under an alphabetically arranged task order, demonstrating robustness to different task orderings.The order is GQA, ImageNet, ScienceQA, TextVQA, and VizWiz.Source: Table 6 · p. 18, §E.1 · p. 18

E.2 Different Number of Experts

  • Four T-MoE experts offer a good trade-off between performance and computational cost, as gains from increasing expert count are modest.Increasing the number of experts slightly improves text alignment and accuracy.Source: §E.2 · p. 19, Table 7 · p. 19

E.3 Ablation on Knowledge Distillation and Modality Decoupling

The ablations report consistently superior performance only when modality decoupling and knowledge distillation are combined. MoDE also reduces forgetting when image-generation training instead threatens multimodal understanding.

E.3 Ablation on Knowledge Distillation and Modality Decoupling

  • Adding knowledge distillation to MoELoRA and CL-MoE mitigates visual-generation forgetting but hurts multimodal understanding performance.Adding it to vanilla LoRA gives modest improvements in both areas; these results come from a single run.Source: Table 8 · p. 19, §E.3 · p. 19
  • Decoupling textual and visual LoRA without knowledge distillation yields only modest gains and is insufficient to maintain rich cross-modal alignment for high-quality generation.The ablation also identifies knowledge distillation as critical to MoDE's performance improvements.Source: Table 9 · p. 19, §E.3 · p. 19
  • Only combining modality decoupling and knowledge distillation leads to consistently superior performance.The paper characterizes MoDE as more than the sum of its parts.Source: §E.3 · p. 19, Table 8 · p. 19, Table 9 · p. 19

F Multimodal Understanding Forgetting

  • MoDE significantly reduces understanding-performance drops when image-generation fine-tuning is applied, extending its reported forgetting mitigation to the reverse fine-tuning direction.The experiment fine-tunes on LAION-5B and uses GQA understanding sequences as reference data.Source: Table 10 · p. 20, F Multimodal Understanding Forgetting · p. 20

G.1 Additional Results with Janus-Pro

MoDE outperforms baseline methods in continual instruction tuning for multimodal understanding with both Janus-Pro-1B and Chameleon backbones. The additional evaluations include grounding datasets and use early stopping.

G.1 Additional Results with Janus-Pro

  • With Janus-Pro-1B, MoDE outperforms baseline methods in continual instruction tuning for multimodal understanding.The additional evaluation includes referring-expression grounding datasets and uses early stopping.Source: Table 11 · p. 20, §G.1 · p. 20

G.2 Additional Results with Chameleon

  • With Chameleon, MoDE outperforms baseline methods in continual instruction tuning for multimodal understanding.The additional evaluation includes referring-expression grounding datasets and uses early stopping.Source: Table 12 · p. 21, §G.2 · p. 21

H Computational Analysis

This section reports MoDE’s computational costs and describes the accompanying baselines, settings, risks, and reproducibility information. It also covers code documentation and states that the research involves neither human subjects nor LLMs as a core method component.

H Computational Analysis

  • MoDE uses 440.3 ms vs. MoELoRA’s 419.2 ms, 84.98 GB vs. 75.27 GB peak GPU memory, and a 0.0211% trainable parameter ratio vs. 0.0171% for MoELoRA and CL-MoE.The passage says TFLOPs remain comparable and attributes MoDE’s gains to modality-decoupled experts and distillation rather than substantially higher resource use.Source: Table 13 · p. 21, H Computational Analysis · p. 21

I.1 Overview of Baselines

  • The baselines include Model Tailor, DualPrompt, MoELoRA [44], CL-MoE, and sequential LoRA.Model Tailor uses a salience-and-sensitivity model patch with Hessian-based compensation; DualPrompt uses shared and task-specific prompts. MoELoRA mixes low-rank experts, CL-MoE combines task- and instance-level routing with momentum updates, and sequential LoRA applies adapters across tasks.Source: §I.1 · p. 22

I.2 Training Settings

  • Baseline hyperparameters follow original codebases, with four experts of rank 8 for MoE-LoRA and CL-MoE, and top-K 2 for CL-MoE.Model Tailor uses 10% sparsity; DualPrompt uses prompt lengths Lg = 5 and Le = 20, injected at specified layers; CL-MoE uses γ = 0.7 and β = 0.5.Source: §I.2 · p. 22

J Ethics Statement

  • The paper states that it uses publicly available datasets under their licensing policies and identifies possible misuse of generated images for disinformation or harmful content.It recommends content filtering, provenance tracking, and visible or invisible watermarking as safeguards.Source: J Ethics Statement · p. 22

NeurIPS Paper Checklist

  • The checklist states that the paper provides theoretical results on modality gradient conflict, with proofs in Appendix A and empirical verification in Appendix B.It also says architecture and experimental details are provided and that code would be released upon acceptance; its open-access answer says code will accompany supplemental materials.Source: NeurIPS Paper Checklist · pp. 24–25
  • The checklist says error bars are not applicable because generative-model evaluation is qualitative or uses metrics not requiring statistical significance testing.It points to the experiment section and supplemental materials for compute-resource information.Source: NeurIPS Paper Checklist · p. 26
  • The paper states that its code is well documented.Source: NeurIPS Paper Checklist · p. 28
  • The paper reports no crowdsourcing or human-subject research, so participant-instruction and IRB disclosures are marked NA.Source: NeurIPS Paper Checklist · p. 29
  • The paper marks LLM-use disclosure NA because LLMs are not an important, original, or non-standard part of its core methods.Source: NeurIPS Paper Checklist · p. 29
Loading 2512.03125v1…