PaperlayerView exact revision on arXiv

Source-linked AI summary

KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls

Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, Qing Li

arXiv:2510.19316v2cs.CL

TL;DR

Large Multimodal Models hold extensive factual knowledge, but it can become outdated, while knowledge injection must also preserve what models already know. KORE pairs structured knowledge augmentation with covariance-based null-space adapter initialization. The authors report superior performance in both adaptation and retention compared with standard fine-tuning and continual-learning methods.

  • Problem

    LMM knowledge is static and can become outdated, while methods that inject new knowledge can struggle to retain prior knowledge.

    Source: §1 · p. 2

  • Method

    KORE expands knowledge items into structured dialogue and instruction data and initializes adapters using weights projected into a covariance matrix's null space.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    KORE exhibits superior performance in both knowledge adaptation and retention compared with standard fine-tuning and continual-learning methods across multiple representative LMMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    KORE addresses the trade-off between knowledge internalization and catastrophic forgetting during LMM knowledge injection.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting.

1. Introduction

LMM knowledge can become outdated, while attempts to add knowledge can forget prior knowledge. KORE combines structured augmentation and covariance-based constraints to balance adaptation and retention.

1. Introduction

  • KORE uses a two-stage process: structured augmentation supports learning new knowledge, while null-space adapter initialization limits interference with previous knowledge.The authors distinguish this structured augmentation from superficial, discrete variations produced by general augmentation techniques.Source: §1 · p. 2
  • KORE outperforms standard fine-tuning and continual-learning methods in both knowledge adaptation and retention across multiple representative LMMs.The authors also report customizable knowledge augmentation and constraints for different retention needs.Source: §1 · p. 2
  • KORE-AUGMENTATION expands individual knowledge items into multi-round dialogues and instruction-following data, and the authors construct the KORE-74K dataset.The dataset is presented as a resource for future knowledge-injection research.Source: §1 · p. 2
  • KORE-CONSTRAINT stores prior knowledge in covariance matrices of linear-layer activations and initializes adapters in their null space to minimize interference.This defines a fine-tuning direction intended to retain previous knowledge.Source: §1 · p. 3

2. Related Works

Prior knowledge-injection work contrasts retrieval at inference with parameter updates, which face challenges in adapting while retaining existing knowledge. Continual-learning approaches span several strategies, while Figure 3 contrasts KORE-AUGMENTATION with general augmentation.

2.1. Knowledge Injection

  • Parameter updates and retrieval-augmented generation are distinct knowledge-injection paradigms; parameter updates can struggle with adaptation and catastrophic forgetting.Retrieval-augmented generation uses an external knowledge base at inference, with effectiveness dependent on retrieval quality and speed.Source: §2.1 · p. 3

2.2. Knowledge Forgetting

  • Continual-learning approaches include parameter regularization, architecture changes, rehearsal, and learnable prompts; Figure 3 compares KORE-AUGMENTATION with general augmentation methods.The caption identifies KORE-AUGMENTATION on the left and general augmentation methods on the right.Source: §2.2 · p. 3, Figure 3 · p. 4

3. Knowledge-Oriented Controls

KORE combines structured, knowledge-oriented augmentation with a covariance-based constraint to support new-knowledge adaptation while limiting interference with prior knowledge. Its constraint analysis compares reconstruction retention and covariance patterns across tasks.

3.1. Knowledge-Oriented Augmentation

  • KORE-AUGMENTATION turns each knowledge item into multi-round dialogue and instruction-task data, building structured knowledge intended to support generalization and internalization.Unlike isolated text or image variations, its dialogue trunk and instruction-task branches form a knowledge tree that supports reasoning about and manipulating learned knowledge.Source: §3.1 · p. 4
  • The augmentation pipeline produces 75,710 dialogue examples and 46,468 VQA samples, alongside visual-recognition and image-caption tasks.GPT-4o generates up to 10 dialogues per knowledge item; image retrieval and CLIP similarity are used for the visual tasks.Source: §3.1 · p. 4

3.2. Knowledge-Oriented Constraint

  • KORE estimates activation covariance from sampled pretrained-knowledge inputs, approximates its null space with the r smallest-singular-value vectors, and initializes LoRA from projected pretrained weights.The constraint targets AC≈0; a residual term keeps the model unchanged at fine-tuning start.Source: §3.2 · pp. 5–6

3.3. Analysis of Knowledge-Oriented Constraint

  • CO-SVD yields superior retention to Plain SVD and ASVD after reconstruction, while covariance matrices show similar outlier patterns for related tasks and different patterns for unrelated tasks.The comparison evaluates MME and ScienceQA; covariance heatmaps show four input activations in the 0-th block.Source: Figure 4 · p. 5, §3.3 · p. 6

4. Experiments

KORE is evaluated for knowledge adaptation, retention, and their combined performance, then examined across knowledge types, model scales, constraints, and design choices. The reported results show gains over baselines in these settings.

4. Experiments

  • Evaluations cover EVOKE image-text knowledge injection, 12 retention benchmarks across seven capability dimensions, and Avg and HARS overall metrics.HARS integrates knowledge adaptation and retention to evaluate their equilibrium; baselines include Full-FT, LoRA, Replay, and continual-learning methods.Source: §4 · p. 6

4.2. Analysis of fine-grained results

  • KORE exceeds LoRA by 6.53 in Avg for fine-grained retention and scores strongly on OCRVQA, MMMU, and HallB.The fine-grained retention comparisons cover the benchmarks named in Table 2; the text also reports close or better performance than Replay with rank 256 and a changed covariance-matrix source.Source: Table 2 · p. 7, §4.2 · p. 7
  • Across 20 fine-grained News and Entity knowledge types, KORE consistently outperforms all baselines.Figure 5 compares KORE with baseline methods on those fine-grained types.Source: Figure 5 · p. 7, §4.2 · p. 7

4.1. Analysis of main results

  • KORE (rank=235) improves over the best baseline by 9.15 in Avg and 15.25 in HARS, while also improving EVOKE CEM by 12.63 and F1-Score by 17.14.For retention, KORE also outperforms Full-FT, LoRA, EWC, LwF, SEFE, and CIA in K.R and achieves top scores on OCR, M-DIS, and HAL.Source: §4.1 · p. 7

4.2. Analysis of fine-grained results

  • Specific knowledge-oriented constraints improve targeted retention, with a reported 7.17 gain on MME.The constraints are constructed by sampling 256 data per benchmark across four dimensions.Source: §4.2 · pp. 7–8, Figure 6 · p. 7

4.3. Analysis of various LMM scales and architectures

  • KORE surpasses LoRA across dimensions on a larger LMM and improves over Replay by 11.10 in Avg and 18.73 in HARS; on Qwen2.5-VL (7B), it improves over Replay by 3.19 in Avg and 36.21 in HARS.The reported comparisons cover different model scales and architectures.Source: Table 3 · p. 8, §4.3 · p. 8

4.4. Analysis of ablation experiments

  • Higher rank improves performance on nearly all evaluations, while rank 64 still exceeds Replay in K.A, Avg, and HARS with less than half its parameters.Ablations report a 21.39 decrease in K.A without augmentation; removing the constraint or frozen matrix A impairs retention.Source: §4.4 · p. 8, Figure 7 · p. 8, Table 4 · p. 8

4.5. Comparison with general augmentation methods

  • KORE-AUGMENTATION beats general augmentation methods across all metrics, improving over the best baseline by 18.53 in K.A, 9.72 in Avg, and 9.61 in HARS.Both KORE-AUGMENTATION and general text augmentation use GPT-4o; the authors attribute the performance gap to augmentation design rather than GPT-4o distillation alone.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

The conclusion presents KORE as a way to structure new knowledge and constrain fine-tuning to reduce interference with prior knowledge. The appendices describe how the authors construct data, evaluate retention, define metrics, and compare baseline methods.

5. Conclusion

  • KORE structures each knowledge item and initializes an adapter in a covariance matrix’s null space to support adaptation while minimizing interference with prior knowledge.The conclusion also describes KORE’s performance as architecture-agnostic and stronger on larger-scale LMMs.Source: §5 · pp. 8–9

A. The Use of Large Language Models in KORE

B.1. Knowledge Adaptation Evaluation

  • Knowledge adaptation evaluation follows EVOKE’s settings, which assess learning evolving knowledge without forgetting original capabilities.The passage characterizes EVOKE as revealing limitations in current adaptation methods and catastrophic forgetting.Source: §B.1 · p. 14

B.2. Knowledge Retention Evaluation

  • Retention is evaluated on 12 benchmarks across seven capability dimensions, including perception, OCR, reasoning, instruction following, dialogue, and hallucination.The described benchmark set also includes mathematical reasoning tasks.Source: §B.2 · p. 14

B.3. Evaluation Protocol

  • CEM checks whether the ground-truth answer is contained in the prediction, while F1 measures word-level overlap as the harmonic mean of precision and recall.The protocol defines CEM and F1 for open-domain question answering.Source: §B.3 · p. 15

B.4. Baseline Methods

  • The baseline section describes continual-learning and parameter-efficient approaches, including EWC and LoRA, and says implementations use open-source or integrated repositories.Other described baselines include LwF, OLoRA, MoELoRA, SEFE, and CIA.Source: §B.4 · p. 15

B.5. Training Parameters about KORE

This section points to the training hyperparameter table and reports the hardware used for training and evaluation.

B.5. Training Parameters about KORE

  • Table 6 lists hyperparameter settings for LLaVA-v1.5 7B and 13B, and Qwen2.5-VL 7B.The passage provides the table’s scope but not its parameter values.Source: Table 6 · p. 16, §B.5 · p. 16

B.6. Experiment Resources about KORE

  • Training used four NVIDIA H100 GPUs with 96 GiB each; evaluation used four NVIDIA A100 PCIe GPUs with 40 GiB each.Source: §B.6 · p. 16

C. Proof of KORE

The proof argues that initializing LoRA’s matrix A in the covariance matrix’s null space constrains updates to reduce interference with prior knowledge. It then shows that layer outputs remain approximately unchanged as matrix B is adjusted.

Theorem C.2

  • Theorem C.2 states that when A lies in the covariance null space, each layer’s output remains approximately equal to its pretrained output regardless of adjustments to B.The proof concludes that fine-tuning therefore minimally alters pretrained knowledge.Source: C. Proof of KORE · pp. 16–17

Theorem C.1

  • With full-rank pretraining weights, Theorem C.1 establishes that A’s column space is contained in the covariance null space.The proof links A’s column space to that of the null-space basis using the SVD construction.Source: C. Proof of KORE · p. 16

D.1. Detailed experimental results for capture knowledge

The detailed results report limited sensitivity to sample count and better performance from task-specific samples when many ranks are discarded. Covariance visualizations show task-related patterns, and a further table reports fine-grained knowledge-type results.

D.1. Detailed experimental results for capture knowledge

  • With 32 samples, performance declines more noticeably than with 256, yet remains significantly above Plain SVD and ASVD; sample count otherwise has limited influence.The passage also compares 512 and 256 samples when the smallest 1536 ranks are discarded, with 512 performing slightly lower.Source: Table 7 · p. 17, §D.1 · p. 18
  • At 1536 discarded ranks, MME samples perform better on MME and ScienceQA samples perform better on ScienceQA.This task-specific pattern supports the claim that covariance matrices capture dataset-specific knowledge and preserve structural features.Source: §D.1 · p. 18

D.2. Covariance Visualization Results

  • The 32×32 covariance heatmaps show similar patterns for POPE and HallusionBench, unlike MMBench, across selected projections in layers 2 and 30.The passage says these patterns empirically support that covariance matrices can characterize the triggered task.Source: Figure 8 · p. 18, Figure 9 · p. 18, §D.2 · pp. 18–19

E.1. More Main Results

E.2. More Results on LMM Scales and Architectures

The appendix evaluates KORE on a larger LLaVA model and a different model architecture. It reports retention advantages across these settings and strong adaptation across fine-grained News and Entity types.

E.2. More Results on LMM Scales and Architectures

E.3. More Results on Specific Knowledge-Oriented Constraints

The appendix examines customized knowledge constraints, rank and design ablations, and comparisons with general augmentation methods. Its results report retention gains alongside adaptation performance across fine-grained knowledge types.

E.3. More Results on Specific Knowledge-Oriented Constraints

  • Specific constraints slightly decrease adaptation efficacy but increase retention on INS and M-IDU, resulting in superior overall performance.Source: Table 12 · p. 20, §E.3 · p. 20
  • Specific constraints improve retention of the corresponding knowledge without compromising retention of other knowledge types.Source: §E.3 · p. 20, Table 13 · p. 21
  • Constrained KORE maintains strong adaptation across fine-grained News and Entity types with negligible performance degradation.Source: Table 14 · p. 21, §E.3 · p. 21

E.4.1. RANK ABLATION EXPERIMENTS

E.4.2. SETTING ABLATION EXPERIMENTS

  • Removing augmentation, constraints, or the frozen matrix lowers overall performance by 11.59, 15.97, and 12.83, respectively, relative to KORE.Source: Table 19 · p. 22, §E.4.2 · p. 22
  • Without constraints, KORE achieves superior adaptation across fine-grained News and Entity types; the reported gains are attributed to structured augmentation.Source: Table 20 · p. 22, §E.4.2 · p. 22

E.5. More Results On Comparison With General Augmentation methods

  • KORE-AUGMENTATION surpasses the best general augmentation method by 10.76 in overall performance on knowledge retention evaluations.Source: §E.5 · p. 22, Table 22 · p. 23
  • KORE-AUGMENTATION achieves the best adaptation performance across all knowledge types compared with general augmentation methods.Source: Table 23 · p. 23, §E.5 · p. 23

F. Convergence comparison of various methods via loss curves.

The convergence comparison reports different fitting and generalization patterns across methods, while the construction appendix describes how the automated KORE-74K pipeline generates varied knowledge tasks and image-based data.

F. Convergence comparison of various methods via loss curves.

  • LoRA, EWC, and Full-FT fit EVOKE at low loss but generalize poorly to new knowledge, whereas KORE converges on KORE-74K and shows strong generalization.O-LoRA and SEFE fail to fit EVOKE; the datasets differ, so the loss curves compare convergence under different training data.Source: F. Convergence comparison of various… · p. 23, Figure 10 · p. 24

G.1. More construction process about KORE-AUGMENTATION

  • It generates up to 10 dialogue questions per instance, selects two CLIP-similar images for recognition and captioning, and builds VQA from question-answer-subject-hypernym quadruplets.Source: §G.1 · pp. 24–25
  • The automated pipeline expands EVOKE's injection data into KORE-74K; only the question templates are manually crafted.Source: §G.1 · p. 25, Figure 11 · p. 25

G.2. More statistical analysis about KORE-AUGMENTATION

The appendices reference KORE-74K statistics, OneVision sampling, News and Entity case studies, and prompts for dialogue, recognition, captioning, and VQA tasks.

G.2. More statistical analysis about KORE-AUGMENTATION

G.3. The process of sampling using the OneVision dataset

  • Figure 12 presents the process for sampling OneVision data.Source: §G.3 · p. 26

H. Case Study

I.1. Prompt details regarding Multi-rounds of Dialogue

  • Dialogue-generation prompts ask for questions based on news, while answer prompts instruct the model to rely on the supplied knowledge.Source: §I.1 · pp. 28–29

Prompts and Templates 2: Visual Recognition QA

I.3. Prompt details regarding Image Caption QA

  • Image-caption prompts request concise summaries of no more than 25 words that preserve the knowledge's meaning, with News and Entity description templates.Source: §I.3 · p. 31

I.4. Prompt details regarding VQA

  • VQA prompts request up to five question sets, each with an answer, subject, and hypernym; the subject must be named in the question and differ from the answer.Source: §I.4 · p. 32

Related papers

Loading 2510.19316v2…