PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, Davide Modolo

arXiv:2609.16875v1cs.CV

TL;DR

Upgrading embedding models can make new query embeddings incompatible with existing databases, requiring costly backfilling. MKP-Adapter trains an adapter without updating the backbone and preserves embedding structure with multi-level losses and focal reweighting. It reports strong compatibility across modalities and model-upgrade scenarios, with negligible inference overhead.

  • Problem

    Independently trained embedding models produce incompatible embeddings, so using a new model with an existing database can require costly backfilling; prior BCT approaches may require backbone updates.

    Source: §1 · pp. 1–2

  • Method

    MKP-Adapter trains only an adapter on pre-extracted embeddings, combining a base projection loss with cycle, distance, and angle preservation losses and focal reweighting.

    Source: §1 · pp. 2–3

  • Results

    Across diverse modalities and upgrade scenarios, MKP-Adapter consistently improves over the old model and baseline adapters; average gains are +5.43% on I2T, +7.63% on T2I, and +10.53% on MMEB.

    Source: §1 · p. 3, §5.2 · p. 12

  • Takeaways & Limitations

    The approach achieves backward compatibility across diverse modalities with negligible inference latency and efficient training on pre-extracted embeddings.

    Source: §1 · p. 3, §6 · p. 15

    Training assumes embeddings from both models are already pre-extracted, so it does not include encoder forward passes during adapter training.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Upgrading embedding models typically requires expensive database re-indexing, as new query embeddings are incompatible with existing database embeddings. While Backward Compatible Training (BCT) mitigates this by enforcing compatibility during training, existing approaches often require updating the backbone model. This is impractical because of significant training cost, the risk of performance regression, and limited access to proprietary model weights. We introduce Multi-modal Knowledge Preserving Adapter (MKP-Adapter), the first adapter-only BCT approach for Multi-modal Large Language Models (MLLMs) that requires no backbone updates. We identified that the primary challenge in adapter-only BCT is preserving the knowledge of the new embeddings while enforcing backward compatibility. Hence, we propose a multi-level preservation loss that maintains the geometric structure of the embedding spaces throughout BCT. Furthermore, a focal re-weighting strategy is integrated to prioritize learning from challenging samples. Experiments demonstrate that our method achieves strong backward compatibility across diverse multi-modal benchmarks (image, text, visual document, and video retrieval tasks) and model types. Notably, MKP-Adapter is trained solely on pre-extracted embeddings and requires only negligible additional latency relative to the original backbone forward pass, highlighting its efficiency.

1 Introduction

New embeddings are incompatible with existing database embeddings, forcing costly backfilling, while prior BCT approaches can require retraining or backbone updates. MKP-Adapter trains only an adapter, using multi-level preservation losses and focal reweighting; experiments report compatibility across diverse modalities and upgrade scenarios.

1 Introduction

  • New-model embeddings are not inherently compatible with old database embeddings, forcing expensive backfilling; existing BCT methods impose compatibility during training and may require retraining or backbone updates.XBT uses an adapter followed by LoRA tuning, so it still updates the backbone.Source: §1 · pp. 1–2
  • Training only an adapter on pre-extracted embeddings avoids backbone updates and their repeated passes, preserves the pretrained backbone, and requires no access to model weights.The adapter alone must satisfy compatibility and discriminative-capability objectives while the backbone remains frozen.Source: §1 · p. 2
  • Multi-level preservation combines cycle consistency, pairwise distance, and triplet-angle losses to retain new-embedding structure, while focal reweighting emphasizes harder samples.The preservation losses supplement the base projection loss.Source: §1 · p. 3
  • MKP-Adapter consistently outperforms baseline approaches in backward compatibility across dual-tower-to-MLLM and MLLM-to-MLLM upgrades, spanning image-text, MMEB, document, and video retrieval.The adapter adds 0.01% of the original backbone forward-pass cost and can be trained in 1–2 hours on 558k paired samples using 8 A100 GPUs.Source: §1 · p. 3

2 Related Work

Prior work spans multi-modal embedding models and compatibility strategies. This paper considers MLLM query encoders and proposes adapter-only backward compatibility without backbone modification or weight access.

2.1 Multi-modal embedding models

  • Multi-modal retrieval progressed from dual-tower models to MLLM embedding systems, including contrastively trained, data-curated, and knowledge-distilled variants.The cited examples include E5-V, GME, VLM2Vec, and UniME.Source: §2.1 · p. 4

2.2 Embedding compatibility

  • Forward-compatible methods adapt old database embeddings for a new model, but may depend on auxiliary features or require converting the database at scale.The section also describes online backfilling, unified adapters, and relative-distance methods.Source: §2.2 · p. 4
  • Backward-compatible training avoids database updates but can reduce discriminative capability; prior methods impose training-time constraints, while XBT still updates the backbone through LoRA.MKP-Adapter instead operates on pre-extracted embeddings without backbone modification or weight access.Source: §2.2 · pp. 4–5

3 Problem Settings

The problem setting maps new-model embeddings into the old embedding space using frozen models and pre-extracted, L2-normalized embeddings. The base adapter learns this mapping with an embedding-distance objective, shared across modalities.

3.1 Problem formulation

  • The formulation learns a mapping from the new embedding space to the old one, with both pretrained embedding models frozen and dataset embeddings pre-extracted and L2-normalized.The dataset covers images, text, visual documents, and videos; training requires no additional forward passes through either embedding model.Source: §3.1 · p. 5

3.2 Base adapter

  • The base adapter maps new embeddings to old embeddings by minimizing mean squared error between each projected embedding and its old embedding.The formulation allows alternative distance metrics such as mean absolute error.Source: §3.2 · p. 5
  • Unlike forward-compatible mapping, which transforms old embeddings to new, this backward-compatible adapter projects new embeddings into the old space using one shared mapping across modalities.The training set aggregates pre-extracted embeddings from different modalities into unified mini-batches.Source: §3.2 · p. 5

4 Method

MKP-Adapter combines projection into the old embedding space with regularization that preserves the new embeddings’ semantic structure. Its multi-level losses preserve individual embeddings and their relationships, while focal re-weighting emphasizes challenging samples.

4.1 Motivation

  • Naively fitting new embeddings to the weaker, noisier old space can sacrifice their discriminative capacity, motivating regularization that preserves their semantic structure.The method combines this preservation objective with the base projection loss.Source: §4.1 · p. 6

4.2 Multi-level preservation loss

  • Point-wise cycle consistency reconstructs each new embedding after projection, while distance-wise preservation maintains relative similarities between embeddings.The auxiliary adapter g reconstructs embeddings during training; the learned adapter h is used at inference. Distance comparisons use random pairs and nearest neighbors.Source: §4.2 · pp. 6–7
  • Angle-wise preservation adds triplet-level structure by maintaining angles defined by the new embeddings after projection.The loss is self-supervised, using the original new embeddings rather than an external teacher as its target.Source: §4.2 · p. 7

4.3 Focal re-weighting strategy

  • Focal re-weighting scales preservation losses according to their errors, giving harder samples more emphasis as the focusing parameter increases.The same strategy is applied to point-wise, distance-wise, and angle-wise losses; λ controls the trade-off between projection and preservation.Source: §4.3 · p. 8

5 Experimental Results

The experiments evaluate MKP-Adapter across model-upgrade scenarios and multimodal retrieval tasks, then examine its components, preservation behavior, hyperparameter sensitivity, and computational overhead.

5.1 Experimental settings

  • Experiments cover dual-tower→MLLM and MLLM→MLLM upgrades, using image-text, MMEB, visual-document, and video retrieval benchmarks.Document and video results are reported only for CLIP→GME because UniME is unsuitable for those tasks.Source: §5 · p. 8, §5.1 · p. 10

5.2 Main results

  • MKP-Adapter improves over the old model and adapter baselines, averaging +5.43% on I2T, +7.63% on T2I, and +10.53% on MMEB.In CLIP→UniME, MMEB gains average +9.30% over the old model.Source: Figure 2 · p. 10, Table 1 · p. 11, §5.2 · pp. 12–13
  • MKP-Adapter gains +17.05% on visual documents and +4.80% on videos.The document benchmark uses nDCG@5; video benchmarks use Recall@1.Source: Table 3 · p. 12, §5.2 · p. 13

5.3 Ablation studies

  • Removing any preservation loss or focal re-weighting degrades performance; distance loss particularly affects I2T/T2I, while other components more strongly affect MMEB.Focal re-weighting improves performance across loss combinations.Source: Table 4 · p. 13, §5.3 · p. 14

5.4 Analyses

  • Projected-space retrieval stays within 0.2 points of the new model; performance peaks around λ ∈[0.5, 10], while λ > 100 degrades results.Increasing training data from 558k to 4M pairs provides only marginal gains.Source: Table 5 · p. 13, Figure 3 · p. 13, §5.4 · p. 14
  • The adapter adds 0.25 ms per batch, or 0.01% latency relative to the backbone forward pass, on one A100 with batch size 32.Training takes less than 10 minutes per epoch on a 1M-sample dataset; 10 epochs achieve comparable performance, enabling training in 1–2 hours on 8 A100 GPUs.Source: §5.4 · p. 15

6 Conclusion

MKP-Adapter combines an adapter-only approach with multi-level preservation losses and focal reweighting to retain new-embedding representation quality while achieving backward compatibility.

6 Conclusion

  • Experiments across diverse modalities demonstrate strong backward compatibility, with negligible latency and efficient training.The approach uses multi-level preservation losses with focal reweighting to retain the representational quality of new embeddings after projection.Source: §6 · p. 15

Supplementary Material

The supplementary material details global and local preservation losses, error-based focal weighting, and the embedding and model configurations used in experiments.

A.1 Details on preservation loss

  • Distance-wise and angle-wise preservation losses each combine global and local terms; local terms preserve relationships among each anchor’s nearest neighbors in the new embedding space.The distance-wise loss uses m = 100 nearest neighbors, while angle-wise losses sample at most 32 pairs per anchor to control computational cost.Source: §A.1 · p. 19

A.2 Details on focal re-weighting strategy

  • Focal weights for distance-wise and angle-wise losses are based on their respective preservation errors, with each error scaled by its average before applying γ.The distance- and angle-wise formulas weight individual errors relative to their respective mean errors.Source: §A.2 · p. 20

B.1 Additional implementation details

  • Experiments use CLIP ViT-L/14 with 768-dimensional embeddings; E5-V and UniME share a LLaVA-1.6 (7B) backbone and produce 4096-dimensional embeddings, while GME (7B) uses Qwen2-VL and produces 3584-dimensional embeddings.Implementation uses PyTorch 2.4 and CUDA 12.5.Source: §B.1 · p. 20

B.2 More results on main ablations

The ablations find complementary benefits from the preservation losses and an average performance gain from focal reweighting.

Main ablations

  • Point-wise preservation mainly improves MMEB, distance-wise preservation primarily boosts I2T and T2I, and angle-wise preservation yields more balanced improvements across metrics.Each preservation loss provides a distinct benefit.Source: §B.2 · p. 20, Table S1 · p. 21
  • Combining point-wise, distance-wise, and angle-wise preservation losses consistently achieves the best overall performance among the tested components.The authors describe the three losses as complementary.Source: §B.2 · p. 21, Table S1 · p. 21
  • +2.11% overall improvement is observed on average when focal reweighting is incorporated across preservation-loss variants.The reported average is across various combinations of preservation-loss variants.Source: §B.2 · p. 21, Table S2 · p. 21

B.3 Batch size ablation

Supplementary ablations report stable performance across batch sizes and hyperparameters, marginal gains from larger adapters, and comparisons of adapter designs and composition data.

B.3 Batch size ablation

  • Performance remains stable across batch sizes, with only marginal overall impact from batch size despite a slight benefit to the CL-based method from larger batches.The CL-based method can leverage additional negative samples in contrastive learning.Source: §B.3 · p. 21, Table S3 · p. 22

B.4 Adapter size ablation

  • Increasing hidden size yields only marginal gains, suggesting the current adapter capacity is sufficient for learning the mapping.The hidden-layer analysis is reported in Table S4.Source: §B.4 · p. 22, Table S4 · p. 22

B.5 Different base adapters

  • Our method consistently achieves superior performance across the tested scenarios, while a unified single adapter performs slightly better than separate modality-specific adapters.L2 loss generally outperforms L1; the CL-based adapter is competitive in one setting but shows uneven I2T and T2I performance across model variations.Source: §B.5 · p. 23, Table S5 · p. 23

B.6 Combination with CL-based adapter [40]

  • The proposed loss design remains effective with a contrastive-learning base adapter, indicating generalizability beyond the L2-based adapter.The CL objective replaces the base adapter loss, and a CL-based variant replaces the point-wise L2 preservation loss.Source: §B.6 · p. 23, Table S6 · p. 23

B.7 Use of composition data

  • Composition data improves L2- and EC-style adapters on MMEB in the E5-V → UniME setting, but MKP-Adapter improves considerably with or without it.For CLIP, fused image–text inputs are approximated during training by summing the corresponding image and text features.Source: §B.7 · p. 24, Table S7 · p. 24, Table S8 · p. 24

B.8 Hyper-parameters

  • Performance remains strong across tested λ values, focal parameter γ values, and preservation-loss weightings, and continues to outperform baselines by a clear margin.Some variation occurs across configurations; the weight tuple (a, b, c) represents weights for point-, distance-, and angle-wise losses.Source: §B.8 · p. 25, Table S9 · p. 25, Table S10 · p. 25

B.9 Exclusion of UniME in Tab. 3.

  • UniME is excluded from visual-document and video comparisons because it does not outperform potential old models there, so it does not form a proper model-update scenario.The authors suggest its LLaVA-NeXT backbone’s lack of explicit video pretraining may contribute, whereas GME uses a backbone pretrained on visual documents and video.Source: §B.9 · p. 26

C Comparison with XBT [14]

In the MLLM setting, XBT’s two-stage adapter-and-backbone approach underperformed even base adapters, while additional LoRA tuning could degrade the new model’s own embeddings. MKP-Adapter’s preprocessing and training are substantially shorter than backbone LoRA training, and 10 epochs often match 50-epoch performance.

C Comparison with XBT [14]

  • XBT first pretrains an adapter on text-only data, then LoRA-tunes the new model’s backbone with the adapter frozen except for LayerNorm parameters.The second stage uses a cross-modal contrastive objective.Source: C Comparison with XBT [14] · p. 26
  • XBT’s best-performing MLLM configurations remained weaker than base adapters, and larger-scale training still fell below old embedding models.Reported results were selected from sweeps of LoRA ranks, α values, and learning rates.Source: C Comparison with XBT [14] · p. 26
  • In [14, Table 1], LoRA-tuning improved NoCaps retrieval by +15.71 in T2I R@1 and +3.93 in I2T R@1, which the authors hypothesize reflects the CLIP-to-CLIP setting.The authors suggest those gains may relate to CLIP’s noisy web-scale pretraining and higher-quality captions.Source: C Comparison with XBT [14] · p. 27
  • For MLLM embeddings, further LoRA tuning failed to improve performance and sometimes degraded it; the authors say substantially more sophisticated data curation and training strategies may be needed to avoid degradation.They report that degradation in the new model’s own space is likely amplified in the projected space, compromising compatibility.Source: C Comparison with XBT [14] · p. 27

D Computational analyses

  • MKP-Adapter preprocessing takes approximately 4–5 hours on 8 A100 GPUs, while adapter training takes approximately 1–2 hours; one LoRA epoch exceeds 24 hours on 8 H100 GPUs for 1M samples.Preprocessing comprises feature extraction and neighbor mining.Source: D Computational analyses · pp. 27–28
  • 10-epoch MKP-Adapter training takes approximately 1–2 hours on a 1M-sample CLIP ViT/L-14 → UniME (7B) dataset and is on par with 50 epochs in most cases.Training for 50 epochs yields slightly better performance.Source: D Computational analyses · p. 28, Table S12 · p. 28

D.1 Additional scaling experiments

Across LLaVA-LCS subsampling scales, a moderate training-set size is sufficient, with performance saturating at around 200k samples.

D.1 Additional scaling experiments

  • Performance saturates at around 200k samples, and a moderate dataset scale is sufficient across the tested 5k–500k LLaVA-LCS subsamples.The reported trend is consistent with Tab. 6.Source: §D.1 · p. 28, Table S13 · p. 28

E.1 Five random seed results

Five-seed results show that the observed improvement is consistent, with standard deviation below 0.2 percentage points for average I2T/T2I R@1. Additional results report consistent R@10 trends and strong performance on the full COCO 2014 validation set.

E.1 Five random seed results

  • Standard deviation of average I2T/T2I R@1 across five seeds is below 0.2 percentage points, indicating the observed improvement is consistent.The evaluation uses the same setup as Tab. 1.Source: §E.1 · p. 28

E.2 Additional R@10 results corresponding to Tab. 1

E.3 Large retrieval evaluation (COCO val entire set)

  • MKP-Adapter demonstrates strong performance on the full COCO 2014 validation set of 40,373 images and 201,865 captions.Images without exactly five corresponding captions were excluded for a fair comparison.Source: §E.3 · p. 29, Table S15 · p. 30

Related papers

Loading 2609.16875v1…