PaperlayerView exact revision on arXiv

Source-linked AI summary

Spherical Interpolation for Backward-Compatible Multimodal Representations

Simone Ricci, Niccolò Biondi, Federico Pernici

arXiv:2609.39836v1cs.CVcs.LG

TL;DR

Independently trained vision-language models can produce incompatible representations, making upgrades difficult when a deployed gallery should remain unchanged. The paper combines orthogonal alignment with query-side spherical interpolation, characterizes when an interior direction can improve angular proximity, and reports improved compatibility across evaluated settings.

  • Problem

    Independently trained models can produce incompatible representations, while recomputing embeddings for a deployed gallery can be prohibitively expensive.

    Source: §1 · pp. 1–2

  • Method

    The method aligns new-model queries into the old-model space and interpolates them with old-model queries along a spherical geodesic using a support-set-selected fixed weight.

    Source: §1 · pp. 2–3

  • Results

    Across evaluated settings, SLERP improves over SVD alignment in nearly all cases and often restores empirical backward-compatibility.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The method provides a compatibility mechanism during migration that can keep the deployed gallery usable while full re-indexing is deferred or performed offline.

    Source: §5 · p. 10

    The interpolation weight must be selected on a support set, inference requires both endpoint embeddings, and performance is bounded by the quality and complementarity of available representations.

    Source: §5 · p. 10

Abstract

from arXiv · show

Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval. A practical challenge arises during model upgrades: independently trained models generally produce incompatible representation spaces, so replacing a deployed model typically requires recomputing embeddings for the entire gallery, which is prohibitively expensive at scale. Orthogonal post-hoc alignment can partially mitigate this problem by mapping new-model queries into the old-model gallery space. However, because independently trained models can differ in fine-grained representation structure, the orthogonal alignment remains approximate, leaving a residual angular discrepancy between the old-model query and the aligned new-model query. We study whether interpolation along the spherical geodesic between these two normalized query representations can improve retrieval without re-indexing the gallery. We characterize when this path contains an interior query direction closer to an idealized retrieval-optimal direction than either endpoint, and connect this characterization to Recall@$K$ through a local margin-based certification result. Experiments across multiple benchmarks and model families show that post-alignment spherical interpolation improves over orthogonal alignment alone, recovering backward-compatibility in most evaluated settings. Consistent with our geometric characterization, per-query oracle analysis shows that retrieval-favorable interior points occur frequently in practice. Code is available at https://github.com/miccunifi/SLERP_backward_compatibility .

1 Introduction

The introduction addresses model upgrades that must preserve an existing retrieval gallery, since independently trained models may have incompatible representations and re-indexing can be prohibitively expensive. It proposes query-side spherical interpolation after orthogonal alignment, with a geometric analysis and experiments showing improved compatibility.

1 Introduction

  • Re-indexing can be computationally prohibitive for large galleries and may be infeasible when original raw data are inaccessible.During progressive upgrades, preserving the existing index can provide compatibility throughout migration.Source: §1 · pp. 1–2
  • Orthogonal alignment preserves inner-product geometry but remains approximate when independently trained models differ in fine-grained representation structure.Residual angular mismatch remains after aligning SigLIP2 to CLIP ViT-B/32.Source: §1 · p. 2, Figure 1 · p. 2
  • The analysis characterizes when the geodesic between old and aligned new queries contains an interior direction closer to an ideal retrieval direction than either endpoint.A margin perturbation bound connects that angular condition to local Recall@K certification.Source: §1 · pp. 2–3
  • Post-alignment SLERP improves over orthogonal alignment and frequently restores compatibility across evaluated model families and retrieval benchmarks.Ablations attribute most mean retrieval improvement to combining the endpoints, while support-set weight selection transfers across datasets without target-test tuning.Source: §1 · p. 3
  • The query-side method combines orthogonal Procrustes alignment with fixed-weight SLERP, without retraining and while leaving the deployed gallery unchanged.The interpolation weight is selected on an alignment support set.Source: §1 · p. 3

2 Related Work

Prior backward-compatible representation work largely addresses unimodal upgrades, while the cited vision-language approach requires a dedicated training pipeline. This paper instead uses post-hoc alignment and query-side interpolation, a distinct use of SLERP from prior applications.

2 Related Work

  • Backward-compatible learning spans several approaches, but most cited work addresses unimodal upgrades; XBT extends it to cross-modal retrieval through a dedicated training pipeline.The pipeline uses text pretraining, Gaussian noise, and LoRA fine-tuning with text and image data.Source: §2 · p. 3
  • The proposed approach instead estimates an orthogonal Procrustes map from a small support set and interpolates queries without retraining or re-indexing.Neither model is modified.Source: §2 · p. 4
  • Unlike prior uses of SLERP for composed image retrieval or policy-weight merging, this work interpolates old-model and aligned new-model queries for backward-compatible retrieval.It also characterizes interior improvement and connects the geometry to Recall@K through margin-based guarantees.Source: §2 · p. 4

3 Geometric Characterization of Spherical Interpolation for Compatible Retrieval

The section analyzes spherical interpolation between old-model and aligned new-model queries against a fixed gallery. It establishes when an interior point improves angular proximity to an idealized retrieval direction and relates that improvement to local Recall@K certification.

3 Geometric Characterization of Spherical Interpolation for Compatible Retrieval

  • The setting keeps the old-model gallery fixed while using an independently trained new model for queries, aligned into the old-model coordinate system by orthogonal Procrustes.The main analysis treats equal-dimensional model pairs; rectangular Procrustes for unequal dimensions is described in an appendix.Source: §3 · p. 4
  • SLERP traverses the minor geodesic between normalized query endpoints at constant angular speed, giving its weight a consistent geometric interpretation across queries.The weight represents normalized angular displacement from the old-model query toward the aligned new-model query.Source: §3 · p. 5
  • An interior query has a strictly smaller angular gap to the retrieval-optimal direction exactly when its normalized in-plane projection lies inside the minor arc; otherwise an endpoint is optimal.When the projection magnitude ρ is zero, the angular gap is constant along the arc.Source: §3 · pp. 5–6, Figure 2 · p. 6
  • A margin perturbation bound specifies when angular proximity to the idealized direction locally certifies success on the discrete top-K retrieval event.Endpoint separation alone does not determine whether an interior angular improvement occurs.Source: §3 · p. 6

4 Experimental Results

The experiments evaluate query-side spherical interpolation after Procrustes alignment with the old-model gallery fixed. CC3M-selected SLERP meets the Recall@1 compatibility criterion in more evaluations than SVD alone, with selected weights transferring across test datasets.

4.1 Experimental Setup

  • The evaluation covers CLIP, SigLIP1, and SigLIP2 models, using 12,637 CC3M image–text pairs for alignment support and testing text-only, image-only, and joint support.The test benchmarks are Flickr30k, COCO 2014, and NoCaps; interpolation weights are selected on CC3M separately for I2T and T2I retrieval.Source: §4.1 · pp. 6–7

4.2 Evaluation Protocol

  • Backward compatibility is defined by updated-query performance strictly exceeding old-query performance against the same fixed gallery, using Recall@K for K ∈ {1, 5, 10}.The experiments report both image-to-text and text-to-image retrieval.Source: §4.2 · p. 7

4.3 Compatibility Results

  • CC3M-selected SLERP met the Recall@1 compatibility criterion in 85 of 90 evaluations, versus 45 for SVD alone; it was also competitive with XBT in same-family upgrades.In the harder SigLIP1 → CLIP ViT-H/14 setting, SLERP achieved compatibility in most cases.Source: Table 1 · p. 7, §4.3 · p. 8, Table 2 · p. 8
  • CC3M-selected ˆα closely tracked the dataset-specific oracle across Flickr30k, COCO, and NoCaps, without test-set tuning.The per-query oracle analysis found interior oracle weights for 97.65% of queries achieving Recall@1 at some evaluated weight; this oracle uses test labels and is not deployable.Source: §4.3 · p. 9, Figure 3 · p. 9
  • Text-only support was the most stable alignment choice overall, while image-only and joint support remained competitive in individual cases.A separate target-distribution subset could also select a reliable dataset-specific weight; midpoint, NLERP, and score interpolation showed comparable average improvements.Source: §4.3 · p. 9
  • The best interpolation weight reflects the balance of positive and negative flips: corrections initially exceed regressions, but regressions grow faster beyond the optimum.The reported oracle optimum was α⋆=0.7 for CLIP ViT-H/14 → CLIP ViT-B/32 and α⋆=0.2 for SigLIP1 → CLIP ViT-H/14; CC3M-selected weights were near each peak.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents post-alignment spherical interpolation as a query-side way to improve backward-compatible retrieval while leaving the deployed gallery unchanged. It reports broad gains and identifies practical trade-offs and unequal-dimension alignment behavior.

5 Conclusion

  • Across CLIP, SigLIP1, and SigLIP2 upgrades and three retrieval benchmarks, SLERP improved over SVD alignment in nearly all settings and often restored empirical backward-compatibility.The paper also reports frequent matches or gains over training-based compatibility baselines without compatibility training, with support-set weights transferring across datasets.Source: §5 · pp. 9–10
  • The method’s practical trade-offs are support-set weight selection, computing both endpoint embeddings at query time, and performance bounded by representation quality and complementarity.The paper describes its experiments as finding these trade-offs favorable and presents the method as useful while full re-indexing is deferred or performed offline.Source: §5 · p. 10

A Additional Related Work

  • Related work supports approximate post-hoc alignment, but cross-modal agreement can weaken at realistic scales; this paper targets residual angular discrepancy after alignment.Unlike parameter-space SLERP for same-architecture models [76], this approach interpolates query representations after cross-model alignment.Source: A Additional Related Work · p. 17

B Extension to Unequal Embedding Dimensions via Rectangular Procrustes

  • Rectangular Procrustes is a partial isometry: lower-dimensional new embeddings can be embedded isometrically, while higher-dimensional embeddings contribute only their old-space-aligned component to retrieval.The residual component preserves new-model geometry but has zero interaction with old-gallery embeddings; when dimensions differ, the map is semi-orthogonal.Source: B Extension to Unequal Embedding… · pp. 18–19
  • For higher-dimensional new embeddings, the projected query may not be unit norm, so it is normalized before SLERP; for new dimensions no greater than old, this normalization is redundant.This places both interpolation endpoints on the old-model unit sphere.Source: B Extension to Unequal Embedding… · p. 19

C Relationship Between SLERP and NLERP

SLERP and NLERP trace the same geodesic between the old-model and SVD-aligned query embeddings, differing in how their weights parameterize it. Retrieval curves are nearly identical, while SLERP provides a direct angular interpretation of its weight.

C Relationship Between SLERP and NLERP

  • SLERP and NLERP reach the same directions continuously, but SLERP moves at constant angular speed and its weight represents normalized angular displacement.NLERP follows the same minor geodesic arc under a monotone, non-uniform reparameterization.Source: C Relationship Between SLERP and NLERP · p. 20
  • Across Flickr30k and COCO and both model pairs, SLERP and NLERP have nearly identical retrieval curves and peak in the same interior region.The small differences arise because uniform NLERP weights do not sample uniform angular positions along the shared arc.Source: C Relationship Between SLERP and NLERP · p. 20, Figure 5 · p. 21

D Decomposition of the Retrieval-Optimal Direction

  • The retrieval-optimal direction decomposes into a projection within the interpolation plane and an orthogonal component, with the projection expressed by an angle when nonzero.The lemma defines the projected component’s norm as ρ and gives a unique angle ψ for its normalized direction.Source: D Decomposition of the… · p. 21

E Proof of Theorem 1

The proof characterizes when the SLERP arc can contain a direction closer to an optimal direction than either endpoint. The appendix then connects that geometry to Recall@K, including queries with multiple relevant gallery items.

E Proof of Theorem 1

  • A strict interior improvement over both endpoints occurs exactly when the normalized in-plane projection of the optimal direction lies inside the minor arc.If the projection has zero norm, angular distance is constant along the arc, so no interior point strictly improves on both endpoints.Source: E Proof of Theorem 1 · pp. 22, 24

F.1 Recall@K with Multiple Relevant Items

  • With multiple relevant gallery items, Recall@K is a hit indicator: it succeeds when at least one relevant item appears among the top K.The setup allows multiple relevant items, as in image-to-text retrieval with multiple captions for an image.Source: §F.1 · p. 24

F.2 Top-K Margin

The appendix defines a top-K margin and relates its sign to retrieval success, then uses margin maximization and stability to formulate a local Recall@K certification. In auxiliary zero-shot classification, SLERP improves on SVD alignment alone across evaluated datasets.

F.2 Top-K Margin

  • Away from tie boundaries, positive top-K margin is equivalent to Recall@K success; a negative margin guarantees failure, while zero leaves the outcome to tie-breaking.The margin compares the best relevant score with the K-th largest negative score.Source: §F.2 · p. 25

F.3 Retrieval-Relevant Optimal Direction

  • The retrieval-relevant optimal direction is selected to maximize the top-K margin, and the SLERP arc improves on both endpoints when its normalized in-plane projection lies inside the arc.This applies the geometric criterion to the margin-maximizing direction for each query.Source: §F.3 · p. 25

F.4 Lipschitz Stability of the Top-K Margin

  • The appendix establishes a perturbation bound for the top-K margin when the unit query direction changes.The bound is developed by controlling changes in relevant scores and in the K-th largest negative score.Source: §F.4 · p. 26

F.5 Local Certification of Recall@K

  • The local Recall@K corollary gives a certification condition, with a simpler sufficient condition also implying positive top-K margin.Positive margin corresponds to Recall@K success away from boundary ties.Source: §F.2 · p. 25, §F.5 · pp. 26–27

G Cross-Modal Zero-shot Classification

  • In zero-shot classification, SLERP consistently improves on SVD alignment alone, often closing the gap to or surpassing the old-model classifier.The reported trend covers the evaluated classification datasets.Source: G Cross-Modal Zero-shot Classification · p. 27

H Additional SLERP Arc Results

Additional image-to-text retrieval results show that support-set-selected SLERP transfers across datasets and model families, with best test performance typically at an interior weight.

H Additional SLERP Arc Results

  • Across Flickr30k, COCO, and NoCaps and both model pairs, best image-to-text Recall@1 performance is typically at an interior weight, and support-set-selected weights transfer across datasets and model families.The CC3M validation curves track test curves closely, and the selected weight is near the dataset-specific oracle.Source: H Additional SLERP Arc Results · p. 27

I Detailed Retrieval Performance

Across retrieval settings, SLERP usually improves on SVD alignment alone, including beyond Recall@1; the re-indexing experiments also find gains over the new-model-equivalent SVD endpoint.

I Detailed Retrieval Performance

  • SLERP improves on the corresponding SVD endpoint in nearly all settings and extends gains from Recall@1 to Recall@5 and Recall@10.It is consistently competitive with training-based XBT for same-family CLIP pairs and often makes incompatible SVD rows compatible for cross-family pairs.Source: I Detailed Retrieval Performance · p. 28
  • Text-only alignment is the most stable overall, while image-only and joint support can be competitive for particular retrieval directions.Source: I Detailed Retrieval Performance · p. 32

J Re-indexing

  • With re-indexing allowed, SLERP often improves on the SVD endpoint, which is equivalent to evaluating the new model directly.In several cases it matches or exceeds XBT despite requiring no compatibility training.Source: J Re-indexing · p. 33

K Target-Set Budget Sensitivity

Small deployment-distribution subsets identify a high-performing region of the SLERP curve, although smaller budgets increase estimate variance. Additional flip analyses attribute Recall@1 gains to the balance of corrected and newly failed queries.

K Target-Set Budget Sensitivity

  • Even small target subsets identify the same high-performing region of the SLERP curve, with the maximizer remaining stable and close to the full-set result.Reducing the subset increases Recall@1 estimate variance; this analysis concerns a deployment-distribution subset, separately from the main protocol's CC3M-selected fixed weight.Source: K Target-Set Budget Sensitivity · pp. 33–34

L Additional Flip Analyses

  • Across retrieval directions and datasets, SLERP improves Recall@1 when it corrects more previously incorrect queries than it causes previously correct queries to fail.The optimal weight therefore depends on the balance of positive and negative flips, not the positive-flip rate alone.Source: L Additional Flip Analyses · p. 34

M Per-Query Oracle Analysis

The per-query oracle analysis finds retrieval improvements at interior weights that neither endpoint provides, while distinguishing relevant-item similarity from success against the full gallery. The accompanying baseline descriptions compare alternative alignment maps and ways to combine query endpoints.

M Per-Query Oracle Analysis

  • For 31,014 Flickr30k queries, the per-query oracle reaches 59.15% I2T Recall@1 versus 54.61% for an endpoint-only oracle; 1,408 queries succeed only at interior weights.The per-query oracle uses query-level test labels and is an upper bound, not a deployable weight-selection procedure.Source: M Per-Query Oracle Analysis · pp. 34–36
  • Relevant-caption similarity peaks strictly inside the arc for 99.87% of queries, yet its average peak at α = 0.3 differs from the fixed-weight Recall@1 optimum α⋆ = 0.5.Retrieval success also depends on how similarities to non-relevant items change along the arc.Source: M Per-Query Oracle Analysis · p. 36

N.1 Alignment baselines

N.2 Endpoint-combination baselines

  • The endpoint-combination baselines include a fixed midpoint, NLERP, score interpolation, and query-adaptive NLERP; score interpolation, score ensembling, and NLERP at the same coefficient induce identical rankings.The query-adaptive rule uses endpoint cosine similarity and no test labels.Source: §N.2 · pp. 37–38

N.3 Discussion

The ablation finds that combining old and aligned-new queries supplies most of the Recall@1 improvement, while validation-based position selection makes compatibility more consistent. SLERP provides near-best mean gains with better compatibility than the fixed midpoint, and its weight has a consistent angular interpretation across queries.

N.3 Discussion

  • Affine, ridge/one-sided CCA, and whitened Procrustes lose 15.35, 11.32, and 1.83 mean Recall@1 points relative to Procrustes, respectively.Their compatibility counts are 11, 14, 14, and 37/90 versus Procrustes at 45/90; the authors attribute poorer generalization to distortion of retrieval's inner-product geometry.Source: §N.3 · p. 38
  • SLERP improves mean Recall@1 over SVD by 4.80 points and raises compatibility from 72/90 to 84/90; query-adaptive NLERP reaches 88/90 for only 0.09 more mean points.The adaptive method requires four additional calibrated parameters.Source: Table 10 · p. 38, §N.3 · p. 38
  • The fixed normalized midpoint raises mean I2T and T2I Recall@1 from 51.72 and 33.77 to 57.76 and 37.23, improving compatibility from 45/90 to 72/90.It has no tunable weight and uses no validation data.Source: §N.3 · p. 38
  • Endpoint combination accounts for most mean Recall@1 improvement, while validation-based position selection makes gains consistently satisfy compatibility across evaluated model pairs, datasets, support modalities, and retrieval directions.SLERP, NLERP, and score interpolation show similar improvements, which the authors attribute to residual post-alignment geometry rather than a particular combination operator.Source: §N.3 · pp. 38–39
  • SLERP's α represents the same fraction of angular displacement for every query, whereas a fixed NLERP or score-interpolation coefficient maps to query-dependent angular positions.Source: §N.3 · p. 39

O Limitations

SLERP avoids target-benchmark fitting, but requires an estimated interpolation weight and adds query-side computation. Its performance also depends on the endpoint representations and alignment quality.

Selection of the interpolation weight

  • The interpolation weight is not known a priori and must be estimated from deployment-distribution data or a separate support set.The main experiments select it on CC3M and apply the same value to Flickr30k, COCO, and NoCaps; reported retrieval and classification results show that transferred weights remain close to dataset-specific best performance.Source: O Limitations · p. 39

Query-side computation

  • SLERP requires both old-model and SVD-aligned new-model query representations, adding query-side computation while leaving the gallery unchanged.This overhead is independent of gallery size; the method is framed as a migration-period compatibility mechanism while full gallery re-indexing proceeds offline.Source: O Limitations · p. 39

Dependence on endpoint quality

  • Performance depends on the quality and complementarity of the endpoint queries and Procrustes map; weak endpoints cannot supply new task-specific information through interpolation alone.The closed-form map and interpolation weight are estimated on the alignment support set, without target-benchmark-specific fitting.Source: O Limitations · p. 39

P Deployment Cost and Migration-Period Analysis

This section examines SLERP’s query-side cost and performance during progressive gallery migration, and records the study’s human-subjects and LLM-use disclosures.

P.1 Query-Side Cost and Re-indexing Trade-off

  • Across 105 queries, SLERP encoder compute is 17.09 PFLOPs for I2T and 1.93 PFLOPs for T2I, versus 0.88 and 0.60 PFLOPs for old-model queries.Full re-indexing with new-model queries requires 149.20 PFLOPs for I2T and 1.62 EFLOPs for T2I; the latter is dominated by image-gallery re-encoding.Source: §P.1 · p. 40
  • SLERP computes both old and aligned-new query embeddings, while projection and interpolation are negligible and leave the gallery, index, and search complexity unchanged.The 768 × 512 projection takes approximately 0.79 MFLOPs, and interpolation is linear in embedding dimension.Source: §P.1 · p. 40
  • Sequential single-GPU SLERP adds approximately 5.4 ms latency; concurrent encoders on separate GPUs or cached old queries reduce the gap to full re-indexing to approximately 0.3 ms.Projection and interpolation together contribute less than 1 ms; encoder inference dominates the overhead.Source: §P.1 · p. 40, Table 11 · p. 40

P.2 Migration Period with Partial Backfilling

  • The partial-backfill comparison uses aligned-new queries against a mixed gallery, with 0% backfill equivalent to the SVD baseline and 100% to full re-indexing; completed backfilling favors direct new-model retrieval.The paper notes that combining SLERP for non-backfilled items with new-model queries for backfilled items is a natural extension, also supported by.Source: §P.2 · pp. 40–41
  • On Flickr30k CLIP ViT-L/14 → ViT-B/32 image-to-text Recall@1, SLERP reaches 48.00 without re-encoding, while partial backfilling needs 90% re-encoding to reach 47.98.SLERP recovers 91.1% of the full re-indexing improvement, whose Recall@1 is 48.72.Source: §P.2 · p. 40, Figure 12 · p. 41
  • For SigLIP2 → SigLIP1, SLERP reaches 58.63 Recall@1 with an unchanged gallery; partial backfilling surpasses the old-model 58.09 only after 40–50% re-encoding.The SVD-aligned query scores 45.46 Recall@1, and partial backfilling exceeds the old model only once at least 50% is re-encoded.Source: §P.2 · pp. 40–41, Figure 12 · p. 41

2. Limitations

  • The paper reports no crowdsourcing or research with human subjects.Source: §2 · p. 48
  • LLM use was limited to writing, editing, or formatting rather than the core research methods.Source: §2 · p. 48

Related papers

Loading 2609.39836v1…