PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Alignment from a Few Paired Data via Distilled Optimal Transport

Julien Kordes, Reuben Dorent

arXiv:2610.12781v1cs.AI

TL;DR

Aligning independently pretrained unimodal encoders is difficult when paired examples are scarce. AnchorDOT expresses representations relative to paired anchors, matches unpaired pools with global entropic optimal transport, and distills the plan into encoders. It outperforms competing approaches at the same anchor budget across image-text and molecule-text evaluations.

  • Problem

    Aligning independently pretrained unimodal encoders is challenging with only a few hundred paired examples, while paired multimodal data is scarce and the embedding spaces can differ in dimension, scale, and geometry.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    AnchorDOT compares representations through similarities to paired anchors, solves global entropic optimal transport over unpaired pools, and distills the plan into inductive encoders.

    Source: §3.2 · p. 5, §3.3 · p. 6

  • Results

    At the same anchor budget, AnchorDOT outperforms anchor-relative and graph-based methods by a wide margin, matches SOTAlign at image-text R@1, and outperforms it elsewhere.

    Source: §4 · p. 8, §5 · p. 11

  • Takeaways & Limitations

    Only the anchors require pairing; the unpaired collections may be sampled independently, with no hidden correspondence required in the pool.

    Source: §1 · p. 2, §4 · p. 10

    The uniform-marginal formulation assumes both unpaired collections come from the same joint distribution.

    Source: §5 · p. 11

Abstract

from arXiv · show

Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only $N \sim 100$--$500$ pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the $N$ paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.

1 Introduction

AnchorDOT targets alignment of independently pretrained encoders when only a few hundred paired examples are available. It uses anchor-relative coordinates, global optimal transport, and distilled encoders to address limitations of prior methods.

1 Introduction

  • Paired multimodal data is scarce, while independently pretrained encoders offer abundant unimodal representations whose embedding spaces differ in dimension, scale, and geometry.The introduction contrasts this setting with vision-language models trained on hundreds of millions of paired examples.Source: §1 · p. 1
  • Earlier anchor-relative methods can require very large anchor collections and costly reference storage, while graph-based alignment assumes comparable local geometry across independently trained encoders.SOTAlign uses a linear alignment as teacher, so its student inherits that alignment’s limits.Source: §1 · pp. 1–2
  • AnchorDOT maps both modalities into anchor-relative coordinates, solves a global entropic OT problem over the unpaired pool, then distills its plan into compact encoders.Marginal constraints limit the total mass received by each point, addressing the hubness of nearest-neighbor matching.Source: §1 · p. 2
  • AnchorDOT aligns frozen unimodal encoders from N ∼100–500 paired anchors and distills the transport plan into inductive encoders that need no reference collection at inference.The unpaired collections may be sampled independently from their marginals, with no hidden pairing in the pool.Source: §1 · p. 2

2 Related work

Prior alignment methods differ in their use of anchors, unpaired data, and cross-modal matching assumptions. AnchorDOT combines anchor-relative representations with global transport and distillation to address these constraints.

Anchor-relative representations

  • ASIF retrieves independently against a dense reference collection, using 1.6 million reference pairs and reaching 18% zero-shot ImageNet accuracy with 104 pairs.RELIT uses nearest-neighbor retrieval to mine pairs before fine-tuning on the full Flickr30k training split, about 29k images.Source: §2 · p. 2

Alignment with little or no supervision

  • At comparable pair budgets, SUE relies on similar intra-domain structure, while CSA uses paired features alone and was validated with 5k Flickr30k pairs.SOTAlign [Roschmann et al., 2026] refines a linear teacher alignment; STRUCTURE uses tens of thousands of pairs to regularize neighborhood geometry.Source: §2 · p. 3

Optimal transport as a distillation target

  • OTTER [Wu et al., 2021], SwAV, and S-CLIP distill Sinkhorn-based targets where cross-view or cross-modal similarity is available by construction.For independently trained unimodal encoders, anchor-relative coordinates supply the similarity used for alignment.Source: §2 · p. 4

Hubness

  • AnchorDOT’s global coupling constrains assignments jointly, unlike retrieval-time hubness corrections such as inverted softmax [Smith et al., 2017] and CSLS [Conneau et al., 2017].The overview describes its sequence of anchor-relative embedding, entropic OT matching, and encoder distillation.Source: Figure 1 · p. 3, §2 · p. 4

3 Method

The method uses paired anchors to construct a shared coordinate space, matches independently sampled unpaired pools with global entropic optimal transport, and distills the resulting plan into encoders for inference.

3.1 Problem setup and notation

  • The setup provides N paired anchors from the joint distribution and two unpaired collections drawn independently from the modality marginals, with no map initially given between encoder spaces.The stated procedure constructs a common coordinate system, then a correspondence, then an inductive model of that correspondence.Source: §3.1 · p. 4

3.2 Anchor-relative coordinate space

  • Each representation becomes a vector of similarities to the paired anchors, whose corresponding coordinates measure similarity to the same multimodal concepts.Anchor-orthonormalized similarity compensates for dominant directions represented repeatedly in the anchor set; plain cosine similarity is recovered at γ = 0.Source: §3.2 · p. 5

3.3 Global entropic optimal transport in the anchor-relative space

  • Anchor-relative coordinates make the modalities comparable but do not identify correspondences, and nearest-neighbor matching typically fails through hubness.The method defines a cross-modal cost on the relative representations and solves one entropy-regularized OT problem over the full unpaired pool with uniform marginals.Source: §3.2 · p. 5, §3.3 · p. 5
  • The global coupling fixes the mass each point receives, countering hubs; inverted softmax is a one-sided Sinkhorn half-step, while CSLS applies fixed neighborhood penalties.The coupling can be computed block-by-block from dual potentials without materializing the full cost matrix, and the rectangular problem requires no shared indexing.Source: §3.3 · p. 6

3.4 From a transductive coupling to inductive encoders via distillation

  • Because the transport plan is transductive, two lightweight modality-specific encoders are trained to map original embeddings into a shared space for single-pass inference without anchors or reference collections.Chunk training uses exact restrictions of the global plan as soft targets, while the anchor chunk supplies the known pairs as one-hot supervision.Source: §3.4 · pp. 6–7

4 Experiments

The experiments compare cross-modal retrieval at 500 paired anchors across image-text and molecule-text tasks, alongside tests of hubness, pool composition, and zero-shot classification. AnchorDOT performs strongly across these settings, including when unpaired pools have no hidden pairing.

Experimental setup

  • The protocol freezes and precomputes backbone embeddings, trains only lightweight encoders, and uses 500 paired anchors with a held-out test split.The usual pool is ∼29k disjoint image-only and caption-only items; the 400-pair test set is smaller than standard Flickr30k and MS-COCO test sets, so results are not comparable to CLIP-era retrieval literature.Source: §4 · pp. 7–8

Image-text retrieval

  • AnchorDOT is best in 10 of 12 image-text retrieval columns; at R@5 and R@10 it leads SOTAlign in all eight entries by 0.9–3.9 points.At R@1, the methods are considered tied on MS-COCO; distillation multiplies the contrastive baseline’s R@1 by 1.6–2.0.Source: Table 1 · p. 8, §4 · p. 8

Molecule-text retrieval

  • On ChEBI-20, AnchorDOT ranks first in all six molecule-text retrieval columns, 1.5–3.2 points ahead of SOTAlign.The gain over the contrastive baseline is smaller than on image-text, at R@1 ×1.4–1.5.Source: §4 · p. 8, Table 2 · p. 9

Zero-shot classification

  • In zero-shot classification, AnchorDOT beats SUE by 18–30 points and is comparable to SOTAlign, with a 2.4-point lead on Flickr30k and a 1.7-point deficit on MS-COCO.The evaluation uses 700 ImageNet images from seven classes, with no training on those images, categories, or task.Source: Table 3 · p. 9, §4 · p. 10

Anchor budget and pool size

  • On Flickr30k, AnchorDOT consistently beats SOTAlign and the contrastive baseline across evaluated anchor budgets; its Recall@10 rises from approximately 0.63 to 0.80 as the unpaired pool grows from 102 to 3 × 10^4 items.At N ∼8k, AnchorDOT and the contrastive baseline approach similar performance near 0.9 R@10.Source: Figure 2 · p. 9, §4 · p. 9

Hidden pairing influence

  • On Flickr30k, Recall@10 is essentially unchanged as overlap varies from no hidden pairing to full overlap, indicating the methods work without hidden pool correspondences.At overlap 0, the image-only and caption-only pools are disjoint; at overlap 1, every unpaired item has a counterpart whose pairing is unrevealed.Source: Figure 3 · p. 9, §4 · p. 10

Hubness reduction

  • On Flickr30k, the converged coupling has the lowest N10 skewness and highest Recall@10 among the four matching rules in both directions.Skewness is 0.32/0.37 and Recall@10 is 0.80/0.80; the inverted softmax increases hubness over nearest neighbor in text-to-image retrieval.Source: Figure 4 · p. 10, §4 · p. 10

Further experiments

  • Further experiments analyze representation participation ratio, classical unsupervised methods, matching-cost ablations, cross-modal generation, and zero-shot retrieval.The cost ablations test adding a Fused Gromov–Wasserstein structural term and replacing the anchor-relative representation.Source: §4 · p. 11

5 Conclusion and future work

AnchorDOT aligns frozen unimodal encoders from 100–500 paired examples by matching anchor-relative representations with global entropic optimal transport and distilling the plan into compact encoders. At the same anchor budget, it outperforms anchor-relative and graph-based methods, matches SOTAlign at image-text R@1, and outperforms it elsewhere; its uniform-marginal assumption and encoder quality bound its scope.

5 Conclusion and future work

  • At the same anchor budget, AnchorDOT outperforms anchor-relative and graph-based methods, matches SOTAlign at image-text R@1, and outperforms it elsewhere.The unpaired pool requires no hidden pairing.Source: §5 · p. 11
  • The framework aligns frozen unimodal encoders from N ∼100–500 paired examples by matching anchor-relative representations with global entropic optimal transport and distilling the plan into compact encoders.The two unpaired collections may be sampled independently from their marginals.Source: §5 · p. 11
  • Performance remains bounded by encoder quality, and uniform marginals assume both collections come from the same joint distribution.Unbalanced optimal transport would relax the marginal assumption; anchors are drawn at random.Source: §5 · p. 11

AI use statement

The authors report using generative AI for writing, literature retrieval, ideation, code generation, and drafting some sections, with human review or responsibility stated. They anticipate no ethical concerns from the work.

AI use statement

  • AI assisted writing, literature retrieval, implementation checks, ideation and execution, code generation, and initial drafting of results and appendix sections.The authors say they checked writing and manually reviewed generated code and drafted sections; tables were made manually and figures locally.Source: AI use statement · p. 11
  • The authors take responsibility for the final content, including text, claims, and artifacts produced with generative AI assistance.Source: AI use statement · p. 12

Ethics statement

A Distillation details

Distillation trains inductive encoders from a fixed transport plan, using direction-asymmetric soft targets and a distinguished anchor chunk. The resulting encoders do not require anchors or another reference collection at inference.

A Distillation details

  • The anchor pairs form a distinguished chunk whose transport plan is the identity because their correspondence is known.Source: A Distillation details · p. 14
  • The two distillation directions use row- and column-normalized soft targets, which differ unless the transport plan is symmetric.Source: A Distillation details · p. 14
  • When row-normalized targets are one-hot, the X→Y loss reduces to hard-target InfoNCE; otherwise it distills transport-plan uncertainty.The fixed plan is computed before optimization, and each chunk contributes one stochastic-gradient step per epoch.Source: A Distillation details · p. 14
  • The trained encoders operate on raw embeddings and require no anchor set or reference collection at inference.Anchors supervise training but do not compute a query's final representation.Source: A Distillation details · p. 14
  • Figure 5 compares recall for the transductive OT teacher with recall retained by the inductive encoders after distillation.Source: Figure 5 · p. 15

B.1 Spectral distortion of anchor-relative profiles

The spectral analysis relates anchor-relative profiles to the embedding covariance spectrum, with γ controlling whether it is squared, preserved, or flattened. It also connects one Sinkhorn update to the additive correction shape of CSLS.

B.1 Spectral distortion of anchor-relative profiles

  • With iid anchors, the sample second-moment matrix M_X converges almost surely to Σ_X and is eventually invertible under the stated positive-definiteness assumption.Source: §B.1 · p. 15
  • In practice, eigenvalues of M_X are clipped below at 10^-6 times their mean before powers are taken.The unclipped results apply when the minimum eigenvalue exceeds the threshold, which holds almost surely for large N if s_min > τ.Source: §B.1 · p. 15
  • For γ = 0 the covariance spectrum is squared, γ = 1 preserves its shape exactly at every N, and γ = 2 flattens it in the limit.Source: §B.1 · p. 16
  • For γ ≠ 1, the spectral result is asymptotic, with leading eigenvalues accurate once N is large relative to d_x.Source: §B.1 · p. 16

B.2 Relation to CSLS.

  • As ε→0, one full Sinkhorn iteration has the CSLS additive form with coefficient 1, K = 1, and a vanishing additive term.CSLS uses one correction per side based on average similarity to K nearest neighbors in the opposite space.Source: §B.2 · pp. 16–17

C.1 Participation ratio

On Flickr30k, increasing γ raises the participation ratio (PR) at larger anchor budgets, but the N = 500 curve peaks near γ ≈ 1.5 and then falls. The authors attribute this downturn to finite-anchor estimation noise and restrict γ to [0, 1] in practice.

C.1 Participation ratio

  • At N = 500, PR peaks around γ ≈ 1.5 and drops sharply toward γ = 2, unlike the monotonic increase at N = 2000 and N = 4000.At the two larger budgets, PR saturates near its asymptotic ceiling dx as γ approaches 2; at N = 500 against dx ≈ 384, poorly estimated small eigenvalues can dominate after inversion.Source: §C.1 · p. 17, Figure 6 · p. 17
  • The authors restrict γ to [0, 1] because, with a few hundred anchors, γ > 1 risks amplifying anchor-estimation noise rather than correcting a real bias.As N grows, the participation-ratio curve converges to the monotonic prediction of Proposition 1.Source: §C.1 · p. 17

C.2 Comparison to unsupervised methods

The additional experiments examine geometry-only alignment, alternative transport costs, structural alignment, generation, and transfer to unseen data. They report near-random recall for zero-pair geometry-only methods, stronger transfer retrieval for the proposed method, and qualitative generation results with limits on fine detail.

C.2 Comparison to unsupervised methods

  • Gromov-Wasserstein, Wasserstein Procrustes, and Iterative Closest Point yield near-random recall when aligning image and text clouds using intra-modal geometry alone, supporting the need for paired anchors.The authors distinguish this outcome from a failure of the established matching algorithms and note that independently trained image and text encoders lack enough shared geometric signal.Source: §C.2 · pp. 17–18

C.3.1 Bridge-cost

  • The bridge-cost ablation routes cross-modal distance through a single anchor, while the proposed profile compares full anchor-similarity vectors; recall@10 drops considerably with the bridge cost.Both costs use the same Sinkhorn coupling, isolating the cost construction in this comparison.Source: §C.3.1 · p. 18, §C.3 · p. 18, Figure 8 · p. 19

C.3.3 Fused-Gromov-Wasserstein

  • Structural alignment never improves retrieval: recall is highest at or near α = 1, where the structural term is dropped and the coupling reduces to anchor-relative feature-cost OT.Figure 10 plots recall@k against α in the fused-Gromov-Wasserstein setting.Source: §C.3.3 · p. 19, Figure 10 · p. 20

C.3.2 KPG-RL cost

  • The KPG-RL cost compares softmax-normalized anchor-distance distributions with Jensen-Shannon divergence; the proposed method has similar hubness and slightly better retrieval performance.The comparison uses the same Sinkhorn coupling without KPG-RL’s anchor-pair mask.Source: §C.3.2 · pp. 18–19, Figure 9 · p. 20

C.4 Qualitative cross-modal generation

  • Generated faces capture coarse caption semantics, including hair color and expression, but fine or spatially localized attributes remain imperfect.Captions mentioning hats rarely produce recognizable texture, although the decoder consistently adds a silhouette above the head.Source: §C.4 · p. 20, Figure 11 · p. 21

C.5.1 Zero-shot classification; C.6 Retrieval examples

  • The appendix provides example figures for zero-shot classification on ImageNet and retrieval examples across Flickr30k, MS-COCO, and ChEBI-20, without stating outcomes in these passages.The classification examples compare the proposed method, SUE, SOTAlign, and a pretrained CLIP checkpoint.Source: §C.5.1 · p. 21, Figure 12 · p. 22, Figure 14 · p. 24

C.5.2 Zero-shot retrieval

  • 59.7% recall@10 for the proposed method on unseen Flickr30k, versus 56.7% and 24.3% for the other methods; in-distribution results were 79.3%, 75.9%, and 37.8%, respectively.Methods were fitted on MS-COCO with approximately 29k unpaired samples and 500 anchors, then applied to Flickr30k without fine-tuning, calibration, or optimization.Source: §C.5.2 · p. 23, Figure 13 · p. 23

C.7 Complement

D Hyperparameters

The hyperparameter procedure fixes encoder-training settings across datasets and seeds, while selecting Sinkhorn regularization and the anchor-relative profile parameters per dataset and seed. Selection uses held-out anchor-match ranks from the transport coupling.

D Hyperparameters

  • Encoder-training hyperparameters are fixed across datasets and seeds, while Sinkhorn and profile hyperparameters are selected per dataset and seed.The fixed training settings are summarized in Table 8.Source: D Hyperparameters · p. 27, Table 8 · p. 28
  • ε is set to η times the median anchor-relative cost, allowing η to adapt regularization to the cost scale without tuning ε directly.η is selected per dataset and seed.Source: D Hyperparameters · pp. 27–28
  • The profile uses independently whitened embeddings and γ ∈ [0, 1] to interpolate between the untouched profile and an unmodified spectrum.γ and η are the only profile and Sinkhorn hyperparameters selected per dataset and seed; other profile-construction choices are fixed.Source: D Hyperparameters · p. 28
  • For each candidate (η, γ), the coupling is evaluated using held-out anchors, and the selected pair minimizes their mean true-match rank across both directions.Anchors are split 90/10; the selected configuration is then used for distillation with the full anchor set.Source: D Hyperparameters · p. 28, Figure 17 · p. 29

Related papers

Loading 2610.12781v1…