Anchor Divergence for Semantic Geometry in Contrastive Learning

• A fixed cosine ranking cannot privilege the semantic attribute relevant to a task, such as dog breed, background, clothing, or a clinical finding.For a query image, different anchor choices change its neighborhood while keeping image representations fixed.Source: §1 · p. 1, Figure 1 · p. 2
• Contrastive structure defines a family of semantic geometries on image representations, each induced by a probability distribution over semantically relevant anchors.The resulting anchor divergences specify context-specific geometry without changing the underlying representations.Source: §1 · pp. 1–2
• Images can be compared by their contrastive-softmax distributions over text anchors, so the selected anchors determine which semantic differences the comparison emphasizes.For chest X-rays, anchors could be clinical descriptions of pathologies.Source: §1 · p. 2
• The KL divergence between softmax distributions over sampled anchors has a population limit that is a Bregman divergence on image representations.This moves semantic geometry from a particular finite anchor set to the underlying anchor distribution.Source: §1 · p. 2
• Fitting a parametric anchor-distribution model makes geometry evaluation independent of the number of observed anchors and can approximate the geometry when anchors are limited.The model is fit using observed anchors.Source: §1 · pp. 2–3
• Across tested settings, the resulting geometries improve over task-agnostic cosine and task-specific heuristic approaches, while broad-to-target adaptation reduces target-anchor needs by a full order of magnitude.The adaptation starts from a broad anchor distribution and refines it with a small set of targeted anchors.Source: §1 · p. 3

• A finite set of text anchors defines a softmax distribution for each image embedding, and the KL divergence between those distributions compares embeddings.The anchors are chosen to describe the semantics of interest, such as clinical descriptions of pathology findings.Source: §2 · p. 3
• Viewing observed anchors as samples from an underlying distribution makes the semantic context depend on that distribution rather than any particular finite realization.This addresses sensitivity to individual anchor choices and yields a well-defined population geometry.Source: §2 · p. 3
• As the number of i.i.d. anchors sampled from q grows, the softmax anchor divergence converges to a population Bregman divergence.The proof uses the contrastive softmax as an exponential tilt of the empirical anchor measure, linking KL divergence to the geometry generated by the cumulant function.Source: §2 · p. 4
• The anchor distribution q specifies the semantic context and thereby determines the geometry on the image embedding space.The softmax anchor divergence is the special case induced by the empirical distribution.Source: §2 · p. 4

3 Modeling Structured Semantic Geometries

• Anchor divergences use an anchor distribution to represent the semantics of interest and induce the corresponding Bregman geometry.Source: §3 · p. 4
• The empirical anchor distribution recovers softmax anchor divergence and converges to the target population geometry as the anchor set grows, but few anchors may poorly represent the semantics.Source: §3 · p. 4
• Fitting a parametric anchor model induces structured geometry; once fitted, divergence evaluation need not scale with the number of observed anchors.Source: §3 · p. 4
• For pleural-effusion retrieval, cosine matches device-related structure, whereas softmax and Gaussian parametric anchor divergences retrieve images with the pathology.Source: Figure 2 · p. 5, §3 · p. 5
• Gaussian fitting yields an elliptical geometry, while a Gaussian mixture can represent multiple semantic modes rather than one global covariance structure.Source: §3 · pp. 4–5
• Latent Gaussian and latent spherical divergences project onto the top-k empirical anchor-covariance eigenvectors, then use Euclidean or cosine distance, respectively.Source: §3 · p. 5

4 Adapting Semantic Geometries from Broad Anchors

• In pathology-specific retrieval, abundant general clinical text can provide broad anchors when precise target-pathology descriptions are scarce.Source: §4 · pp. 5–6
• The procedure fits the model to broad anchors to set prior hyperparameters, updates that prior with target anchors, and summarizes the update with an estimate such as a posterior mean or MAP.Source: §4 · p. 6
• The posterior anchor divergence is the anchor divergence induced by the adapted distribution, formed by defining prior information from broad anchors and updating it with target anchors.Source: §4 · p. 6
• Posterior anchor divergence regularizes a structured estimate with broad, related anchors; when target anchors are few, broad anchors stabilize the geometry, while more target anchors increasingly determine it.Source: §4 · p. 6

5.1 Implementation details

• Experiments use MetaCLIP-2 for natural images and CheXzero with CheXpert for clinical images; retrieval is evaluated with AUPRC and compared with cosine similarity and image-text inversion.AUPRC is used because relevant keys are sparse and prevalence varies across semantic targets.Source: §5.1 · pp. 6–7

5.2 Anchor distributions select the semantic coordinate

• Pathology-specific anchors improve retrieval over reference geometries for most clinical findings, with the largest gains for pleural effusion, cardiomegaly, edema, and support devices.Table 2 reports condition-wise AUPRC differences against the reference geometries, averaged over 10 anchor sets and 10 CheXzero checkpoints, with 95% intervals.Source: §5.2 · p. 7, Table 2 · p. 7
• Increasing object-anchor mass on ImageNet-R increases object retrieval while decreasing style retrieval, showing that the same fixed embeddings support different geometries through anchor changes.The object and style anchor distributions are mixed as qp = p qobj + (1 − p) qstyle.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 Parametric modeling is useful with limited anchors

• On ImageNet, structured parametric geometries recover useful neighborhoods at moderate anchor budgets, while empirical softmax improves with coverage and eventually becomes strongest.Similar trends hold across the other datasets.Source: §5.3 · p. 8, Figure 4 · p. 9
• With full dataset-matched anchors on ImageNet-R, CUB-200, and Sketchy, softmax is strongest, while parametric anchor divergences retain much of their gain over cosine similarity.This illustrates the tradeoff: parametric structure is useful with limited anchors, while empirical geometry becomes more effective as coverage grows.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 Posterior modeling adapts broad geometry with few target anchors

• Posterior anchor models are most useful when target anchors are scarce, and their gap to target-only models narrows as target-anchor coverage grows.The experiment uses WordNet for broad semantic structure and ImageNet class names as target-specific anchors.Source: §5.4 · pp. 8–9
• Changing the anchor distribution shifts retrieval toward different semantic factors; parametric models help with limited anchors, empirical softmax improves with coverage, and posterior models adapt broad structure from few target anchors.Together, the experiments support anchor divergences as a flexible framework for context-dependent semantic geometry.Source: §5.4 · p. 9

Semantic geometry in contrastive representations

• Anchor divergences represent semantic context with an anchor distribution that induces a Bregman geometry on a fixed representation space.This addresses which semantic factors should determine similarity when representations encode several factors simultaneously.Source: §6 · p. 9

Future directions

• For Figure 5, posterior models adapted from WordNet are strongest at small and moderate target-anchor budgets, with their advantage narrowing as coverage grows.The figure contrasts dotted target-only curves with solid posterior-model curves using the same target anchors.Source: Figure 5 · p. 10

Relation to TIP-X

• TIP-X's KL comparison of image affinity profiles over downstream class-text prompts is the same finite-anchor construction, while this work studies the population Bregman geometry induced by an anchor distribution.The paper uses that perspective to model and adapt semantic geometry.Source: §6 · p. 10

Information geometry and metric learning

• Unlike metric learning from supervised image-pair constraints, anchor divergences specify context through semantic text-anchor distributions without labeled pairs or representation fine-tuning.Parametric modeling imposes corresponding geometric structure, and Bayesian updating provides a mechanism for semantic adaptation.Source: §6 · p. 10

Future directions

• The construction is not intrinsically image-specific: a softmax distribution over anchors in one space can induce geometry on representations in another space.The discussion suggests extension to other contrastive and multimodal settings with the same structure.Source: §6 · p. 10

A Proof of Theorem 3

• With i.i.d. anchors from q, the softmax anchor divergence converges to the population Bregman divergence as the anchor count tends to infinity.Source: A Proof of Theorem 3 · p. 13

B Structured approximations to the population anchor geometry

• Gaussian and low-rank models are tractable structured surrogates, not literal distributions of normalized embeddings, trading fidelity, robustness, and sample efficiency.Gaussian models place mass off the unit sphere, while low-rank models also idealize semantic variation as concentrated in a lower-dimensional subspace.Source: B Structured approximations to the… · p. 17

B.1 General framework

• A latent exponential family defines semantic geometry through a Bregman divergence, which is pulled back to ambient embeddings through score-aligned parameterizations and, for low-rank models, projection or extension maps.The ambient Gaussian case recovers the population anchor divergence; low-rank semantics are realized in an affine subspace.Source: §B.1 · pp. 17–18

B.2 Gaussian anchor model; B.3 Latent Gaussian model

• The Gaussian anchor model is the identity case in ambient space, while the latent Gaussian model pulls its geometry back to a lower-dimensional semantic subspace.Source: §B.2 · p. 18, §B.3 · p. 19

B.4 Latent spherical model

• The latent spherical model retains directional variation in the semantic subspace and fixes the latent radius for general ambient embeddings; its divergence assumes both projected parameters are nonzero.Its expression uses the modified Bessel function of the first kind.Source: §B.4 · p. 20

B.5 Gaussian mixture model

• A Gaussian mixture provides a more expressive, multimodal ambient-space approximation to the population anchor geometry.Source: §B.5 · p. 21

C Parameter estimation for structured anchor models

• Parameter fitting uses maximum likelihood for Gaussian mean and covariance, top-k anchor-covariance eigenvectors explaining 95% of variance for low-rank models, and EM for Gaussian mixtures.Source: C Parameter estimation for structured… · p. 22

D Model-specific posterior and posterior-like updates for posterior anchor modeling; D.1 General adaptation framework; D.2 Gaussian anchor model with Normal–Inverse-Wishart update

• Adaptation fits broad anchors, centers a prior update on their semantic structure, and refines it with precise anchors; the Gaussian specialization uses a Normal–Inverse-Wishart prior to update covariance and induce adapted geometry.A posterior summary defines the adapted Gaussian model and its geometry.Source: D Model-specific posterior and… · p. 22, §D.1 · pp. 22–23, §D.2 · p. 23

D.3 Latent Gaussian model with matrix Bingham update

• The latent Gaussian update adapts geometry by updating a low-dimensional subspace, using a matrix-Bingham prior initialized from broad anchors and a MAP estimate informed by precise anchors.The adapted subspace is given by the top-k eigenvectors of the updated parameter matrix.Source: §D.3 · pp. 24–25

D.4 Latent spherical model with posterior-like Bingham update

• The latent spherical update likewise adapts the semantic subspace using a tractable posterior-like matrix-Bingham update that combines the broad-anchor subspace with precise-anchor covariance.The parameter β controls the influence of precise anchors, and the adapted subspace is the top-k eigenvectors of the updated matrix.Source: §D.4 · p. 25

D.5 Gaussian mixture model with Dirichlet–NIW update

• Unlike a single-Gaussian geometry, the Gaussian-mixture geometry is generally nonquadratic and can represent multiple semantic modes.Broad anchors determine semantic modes and local covariance scales, while soft assignments let precise anchors refine multiple modes simultaneously.Source: §D.5 · p. 26

D.6 Discussion

• Posterior anchor modeling combines broad-anchor structure with precise-anchor evidence to update the induced Bregman geometry, with richer mixtures enabling semantic transfer beyond projection heuristics.The models transfer a single global covariance, a shared low-dimensional subspace, or multiple reweighted semantic modes.Source: §D.6 · p. 27

E.1 Datasets

• Retrieval spans ImageNet, ImageNet-R, CUB-200, Sketchy, and CheXpert, with relevance and query/gallery splits defined for each task.The datasets include class-based, style-based, and pathology-based retrieval settings.Source: §E.1 · pp. 27–28

E.2 Models

• Natural-image experiments use MetaCLIP-2, while CheXpert uses CheXzero results averaged over 10 released model weights.Image and text embeddings are unit-normalized, with each model’s provided logit scale used as inverse temperature.Source: §E.2 · p. 28

E.3 Image-text inversion baseline

• Image-text inversion selects the anchor most compatible with a query, then ranks gallery images by cosine similarity to that single text proxy.The anchor set is used to choose the proxy, rather than to compare distributions induced over anchors.Source: §E.3 · pp. 28–29

E.4 Evaluation metric

• AUPRC is the area under the precision–recall curve traced as the retrieval-score threshold varies over ranked keys.The discrete implementation uses a finite-sum approximation over ranked thresholds.Source: §E.4 · p. 29

E.5 Parametric models and adaptation

• Gaussian anchors are fit by maximum likelihood, low-rank subspaces by top-k covariance eigenvectors, and Gaussian mixtures by expectation–maximization.Adaptation uses Normal–Inverse-Wishart, matrix-Bingham, posterior-like Bingham, or Dirichlet–NIW updates for the corresponding models.Source: §E.5 · pp. 29–30

E.6 CheXpert anchor generation

• CheXpert pathology anchors are generated as report-style positive and negative notes, embedded with CheXzero, and used as pathology-specific retrieval anchor sets.The prompt targets roughly 50 strings per condition.Source: §E.6 · p. 30

F.1 Parametric anchor models on CheXpert

• Gaussian anchor divergence performs comparably to softmax on average, while latent models improve over cosine on CheXpert.Table 3 reports pathology-wise mean AUPRC differences relative to cosine, averaged across independently generated anchor sets and CheXzero checkpoints, with 95% confidence intervals.Source: §F.1 · p. 30, Table 3 · p. 31
• Parametric anchor divergences are strongest across much of the limited- and moderate-anchor regime on ImageNet-R and CUB-200.Softmax improves as class-name anchor coverage increases and is strongest with the full class-anchor set.Source: Figure 6 · p. 31

F.2 Parametric models across anchor budgets on additional datasets

• Parametric anchor divergences are strongest with limited or moderate anchor coverage on ImageNet-R and CUB-200, while softmax becomes strongest with the full class-anchor set.The same qualitative anchor-coverage tradeoff was observed on ImageNet, so it is not specific to that dataset.Source: §F.2 · p. 31

F.3 Exploratory model-complexity diagnostic

• More mixture components tend to help on ImageNet, but gains are smaller and less consistent on CUB-200, where simpler geometries may capture much of the anchor variation.The diagnostic contrasts heterogeneous ImageNet class-name anchors with CUB-200 anchors concentrated around fine-grained bird categories; it is not a complete model-selection study.Source: §F.3 · pp. 31–32, Figure 7 · p. 32

F.4 Computational scaling with anchor-set size

• After fitting, parametric anchor models make query and gallery preprocessing and pairwise evaluation independent of the original anchor-set size.Direct softmax computation costs O(QGN + (Q + G)Nd); cached statistics reduce comparison to O((Q + G)Nd + QGd), leaving N-dependent preprocessing.Source: §F.4 · p. 32
• In timing tests with 10,000 queries and 50,000 gallery images on one NVIDIA H100 NVL GPU, softmax runtime increases with anchor count while Gaussian anchor divergence stays nearly constant after fitting.Runtimes include fitting, preprocessing, and retrieval, and are averaged over 10 runs after 3 warm-up runs.Source: §F.4 · p. 32, Table 4 · p. 33

F.5 Qualitative object–style retrieval examples

• Object anchors retrieve images sharing an object across styles, while style anchors retrieve images sharing a style across object categories, for both empirical softmax and parametric divergences.The examples suggest parametric models preserve the semantic coordinate selected by the anchors.Source: §F.5 · p. 33, Figure 8 · p. 34