PaperlayerView exact revision on arXiv

Source-linked AI summary

Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models

Lauren Hyoseo Yoon, Yisong Yue, Been Kim

arXiv:2507.01201v7cs.LGcs.CV

TL;DR

Independently trained vision and language models may share broad representational structure, but existing evidence is largely observational and coarse, leaving fine-grained alignment as a key challenge. JAM jointly trains modality-specific autoencoders to align frozen models, and the paper reports that Spread Loss outperforms classic contrastive baselines while JAM can rival or outperform generalist architectures in specialist settings.

  • Problem

    Most evidence for alignment is observational and coarse, while fine-grained distinctions between descriptions with shared semantics remain a central challenge.

    Source: §1 · pp. 1–2, §1.3 · p. 2

  • Method

    JAM aligns frozen, independently trained vision and language models through modality-specific autoencoders with coordinated reconstruction and cross-modal alignment objectives.

    Source: §1.2 · p. 2, §3.1 · p. 6

  • Results

    Spread consistently outperforms contrastive baselines across tasks, and JAM can rival or outperform generalist architectures in specialist settings.

    Source: §4.1 · p. 7, §5 · pp. 9–10

  • Takeaways & Limitations

    JAM offers a practical recipe for building specialist multimodal models on frozen unimodal foundations through lightweight post-hoc adaptation and structured supervision.

    Source: §5 · p. 9

    Future work includes testing domains requiring highly specific reasoning, larger datasets and models, and more than two modalities.

    Source: §5 · p. 10

Abstract

from arXiv · show

Independently trained vision and language models inhabit disjoint representational spaces, shaped by their respective modalities, objectives, and architectures. The Platonic Representation Hypothesis (PRH) suggests these models may nonetheless converge toward a shared statistical model of reality. This raises a fundamental question: can we move beyond post-hoc detection of such alignment and explicitly optimize for it? We argue this challenge is most critical in fine-grained contextual distinctions-where multiple descriptions share global semantics but differ in subtle compositional details. We address this with the Joint Autoencoder Modulator (JAM), which aligns frozen unimodal models by jointly training modality-specific autoencoders with coordinated reconstruction and cross-modal alignment objectives. We systematically evaluate JAM across three design axes: (i) alignment objectives, introducing our multimodal Spread Loss that outperforms classic contrastive methods; (ii) the layer depth at which alignment is most effective; and (iii) the role of foundation model scale in representational convergence. Our findings show that JAM reliably induces alignment even across independently trained representations, offering both theoretical insight into the structure of shared semantics and practical guidance for transforming generalist unimodal foundations into specialist multimodal models.

1 Introduction

The introduction asks whether independently trained vision and language models can be explicitly aligned, especially where descriptions share global semantics but differ in fine-grained details. It presents JAM as a post-hoc approach and positions it against diagnostic tests and models trained with multimodal objectives.

1 Introduction

  • Fine-grained alignment is difficult because descriptions can share global semantics while differing in compositional details, beyond what coarse observational metrics diagnose.The paper contrasts coarse judgments, such as whether an image contains a dog, with distinctions such as the color of the ball being chased.Source: §1 · pp. 1–2

1.1 Formulation of Plato’s Cave

  • The paper frames unimodal representations as shadowed projections and defines context as shared high-level semantics, with fine-grained understanding resolving distinctions within it.Source: §1.1 · p. 2

1.2 Escaping Plato’s Cave (Platonic Alignment)

  • JAM aligns frozen, independently trained vision and language models through coordinated reconstruction and alignment, while Spread Loss uses contextual structure for fine-grained alignment.The introduction says Spread Loss outperforms classic contrastive objectives for fine-grained alignment.Source: §1.2 · p. 2

1.3 Related Work

  • Prior tests passively measure feature correlations, while paired-corpus models impose multimodal objectives; JAM probes post-hoc alignment from truly unimodal foundations.The related-work discussion distinguishes JAM from systems whose vision encoders already inherit a multimodal alignment objective.Source: §1.3 · pp. 2–3

2 Statistical Tests for Representation Alignment

The section tests whether statistical alignment metrics distinguish true image–caption matches from unrelated and fine-grained hard negatives. They reveal broad compatibility between independently trained models, but do not reliably diagnose fine-grained distinctions.

2 Statistical Tests for Representation Alignment

  • The evaluation applies CCA, CKA, SVCCA, and CKNNA to image–caption matches, unrelated captions, and semantically similar captions with fine-grained differences.The hard negatives use localized compositional changes to test sensitivity beyond coarse similarity.Source: §2 · p. 3
  • The experiments use unimodally pretrained vision and language backbones with SugarCrepe, extracting CLS-token image and penultimate-layer text embeddings.SugarCrepe supplies controlled Replace, Swap, and Add caption perturbations.Source: §2 · p. 3

2.1 Statistical tests detect alignment, but not fine-grained context.

  • Across metrics, match pairs separate from easy non-matches, but hard non-matches score nearly like true matches, limiting diagnosis to coarse compatibility.The authors say hard negatives share global semantics with positives and differ in fine-grained attributes.Source: §2.1 · pp. 3–4, Figure 2 · p. 4

3 Our method: Joint Autoencoder Modulator (JAM)

JAM jointly trains modality-specific autoencoders with reconstruction and alignment objectives. Its Spread Loss combines context-level grouping with fine-grained contrast to retain shared meaning while distinguishing local differences.

3 Our method: Joint Autoencoder Modulator (JAM)

  • JAM jointly trains autoencoders for alignment alongside a reconstruction objective.The described architecture uses three hidden layers of dimension 512 and a bottleneck of dimension 256.Source: §3.1 · p. 5, §3 · p. 5

3.1 Loss functions

  • NegCon adds hard-negative captions to image-to-text comparisons while leaving text-to-image matching unchanged.The loss is symmetric across directions, but asymmetric in negative usage because hard negatives lack associated images.Source: §3.1 · p. 5
  • Spread Loss pulls together the true caption and selected hard negatives, then separates fine-grained distinctions within that similar-context group.Its vision-to-language formulation weights context contrastive and contextNCE terms, with α = 0.5 in the experiments.Source: §3.1 · p. 6
  • Context Contrastive groups the positive caption and hard negative as similar, then pushes this group away from other text embeddings.This term provides global learning focused on what is easily differentiable.Source: §3.1 · p. 6
  • ContextNCE treats the true positive as the only correct text, keeping the similar-context hard negative in the denominator as incorrect.Together with Context Contrastive, it helps prevent representation collapse within similar-context embeddings.Source: §3.1 · p. 6

4 Experiments

Experiments evaluate JAM on fine-grained retrieval across SugarCrepe and Winoground and varied pretrained backbones. They compare alignment objectives, probe layer depth, and test whether language-model scale improves alignment.

4.1 Does JAM enable escaping Plato’s Cave?

  • Spread consistently outperforms contrastive baselines across tasks and backbones, and matches or surpasses several pretrained and finetuned CLIP variants.The reported comparisons include SugarCrepe and Winoground; Spread uses structured supervision to reflect local clusters and semantic proximity.Source: Table 2 · p. 7, §4.1 · pp. 7–8
  • JAM with Spread outperforms projection baselines, while removing reconstruction leads to rapid overfitting and sharply rising validation loss.The authors describe reconstruction as a regularizer preserving generalizable structure across modalities; they also characterize the post-hoc approach as lightweight and suitable for low-resource or specialized domains.Source: §4.1 · p. 8

4.2 What kind of supervision is useful for different layer depth?: Layer-Wise Probing via Curriculum Learning

  • Earlier layers require higher α for peak retrieval, while retrieval improves steadily from early to late layers.The curriculum varies α to balance broad semantic alignment and fine-grained contrast; later layers align effectively with weaker supervision.Source: Figure 5 · p. 8, Figure 6 · p. 8, §4.2 · p. 9

4.3 Does JAM performance scale with backbone models’ scale?

  • Scaling language models does not consistently improve performance, and gains from larger models are minimal in the tested setting.The authors suggest that 200–1000 image–text pairs limit the gradient signal available to larger models.Source: §4.3 · p. 9, Table 4 · p. 9

5 Conclusion / Future Directions

The authors present post-hoc joint autoencoding as evidence that independently trained unimodal representations can be aligned, and propose JAM as a recipe for specialist multimodal models. They identify broader domains, larger datasets and models, other alignment approaches, and more modalities as future directions.

5 Conclusion / Future Directions

  • JAM provides empirical evidence of representational convergence and a Pareto-efficient framework for building specialist multimodal models on frozen unimodal foundations.The authors report that lightweight adaptation and structured supervision can rival or outperform generalist architectures in specialist settings.Source: §5 · pp. 9–10
  • Post-hoc alignment with lightweight adaptation and structured supervision can rival or outperform generalist architectures in specialist settings.JAM is also proposed as a flexible testing ground for modality alignment under varying supervision regimes.Source: §5 · pp. 9–10
  • Future work includes specialized-reasoning domains, larger datasets and models, more sophisticated alignment approaches, and settings with more than two modalities.The authors also suggest comparing accuracy-efficiency tradeoffs against adapter-based and joint-training approaches at larger scales.Source: §5 · p. 10

A Impact Statement

JAM is presented as a lightweight post-hoc adaptor that aligns frozen unimodal models while preserving modality-specific information. Its contributions include multimodal Spread Loss and evaluation in specialist settings.

A Impact Statement

  • JAM aims to surface shared structure from independently trained vision and language representations while keeping pretrained unimodal backbones frozen.The framing invokes the Platonic Representation Hypothesis [1].Source: A Impact Statement · p. 12
  • JAM uses modality-specific autoencoders for reconstruction and a shared alignment objective, without end-to-end multimodal fine-tuning or pretraining.The autoencoders guard task-relevant features, while Spread Loss amplifies latent commonality.Source: A Impact Statement · p. 12
  • The work tests Platonic alignment in specialist tasks requiring fine-grained contextual understanding and reports that frozen backbones can rival inherently multimodal models.The authors present this as a potential alternative strategy for specialist multimodal systems.Source: A Impact Statement · p. 12

B Statistical Metrics to Measure Platonic Alignment

This section describes statistical measures used to assess alignment between vision and language representations, covering global and local structure as well as projection-based and fixed-space comparisons. It also specifies metric formulations and the supplemental task setup.

B.1 Thematic analysis of statistical representation alignment tests

  • The analysis uses CCA, SVCCA, CKA, and CKNNA, which are unsupervised, second-order metrics; CKNNA captures neighborhood-level similarity that global measures may obscure.The metrics are characterized by whether they learn projections or assess fixed representations, and whether they measure global or local structure.Source: §B.1 · p. 12
  • The metric comparison distinguishes data-driven projections, which estimate best-case alignment after adaptation, from canonical measures of inherent similarity in fixed representations.CCA and SVCCA are data-driven; CKA and CKNNA assess representations without transformation.Source: §B.1 · p. 12

B.2 Metrics Formulation

  • Paired image and text features are represented through kernels, then centered to remove mean effects before alignment calculations.The section introduces inner-product kernels for the two modalities and defines a centering operation.Source: §B.2 · p. 13

B.2.1 Centered-Kernel Alignment (CKA)

  • CKA normalizes HSIC between centered kernels and measures global geometric correspondence, but aggregating all pairwise relations can make it sensitive to a small misaligned region.The score is described as scale-invariant and bounded in [0, 1].Source: §B.2.1 · p. 13

B.2.2 Centered-Kernel k-NN Alignment (CKNNA)

  • CKNNA restricts alignment to pairs that are mutual k-nearest neighbors in both modalities, emphasizing local structure rather than all pairwise relations.As k approaches n, the formulation recovers CKA.Source: §B.2.2 · p. 14

B.2.3 CCA (Canonical Correlation Analysis)

  • CCA finds projections of the two feature matrices that maximize Pearson correlation, and reports the first canonical correlation as its alignment score.The section also describes PCA-assisted CCA to address high-dimensional feature spaces.Source: §B.2.3 · p. 14

B.2.4 SVCCA (Singular Vector Canonical Correlation Analysis)

  • SVCCA uses singular-value decomposition to retain components explaining a set proportion of variance, then runs linear CCA and averages the resulting correlations.The example retained variance proportion is 99%.Source: §B.2.4 · p. 15

B.3 Supplemental Results for Statistical Alignment Tests

  • The supplemental statistical alignment tests use Replace, Add, and Swap cases from Sugarcrepe, with reduced feature dimensions and k=10 for nearest neighbors.CCA uses dimension 50; SVCCA, CKA, and CKNNA use dimension 10.Source: §B.3 · p. 15

B.3.1 Replace Task

The Replace task distinguishes correct matches, easy non-matches, and hard non-matches that preserve broad context while changing a specific detail. A table is identified as reporting statistical alignment test results for this task.

B.3.1 Replace Task

  • Hard non-matches alter a specific attribute, object, or relation while retaining context similar to the correct text, requiring fine-grained discrimination.The setup also includes correct matches and easy non-matches with clear mismatches.Source: §B.3.1 · p. 15
  • Table 7 is identified as presenting statistical alignment test results for the Replace task.The supplied passage identifies the table's scope but provides no result values.Source: Table 7 · p. 15

B.3.2 Add task

The Add and Swap tasks define fine-grained mismatches, while the multimodal Spread Loss combines context-level alignment with intra-context contrast. The accompanying comparison maps these objectives to the visual-domain loss formulation.

B.3.2 Add task

  • The Add task treats an added attribute or object that creates an incorrect correspondence as a hard non-match, distinct from clear easy non-matches.The setup also includes correct matches.Source: §B.3.2 · p. 15

B.3.3 Swap task

  • The Swap task treats a specific object being swapped as a hard non-match, alongside correct matches and easy non-matches.Its hard negatives differ by the swapped object.Source: §B.3.3 · p. 16

C.1 Our Formulation of Spread Loss for Multimodal Representation Learning

  • The VL spread loss weights context-level alignment and intra-context contrast, and its batch objective averages per-example losses.The LV direction omits hard-negative texts because they have no correct corresponding images.Source: §C.1 · p. 16

C.2 Spread Loss Formulation in Visual Representation Learning Domain

  • The visual-domain Spread Loss combines supervised contrastive and class-conditional InfoNCE losses, clustering same-class samples while retaining instance-level attraction.The combination spreads same-class points while preserving attraction to augmentations.Source: §C.2 · pp. 16–17

C.3 Side-by-Side Comparison

  • The comparison maps visual classes to collections of captions sharing a visual context, with context clustering paired with a sharpening loss that contrasts a primary caption against hard negatives.The tables compare notation and each objective's anchor, positives, negatives, and motivation.Source: §C.3 · p. 17, Table 10 · p. 18, Table 11 · p. 18

D Model Analysis & Training Configuration

The experiments use specified pretrained backbones and a shared JAM architecture, trained with multiple alignment losses and evaluated across two retrieval settings.

D.1 Pretrained Unimodal Backbones

  • MAE uses ViT as its backbone, while Swav and Swin use RestNet50 backbones.These are among the pretrained unimodal models configured for the experiments.Source: Table 12 · p. 18

D.2 JAM (Joint Autoencoder Modulator) Framework

  • JAM uses three 512-dimensional hidden layers, a 256-dimensional latent layer, 0.1 dropout, batch size 32, and SwiGLU activation.Table 13 reports JAM framework analysis attached to pretrained backbones.Source: §D.2 · p. 18, Table 13 · p. 19
  • For each task, JAM trains with Lspread, Lcon, and LNegCon for 100 epochs on 70-15-15 splits, averaging recall scores across seeds 5, 42, and 55.Both autoencoders are optimized jointly with AdamW, gradient clipping, and cosine annealing; the reconstruction-loss weight decays from 1.0 to 0.1.Source: §D.2 · pp. 18–19
  • Validation image-to-text Recall@1 is checked every five epochs, with early stopping after five consecutive validations without improvement; evaluation uses binary and 5-way choices.The 5-way setting adds three distractors.Source: §D.2 · p. 19

E Further Results

Further results cover image-to-text retrieval with a wider set of pretrained backbones and Winoground text-score comparisons for JAM with Spread.

E Further Results

  • Table 14 reports image-to-text retrieval results for JAM on vision-language compositionality with a wider set of pretrained backbones.The passage provides the table's scope but no specific values or winner.Source: Table 14 · p. 19
  • JAM with Spread shows the best Winoground text-score performance compared with the pretrained CLIP baseline.Table 15 presents the comparison; the passage also notes that finetuning CLIP in a small-data regime decreases performance on Sugarcrepe.Source: Table 15 · p. 20

F Compute Systems/Resources

Experiments used Apple M2 and NVIDIA RTX 3080 systems, with feature extraction and some model runs assigned to specific hardware.

F Compute Systems/Resources

Loading 2507.01201v7…