PaperlayerView exact revision on arXiv

Source-linked AI summary

RankAlign: Vision-Language Representation Alignment via Rank Transformation

Enzhe Zhao, Marco Fiorucci, Lamberto Ballan

arXiv:2610.31145v1cs.CV

TL;DR

Class-level unpaired vision-language matching asks whether unknown correspondences can be recovered without cross-modal supervision, even when similarity values use different numerical parameterizations. RankAlign compares rank-based relational representations before permutation matching, retaining the matching framework. The paper reports more stable and reliable correspondence alignment across diverse encoders and datasets, with exact-recovery and stability guarantees under stated ordinal conditions.

  • Problem

    Class-level unpaired vision-language matching seeks unknown visual-to-textual category correspondences without paired supervision, while direct kernel comparisons can be unreliable when modalities use different numerical parameterizations.

    Source: §1 · p. 2

  • Method

    RankAlign replaces within-modality kernel values with ordinal rank representations before permutation matching, without changing the correspondence space or solver.

    Source: §1 · p. 2, §4 · p. 10

  • Results

    Experiments across diverse vision and language encoders and datasets show more stable and reliable cross-modal correspondence alignment with RankAlign.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Ordinal correspondence recovery can be studied without rigid metric equivalence, using shared relational order while retaining the existing matching framework.

    Source: §1 · p. 2, §4 · p. 10

    Value-based relational alignment requires cross-modal numerical calibration, which independently trained encoders are not guaranteed to provide.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

While the Platonic Representation Hypothesis posits that vision and language models converge toward compatible latent geometries, recovering class-level correspondences between unpaired modalities without cross-modal supervision remains challenging. Existing relational matching methods compare within-modality similarity kernels directly, although the same semantic structure may be represented through different numerical parameterizations across modalities. We propose RankAlign, a rank-based normalization strategy for class-level unpaired vision-language matching. RankAlign transforms each within-modality similarity kernel into an ordinal relational representation before permutation matching, while leaving the correspondence space and matching solver unchanged. The resulting representation is invariant to strictly increasing reparameterizations of similarity values while preserving the ordinal structure used for matching. We provide exact-recovery and stability guarantees that characterize correspondence recovery under both exact and partial ordinal agreement. Experiments across diverse frozen vision and language encoders and multiple datasets demonstrate improved and more stable class-level correspondence recovery, including under label corruption and domain shift.

1 Introduction

Class-level unpaired vision-language matching seeks correspondences between known visual and textual categories without paired supervision, but direct kernel-value comparisons can fail when modalities encode similar relational structures with different numerical scales. RankAlign compares ordinal relations instead, preserving the matching framework while improving correspondence recovery across encoders and datasets.

1 Introduction

  • Without paired supervision, class-level matching must recover unknown visual-to-textual category correspondences from within-modality structure alone.The setting has known within-modality groups but no paired image–text samples, cross-modal anchors, or verified correspondences.Source: §1 · p. 2
  • Direct numerical kernel comparisons can be weakly discriminative when modalities agree on similarity ordering but assign different values to corresponding relations.Value-based measures are sensitive to modality-specific scale distortions and biases.Source: §1 · p. 2
  • RankAlign compares within-modality relational order rather than raw similarity values, leaving the correspondence problem unchanged while removing sensitivity to strictly increasing reparameterizations.The same matching objective can use the transformed kernels.Source: §1 · p. 2
  • Experiments across diverse vision and language encoders and datasets show that RankAlign achieves more stable and reliable cross-modal correspondence alignment.The introduction identifies empirical validation across multiple encoders and datasets.Source: §1 · p. 3

2 Method

RankAlign replaces numerical kernel values with ordinal relational representations before permutation matching, while retaining the correspondence space and solver. The paper characterizes when ordinal agreement enables exact recovery, how partial disagreement affects stability, and the preprocessing cost.

2 Method

  • RankAlign transforms within-modality similarity kernels into rank-based kernels, then applies permutation matching without changing the matching space or solver.The pipeline refines ordinal representations using local neighbor order before matching.Source: §2 · p. 3, §2.3 · p. 5

2.1 Preliminary

  • Blind relational matching represents candidate cross-modal correspondences as permutations and optimizes a quadratic assignment objective over the two within-modality kernels.The objective can be written as a maximization over permutations.Source: §2.1 · pp. 3–4

2.2 Limitations of value-based relational alignment

  • Value-based matching compares numerical kernel entries directly, implicitly requiring cross-modal calibration that independently trained encoders are not guaranteed to provide.Different modality-specific parameterizations can penalize a correct correspondence despite shared underlying relational structure.Source: §2.2 · p. 4

2.3 Rank-based relational alignment

  • Rank transformation replaces pairwise similarity spacing with relative positions, preserving strict ordering and ties while removing strictly increasing value reparameterizations.The correspondence problem itself remains unchanged.Source: §2.3 · p. 5

2.4 Recovery guarantees

  • Under strict agreement in relational order and N ≥3, the ground-truth correspondence is the unique maximizer of the ordinal matching objective.The guarantee depends on ordering rather than numerical similarity values.Source: §2.4 · p. 5
  • Under partial ordinal disagreement, the stability result separates cross-modal residual disagreement from how distinguishable the relational structure is under an incorrect relabeling.The representation-independent bound is stated for any candidate permutation.Source: §2.4 · p. 6

2.5 Computational complexity

  • RankAlign adds O(N^2 log N) time and O(N^2) memory for preprocessing, excluding feature extraction and the chosen QAP solver.Kernel construction already costs O(N^2d) time; the solver's running time remains a separate term.Source: §2.5 · p. 6

3 Experiments

Experiments evaluate frozen encoders and class-level matching across diverse datasets, reporting performance under increasing problem size, visual-label corruption, noisy pseudo-classes, and medical-domain shift.

3 Experiments

  • Frozen encoders and class prototypes support matching withheld correspondences across four datasets, with comparisons against same-solver baselines and normalization ablations.The evaluated encoders span diverse visual and textual models, and matching uses within-modality relational structure.Source: §3 · pp. 6–7

3.1 Computational Cost

  • With the same solver, RankAlign adds modest runtime overhead while substantially improving accuracy; FHG hits its runtime limit at larger sizes.The results identify permutation optimization, rather than rank transformation, as the main scalability bottleneck.Source: Table 1 · p. 7, §3.1 · p. 7

3.2 Scalability

  • At N = 100, RankAlign retains approximately 30%–60% matching accuracy across representative backbones, while BLINDMATCH(GW) falls to near-chance levels.The improvement is roughly 45%–70% at the largest evaluated problem size; selected class subsets favor cross-modal relational agreement.Source: §3.2 · pp. 7–8, Figure 2 · p. 8

3.3 Robustness to label corruption

  • On CIFAR-100 and ImageNet-100, matching accuracy declines gradually as visual-label corruption rises, with reasonable performance under moderate noise.The reported comparison uses matching accuracy and relative performance drop against the clean setting.Source: §3.3 · p. 9, Table 2 · p. 9

3.4 LAION Pseudo-Class Matching

  • On LAION pseudo-classes, RankAlign outperforms BLINDMATCH(GW) across evaluated visual backbones, with gains not tied to one backbone family.The class definitions derive from weakly supervised image–text signals rather than manually curated labels.Source: Table 3 · p. 9, §3.4 · p. 9

3.5 Medical Domain Evaluation

  • On CheXpert, RankAlign achieves substantially higher matching accuracy than BLINDMATCH(GW) across evaluated backbones under pronounced domain shift.The evaluation includes subtle inter-class medical distinctions and class imbalance.Source: Table 4 · p. 10, §3.5 · p. 10

4 Conclusions

The conclusion presents RankAlign as an ordinal alternative to magnitude-based correspondence matching and identifies prototype quality, computational scaling, and task scope as limitations.

4 Conclusions

  • RankAlign changes the relational representation rather than the permutation-matching framework, using relational order to support correspondence recovery without paired supervision.The authors argue that the findings support cross-modal convergence through shared relational topologies rather than rigid metric equivalence.Source: §4 · p. 10
  • RankAlign depends on informative class prototypes; noisy clustering can reduce ordinal agreement, matching grows costly with class count, and instance-level and open-vocabulary alignment remain future work.The method intentionally discards absolute similarity magnitudes; hybrid ordinal–magnitude alignment is also left for future work.Source: §4 · p. 10

A Technical appendices and supplementary material

  • The appendices provide theoretical proofs and additional empirical analyses, including recovery guarantees and computational-complexity details.They also describe the full ordinal refinement used in experiments.Source: A Technical appendices and… · p. 13

A.1 Properties of the ordinal representation

Rank-based representation preserves the kernel’s ordinal relations while removing dependence on strictly increasing changes to similarity values. The section also relates the matching distortion objective to an agreement score and describes an ordinal refinement with the same invariance properties.

A.1 Properties of the ordinal representation

  • For standardized kernels, minimizing relational distortion is equivalent to maximizing agreement, which is also equivalent to maximizing correlation between relational structures.The distortion is Dℓ(Kx, Ky; π) = 2N(N −1)(1 −ρ(π)).Source: §A.1 · p. 13
  • Strictly increasing transformations preserve pairwise rank order, and ties in the original kernel remain ties rather than creating new distinctions.With distinct off-diagonal values, ranks run from 1/M to 1.Source: §A.1 · p. 14
  • The practical refinement T(K) combines kernel-level and anchor-wise ordinal information, then standardizes the components before aggregation.It remains invariant to strictly increasing reparameterizations and equivariant to class permutations.Source: §A.1 · p. 14

A.2 Recovery guarantees and approximate optimization

The section establishes exact recovery under strict ordinal agreement and analyzes stability when agreement is partial or optimization is approximate. It also describes RankAlign’s computational overhead and compatibility with existing QAP solvers.

A.2 Recovery guarantees and approximate optimization

  • Under strict ordinal consistency and N ≥ 3, the ground-truth correspondence uniquely maximizes the rank objective, with an unordered-pair margin of at least 1/M^2.The ordered off-diagonal objective has margin at least 2/M^2.Source: §A.2 · p. 15
  • Stability separates cross-modal disagreement from relational-profile distinguishability, while approximate optimization adds a solver objective deficit as a third difficulty.The analysis identifies these sources as ∥EZ∥F, κ(Zx), and η.Source: §A.2 · p. 16

A.3 Computational complexity

  • RankAlign’s preprocessing adds O(N^2 log N) time and O(N^2) memory beyond kernel construction, while leaving the QAP input structure unchanged.The complete pipeline takes CS(N) + O(N^2d + N^2 log N) time.Source: §A.3 · p. 17

B Additional experiments

This section introduces supplementary experimental results as support for the main paper.

B Additional experiments

B.1 Datasets

The evaluation spans benchmarks with different data sources and semantic settings, including class-count scalability tests, pseudo-classes, and medical imaging.

B.1 Datasets

  • RankAlign is evaluated across web-scale pseudo-label data, object-recognition datasets at different semantic granularities, and a medical-imaging dataset.Source: §B.1 · p. 17
  • CIFAR-100 and ImageNet-100 subsets test matching accuracy as the number of classes varies, assessing performance under increasing task difficulty.Source: §B.1 · p. 17
  • The LAION-Aesthetic pseudo-class dataset uses 20 visual categories and about 7,000 samples, with 200–500 images per class.Samples are retained when original and auxiliary captions yield a unique, consistent top-scoring assignment.Source: §B.1 · p. 17
  • The CheXpert evaluation uses a 12-class single-label subset of about 62,000 radiographs, selected from a 220,000-image benchmark.The setting includes subtle inter-class differences, class imbalance, and a medical image-text domain gap.Source: §B.1 · p. 17

B.2 Kernel Transformation Ablation

The CIFAR-100 ablation isolates kernel transformation while holding prototypes, encoders, class subsets, kernels, and solvers fixed. RankAlign outperforms generic normalization and local ranking across problem sizes.

B.2 Kernel Transformation Ablation

  • The ablation changes only the relational representation supplied to the solver, keeping prototypes, class subsets, kernels, encoders, and permutation solvers identical.It is conducted on CIFAR-100.Source: §B.2 · p. 17
  • Compared variants are the raw kernel, z-score normalization, row-wise ranking, quantile normalization, and RankAlign.Z-score preserves relative spacing; row-wise ranking captures local neighbor order; quantile normalization aligns marginal value distributions.Source: §B.2 · p. 17
  • RankAlign achieves the highest matching accuracy across all problem sizes, combining kernel-level ordinal normalization with anchor-wise neighborhood order.Its advantage over generic normalization or local ranking alone persists as problem size grows.Source: §B.2 · pp. 17–18
  • The ablation table covers kernel transformations on CIFAR-100, with other experimental components held fixed.Source: Table 5 · p. 18

B.3 Comparison with Classical Structural Alignment Methods

On CIFAR-100, RankAlign substantially outperforms classical structural alignment baselines across evaluated backbones and problem sizes. The comparison uses the same class-level representations and matching instances.

B.3 Comparison with Classical Structural Alignment Methods

  • All methods use the same class-level visual and language representations and CIFAR-100 matching instances, isolating differences in alignment objectives.Source: §B.3 · p. 18
  • Neither Orthogonal Procrustes nor vanilla GW-OT approaches RankAlign across evaluated backbones and problem sizes; both remain near chance as N increases.Table 6 reports matching accuracy across problem sizes.Source: §B.3 · p. 18, Table 6 · p. 18
  • The results indicate that global linear alignment and direct matching of original metric structures are insufficient for these experiments.The comparison does not replace a comparably accurate simpler baseline.Source: §B.3 · p. 18

B.4 Full larger-scale matching results

Full larger-scale results show RankAlign outperforming BlindMatch with GW across backbones on CIFAR-100 and ImageNet-100. The supplement also identifies the public datasets and pretrained models used and commits to following their licenses and terms.

B.4 Full larger-scale matching results

  • Full results cover CLIP, DINOv2, DeiT, SigLIP, and SigLIP2, beyond the representative backbone subset shown in the main paper.Source: §B.4 · p. 18
  • Across CIFAR-100 and ImageNet-100, RankAlign outperforms BlindMatch with GW across backbone families and degrades more gradually as matched classes increase.Figure 3 plots matching accuracy against problem size, with color identifying the vision backbone and line style distinguishing methods.Source: §B.4 · p. 18, Figure 3 · p. 19

B.5 Existing Assets and Licenses

  • The evaluation uses existing public datasets and pretrained models, with licenses and terms of use to be followed and detailed in released materials.Listed assets include LAION-Aesthetic, CIFAR-100, ImageNet-100, CheXpert, and several pretrained encoders.Source: §B.5 · p. 19

Related papers

Loading 2610.31145v1…