PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Visual Representation Enhancement of VLMs via Kernel Canonical Correlation Analysis

Peilin Yang, Xiaoyu Liu, Jian Sun, Qinghua Tao

arXiv:2610.02718v1cs.CVcs.AIcs.LG

TL;DR

CLIP’s semantic generalization does not ensure fine-grained visual perception, while KUEA’s kernel-matrix alignment may be insufficient for improving it. The paper uses KCCA for subspace-correlation alignment, incorporating DINOv2 and frozen CLIP text projections in an efficient 3-view scheme. On MMVP-VLM, accuracy rises from 17.8 to 25.9 while strong zero-shot image–text retrieval is preserved.

  • Problem

    CLIP’s semantic generalization does not necessarily yield fine-grained visual perception, and KUEA’s element-wise kernel alignment may be insufficient to enhance it.

    Source: §1 · pp. 1–2

  • Method

    The method aligns feature-subspace projections through KCCA, adds frozen CLIP text projections as a third view alongside DINOv2, and derives an end-to-end scheme using KKT conditions.

    Source: §1 · p. 2, §3.2 · p. 4, §5 · p. 5

  • Results

    MMVP-VLM accuracy rises from 17.8 to 25.9 with 3vKCCA, while strong zero-shot image–text retrieval is preserved.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported ablations support KCCA alignment to DINOv2 and the use of frozen text semantics in the 3-view formulation for fine-grained visual enhancement.

    Source: §1 · p. 2

    The study is limited to CLIP–DINOv2 setups; other VLM backbones, vision-centric teachers, and broader perceptual benchmarks remain for future exploration.

    Source: §5 · p. 5

Abstract

from arXiv · show

Vision-language models such as CLIP exhibit strong semantic generalization, but remain limited in fine-grained visual perception. A recent work named KUEA presents a natural remedy by finetuning the image encoder under the supervision of the vision-centric DINOv2 to align their kernel matrices element-wisely, while regularizing the embeddings to remain close to the pretrained visual encoder for preserving image-text semantics in CLIP. However, we show that diminishing the role of the alignment loss to DINOv2 does not necessarily degrade its fine-grained visual performance, suggesting that the kernel-matrix discrepancy may be insufficient for further visual representation enhancement, motivating us to revisit the alignment formulation. In this work, we present a novel perspective to characterize representation alignment on feature subspaces through Kernel Canonical Correlation Analysis (KCCA), which maximizes the projection correlations. In optimization, we derive an efficient end-to-end training scheme upon KKT conditions, avoiding the eigenvalue problem in KCCA. Further, we extend our method into a 3-view formulation, i.e., 3vKCCA, in which the projections from the pretrained text encoder are also incorporated under a unified optimization framework for joint alignment. With CLIP ViT-L/14 on ImageNet-1K, our 3vKCCA improves the MMVP-VLM accuracy from 17.8 to 25.9, substantially outperforming the existing methods, and meanwhile maintains zero-shot image--text retrieval performance.

1 Introduction

CLIP’s strong semantic generalization does not ensure fine-grained visual perception, motivating work on visual enhancement that preserves image–text alignment. The paper revisits kernel alignment and proposes KCCA-based joint alignment with DINOv2 and the frozen CLIP text encoder.

Motivation

  • CLIP’s semantic generalization does not ensure fine-grained perception, including spatial relations that humans can readily distinguish.The challenge is to enhance visual representations without downgrading pretrained image–text alignment.Source: §1 · p. 1

Revisiting KUEA

  • In KUEA [9], removing kernel alignment leaves MMVP-VLM accuracy similar to the full objective, and further reducing alignment loss does not improve it.The observations suggest element-wise kernel alignment with DINOv2 can be insufficient for enhancing fine-grained visual capabilities.Source: §1 · pp. 1–2

Proposed perspective

  • The paper uses KCCA to align feature-subspace projections by maximizing correlations, then adds frozen CLIP text projections in a 3-view formulation.The joint framework aligns the finetuned visual encoder with DINOv2 while using the text encoder to preserve image–text semantics.Source: §1 · p. 2

Results

  • MMVP-VLM accuracy rises from 17.8 to 25.9 with 3vKCCA, while strong zero-shot image–text retrieval is preserved.The authors also report that 3-view KCCA consistently outperforms the 2-view alternative.Source: §1 · p. 2

2 Related Work

Prior work enhances fine-grained VLM representations through localized supervision, vision-centric teachers, and image-only finetuning. This paper’s context also includes KCCA and its multi-view extensions.

Enhancing Visual Representations of CLIP

  • Prior approaches use region–text supervision, detailed captions and annotations, DINO-derived localization, diffusion feedback, or kernel matching.Examples include RegionCLIP [25], FG-CLIP [22], CLIP-DINOiser, Talk2DINO [4], DIVA [20], and KUEA [9]; KALE [14] adaptively balances reweighted loss terms.Source: §2 · p. 2

Kernel CCA and Multi-View Extensions

  • CCA maximizes correlations between projections of two views, while KCCA extends this principle to nonlinear relations through kernel tricks.Deep CCA uses neural networks for nonlinear transformations [6], and prior work connects CCA to Barlow Twins [24] and VICReg [3].Source: §2 · p. 2
  • This work formulates KCCA over three functionally distinct views using LSSVM, with KKT stationary solutions supporting an unconstrained zero-value objective.The authors use this formulation for efficient end-to-end training.Source: §2 · p. 2

3 Method

The method aligns representations through correlations between feature-subspace projections rather than sample-wise distances. It extends KCCA to jointly use DINOv2 supervision and frozen CLIP text semantics, with a KKT-based end-to-end optimization scheme.

3 Method

  • KCCA characterizes alignment through maximal correlations between projected representations in feature subspaces, unlike sample-wise distance objectives.The method proposes an efficient end-to-end training scheme under KKT conditions.Source: §3 · p. 3

3.1 Aligning Subspace Projections with Kernel CCA

  • The KCCA formulation uses feature mappings and kernels for the finetuned CLIP image encoder and DINOv2, with projections into r-dimensional subspaces.Projection scores are maximally correlated in the optimization problem.Source: §3.1 · p. 3
  • 3-view KCCA jointly aligns the finetuned CLIP image encoder with DINOv2 and the frozen CLIP text encoder to consider visual enhancement and image–text semantics together.The text encoder supplies the third view in the KCCA framework.Source: §3.1 · p. 4

3.2 An End-to-End Optimization Scheme

  • KKT stationary solutions give a zero-value primal objective, enabling end-to-end minimization of the primal residual instead of repeatedly solving a dual eigenvalue problem.The method retains an ℓ2 regularizer that keeps finetuned visual embeddings close to pretrained CLIP embeddings; the resulting scheme avoids the KCCA eigenvalue problem.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate fine-grained visual perception and image–text retrieval, then test the roles of DINOv2 alignment and multi-view choices. 3vKCCA leads on MMVP-VLM while improving aggregate retrieval, and its gains vary with the alignment and third-view components.

4.1 Setups

  • Experiments fine-tune OpenAI CLIP ViT-L/14's visual encoder on ImageNet-1K, freeze its text encoder, and use DINOv2 ViT-L/14 supervision.MMVP-VLM [18] assesses fine-grained abilities, while Flickr30K and MSCOCO are used for zero-shot image–text retrieval.Source: §4.1 · p. 4

4.2 Results and Analysis

  • 25.9 vs 17.8 average MMVP-VLM accuracy, 2.9 and 5.2 above DIVA and KUEA, respectively; aggregate retrieval rises from 72.7 to 73.3.The main-results table reports fine-grained perception and image–text retrieval performances.Source: Table 1 · p. 5, §4.2 · p. 5
  • Removing KUEA's kernel-alignment loss leaves MMVP-VLM accuracy at 20.7; removing 3vKCCA's KCCA-alignment loss lowers it from 25.9 to 20.7.Shuffling CLIP–DINOv2 correspondences and replacing DINOv2 with a random teacher yield 21.5 and 20.0, respectively.Source: §4.2 · p. 5, Table 2 · p. 5
  • The 2-view KCCA setup raises MMVP accuracy from 20.7 to 23.0; adding frozen CLIP text embeddings raises it to 25.9, unlike pretrained image features, which leave it at 20.7.The comparison tests the contribution of the third view.Source: §4.2 · p. 5

5 Conclusion and Outlook

The conclusion presents 3vKCCA as a KCCA-based, three-view alignment method with an efficient end-to-end training scheme. The study is limited to CLIP–DINOv2 setups, leaving other backbones, teachers, and broader perceptual benchmarks for future investigation.

5 Conclusion and Outlook

  • 3vKCCA aligns feature-subspace representations by maximizing projection correlations, adds frozen CLIP text-encoder embeddings as a third view, and derives end-to-end training from Lagrangian duality and KKT conditions.The training scheme uses a necessary condition satisfied by stationary solutions.Source: §5 · p. 5
  • The study is limited to CLIP–DINOv2 setups; other VLM backbones, vision-centric teachers, and broader perceptual benchmarks remain future-work directions.The authors identify these as areas for further exploration.Source: §5 · p. 5

A Proof of Lemma 1: Zero-Value Property of the Three-View Primal

The appendix proves a zero-value property for stationary KKT solutions of the three-view primal and explains how this property supports an eigenvalue-free, end-to-end training loss.

A Proof of Lemma 1: Zero-Value Property of the Three-View Primal

  • The three views are CLIP image, DINOv2, and CLIP text, with centered features used to form each sample’s projection scores.The centering operation is absorbed into the feature representation.Source: A Proof of Lemma 1: Zero-Value… · p. 8
  • The proof derives stationarity conditions for projection matrices and scores, then combines the resulting relations with primal feasibility.It sums stationarity relations across views before substituting the result into the primal objective.Source: A Proof of Lemma 1: Zero-Value… · pp. 8–9
  • Every stationary solution satisfying the three-view primal’s KKT conditions has objective value zero.The paper uses this zero-value property to define J2, which it minimizes jointly with the regularization term.Source: A Proof of Lemma 1: Zero-Value… · p. 9
  • Minimizing J2 jointly with regularization avoids KCCA’s eigenvalue problem and keeps training end-to-end.The paper describes this setup as compatible with pretraining or post-training generic neural networks.Source: A Proof of Lemma 1: Zero-Value… · p. 9

B More Detailed Numerical Results

This appendix reports zero-shot retrieval and classification results to assess whether post-training preserves CLIP’s image–text semantics and capabilities. It also compares training-loss behavior for KUEA and 3vKCCA.

B More Detailed Numerical Results

  • 73.35 vs 72.66: 3vKCCA has the higher overall zero-shot retrieval average than original CLIP across Flickr30K and MSCOCO.The reported metrics are Recall@1/5/10 for image-to-text and text-to-image retrieval.Source: B More Detailed Numerical Results · p. 9, Table 4 · p. 10
  • 68.06 vs 68.08: 3vKCCA’s average zero-shot classification accuracy closely matches original CLIP across nine datasets.The passage says this nearly unchanged average indicates that 3vKCCA retains CLIP’s overall zero-shot classification performance.Source: Table 5 · p. 10, B More Detailed Numerical Results · p. 10
  • KUEA’s alignment loss oscillates without a clear downward trend, while 3vKCCA’s falls early and converges to a low, stable range.The KUEA trace is consistent with the ablation finding that removing its alignment term leaves MMVP performance unchanged.Source: B More Detailed Numerical Results · p. 10

C Implementation Details

The implementation normalizes and minibatch-centers each view’s encoder embeddings before projection, then stabilizes the visual projection matrices after optimizer updates. Training uses a residual-based alignment objective and fixed three-view settings.

C Implementation Details

  • Each view’s encoder embeddings are sample-wise ℓ2-normalized and minibatch-centered before projection by Wv.Centering is performed separately for each view.Source: C Implementation Details · p. 10
  • Figure 2 records alignment and regularization losses as loss_align and loss_clean, respectively, for KUEA and 3vKCCA.The left and right columns show alignment and regularization losses, respectively.Source: Figure 2 · p. 11
  • After each optimizer update, the visual projection columns are normalized to control canonical-direction scale; the text projection remains unconstrained.This keeps the two visual projection scales stable while retaining scale flexibility for cross-modal correlation optimization.Source: C Implementation Details · pp. 10–11
  • The alignment objective uses J^2 + ϵ, with ϵ = 10^-8; this stabilizes the loss and its gradient as J approaches zero while preserving J = 0 as the zero-residual target.Source: C Implementation Details · p. 11
  • Training uses ρ = 5.0, γV = γD = γT = 1.0, r = 8, AdamW at learning rate 10^-5, and weight decay 10^-4.The model is trained for 40,000 steps with batch size 64 and 2,800 warmup steps on 2 × RTX 5090 GPUs.Source: C Implementation Details · p. 11
Loading 2610.02718v1…