Source-linked AI summary
SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis
Ehud Gordon, Meir Yossef Levi, Guy Gilboa
TL;DR
Vision-language concept explanations are limited by visual-only methods and a modality gap between image and text embeddings. SCoCCA combines CCA-based shared-space alignment with sparse concept decomposition, achieving strong reconstruction, purity, and editing results while generalizing beyond discovery data.
Problem
Existing concept-based explainability largely focuses on images, while CLIP-like image and text embeddings have mismatched distributions that limit multimodal concept analysis.
Method
SCoCCA aligns image and text embeddings with CCA and enforces sparse concept decomposition, using Lasso under a linear relationship between concepts and embeddings.
Results
SCoCCA achieves state-of-the-art performance in reconstruction, purity, and editing metrics, and its multimodal concept space supports retrieval and editing on out-of-distribution images.
Takeaways & Limitations
SCoCCA provides a training-free way to expose and control shared conceptual structure in vision-language models through decomposition and manipulation.
Takeaways & Limitations
The decomposition framework assumes a linear relationship between the concept matrix C and the embedding matrix X.
Abstract
from arXiv · showhide
Interpreting the internal reasoning of vision-language models is essential for deploying AI in safety-critical domains. Concept-based explainability provides a human-aligned lens by representing a model's behavior through semantically meaningful components. However, existing methods are largely restricted to images and overlook the cross-modal interactions. Text-image embeddings, such as those produced by CLIP, suffer from a modality gap, where visual and textual features follow distinct distributions, limiting interpretability. Canonical Correlation Analysis (CCA) offers a principled way to align features from different distributions, but has not been leveraged for multi-modal concept-level analysis. We show that the objectives of CCA and InfoNCE are closely related, such that optimizing CCA implicitly optimizes InfoNCE, providing a simple, training-free mechanism to enhance cross-modal alignment without affecting the pre-trained InfoNCE objective. Motivated by this observation, we couple concept-based explainability with CCA, introducing Concept CCA (CoCCA), a framework that aligns cross-modal embeddings while enabling interpretable concept decomposition. We further extend it and propose Sparse Concept CCA (SCoCCA), which enforces sparsity to produce more disentangled and discriminative concepts, facilitating improved activation, ablation, and semantic manipulation. Our approach generalizes concept-based explanations to multi-modal embeddings and achieves state-of-the-art performance in concept discovery, evidenced by reconstruction and manipulation tasks such as concept ablation.
1. Introduction
Existing concept-based explainability is largely visual, while multimodal embeddings face a modality gap that limits concept analysis. SCoCCA addresses this by combining CCA-based cross-modal alignment with sparse, interpretable concept decomposition.
- SCoCCA’s concept decomposition supports controllable manipulation by swapping concept magnitudes and recomposing embeddings for image synthesis.
- Multimodal concept explanations remain underdeveloped because existing concept-based methods largely focus on visual representations.
- CLIP image and text features follow distinct distributions and geometric structures, constraining interpretability and concept reconstruction quality.
- CCA aligns distinct observations by maximizing cross-modal correlation, but conventional CCA-based approaches do not address concept-level decomposition.
- The paper links CCA and InfoNCE, showing that optimizing CCA correlates with optimizing InfoNCE’s alignment component for CLIP-like networks.
- CoCCA unifies C-XAI with CCA, while SCoCCA adds sparsity to produce sharper, more discriminative concepts for activation, ablation, and swapping.
2. Related Work
Prior work develops interpretable concepts within individual modalities and increasingly explores text–image embeddings. However, multimodal methods commonly overlook the modality gap between image and text representations.
- Vision-only approaches discover or represent concepts through activation vectors, concept bottlenecks, clustering, factorization, projections, or rotations.
- SpLiCE decomposes CLIP vision embeddings into sparse additive mixtures of textual concepts, while other studies examine generic visual attributes and cross-modal concept alignment.
- CCA, deep CCA, and sparse CCA learn shared subspaces across modalities as part of multiview representation learning.
- Vision–language embeddings can occupy disjoint, non-isotropic distributions across modalities, creating a modality gap that dedicated concept methods typically neglect.
3. Method
The method aligns paired image–text embeddings with CCA to discover interpretable concept directions, then decomposes unseen image embeddings into sparse concept combinations. It also supports semantic concept matching and invertible manipulation through activation modification, ablation, swapping, and recomposition.
- Concept Decomposition Framework: Concept decomposition represents embeddings with a dictionary of Concept Activation Vectors and coefficient matrices, targeting reconstruction, sparsity, and semantic purity.Purity is evaluated through concept ablation, concept swapping, and linear-probe performance.
- CCA and InfoNCE Alignment: CCA seeks orthogonal linear projections of image and text embeddings that maximize cross-modal correlation, equivalently aligning whitened observations.The method uses centered embeddings and whitening transformations before relating CCA to InfoNCE.
- CCA and InfoNCE Alignment: The alignment term of InfoNCE on whitened embeddings is proportional to the CCA objective, allowing closed-form CCA optimization to enhance pretrained alignment without additional training.This connection motivates CCA as an analytical alternative to another training phase.
- Concept Discovery and Decomposition: Concept discovery constructs an interpretable dictionary once from paired embeddings, while decomposition applies Lasso to express each unseen image embedding as a sparse combination of learned concepts.The sparse coefficients support disentangled representations and downstream manipulation.
- Concept CCA (CoCCA): CoCCA learns k-dimensional image and text projections that maximize correlation under orthogonality constraints, producing aligned concept directions through a closed-form solution.The projections are obtained using covariance whitening and SVD, after which image projections form the concept bank.
- Concept Matching: Concept directions are semantically grounded by matching them to labeled class prototypes through cosine similarity and a one-to-one Hungarian assignment.Class prototypes are computed from centered image embeddings grouped by their labels.
4. Experiments
SCoCCA is evaluated through concept discovery, semantic purity and editing, sparsity, reconstruction, and cross-dataset generalization experiments. It consistently achieves strong performance, including high semantic control, near-perfect reconstruction, and improved accuracy under moderate sparsity.
- Evaluation setup: The evaluation measures purity, editing, sparsity, reconstruction accuracy, and concept-retrieval generalization across multiple experiments.The experiments use CLIP embeddings and primarily evaluate a subset of 500 ImageNet classes, with additional MSCOCO retrieval testing.
- Overall results: Table 1 reports that SCoCCA achieves state-of-the-art or comparable reconstruction, purity, and editing performance while retaining CLIP-level accuracy.The comparison covers dual-modality and single-modality concept decomposition methods on 500 ImageNet classes.
- Purity and concept editing: SCoCCA achieves the highest residual cosine similarity of 0.76, target probability gain of 0.95, and source probability drop of 0.87.These results indicate that reconstructed embeddings preserve fine-grained semantics while concept activation edits affect associated class predictions.
- Reconstruction: SCoCCA attains cosine reconstruction similarity of 0.99 and relative ℓ2 error of 0.02, while reaching zero-shot accuracy of 0.74 and precision@5 of 0.85.The reconstruction results are reported as on par with or surpassing the CLIP baseline for discriminative performance.
- Generalization: A concept bank calibrated on ImageNet retrieves MSCOCO images with clear Microwave and Traffic Light evidence, demonstrating concept-retrieval generalization.The retrieval compares the top four images ranked by concept activation and shows stronger semantic evidence for SCoCCA than for some other methods.
- Sparsity: Increasing λ produces fewer active concepts, and the best performance occurs when roughly half of the concept weights are zeroed out.The ablation identifies moderate sparsity as the strongest balance between sparsity and reconstruction terms.
5. Conclusion
SCoCCA combines CCA-based cross-modal alignment with sparse concept-based explainability for vision-language representations. The method supports precise concept manipulation, strong reconstruction and editing performance, and generalization beyond the discovery data.
- Method: SCoCCA discovers a shared latent subspace between image and text embeddings while enforcing sparsity for interpretable multimodal concepts.Its CCA formulation derives shared Concept Activation Vectors and implicitly enhances the alignment term of InfoNCE without additional training.
- Results: The method achieves state-of-the-art reconstruction, purity, and editing performance while producing faithful and controllable concept representations.The conclusion attributes these outcomes to extensive experiments and reports precise concept decomposition and manipulation.
- Generalization: The learned multimodal concept space generalizes beyond discovery data to retrieval and editing on out-of-distribution images.This supports exposing and controlling conceptual structure in vision-language models within the demonstrated scope.
6. Ablation of k hyperparameter
For SCoCCA on ImageNet-500, zero-shot accuracy increases monotonically with the number of concepts, while each additional concept yields a smaller classification gain.
- Zero-shot accuracy rises monotonically as SCoCCA computes more concepts.More concepts improve separation on the ImageNet-500 test set.
- The accuracy curve becomes less steep at larger k values.Additional concepts provide diminishing returns for classification.
7. Comparison on CLIP B-32 model
On a 500-class ImageNet subset using CLIP B-32, decomposition methods show trends similar to Table 1, with improved reconstruction but reduced purity at k = 500.
- At k = 500, reconstruction metrics improve for the smaller CLIP B-32 model.The comparison uses a subset of 500 random ImageNet classes.
- At k = 500, Purity metrics decrease for CLIP B-32 relative to the corresponding comparison.The table reports trends similar to Table 1.
- Table 2 compares concept decomposition methods on 500 random ImageNet classes using CLIP B-32.Best results are bolded and second-best results are underlined.
8. Derivation of CCA via whitening and SVD
The CCA derivation whitens paired embeddings, converts the objective into orthogonal trace maximization, and solves it through the singular value decomposition of the whitened cross-covariance.
- Whitening: CCA starts from centered paired samples X and Y with empirical covariance and cross-covariance matrices.The derivation assumes positive-definite covariance matrices so their symmetric inverse square roots are defined.
- CCA formulation: The target dimension k determines the number of canonical directions selected by CCA.CCA solves the constrained optimization problem for k dimensions.
- Whitening: Whitening transforms the CCA constraints into orthonormality constraints on the transformed projection matrices.The resulting columns satisfy A⊤A = Ik and B⊤B = Ik.
- Orthogonal reduction: The whitened cross-covariance matrix reduces the CCA objective to an orthogonal trace-maximization problem.This reformulation isolates cross-modal correlation after covariance normalization.
- SVD solution: The solution uses the leading singular vectors of the whitened cross-covariance matrix.Selecting the first k columns yields the maximum possible objective value, up to a common orthogonal rotation.
- SVD solution: Back-substitution produces closed-form CCA projection matrices from the whitening factors and singular-vector matrices.The closed-form projections follow after solving the orthogonal problem.
9. Methods
The methods section reviews rotation-, dictionary-, clustering-, factorization-, and supervision-based concept decompositions used as comparison methods.
- Varimax: Varimax applies an orthogonal rotation after k-truncated PCA to concentrate loadings for interpretability.The resulting concept bank is C = VkR, with coefficients W = (UkDk)R.
- SpLiCE: SpLiCE uses a CLIP text-embedding dictionary built from frequent LAION-400m caption bigrams and dataset class names.It decomposes image embeddings by solving a LASSO problem, varying λ to control sparsity and reconstruction.
- K-Means: The k-means baseline uses image-embedding centroids as concept vectors and assigns each new image to its nearest centroid.Each coefficient vector is one-hot for the selected centroid.
- NMF: NMF shifts image embeddings to nonnegative values, solves the factorization with multiplicative updates, and reverses the shift for reconstruction.The method uses sklearn’s multiplicative-updates solver.
- TCAV: TCAV learns a concept activation vector by linearly separating positive concept examples from randomly sampled negative images.The separating hyperplane’s normal defines the vector for the concept.