Source-linked AI summary
Reconsidering Representation Alignment for Multi-view Clustering
Daniel J. Trosten, Sigurd Løkse, Robert Jenssen, Michael Kampffmeyer
TL;DR
Naïve distribution alignment in deep multi-view clustering can reduce cluster separability and hinder view prioritization. The paper proposes SiMVC without alignment and CoMVC with selective contrastive alignment; experiments show that selective alignment improves performance over the baseline and prior methods on several datasets.
Problem
Distribution alignment can make clusters less separable and make it harder for models to prioritize views in multi-view clustering.
Method
The paper develops SiMVC, an alignment-free baseline, and CoMVC, which adds selective contrastive alignment while preserving view prioritization.
Results
Relative ACC gain exceeded 29% on E-MNIST when representations were not aligned versus adversarial alignment, while selective alignment always improved performance.
Takeaways & Limitations
Selective alignment combines the benefits of alignment with the ability to suppress views that cannot be reliably aligned or are uninformative.
Takeaways & Limitations
The model computes similarity directly on representations because adding a projection head did not improve its learned representations in this study.
Abstract
from arXiv · showhide
Aligning distributions of view representations is a core component of today's state of the art models for deep multi-view clustering. However, we identify several drawbacks with naïvely aligning representation distributions. We demonstrate that these drawbacks both lead to less separable clusters in the representation space, and inhibit the model's ability to prioritize views. Based on these observations, we develop a simple baseline model for deep multi-view clustering. Our baseline model avoids representation alignment altogether, while performing similar to, or better than, the current state of the art. We also expand our baseline model by adding a contrastive learning component. This introduces a selective alignment procedure that preserves the model's ability to prioritize views. Our experiments show that the contrastive learning component enhances the baseline model, improving on the current state of the art by a large margin on several datasets.
1. Introduction
Deep multi-view clustering learns a shared clustering structure from heterogeneous views, but adversarial representation alignment can reduce cluster separability and hinder view prioritization. The paper introduces alignment-free and selectively aligned alternatives.
- Multi-view clustering discovers common cluster structure by learning simultaneously from heterogeneous views such as visual, audible, image, and text data.
- Deep models transform each view with a view-specific encoder and fuse the resulting representations before clustering.
- Distribution alignment makes it harder to prioritize views because it pushes their representation distributions toward similarity despite unequal clustering importance.
- Adversarial alignment can make clusters less discriminative because clusters from different views may be aligned incorrectly.
- SiMVC avoids alignment and uses a learned linear combination that can suppress uninformative views and emphasize important ones.
- CoMVC adds selective contrastive alignment to SiMVC, aligning sample-level angles while allowing the model to ignore the objective when one-to-one alignment is unavailable.
2. Pitfalls of distribution alignment in multiview clustering
The paper formalizes how aligned representations can collapse distinct clusters when views contain incomplete clustering structure, then illustrates the effect on a controlled toy dataset. Alignment-free fusion can preserve separability under these conditions.
- Proposition 1: The proposition assumes cluster samples coincide within each view, views have k_v distinct input points, and encoded views are fused by a weighted linear combination.
- Proposition 1: The maximum number of unique fused clusters depends on whether representation distributions are perfectly aligned or left unaligned.
- Proposition 1: If a view groups all clusters together, alignment yields κ_fused aligned = 1 because every other view is aligned to that uninformative view.
- Proposition 1: When every view separates all k clusters, both aligned and unaligned models can perfectly cluster the data if the clustering module is sufficiently capable.
- Experiments on toy data: The toy dataset contains five elliptical clusters across two two-dimensional views, with overlapping class groups differing between views.
- Experiments on toy data: Adversarial alignment reduced visible fused clusters from 5 to 4 by preventing separation between clusters 1 and 4.
- Experiments on toy data: CoMVC aligned observation angles without causing class overlap, whereas EAMC produced a fused representation in which classes were hard to separate.
3. Methods
The methods encode each view, fuse representations with learned weights, and cluster the fused representation. CoMVC extends the alignment-free SiMVC baseline with selective sample-level contrastive learning.
- Simple Multi-View Clustering (SiMVC): Each view x_i^(v) is transformed by a view-specific encoder into representation z_i^(v), which are fused by a weighted average.
- Simple Multi-View Clustering (SiMVC): Trainable softmax-normalized fusion weights let SiMVC prioritize views without an auxiliary attention network.The weights are positive and sum to one, while the parameterization also improves efficiency.
- Simple Multi-View Clustering (SiMVC): The fused representation is passed through fully connected layers to produce hidden representations and k-dimensional soft cluster assignments.SiMVC minimizes a DDC-based clustering loss comprising separability and compactness, orthogonality, and simplex-corner terms.
- Contrastive Multi-View Clustering (CoMVC): The model computes contrastive similarity directly on representations because adding a projection head did not improve its linear classification accuracy.Comparisons with and without a projection head are reported in the supplementary material.
- Contrastive Multi-View Clustering (CoMVC): CoMVC adds a contrastive module to SiMVC, using cosine similarity to align angles between representations rather than the representations themselves.The generalized NT-Xent framework handles an arbitrary number of views, with negatives sampled from objects assigned to different clusters.
- Contrastive Multi-View Clustering (CoMVC): CoMVC weights the contrastive loss by the smallest fusion weight, automatically reducing its strength when the least-prioritized view receives low weight.The total objective is Lcluster + δ · min{w1, . . . , wV}Lcontrastive.
4. Related work
Related work spans traditional and deep multi-view clustering, with deep end-to-end methods adapting representation learning to the clustering objective. Distribution alignment has mainly been studied in supervised multimodal learning and domain adaptation outside this field.
- Traditional methods: Traditional methods learn common representations before clustering, factorize cluster assignments directly, or use subspace assumptions about the observations.
- Deep learning based methods: Deep two-stage methods use neural networks for common-representation learning, while deep end-to-end methods jointly adapt representation learning to clustering.Deep graph-based and subspace methods are additional deep-learning approaches.
- Distribution alignment: Distribution alignment has produced more efficient fusion techniques, but related work has largely focused on supervised multimodal learning and domain adaptation.
5. Experiments
Experiments evaluate the proposed models across several datasets, showing that avoiding naïve representation alignment improves clustering and that selective alignment strengthens performance while preserving view prioritization.
- 5.1. Setup: The evaluation covers VOC, CCV, E-MNIST, E-FMNIST, COIL-20, and RGB-D using comparisons with traditional and deep multi-view clustering baselines.The datasets include heterogeneous views such as images, edges, video features, object views, and image descriptions.
- 5.2. Results: Not aligning representations yields a relative ACC gain larger than 29% on E-MNIST compared with adversarial alignment, while selective alignment always improves performance.Table 2 reports results on VOC, CCV, and E-MNIST; E-MNIST entries for traditional approaches are omitted because computation is infeasible.
- 5.2. Results: Naïve feature alignment tends to worsen performance on E-FMNIST, COIL-20, and RGB-D compared with SiMVC and CoMVC.These comparisons are reported against the previous state-of-the-art method EAMC.
- 5.2. Results: Removing either adaptive weighting or negative pair sampling negatively affects CoMVC, supporting inclusion of both components in its contrastive loss.The ablation study evaluates CoMVC on E-MNIST and VOC.
- 5.2. Results: EAMC produces nearly uniform fusion weights, whereas SiMVC and CoMVC suppress uninformative views; on COIL-20, near-uniform weights are also observed where views are assumed equally important.Fusion weights are averaged over batches of size 100 for EAMC.
- 5.2. Results: With a noisy E-MNIST edge view, CoMVC sets that view’s fusion weight to 0, disables alignment, and preserves the fused cluster structure.In the noise-free case, CoMVC aligns representations while maintaining well-separated clusters.
6. Conclusion
The conclusion argues that representation alignment must be considered carefully in multi-view clustering. SiMVC shows that naïve adversarial alignment can harm clustering, while CoMVC uses selective alignment to retain its benefits and view prioritization.
- 6. Conclusion: Naïve adversarial alignment can prevent models from learning good clusterings, whereas CoMVC illustrates the benefit of selective alignment.The conclusion frames selective alignment as combining the advantages of alignment with view prioritization.
Supplementary material
The supplementary analysis formalizes how fusion of view representations can limit the number of distinct clusters. It contrasts perfectly aligned and unaligned representations under an idealized multi-view setting.
- Supplementary material: The analysis assumes that each view maps cluster-containing samples to representation-space points and studies the number of unique clusters after fusion.The setting is idealized to formalize observations about representation alignment.
- Supplementary material: Under the stated assumptions, fused clusters are bounded by the number of unique linear combinations of view-specific representation points.The bound is expressed through the product c1 · c2 · · · cV = QV v=1 cv and the dataset’s total cluster count k.
- Supplementary material: Perfect alignment forces each view’s number of unique representation points to equal the smallest cluster count among views.The relation is given as cv = min w=1,...,V {kw} for v = 1, . . . , V.
- Supplementary material: Without alignment, each encoder can map its view’s separable clusters to distinct representations that do not coincide with representations from other views.In this case, the analysis gives cv = kv.
A.2. Experiments with toy data
Toy experiments show that representation alignment can produce different cluster-separability outcomes: CoMVC preserves separability, while EAMC mixes clusters after fusion. Adding a projection head does not improve CoMVC overall.
- SiMVC + Adv. partially aligns distributions, yet the reduced three-cluster experiment still permits separation after fusion.
- CoMVC aligns angles between representations and produces separable clusters after fusion.
- EAMC mixes all clusters together after fusion when attempting to align representation distributions.Approximately equal fusion weights can further reduce separability in the fused representation space.
- Some CoMVC configurations benefit marginally from a projection head, but it does not improve overall performance.The final CoMVC model therefore omits the projection head.
B.2. Ablation study: clustering loss
The clustering-loss ablation shows that retaining all loss terms gives the best performance for both SiMVC and CoMVC. The surrounding experimental setup specifies the datasets, architectures, and projection-head comparison used in this evaluation.
- Ablation study: clustering loss: The ablation study evaluates SiMVC and CoMVC by systematically dropping terms from the DDC clustering loss.Models are selected using the sum of the included terms when the L1 term is absent.
- Ablation study: clustering loss: All clustering-loss terms produce the best performance for both SiMVC and CoMVC on E-MNIST.Dropping terms reduces performance in both models.
- Pre-trained features: The experiments use pre-trained ResNet-50 features for RGB-D View 1 and pre-trained Doc2Vec features for View 2.
- Projection-head ablation: Table 6 compares CoMVC configurations with and without a two-layer projection head before cosine similarity.The projection head uses fully connected layers, ReLU nonlinearities, and batch normalization.
- Model architectures: SiMVC and CoMVC use fully connected encoders on VOC, CCV, and RGB-D, and convolutional encoders on E-MNIST, E-FMNIST, and COIL.The clustering module is shared across SiMVC and CoMVC experiments.
C.4. Evaluation protocol
The evaluation compares the proposed models with prior methods using reported baseline results and repeated-run procedures. On NUS-WIDE-Animal, SiMVC is comparable to additional competitors while CoMVC outperforms them.
- Evaluation protocol: For VOC, CCV, and E-MNIST, baseline results are taken from under its reported evaluation procedures.Most baseline models are run 10 times and averaged over ACC and NMI, while EAMC reports its lowest-loss run from 20 trainings.
- Model architecture: The clustering module architecture uses k to denote the number of clusters.
- NUS-WIDE-Animal comparison: Table 10 compares SiMVC and CoMVC with two additional models on NUS-WIDE-Animal, a subset of NUS-WIDE containing animal classes.
- NUS-WIDE-Animal comparison: SiMVC performs comparably to the NUS-WIDE-Animal competitors, while CoMVC outperforms them.
C.6. Training times
Training-time comparisons show that SiMVC and CoMVC are substantially faster per epoch than EAMC. SiMVC is slightly faster than CoMVC because CoMVC adds contrastive-loss computations.
- Training times: SiMVC and CoMVC are both more than twice as fast per training epoch as EAMC.The authors attribute this difference to EAMC’s additional attention-network and discriminator components.
- Training times: SiMVC trains slightly faster per epoch than CoMVC.CoMVC incurs extra computations from the contrastive loss.
- Training times: Table 11 reports average training time per epoch for EAMC, SiMVC, and CoMVC on E-FMNIST.