PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multimodal Subspace Clustering Networks

Mahdi Abavisani, Vishal M. Patel

arXiv:1804.06498v3cs.LGcs.AIcs.CVstat.ML

TL;DR

The paper addresses the lack of deep multimodal methods specifically designed for unsupervised subspace clustering. It proposes CNN-based spatial- and affinity-fusion networks with encoder, self-expressive, and decoder components, and reports state-of-the-art results across conducted experiments.

  • Problem

    Deep multimodal learning had improved supervised tasks, but no method was specifically designed for unsupervised subspace clustering.

  • Method

    CNN-based networks combine multimodal encoders, latent-space self-expressive layers, and multimodal decoders using spatial or shared-affinity fusion.

  • Results

    The proposed affinity-fusion method provides state-of-the-art results in all conducted experiments.

  • Takeaways & Limitations

    The study supports shared-affinity fusion as an effective approach for deep multimodal subspace clustering within the evaluated experiments.

Abstract

from arXiv · show

We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representation. The self-expressive layer is responsible for enforcing the self-expressiveness property and acquiring an affinity matrix corresponding to the data points. The decoder reconstructs the original input data. The network uses the distance between the decoder's reconstruction and the original input in its training. We investigate early, late and intermediate fusion techniques and propose three different encoders corresponding to them for spatial fusion. The self-expressive layers and multimodal decoders are essentially the same for different spatial fusion-based approaches. In addition to various spatial fusion-based methods, an affinity fusion-based network is also proposed in which the self-expressive layer corresponding to different modalities is enforced to be the same. Extensive experiments on three datasets show that the proposed methods significantly outperform the state-of-the-art multimodal subspace clustering methods.

I. INTRODUCTION

The paper introduces a CNN-based framework for unsupervised multimodal subspace clustering that encodes self-expressiveness in latent representations. It investigates spatial fusion designs and affinity fusion for combining modalities.

  • Motivation and framework: The framework uses a multimodal encoder, self-expressive layer, and multimodal decoder as its three principal blocks.The self-expressive layer enforces self-expressiveness and the decoder reconstructs the original data.
  • Motivation and framework: High-dimensional multimodal data are addressed through a CNN-based autoencoder with a fully connected layer that mimics self-expressiveness.The decoder reconstructs the original input, while the self-expressive layer supports affinity-matrix construction.
  • Fusion strategies: Three encoder architectures investigate late, early, and intermediate spatial fusion of multimodal data.These designs fuse modality representations across spatial positions.
  • Fusion strategies: Affinity fusion enforces the same self-expressive-layer weights across latent representations of different modalities.The spatial and affinity fusion approaches are trained with end-to-end objective losses.
  • Contributions: The paper reports state-of-the-art results across multimodal subspace-clustering datasets.The introduction presents this as a principal outcome of the proposed methods.

II. RELATED WORK

Subspace clustering seeks to partition signals drawn from a union of linear subspaces by exploiting self-expressiveness. Related methods range from classical sparse or low-rank formulations to deep encoder-decoder networks with trainable self-expressive layers.

  • Classical subspace clustering: Subspace clustering partitions signals from a union of linear subspaces into sub-matrices associated with individual subspaces.The data are modeled as signals in R^D drawn from n subspaces with dimensions d_ℓ.
  • Classical subspace clustering: SSC and LRR exploit self-expressiveness, representing each data point as a linear combination of other data points.SSC seeks sparse representations, whereas LRR uses low-rank representations.
  • Classical subspace clustering: The classical optimization uses regularization and, for SSC, diag(C) = 0 to prevent the trivial solution C = I.The regularization norm is the ℓ1-norm for SSC and the nuclear norm for LRR.
  • Classical subspace clustering: After estimating C, spectral clustering operates on the affinity matrix W = |C| + |C|T to segment the data.This affinity construction follows the representation-learning stage of SSC or LRR.
  • Related extensions: Non-linear extensions of SSC and LRR have also been proposed for subspace clustering.
  • Deep subspace clustering: DSC embeds data in a latent space using an encoder-decoder network and approximates C with a trainable dense self-expressive layer.Its parameters include encoder, decoder, and self-expressive-layer parameters, with two regularization parameters in the objective.

C. Multimodal Subspace Clustering

Multimodal subspace clustering combines complementary views to cluster data according to subspaces, but prior deep multimodal learning largely focused on supervised tasks. The paper addresses the lack of a deep multimodal method specifically designed for unsupervised subspace clustering.

  • Existing multimodal approaches include kernel-based methods, multiview subspace learning, tensor-constrained clustering, and diversity-induced formulations.
  • Deep multimodal learning has reported improvements mainly in supervised tasks such as classification and recognition.
  • There was no deep multimodal learning method specifically designed for unsupervised subspace clustering.

III. SPATIAL FUSION-BASED DEEP MULTIMODAL SUBSPACE CLUSTERING

The spatial fusion framework merges multimodal feature maps into a joint representation before self-expressive clustering and reconstructs each modality through separate decoder branches. It studies early, late, and intermediate fusion using sum, maxpooling, and concatenation functions.

  • Spatial fusion produces a joint representation containing complementary information with spatial correspondence across modalities.
  • The framework uses an encoder, fully connected self-expressive layer, and decoder, with the fused representation passed through the self-expressive layer.
  • For M modalities, the decoder has M branches, each reconstructing one modality.
  • Early fusion integrates modalities before network processing, late fusion combines them at the final stage, and intermediate fusion combines intermediate feature maps.
  • Fusion functions operate on corresponding feature-map locations and include sum, maxpooling, and concatenation.

C. End-to-End Training Objective

The spatial fusion models are trained end-to-end using network parameters for the encoder, self-expressive layer, and decoder, together with reconstruction and regularization terms.

  • The spatial fusion models use encoder, self-expressive-layer, and decoder parameters in a unified training objective.
  • The joint representation is produced by the multimodal encoder, while the decoder reconstructs each modality from the learned representation.
  • The objective includes two regularization parameters, λ1 and λ2, and uses either an ℓ1 or ℓ2 norm.

IV. AFFINITY FUSION-BASED DEEP MULTIMODAL SUBSPACE CLUSTERING

Affinity fusion combines modalities through a shared affinity structure rather than spatially aligned feature fusion. Separate encoder-decoder networks retain modality-specific latent representations while sharing the self-expressive layer.

  • Affinity fusion combines similarities across modalities by enforcing a joint affinity matrix.
  • Unlike spatial fusion, affinity fusion avoids requiring aligned samples or increasing fused-output dimensionality through concatenation.
  • The modalities have different encoders, decoders, and latent representations but a common self-expressive-layer parameter Θs.
  • The affinity-fusion model stacks M parallel encoder-decoder networks that share one self-expressive layer.
  • After training, the method extracts Θs, normalizes its columns, forms a similarity graph, and applies spectral clustering.

B. End-to-End Training

The experiments evaluate multimodal subspace clustering across several datasets, fusion structures, baselines, and training settings. They use end-to-end networks with shared or modality-specific components and assess clustering using ACC, NMI, and ARI.

  • Experimental setup: The experiments cover multiview digits, heterogeneous faces, facial regions, and multiple spatial and affinity fusion configurations.The evaluation includes MNIST-USPS digits, ARL faces, and Extended Yale-B facial components, with comparisons across proposed fusion structures and baselines.
  • Baselines: The study compares proposed multimodal methods with classical SSC and LRR, DSC, AE+SSC, and single-modality results.AE+SSC isolates the contribution of end-to-end deep learning by clustering encoder-decoder features with SSC.
  • Training protocol: All methods use four-layer encoders and three-layer deconvolutional decoders under a common experimental protocol.Inputs are resized to 32×32, and the networks are pretrained before self-expressive-layer training.
  • Affinity fusion: The affinity-fusion model uses separate encoder-decoder networks for each modality connected through a shared self-expressive layer.For ARL with five modalities, the model contains five encoders and decoders linked by one shared self-expressive layer.
  • Training protocol: The self-expressive-layer stage uses full-batch training with dataset-specific batch sizes of 2000, 2160, and 2432 for digits, ARL faces, and Yale-B components.Pretraining instead uses a batch size of 100, while the self-expressive stage feeds all data points as a batch.
  • Evaluation: Clustering performance is evaluated using accuracy rate, normalized mutual information, and Adjusted Rand Index.The experiments use ACC, NMI, and ARI as the reported clustering metrics.

A. Handwritten Digits

The MNIST-USPS experiments treat corresponding handwritten-digit images as two spatially related but unaligned modalities. Multimodal fusion generally improves over unimodal clustering, with affinity fusion and late-addition achieving the strongest reported digit results.

  • Dataset and protocol: The digit experiment uses 10 MNIST and USPS classes, randomly sampling 200 images per class to reduce computation and balance the tests.Samples from the two datasets are bundled by class and treated as two views of the same digit.
  • Unimodal results: MNIST is easier than USPS for single-modality subspace clustering across the evaluated methods.The paper reports that this pattern agrees with results from other methods.
  • Multimodal results: Most multimodal methods outperform their unimodal counterparts by integrating complementary information from the two digit modalities.The comparison reports performance using accuracy, NMI, and ARI.
  • Multimodal results: 95.15% accuracy is achieved by affinity fusion and late-addition, with NMI and ARI both above 90%.These are the strongest digit results highlighted in the multimodal comparison.

B. ARL Heterogeneous Face Dataset

The ARL dataset contains five well-aligned facial modalities from 60 individuals across visible and polarimetric thermal domains. Multimodal methods generally improve clustering, and affinity fusion achieves the best performance in the reported comparison.

  • Dataset: ARL contains facial images from 60 individuals across visible and four polarimetric thermal modalities.Each subject has several well-aligned facial images in every modality.
  • Unimodal results: Because ARL modalities are well aligned, most subspace clustering methods perform well across the different spectrums.LRR is reported as comparable to DSC because it benefits from aligned data points.
  • Fusion design: The intermediate fusion design integrates S0, S1, and S2 in one layer, combines them with DP in another, and then fuses the result with visible features.The visible domain is treated as the main modality for this structure.
  • Multimodal results: Affinity fusion has the best performance, while late-concat and early-concat provide comparable results.Most deep multimodal methods leverage complementary information and outperform the corresponding unimodal results.

C. Facial Components

On Extended Yale-B facial components, weak individual modalities perform poorly, while affinity fusion avoids reliance on spatial correspondence and achieves strong multimodal clustering performance.

  • Nose and mouth modalities fail to provide good clustering results, whereas DSC and AE+SSC perform well on eye and entire-face modalities.
  • Spatial fusion: Intermediate fusion combines the two eyes, mouth, and nose before fusing these features with face features.
  • Spatial fusion: Late-mpool and intermediate-mpool fail because max pooling assumes spatial correspondence that the modalities do not share.
  • Spatial fusion: Additive fusion performs better with intermediate fusion, whereas concatenation works better with late fusion.
  • Affinity matrices: Late-concat and affinity fusion produce solid block-diagonal affinity matrices, unlike the failing late-mpool method.

D. Convergence study

The affinity fusion method converges in a few iterations and remains effective across a wide range of regularization parameters, while very small nonconvex norms can destabilize optimization.

  • Convergence: The affinity fusion algorithm converges in a few iterations while its objective function and clustering metrics are monitored across training.
  • Parameter sensitivity: Across a wide range of λ1 and λ2 values, the method’s final performance is not sensitive to the regularization-parameter choices.
  • Parameter sensitivity: The experimental setting suggested in performs well in all experiments.
  • Regularization norms: p = 1, p = 1.5, and p = 2 yield comparable clustering performance, whereas p = 0.3 is insufficient.
  • Regularization norms: For 0.3 < p < 1, experiments show instability, and for p < 0.3, minimizing the objective does not converge.

APPENDIX: NETWORK ARCHITECTURES

The appendix lists the network architectures used for digits and ARL experiments, including affinity, early, late, and intermediate spatial-fusion networks and multimodal decoders.

  • Architecture details: The experimental convolutional layers use ReLU activations.
  • Digits experiments: The digits experiments include affinity-fusion, early-fusion, late-fusion, and multimodal-decoder architecture tables.
  • ARL experiments: The ARL experiments include early-fusion, late-fusion, and intermediate spatial-fusion network architecture tables.
Loading 1804.06498v3…