PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Multi-View Representation Learning

Yingming Li, Ming Yang, Zhongfei Zhang

arXiv:1610.01206v5cs.LGcs.CVcs.IR

TL;DR

Learning useful associations across multiple data views remains an open problem. This survey organizes methods into alignment and fusion, reviews representative theories and models, and discusses applications to clarify available approaches.

  • Problem

    Learning a good association between multiple data views remains an open problem despite extensive algorithmic activity.

  • Method

    The survey categorizes approaches as representation alignment or fusion and reviews methods spanning CCA extensions, generative models, and neural networks.

  • Results

    The survey synthesizes representative theories, methods, and applications across alignment and fusion, including sparse coding, autoencoders, CNNs, and RNNs.

  • Takeaways & Limitations

    The alignment–fusion framework provides an overview for understanding multi-view representation learning approaches and selecting tools for particular applications.

  • Takeaways & Limitations

    CCA ignores nonlinearities in multi-view data, while KCCA's closed-form solution does not scale to large training sets because of time complexity.

Abstract

from arXiv · show

Recently, multi-view representation learning has become a rapidly growing direction in machine learning and data mining areas. This paper introduces two categories for multi-view representation learning: multi-view representation alignment and multi-view representation fusion. Consequently, we first review the representative methods and theories of multi-view representation learning based on the perspective of alignment, such as correlation-based alignment. Representative examples are canonical correlation analysis (CCA) and its several extensions. Then from the perspective of representation fusion we investigate the advancement of multi-view representation learning that ranges from generative methods including multi-modal topic learning, multi-view sparse coding, and multi-view latent space Markov networks, to neural network-based methods including multi-modal autoencoders, multi-view convolutional neural networks, and multi-modal recurrent neural networks. Further, we also investigate several important applications of multi-view representation learning. Overall, this survey aims to provide an insightful overview of theoretical foundation and state-of-the-art developments in the field of multi-view representation learning and to help researchers find the most appropriate tools for particular applications.

1 INTRODUCTION

This survey organizes multi-view representation learning into alignment and fusion, both of which exploit complementary knowledge across views to represent data comprehensively. It reviews theoretical foundations, representative models, applications, and challenges to guide method selection for particular tasks.

  • Motivation: Multi-view representation learning learns features from multi-view data that facilitate extracting useful information for prediction models.Such data may combine modalities including image and text, or audio and video.
  • Alignment perspective: It reviews alignment through distance-based, similarity-based, and correlation-based approaches, with CCA and kernel CCA extensions representing early techniques.Alignment maps each view and regularizes the resulting representations with constraints.
  • Challenges: The survey addresses an open problem in learning good associations among multi-view data amid growing research activity and numerous proposed algorithms.It also frames the field around challenges including noisy or missing inputs, unsuitable objectives, scalability, and view disagreement.
  • Survey taxonomy: The survey’s taxonomy comprises alignment, which captures relationships among views, and fusion, which combines separate view features into one compact representation.Both strategies seek to exploit complementary knowledge contained in multiple views.
  • Scope and contribution: Unlike broader multi-view learning surveys, this work formulates multi-view embeddings through alignment and fusion perspectives to clarify joint representation learning.Its stated goal is to provide a global view of theoretical foundations and key advances while helping researchers choose approaches for particular applications.

2 A TAXONOMY ON MULTI-VIEW REPRESENTATION LEARNING

The survey organizes multi-view representation learning into multi-view representation alignment and multi-view representation fusion. Alignment jointly relates view-specific embeddings under distance, similarity, or correlation constraints, whereas fusion integrates multiple views into a single representation using graphical models or neural networks.

  • Taxonomy: The survey focuses on jointly optimized mid-level, view-specific embeddings and divides existing methods into representation alignment and representation fusion.The two categories reflect whether views are related through constraints or integrated into a shared representation.
  • Multi-view representation alignment: Alignment learns view-specific embedding functions that map original data into a common space under distance, similarity, or correlation constraints.Examples include CFA, correspondence autoencoders, visual-semantic embeddings, and cross-modal alignment models.
  • Multi-view representation alignment: Correlation-based alignment maximizes correlations between projections of different views, with CCA and deep CCA providing linear and deep nonlinear mappings, respectively.The learned basis vectors or mappings produce embeddings for two-view data.
  • Multi-view representation fusion: Fusion integrates complementary information from multiple views into a single representation and includes graphical model-based and neural network-based approaches.Generative methods learn latent variables over observed multiview data, while neural models fuse learned feature maps or representations.
  • Multi-view representation fusion: Multi-view convolutional networks fuse tied-convolution feature maps using sum, max, or concatenation operations.The feature maps are fused at the convolution layer rather than at the softmax layer.

3 MULTI-VIEW REPRESENTATION ALIGNMENT

Multi-view representation alignment methods align representations learned from multiple views. The section reviews correlation-based techniques, including canonical correlation analysis and extensions from traditional models to nonlinear deep embeddings, alongside distance- and similarity-based alignment.

  • 3 MULTI-VIEW REPRESENTATION ALIGNMENT: Multi-view representation alignment aligns representations learned from multiple different views through correlation-based or distance- and similarity-based techniques.The section organizes representative methods around these two alignment aspects.
  • Correlation-based alignment: Correlation-based alignment includes canonical correlation analysis and extensions spanning traditional modeling to nonlinear deep embedding.CCA and its extensions are reviewed first within the alignment methods.

3.1 Correlation-based Alignment

Correlation-based alignment learns shared multi-view representations by maximizing cross-view correlations, with CCA providing the foundational linear formulation. Sparse CCA addresses variable selection and regularization, while KCCA extends CCA to nonlinear data through kernelized feature spaces.

  • Canonical Correlation Analysis: CCA learns a shared embedding of two or more variable sets by maximizing correlations among the views.It is widely used to generate low-dimensional multi-view representations.
  • Canonical Correlation Analysis: For paired datasets X and Y, CCA finds linear projections wx and wy that make corresponding examples maximally correlated in the projected space.Because correlation is invariant to scaling, CCA can be posed as a constrained optimization problem whose solution is obtained through generalized eigenvalue problems.
  • Kernel CCA: CCA’s linear embedding cannot capture all properties of multi-view data with nonlinearities, motivating nonlinear extensions such as KCCA.KCCA introduces nonlinearity by embedding data into higher-dimensional feature spaces associated with reproducing kernel Hilbert spaces.
  • Sparse CCA: Sparse CCA finds linear combinations of wx and wy with prescribed cardinality that maximize correlation.Its motivations include identifying crucial variables and providing regularization, and many approaches use the LASSO shrinkage-and-selection method.
  • Kernel CCA: KCCA replaces the original CCA projections with kernel-induced functions and uses corresponding kernel covariance matrices in its objective.Regularization is introduced because the unregularized optimization can produce degenerate solutions when either kernel matrix is invertible.

3.2 Distance and Similarity-based Alignment

Distance- and similarity-based alignment methods learn shared representations by modeling covariance, semantic ranking, cross-modal similarity, or deep cross-view correspondences. The section reviews PLS and CFA, ranking models such as SSI, multimodal hashing, and deep cross-view embeddings.

  • 3.2 Distance and Similarity-based Alignment: The section organizes distance- and similarity-based alignment around partial least squares, cross-modal ranking, cross-modal hashing, and deep cross-view embedding models.
  • 3.2 Distance and Similarity-based Alignment: PLS assumes observations are driven by a small number of latent variables and creates orthogonal latent vectors by maximizing covariance between views.Unlike CCA, which finds directions of maximum correlation, PLS finds directions of maximum covariance.
  • 3.2 Distance and Similarity-based Alignment: Cross-modal factor analysis is a least-square form of PLS that finds orthogonal transformations, with extensions incorporating non-linearity and supervised information.
  • 3.2 Distance and Similarity-based Alignment: SSI learns nonlinear multimodal scoring functions that rank relevant images above irrelevant ones using a margin ranking loss, while stochastic training improves scalability.The model maps text-image pairs to ranking scores based on semantic relevance and has improvements including low-rank representations.
  • 3.2 Distance and Similarity-based Alignment: Multimodal hashing maps high-dimensional data into common binary codes, while deep cross-view embeddings support applications such as cross-media retrieval and multimodal semantic learning.CMSSH preserves cross-modality similarity through Hamming embeddings, whereas CVH additionally preserves intra-view similarity.

4 MULTI-VIEW REPRESENTATION FUSION

Multi-view representation fusion integrates multiview inputs into a single compact representation. The section reviews fusion methods through graphical models and neural network-based models, covering generative techniques and deep joint representations.

  • Multi-view representation fusion integrates multiview inputs into a single compact representation.
  • Fusion methods are reviewed from two perspectives: graphical models and neural network-based models.
  • The section first surveys generative multi-view representation learning techniques, then examines neural network-based fusion methods and the expressive power of deep multi-view joint representation.

4.1 Graphical Model-based Representation Fusion

Graphical-model-based representation fusion includes generative approaches based on multi-modal LDA, multi-view sparse coding, latent-space Markov networks, and deep Boltzmann machines. These methods fuse views through shared latent variables, probabilistic correspondences, sparse codes, or graph-based dependencies.

  • Multi-modal LDA: Corr-LDA jointly reduces image-region descriptions and caption words while modeling conditional correspondence between their reduced representations.It generates region features from latent topics and caption words conditioned on the corresponding latent region topic.
  • Multi-modal LDA: Corr-LDA represents each image-caption pair with region features, caption words, latent topic variables, and discrete word-to-region indexing variables.The indexing variables are sampled uniformly from the image regions, while words are generated from the selected region’s latent factor.
  • Multi-modal LDA: Exact inference for Corr-LDA is intractable, so variational inference approximates the posterior over latent variables for an image-caption pair.Supervised multi-modal LDA extensions additionally model discriminative information such as image class labels and local annotation terms.
  • Multi-view sparse coding: Multi-view sparse coding learns a shared sparse representation through view-specific dictionaries, selecting appropriate bases while zeroing others.Dictionary columns are usually constrained to have unit norm, and expectation-maximization can alternate dictionary and representation learning in probabilistic formulations.
  • Multi-view latent-space Markov networks: Multi-view latent-space Markov networks couple multiple modalities through shared hidden units, with multi-wing harmonium models providing an undirected formulation.Extensions include large-margin models under weak conditional independence assumptions and multi-modal distance metric learning.

4.2 Neural Network-based Representation Fusion

Neural network-based multi-view representation fusion includes multi-modal deep autoencoders, multi-view convolutional neural networks, and multi-modal recurrent neural networks. These methods learn shared or compact representations by reconstructing multimodal inputs, pooling multiple views, or encoding and decoding multimodal sequences.

  • The section reviews multi-modal deep autoencoders, multi-view convolutional neural networks, and multi-modal recurrent neural networks.
  • Multi-modal deep autoencoders: Autoencoders learn latent representations by encoding inputs and decoding them to minimize reconstruction error, while bimodal architectures fuse audio and video into a shared representation.The fused representation models relationships between modalities and can be trained with modality-specific reconstruction losses.
  • Multi-modal deep autoencoders: After training on bimodal inputs, autoencoders can use one modality to recover missing data from another, supporting camera-depth fusion for scene understanding.
  • Multi-view convolutional neural networks: Multi-view CNNs process separate views with shared convolutions, fuse them by view pooling, and produce a compact representation for tasks such as 3D object recognition.The view-pooling layer performs an element-wise maximum across views, acting like data augmentation.
  • Multi-view convolutional neural networks: For video action recognition, fusing spatial and temporal CNN representations at a convolutional layer saves parameters without loss of performance compared with softmax-layer fusion.The convolutional fusion captures informative spatio-temporal information and learns channel correspondences between feature maps.
  • Multi-modal recurrent neural networks: RNN encoder-decoder models connect multimodal sequences by encoding a variable-length source into a fixed-length vector and decoding it into a variable-length target sequence.

5 APPLICATIONS

Multi-view representation learning exploits complementary views to produce informative, compact representations that improve predictor performance. Its applications span cross-media retrieval, NLP, video analysis, and recommender systems.

  • Cross-media retrieval: CCA and its extensions, including KCCA, support label-free cross-media retrieval by modeling relationships between textual and visual modalities.KCCA has been used to retrieve images from multiple text queries and map textual words to visual words.
  • Cross-media retrieval: Deep multi-view models learn expressive cross-view representations for cross-media retrieval by mapping images and queries into a common vector space.The mappings use convolutional and query-embedding components.
  • Natural language processing: Multi-view representation learning improves NLP by combining linguistic and visual information for semantic tasks and by fusing sequential representations in encoder-decoder models.Applications include semantic relatedness, compositional prediction, machine translation, text summarization, and dialogue systems.
  • Video analysis: Video analysis applications fuse representations across time steps into sequential representations for action recognition, temporal action detection, and video captioning.Approaches include convolutional temporal pooling, sequence-to-sequence learning, and recurrent convolutional networks.
  • Recommender systems: Recommender systems encode ratings alongside user and item content, using joint representation learning to improve generalization performance.Examples combine topic modeling or stacked denoising autoencoders with probabilistic matrix factorization.

6 CONCLUSION

The paper organizes multi-view representation learning into two major categories: representation alignment and representation fusion. It reviews representative methods and theories from the alignment perspective, then investigates advances from the fusion perspective.

  • The paper defines two major categories of multi-view representation learning: representation alignment and representation fusion.
  • It reviews representative methods and theories for multi-view representation learning from the alignment perspective.
  • It investigates advances in multi-view representation learning from the fusion perspective.
Loading 1610.01206v5…