PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Multi-view Learning

Chang Xu, Dacheng Tao, Chao Xu

arXiv:1304.5634v1cs.LG

TL;DR

Conventional methods often concatenate multiple views, risking overfitting and ignoring view-specific statistical properties. This survey classifies multi-view learning approaches and finds promising performance through consensus or complementary principles.

  • Problem

    Concatenating views can overfit small training samples and ignore each view’s specific statistical properties, motivating multi-view learning approaches.

  • Method

    The survey organizes multi-view learning into co-training, multiple kernel learning, and subspace learning, while reviewing view construction and evaluation.

  • Results

    The survey reports promising performance compared with single-view learning and finds that successful approaches mainly use consensus or complementary principles.

  • Takeaways & Limitations

    Multi-view learning is effective and promising in practice, but better processing of multi-view data remains needed across applications.

  • Takeaways & Limitations

    Future work should better construct, analyze, and evaluate views and develop a general framework integrating the three algorithmic groups.

Abstract

from arXiv · show

In recent years, a great many methods of learning from multi-view data by considering the diversity of different views have been proposed. These views may be obtained from multiple sources or different feature subsets. In trying to organize and highlight similarities and differences between the variety of multi-view learning approaches, we review a number of representative multi-view learning algorithms in different areas and classify them into three groups: 1) co-training, 2) multiple kernel learning, and 3) subspace learning. Notably, co-training style algorithms train alternately to maximize the mutual agreement on two distinct views of the data; multiple kernel learning algorithms exploit kernels that naturally correspond to different views and combine kernels either linearly or non-linearly to improve learning performance; and subspace learning algorithms aim to obtain a latent subspace shared by multiple views by assuming that the input views are generated from this latent subspace. Though there is significant variance in the approaches to integrating multiple views to improve learning performance, they mainly exploit either the consensus principle or the complementary principle to ensure the success of multi-view learning. Since accessing multiple views is the fundament of multi-view learning, with the exception of study on learning a model from multiple views, it is also valuable to study how to construct multiple views and how to evaluate these views. Overall, by exploring the consistency and complementary properties of different views, multi-view learning is rendered more effective, more promising, and has better generalization ability than single-view learning.

1. Introduction

Multi-view learning addresses heterogeneous data represented by distinct variable groups or information sources, avoiding the limitations of concatenating views into one representation. This survey reviews its main paradigms, view construction and evaluation, and relationships to related learning topics.

  • Motivation: Multi-view data arise when examples contain heterogeneous variable groups or information from diverse sources, including text, images, viewpoints, audio, video, and languages.Each variable group is treated as a particular view.
  • Motivation: Concatenating all views into one representation can overfit with small training samples and ignore each view’s specific statistical properties.Conventional methods include support vector machines, discriminant analysis, kernel machines, and spectral clustering.
  • Surveyed approaches: Co-training alternately trains learners on two distinct views to maximize their mutual agreement on unlabeled data.The survey also discusses variants combining co-training with expectation-maximization and active learning.
  • Surveyed approaches: Multiple kernel learning combines kernels corresponding to different views, either linearly or non-linearly, to improve learning performance.Its theoretical analyses include generalization bounds for convex combinations of base kernels and tighter bounds under uncorrelated views.
  • Surveyed approaches: Subspace learning seeks a lower-dimensional latent subspace shared by multiple views, supporting subsequent classification and clustering while reducing the curse of dimensionality.It assumes the input views are generated from the shared latent subspace.
  • Survey scope: The survey provides a comprehensive overview covering underlying principles, construction and evaluation of multiple views, view combination, and detailed multi-view learning algorithms.It also connects multi-view learning with active learning, ensemble learning, and domain adaptation.

2. Principles for Multi-view Learning

Multi-view learning benefits from multiple views only when their relationships are modeled appropriately, chiefly by maximizing consensus and exploiting complementary information. These principles explain why agreement can reduce errors while view diversity can improve learning.

  • Motivation: Multiple views provide abundant information, but mishandling their relationships can degrade multi-view learning performance.The contrast with single-view learning arises from the demand for redundant views and the need to cope with them appropriately.
  • Consensus principle: The consensus principle maximizes agreement across distinct views; under mild assumptions, disagreement between independent hypotheses upper-bounds either hypothesis’s error rate.Therefore, minimizing disagreement can minimize each hypothesis’s error rate.
  • Consensus principle: Co-training applies consensus by alternating between views to reduce labeled-data error and increase agreement on unlabeled examples.Other methods, including multi-view embedding, also enforce consistency while considering relationships among views.
  • Complementary principle: The complementary principle exploits knowledge present in one view but absent from others, enabling multiple views to describe data more comprehensively and accurately.Co-training can succeed without redundant views when learner diversity exceeds the amount of errors.
  • Complementary principle: Multiple kernel learning combines similarity notions from different kernels, while shared low-dimensional representations avoid over-fitting and preserve view-specific statistical properties better than concatenation.Kernel combination follows complementarity, whereas shared subspaces address the limitations of directly concatenating long feature vectors.
  • Combined principles: Both consensus and complementary principles should guide multi-view learning to fully exploit multiple views.Consensus reduces disagreement-related errors, while diversity supplies complementary information that influences co-training performance.

3. View Generation

View generation constructs multiple views that represent the data and satisfy multi-view learning assumptions, then evaluates their effectiveness and robustness to incompatibility or noise. Methods include random construction, feature reshaping or decomposition, and automatic feature partitioning.

  • View generation goals: View generation prioritizes acquiring redundant views while ensuring they represent the data and satisfy assumptions required for learning.It includes both constructing multiple views and evaluating them.
  • View construction: Feature set partitioning decomposes an original feature set into multiple disjoint subsets, with random splitting providing a simple single-view-to-multi-view conversion.This generalizes feature selection by producing multiple feature subsets rather than one representative set.
  • View construction: Random subspaces sample feature subsets to build multiple SVMs whose combination can reduce over-fitting, while matrix reshaping can save memory and improve classifier performance.These approaches exploit alternative feature-space representations as multiple views.
  • View construction: View construction methods fall into random approaches using metadata, reshaping or decomposing original features, and automatic feature set partitioning.Examples include matrix representations, different kernel functions, and algorithms such as PMC.
  • View evaluation: View evaluation analyzes relationships among views and addresses violations of view assumptions or noise that can undermine multi-view learning.View sufficiency may fail when low-level features do not adequately represent concepts, and kernels can reflect information sources corrupted by disparate noise.

4. View Combination

Section 4 surveys advanced strategies for combining multiple views, emphasizing co-training, multiple kernel learning, and shared-subspace learning over simple concatenation. These approaches integrate views through cross-view consistency, kernel combination, or a latent shared representation.

  • Motivation: Simple view concatenation can overfit small training samples and ignores the distinct statistical properties of individual views.The section therefore motivates more advanced combination methods to improve performance over single-view learning.
  • Co-training: Co-training algorithms train separate learners on distinct views and iteratively maximize agreement, allowing information to pass between views.Co-regularization formally measures cross-view agreement, while unsupervised variants validate view combinations on the training data.
  • Multiple kernel learning: Multiple kernel learning combines kernels associated with different similarities, representations, sources, or modalities through linear or nonlinear combination methods.Linear methods include direct and weighted summation, whereas nonlinear methods use products, exponentiation, powers, or polynomial combinations.
  • Multiple kernel learning: Kernel weights may be unrestricted, non-negative, convex, binary, or input-dependent through gating functions.Weighting schemes can therefore support kernel selection and local adaptation, rather than assigning one fixed weight across the whole input space.
  • Multiple kernel learning: Nonlinear kernel combinations provide richer representations, but empirical results do not show consistent performance improvement, questioning their necessity or efficiency.The section contrasts this result with the broader motivation for combining multiple kernels beyond linear combinations.
  • Subspace learning: Subspace learning assumes that multiple views are generated from a shared latent subspace, with CCA serving as a multi-view counterpart to PCA.The section also characterizes co-training as late combination because views are handled independently while base learners are trained.

5. Co-training Style Algorithms

Co-training learns from two distinct views by iteratively training classifiers to maximize agreement on unlabeled data, relying on assumptions such as sufficiency, compatibility, and conditional independence. Later work relaxes these assumptions and extends co-training through expansion conditions, disagreement-based methods, and graph-based models.

  • Agreement Principle: Agreement between classifiers on unlabeled data upper bounds one classifier’s error, motivating co-training to maximize cross-view agreement.With sufficiently large samples and near-perfect agreement, the error-bound terms become favorable.
  • Relaxed Assumptions: Later analyses weaken the original requirements: weak dependence can suffice, while expansion of the data distribution can support iterative co-training with strong PAC learners.These results address the difficulty of satisfying conditional independence in real applications.
  • Foundations: Co-training partitions each example into two distinct views and assumes sufficiency, compatibility, and conditional independence for successful learning.Conditional independence is identified as critical to the original formulation.
  • Algorithm: The original algorithm alternately trains classifiers on labeled data, adds their most confidently labeled examples, and replenishes the unlabeled pool.Each view-specific classifier contributes positive and negative examples to the shared labeled set.
  • Extensions: Most co-training style algorithms are disagreement-based, while graph-based variants model multiple views through Bayesian Gaussian-process frameworks and view-specific kernels.The graph-based formulation provides an interpretation of how kernels from different views are combined.

6. Multiple Kernel Learning

Multiple kernel learning treats kernels as views and combines them to improve learning while controlling kernel-search capacity. Representative approaches use ensemble, SDP, SOCP, SILP, and adaptive regularization formulations, with theory characterizing generalization and kernel-number dependence.

  • Motivation: MKL controls the search-space capacity of possible kernel matrices and applies naturally to multi-view data because kernels correspond to different views.Appropriate kernel combination may improve learning performance.
  • Representative algorithms: MARK combines a large library of heterogeneous kernel matrices, treating kernel columns as hypotheses that can be generated on the fly.Gradient-based ensemble methods can be adapted to optimize this linear kernel combination.
  • Representative algorithms: SDP, SOCP, and related formulations learn or combine kernels through constrained optimization, including label alignment, block sparsity, and regularized support kernel machines.The SKM uses 1-norm and 2-norm penalties, while its SOCP formulation is equivalent to Lanckriet et al.’s QCQP formulation.
  • Computational efficiency: SILP has lower computational complexity than SDP and QCQP and can handle more than a hundred thousand examples or several hundred kernels.It can be solved using an off-the-shelf LP solver and a standard SVM implementation.
  • Theoretical analysis: Generalization analyses derive bounds whose dependence on the number of base kernels may be multiplicative, additive, logarithmic, or tied to eigenvalue decay.These results examine margin complexity, candidate-kernel pseudo-dimension, Rademacher chaos, and minimum eigenvalue decay.

7. Subspace Learning-based Approaches

Subspace learning-based approaches seek a latent subspace shared across multiple views, assuming the views are generated from it. Beyond CCA, newer methods offer more effective ways to construct these subspaces.

  • Subspace Learning-based Approaches: Subspace learning-based approaches obtain a latent subspace shared by multiple views by assuming the input views are generated from that subspace.The approach focuses on constructing a common representation underlying the views.
  • Subspace Learning-based Approaches: Besides canonical correlation analysis (CCA), newer methods have become available for constructing subspaces more effectively.CCA is identified as a well-known method, while recent alternatives aim to improve subspace construction.

7.1 Algorithms based on CCA

CCA-based algorithms seek projections that maximize correlation between views, while kernelization extends CCA to nonlinear data through higher-dimensional feature mappings. These methods support common-information extraction, classification preprocessing, and theoretically analyzed generalization and convergence.

  • CCA formulation: CCA finds projection vectors for two or more variable sets that maximize the correlation between corresponding projected examples.The method is formulated through an eigenvalue problem involving the two views.
  • Kernel CCA: Kernel CCA maps nonlinear data into higher-dimensional spaces and applies CCA using the kernel matrices of the two views.For a linear kernel, KCCA reduces to standard CCA.
  • Applications: KCCA isolates directions that correlate across views and can improve classification preprocessing, while SVM-2K jointly optimizes two SVMs with an ϵ-insensitive 1-norm.SVM-2K treats the two feature spaces through distinct SVMs in a single optimization.
  • Theory and regularization: Regularization is central to KCCA: consistency analysis establishes conditions on the regularization decay for convergence in the L2 norm, and generalization quality is controlled by squared weight-vector norms.These analyses motivate regularizing KCCA and provide convergence and error-bound results.
  • Extensions and applications: CCA extensions apply to supervised feature extraction, clustering, semi-supervised learning with one labeled example, and multiple kernel learning.Examples include label-directed projections, mean-based clustering subspaces, correlation-based similarity, and classifiers built from multiple kernel-induced views.

7.2 Multi-view Fisher Discriminant Analysis

Multi-view Fisher discriminant analysis extends Fisher’s discriminant analysis to supervised multi-view data, unlike CCA, which ignores labels. For two views of the same semantic object, it learns view-specific weight vectors through a constrained optimization problem.

  • Motivation: Diethe et al. (2008) generalized Fisher’s discriminant analysis to find informative projections for multi-view data in a supervised setting.This contrasts with CCA’s omission of label information.
  • Method: For two views of the same underlying semantic object, the method selects two corresponding sets of view-specific weights, w_a and w_b.The views are denoted X_a and X_b.
  • Method: The optimization is constrained because rescaling either weight vector does not affect the objective.The supplied formulation imposes normalization constraints on the view-specific weight vectors.
  • Optimization: The resulting constrained problem is expressed through a Lagrangian and solved by differentiating with respect to the weight vectors.The passages describe differentiation with respect to w_a and w_b, and equivalently α and β.

7.3 Multi-view Embedding

Multi-view embedding methods reduce high-dimensional features into shared low-dimensional representations while preserving relationships across views. Representative approaches include MSE, m-SNE, and SSMVE, which exploit locality, complementary information, or sparse consensus structure.

  • Motivation: Embedding multiple features jointly is preferable to separate reductions because it captures their underlying connections in a meaningful low-dimensional representation shared across views.High dimensionality can increase variance, noise, over-fitting, complexity, and inefficiency.
  • Multi-view spectral embedding: MSE builds view-specific sample patches and aligns them through a global coordinate to produce a smooth embedding preserving each view’s locality simultaneously.Non-negative view weights explore complementary properties while globally consistent coordinates link embeddings across views.
  • Multi-view stochastic neighbor embedding: m-SNE generalizes SNE to multi-view data by assigning each view a combination coefficient that exploits complementary information and suppresses noise.A 2-norm regularization term balances coefficients across views, with λ as the tradeoff coefficient.
  • Sparse multi-view embedding: SSMVE learns view-specific low-dimensional PCA patterns, combines them into a unified pattern, and uses sparse loading and orthogonal base constraints to obtain a consensus embedding.The 1-norm selects discriminative features for multi-view spectral embedding.

7.4 Multi-view Metric Learning

Multi-view metric learning maps different representations into a shared feature space where distances are meaningful within and across views. Representative methods enforce cross-view similarity and dissimilarity constraints, while semi-supervised approaches learn metrics using graphs, labels, and shared latent structure.

  • The goal is to construct projections from different representations into a shared feature space whose Euclidean distances are meaningful within and across views.
  • Cross-view metric learning pulls related samples together and pushes unrelated samples apart through similarity and dissimilarity loss terms.The resulting objective can be solved efficiently using the concave convex procedure (CCCP).
  • SSM-DML addresses uncertainty over visual features by learning an accurate multi-view distance metric from multiple feature sets and unlabeled-data labels.It combines graphs constructed from multi-view feature sets using weights α and uses alternating optimization.
  • MVML-GL jointly considers global consistency and local smoothness by seeking a shared latent feature space that relates multi-view observations using labeled instance pairs.

7.5 Latent Space Models

Latent space models connect multiple views through shared latent variables, enabling information propagation between observation spaces. Representative methods learn shared nonlinear embeddings, factorize shared and private information, or jointly learn predictive latent representations and model parameters.

  • Latent Space Models: Latent space models connect multiple views through latent variables and propagate information from one observation space to another.These methods focus on relationships between views rather than only dimensional reduction.
  • Latent Space Models: SGPLVM learns separate kernels for each observation space while inferring a common set of latent points.Given one observation, it first infers its most likely latent coordinate and then predicts the corresponding observation in the other space.
  • Latent Space Models: Shared KIE constructs a two-view joint embedding by maximizing mutual information between the views and their shared latent representation.Under conditional independence, the joint mutual information decomposes into two view-specific mutual information terms.
  • Latent Space Models: FOLS factorizes the latent space into shared and private components using orthogonality constraints that penalize redundant representations.The goal is to preserve common information in the shared space while reducing redundancy across private spaces.
  • Latent Space Models: Multi-view latent space Markov networks jointly learn a predictive latent representation and prediction parameters by maximizing data likelihood while minimizing training loss.They assume views and responses are conditionally independent given latent variables.

8. Applications

Multi-view learning has been applied successfully in real-world settings by exploiting consistency and complementarity across views. Applications span co-training in natural language processing, multiple kernel learning for visual tasks, and subspace learning for cross-view analysis.

  • Exploiting consistency and complementarity among views improves learning performance and supports successful applications of multi-view learning.
  • Co-training: Co-training was first applied to web document classification and subsequently widely used in natural language processing.
  • Multiple kernel learning: Multiple kernel learning improved object classification by linearly combining image similarity functions and supported object detection through feature-channel-specific exponential χ2 kernels.
  • Subspace learning: Subspace learning applications include active appearance model search, facial expression recognition, and low-dimensional context-specific word representations from unlabeled data.

9. Performance Evaluation

The section introduces widely used multi-view learning datasets and reviews empirical comparisons between representative multi-view and single-view algorithms. Evaluations span WebKB, UCI benchmark datasets, and other multimedia datasets, using measures including accuracy and time cost.

  • Empirical Evaluation: Performance evaluation uses widely used datasets to compare representative multi-view learning algorithms with single-view learning algorithms.The comparison is presented as an empirical assessment of multi-view learning benefits relative to traditional single-view learning.
  • Data Sets for Multi-view Learning: WebKB is a prominent multi-view dataset containing 8282 academic web pages from four universities, grouped into six classes and represented by two views.The views include page text and another view described in the supplied passage fragment.
  • Data Sets for Multi-view Learning: Citeseer contains 3312 scientific-publication documents in six classes, with text, inbound-reference, and outbound-reference views.The text view consists of each paper’s title and abstract, while the other two views are link views.
  • Empirical Evaluation: WebKB experiments evaluated co-training, co-EM, SVM-based co-EM, co-regularization, and subspace-learning approaches against single-view methods across multiple WebKB variants.Reported comparisons include naive Bayes, single-view SVM, and co-trained naive Bayes.
  • Empirical Evaluation: UCI benchmark experiments evaluated representative multiple kernel learning methods, including localized MKL and simple MKL, using accuracy and time cost.The studies used four groups denoted UCI1 through UCI4.

10. Conclusions

The survey classifies multi-view learning methods into co-training, multiple kernel learning, and subspace learning, finding that they mainly rely on consensus or complementary principles. It also identifies view construction, analysis, and evaluation as important future research directions.

  • The survey organizes multi-view learning algorithms into co-training, multiple kernel learning, and subspace learning.
  • These approaches mainly ensure success through either the consensus principle or the complementary principle.
  • Future work should emphasize constructing, analyzing, and evaluating views because their properties strongly influence multi-view learning performance.
  • Multi-view learning is effective and promising in practice, but better processing of multi-view data remains necessary across diverse applications.
Loading 1304.5634v1…