PaperlayerView exact revision on arXiv

Source-linked AI summary

Correlational Neural Networks

Sarath Chandar, Mitesh M. Khapra, Hugo Larochelle, Balaraman Ravindran

arXiv:1504.07225v3cs.CLcs.LGcs.NEstat.ML

TL;DR

Common Representation Learning must reconcile correlated cross-view embeddings with reconstruction and scalability. CorrNet addresses this by adding an explicit correlation objective to an autoencoder, and the paper reports better representations than other methods on cross-language document classification and transliteration equivalence detection.

  • Problem

    Existing CCA and autoencoder approaches provide complementary benefits for common representations: CCA emphasizes correlation but is less scalable, while autoencoders emphasize reconstruction without guaranteeing correlated representations.

  • Method

    CorrNet is an autoencoder-based method that jointly supports self/cross reconstruction and explicitly maximizes correlation between the views’ common representations.

  • Results

    CorrNet representations perform better than other methods on cross-language document classification and transliteration equivalence detection.

  • Takeaways & Limitations

    CorrNet provides correlated, reconstructive, scalable common representations that can exploit additional single-view data for cross-view applications.

  • Takeaways & Limitations

    The authors report no strong empirical evidence that their deep CorrNet training procedure is superior to alternative methods and leave detailed comparisons for future work.

Abstract

from arXiv · show

Common Representation Learning (CRL), wherein different descriptions (or views) of the data are embedded in a common subspace, is receiving a lot of attention recently. Two popular paradigms here are Canonical Correlation Analysis (CCA) based approaches and Autoencoder (AE) based approaches. CCA based approaches learn a joint representation by maximizing correlation of the views when projected to the common subspace. AE based methods learn a common representation by minimizing the error of reconstructing the two views. Each of these approaches has its own advantages and disadvantages. For example, while CCA based approaches outperform AE based approaches for the task of transfer learning, they are not as scalable as the latter. In this work we propose an AE based approach called Correlational Neural Network (CorrNet), that explicitly maximizes correlation among the views when projected to the common subspace. Through a series of experiments, we demonstrate that the proposed CorrNet is better than the above mentioned approaches with respect to its ability to learn correlated common representations. Further, we employ CorrNet for several cross language tasks and show that the representations learned using CorrNet perform better than the ones learned using other state of the art approaches.

1 Introduction

Common Representation Learning seeks shared embeddings for multiple views that support reconstruction, transfer, matching, and single-view improvement. CorrNet combines autoencoder reconstruction with explicitly correlated representations, scalable training, and use of additional single-view data.

  • 1 Introduction: CRL learns common-subspace projections whose representations should be correlated and support reconstructing either view from the other.The formal task uses functions hX and hY that map two views into R^k while requiring correlation and bidirectional reconstruction.
  • 1 Introduction: CCA produces correlated representations but is difficult to scale, may reconstruct poorly, and cannot use additional non-parallel single-view data.These limitations constrain CCA in settings with large datasets, missing reconstruction requirements, or abundant single-view data.
  • 1 Introduction: MAE supports self- and cross-reconstruction but does not explicitly ensure that its shared hidden representation is correlated across views.CCA and MAE therefore provide complementary capabilities rather than satisfying both goals directly.
  • 1 Introduction: CorrNet combines self/cross reconstruction with correlated common representations, making it useful for missing-view reconstruction and cross-view matching.Its training objective is designed to retain predictive capabilities while aligning the representations of both views.
  • 1 Introduction: CorrNet is trained with gradient-based methods, can scale through mini-batch SGD, and can incorporate additional single-view data.The paper evaluates its reconstruction, correlation, transfer-learning, and cross-language applications, reporting better representations than other state-of-the-art approaches for cross-language tasks.

2 Correlational Neural Network

CorrNet uses a three-layer neural architecture whose shared hidden representation reconstructs both views from complete or single-view inputs. Its objective combines self-reconstruction, cross-reconstruction, and hidden-representation correlation, with training extensions for abundant single-view data.

  • 2.1 Model: The model uses an input layer, a shared hidden layer, and an output layer that reconstructs the concatenated two-view input.For z=(x,y), the hidden representation combines view-specific projections before the output layer reconstructs z.
  • 2.2 Training: The training objective uses reconstruction loss plus a scaled correlation term, with cross-entropy for binary inputs and squared error otherwise.The correlation term is computed from the hidden representations, while λ scales it relative to the reconstruction terms.
  • 2 Correlational Neural Network: CorrNet minimizes self-reconstruction and cross-reconstruction errors while maximizing correlation between the hidden representations of both views.The objective includes reconstruction from the combined input and from either view alone, plus a correlation term that encourages shared hidden units.
  • 2 Correlational Neural Network: With only one view available, CorrNet computes that view’s hidden representation and uses it as the data representation that can generalize across views.The representation is h(x) when x is observed and h(y) when y is observed.
  • 2.3 Using additional single view data: Additional single-view data is incorporated by adding conventional autoencoder-style self-reconstruction objectives for the available x-only and y-only examples.When all data types are available, the procedure samples mini-batches from X, Y, and Z and updates the model using their corresponding objectives.

3 Deep Correlational Neural Networks

Deep CorrNet extends CorrNet with multiple hidden layers to seek more nonlinear cross-view correlations. Its training procedure progressively stacks and retrains CorrNet layers, while the authors note that alternatives remain insufficiently compared.

  • Deep CorrNet adds multiple hidden layers because more nonlinear representations may improve correlation between data views.
  • Training starts with a shallow CorrNet, separates view-specific hidden layers, adds a new common hidden layer, and retrains the deeper model.The procedure repeats the layer-construction and retraining steps as needed.
  • Figure 2 illustrates the progressive stacking of CorrNet components to construct a deeper correlational network.
  • The authors describe this procedure as similar to greedy layerwise pre-training and do not provide strong evidence that it outperforms alternative deep-training methods.They leave detailed comparisons for future work and note that another method may make more sense with less parallel data.

4 Related Models

Related models for common representation learning include CCA-based and neural-network approaches. CorrNet differs by combining correlation maximization with self- and cross-reconstruction in a unified training objective.

  • CCA and its variants learn common representations through correlated projections, while kernel and deep variants extend this idea to nonlinear or deep mappings.
  • Neural CCA uses separate networks and staged objectives for correlation maximization and reconstruction, whereas CorrNet uses one network and one objective.
  • Unlike Neural CCA and DCCA, CorrNet combines correlation maximization with reconstruction, including both self-reconstruction and cross-reconstruction.
  • MAE minimizes self- and cross-reconstruction errors but lacks an explicit objective term forcing correlated common representations.Its training also considers the reconstruction terms separately, whereas jointly considering them performed better empirically.

5 Analysis of Correlational Neural Networks

CorrNet is evaluated against CCA, KCCA, and MAE on reconstruction, correlated representations, and transfer learning using MNIST views. It matches MAE on cross-reconstruction, produces stronger correlations, and improves transfer learning, while the correlation loss contributes to performance.

  • 5.1 Data Description: The evaluation compares CorrNet, CCA, KCCA, and MAE on self-reconstruction, cross-reconstruction, representation correlation, and transfer learning using MNIST image halves.MNIST images are split into left and right halves, with 50,000 training images and 10,000 validation images.
  • 5.2 Performance of Self and Cross Reconstruction: CorrNet and MAE are equally good at cross-reconstructing the right image half from the left, although CorrNet has higher self-reconstruction error.CorrNet’s correlation objective interferes with self-reconstruction relative to MAE, while cross-reconstruction remains nearly the same.
  • 5.4 Transfer Learning across views: CorrNet clearly outperforms the other models in transfer learning, including when common representations are learned from only 10,000 MNIST instances.The experiments use representations learned from MNIST image halves and compare transfer-learning accuracy across models.
  • 5.6 Importance of Terms in the Objective Function: Adding the correlation loss term produces representations that yield better transfer learning than otherwise comparable loss functions.The comparison pairs loss functions with and without the correlation term and examines transfer-learning performance.

6 Experiments using Deep Correlational Neural Network

Deep CorrNet is evaluated against DCCA on MNIST using correlation and transfer-learning metrics, with deeper CorrNet models outperforming corresponding DCCA models.

  • Deep CorrNet models outperform the corresponding DCCA models on both sum correlation and transfer-learning performance.The comparison uses MNIST images and reports transfer in both left-to-right and right-to-left directions.
  • The three-layer CorrNet-500-300-50 performs better than the two-layer CorrNet-500-50 on both transfer-learning tasks.
  • The two-layer CorrNet-500-50 achieves higher sum correlation than the three-layer CorrNet-500-300-50.

7 Cross Language Document Classification

CorrNet learns bilingual word representations from parallel or aggregated sentence data and applies them to cross-language document classification, where it generally achieves the strongest reported performance.

  • Representation learning: CorrNet can learn bilingual word representations from sentence-level bag-of-words pairs without requiring word-level alignments.The model encodes bag-of-words inputs through word-specific weight columns and learns aligned representations from translated sentence pairs.
  • Representation learning: Document vectors are formed as tf-idf-weighted sums of word representations and then used to train cross-language classifiers.
  • Experiments: CorrNet is the best-performing method for nearly all evaluated language-pair classification directions and often substantially outperforms Klementiev et al. (2012).The results use 1000 RCV training examples and report both translation directions.
  • Experiments: With 500k EN/DE sentence pairs plus monolingual RCV documents, CorrNet reaches 87.9% for EN →DE and 76.7% for DE →EN.Without monolingual data, performance is 86.1% for EN →DE and 68.8% for DE →EN.
  • Experiments: CorrNet outperforms other models at almost all data sizes and remains strong with only 100 labeled examples.The method can still benefit from additional labeled data, as observed for DE →EN.
  • Experiments: Merging 5, 25, or 50 adjacent sentences into one bag-of-words does not necessarily reduce embedding quality, indicating exact sentence alignment is not essential.

8 Transliteration Equivalence

CorrNet is applied to English–Hindi transliteration mining by learning common representations for character-bigram views of paired words.

  • CorrNet learns common representations for English–Hindi word pairs represented as bags of character bigrams.Hindi words use 2860 bigram-character features, while English words use 651.
  • The transliteration-equivalence task identifies word pairs forming transliterations within paired English and Hindi titles.Testing uses approximately 1000 Wikipedia title pairs from the NEWS 2010 dataset.

9 Bigram similarity using multilingual word embedding

The paper also uses multilingual word representations from CorrNet to improve monolingual English bigram similarity, where CorrNet exceeds the previous state of the art on average.

  • Bigram representations are formed by adding the vectors of the two constituent words, and pair similarity is measured with cosine similarity.The evaluation covers adjective-noun and verb-object subsets.
  • CorrNet performs better than the previous state-of-the-art DCCAE on average Spearman correlation for bigram similarity.The best results are obtained with CorrNet-500-384.
  • The experiment shows that multilingual data can improve a single-view task such as monolingual bigram similarity.

10 Conclusion and Future Work

CorrNet learns aligned, correlated common representations for two views while reconstructing one view from another and training scalably. It outperforms other methods on cross-language document classification and transliteration equivalence detection, while extension to multiple views remains future work.

  • CorrNet representations perform better than other methods on cross-language document classification and transliteration equivalence detection.
  • CorrNet combines cross-view reconstruction with aligned, correlated common representations and scalable training.
  • The proposed model can use additional single-view data, which is often available in real-world applications.
  • Extending CorrNet to multiple views is identified as future work for settings with unevenly available data across views.
Loading 1504.07225v3…