PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Deep Latent Spaces for Multi-Label Classification

Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, Yu-Chiang Frank Wang

arXiv:1707.00418v1cs.LG

TL;DR

Multi-label classification must assign multiple labels to each instance, while existing approaches can miss label dependencies or incur computational costs. C2AE combines deep canonical correlation analysis and an autoencoder for joint feature-label embedding, with label-correlation-sensitive prediction losses. The authors report favorable results against baseline and state-of-the-art methods across multiple datasets, including settings with missing labels.

  • Problem

    Multi-label classification requires assigning several labels to each instance, and independent classifiers do not capture correlations among labels.

    Source: Introduction · p. 1

  • Method

    C2AE integrates deep canonical correlation analysis and an autoencoder for joint feature-label embedding, with label-correlation-aware losses at the prediction outputs.

    Source: Introduction · p. 2

  • Results

    C2AE performed favorably against baseline and state-of-the-art methods on multiple datasets, including tasks with varying amounts of missing labels.

    Source: Conclusion · p. 6

  • Takeaways & Limitations

    The authors report that C2AE can be applied to learning tasks with missing labels without modifying its architecture.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Multi-label classification is a practical yet challenging task in machine learning related fields, since it requires the prediction of more than one label category for each input instance. We propose a novel deep neural networks (DNN) based model, Canonical Correlated AutoEncoder (C2AE), for solving this task. Aiming at better relating feature and label domain data for improved classification, we uniquely perform joint feature and label embedding by deriving a deep latent space, followed by the introduction of label-correlation sensitive loss function for recovering the predicted label outputs. Our C2AE is achieved by integrating the DNN architectures of canonical correlation analysis and autoencoder, which allows end-to-end learning and prediction with the ability to exploit label dependency. Moreover, our C2AE can be easily extended to address the learning problem with missing labels. Our experiments on multiple datasets with different scales confirm the effectiveness and robustness of our proposed method, which is shown to perform favorably against state-of-the-art methods for multi-label classification.

Introduction

Multi-label tasks require models to relate features to multiple labels and account for dependencies among labels. C2AE combines deep feature-aware label embedding with label-correlation-sensitive prediction losses, and is reported to perform favorably with and without missing labels.

Introduction

  • Multi-label classification assigns several labels to each instance, while binary relevance treats labels as independent tasks and does not capture their correlations.The passage also notes the computational costs of this approach.Source: Introduction · p. 1
  • Label-embedding approaches map labels into latent spaces to exploit label correlations, with both reduced- and high-dimensional embeddings described.The passage notes that latent subspaces can reduce classification time and that decoding maps predictions back to the original label space.Source: Introduction · p. 1
  • Earlier neural approaches include deep-network extensions and CNN-RNN's linear label embedding, which may fail to capture higher-order label dependencies.CNN-RNN uses recurrent networks to model label co-occurrence, but the cited passage cautions that its linear embedding may not discover higher-order dependencies.Source: Introduction · pp. 1–2
  • C2AE integrates deep canonical correlation analysis and an autoencoder to learn a feature-aware latent subspace and use label-correlation-aware losses in embedding and prediction.Jointly learning DCCA and the autoencoder encoding stage supports feature-aware embedding; losses at the decoding outputs support label-correlation-aware prediction.Source: Introduction · p. 2
  • C2AE is reported to perform significantly better than state-of-the-art approaches on multi-label tasks both with and without missing labels.The authors say the architecture can be extended to missing-label problems without modification.Source: Introduction · p. 2

Related Work

Prior multi-label approaches include independent classifiers, classifier chains, and label embedding. C2AE is presented as a deep, feature-aware embedding framework that exploits label interdependency during both embedding and prediction.

Related Work

  • Binary relevance treats labels independently, while classifier-chain methods capture dependencies but can be computationally expensive and difficult to scale to many labels.The passage also notes that parallel implementation is not applicable to the series of classifications when reducing computational loads is desirable.Source: Related Work · p. 2
  • Label embedding maps label vectors into a latent subspace, where label correlations can be described implicitly; approaches include random and principal-component projections as well as CNN-RNN.Mapping inputs into the latent space and decoding predictions back to labels are described as reducing computation costs.Source: Related Work · p. 2
  • C2AE is distinguished from typical linear label embeddings by using deep neural networks for feature-aware label embedding and exploiting label interdependency during embedding and prediction.The passage notes that only a few existing methods jointly use input features for label embedding.Source: Related Work · p. 2

Our Proposed Method

C2AE combines DCCA and an autoencoder to jointly embed features and labels and recover label outputs. Its label-pair loss and missing-label handling support correlation-aware prediction.

Canonical-Correlated Autoencoder (C2AE)

Optimization

Canonical-Correlated Autoencoder (C2AE)

  • Identity constraints make the latent objective equivalent to standard CCA correlation maximization and allow efficient network-loss and gradient computation.The latent representations are the transformed feature and label data.Source: Learning Deep Latent Spaces for Joint… · p. 3
  • Unlike approaches that separate label embedding from prediction, C2AE jointly learns feature and label embeddings with classification guarantees and autoencoder-based output recoverability.The paper contrasts this unified approach with existing label-embedding methods that learn an additional feature-to-subspace model for prediction.Source: Learning Deep Latent Spaces for Joint… · p. 3, Learning and Recovering… · p. 3
  • The output loss maximizes positive-label outputs over negative-label outputs, implicitly preserving label co-occurrence information that mean-square-error or cross-entropy losses do not identify.The loss is evaluated over positive-negative label pairs.Source: Learning and Recovering… · p. 3
  • For missing labels, the loss uses only known positive-negative pairs; preprocessing sets missing labels to zero so they are not fed into the DNN.The authors describe this as suppressing noise from missing labels in the latent space.Source: Learning from Data with Missing Labels · p. 4

Experiments

C2AE performed favorably against label-embedding and DNN-based alternatives, and its performance remained strong in experiments with missing labels. Evaluations covered five datasets and varied latent-space dimensions and missing-label rates.

Datasets and Settings

  • Experiments covered iaprtc12, ESPGame, mirflickr, tmc2007, and NUS-WIDE, using image and text datasets.For NUS-WIDE, 150,000 instances were randomly selected for training; 4096-dimensional AlexNet fc-7 features were used for fair comparison with CNN-based methods.Source: Datasets and Settings · pp. 4–5

Comparisons with Label Embedding based Approaches

Comparisons with DNN-based Approaches

  • Among the DNN methods evaluated on NUS-WIDE, C2AE produced promising performance; DNN-BCE and CNN-WARP did not exploit label co-occurrence.Training C2AE on NUS-WIDE took 10–15 minutes on a Titan X GPU, reported as more efficient than other DNN approaches, especially those requiring RNN learning.Source: Comparisons with DNN-based Approaches · p. 6, Table 2 · p. 6

Performance Evaluation with Missing Labels

Conclusion

C2AE integrates DCCA and an autoencoder to jointly embed features and labels, with label-correlation-sensitive losses to exploit dependencies. Experiments found favorable performance across multiple datasets and applicability to settings with varying amounts of missing labels.

Conclusion

  • C2AE integrates DCCA and an autoencoder in a unified DNN model for joint feature and label embedding.Label-correlation-sensitive losses at the outputs add the ability to exploit cross-label dependency.Source: Conclusion · p. 6
  • Across multiple datasets, C2AE performed favorably against baseline and state-of-the-art methods.The conclusion also reports that it can be applied to learning tasks with varying amounts of missing labels.Source: Conclusion · p. 6
  • The authors conclude that experiments verify C2AE’s effectiveness and robustness, including for learning with missing labels.Source: Conclusion · p. 6
Loading 1707.00418v1…