Source-linked AI summary
Deep Linear Discriminant Analysis
Matthias Dorfer, Rainer Kelz, Gerhard Widmer
TL;DR
DeepLDA addresses the lack of direct latent-space separability constraints in standard classification objectives by placing an LDA-inspired objective on a deep neural network. It learns representations with low within-class and high between-class variance, achieving competitive MNIST and CIFAR-10 results and outperforming categorical cross-entropy on supervised STL-10.
Problem
Standard categorical cross-entropy does not directly constrain latent representations to be linearly separable, motivating a deep method that exploits LDA’s class-separation properties.
Method
DeepLDA places an LDA eigenvalue-based objective on a DNN and trains it end-to-end to reduce within-class variance while increasing between-class variance.
Results
DeepLDA achieves competitive results on MNIST and CIFAR-10 and outperforms categorical cross-entropy by more than 9% test-set accuracy in fully supervised STL-10.
Takeaways & Limitations
DeepLDA produces discriminative latent representations that improve classification accuracy within the evaluated benchmark settings.
Abstract
from arXiv · showhide
We introduce Deep Linear Discriminant Analysis (DeepLDA) which learns linearly separable latent representations in an end-to-end fashion. Classic LDA extracts features which preserve class separability and is used for dimensionality reduction for many classification problems. The central idea of this paper is to put LDA on top of a deep neural network. This can be seen as a non-linear extension of classic LDA. Instead of maximizing the likelihood of target labels for individual samples, we propose an objective function that pushes the network to produce feature distributions which: (a) have low variance within the same class and (b) high variance between different classes. Our objective is derived from the general LDA eigenvalue problem and still allows to train with stochastic gradient descent and back-propagation. For evaluation we test our approach on three different benchmark datasets (MNIST, CIFAR-10 and STL-10). DeepLDA produces competitive results on MNIST and CIFAR-10 and outperforms a network trained with categorical cross entropy (same architecture) on a supervised setting of STL-10.
1 INTRODUCTION
The paper adapts LDA’s class-separating properties to deep neural networks by introducing an end-to-end method that learns linearly separable latent representations.
- The method aims to preserve LDA’s low within-class variability, high between-class variability, and linear decision boundaries in learned representations.
- 1.1 MAIN IDEA OF THIS PAPER: DeepLDA is proposed as an end-to-end DNN version of LDA, extending a family of neural methods based on multivariate statistics.
- 1.1 MAIN IDEA OF THIS PAPER: DeepLDA replaces sample-wise label-likelihood optimization with an LDA eigenvalue objective that trains feature distributions through back-propagation.
- The paper evaluates DeepLDA on three benchmark datasets and compares it with existing approaches and categorical cross-entropy training.
2 DEEP NEURAL NETWORKS
Standard DNN classification optimizes sample-wise categorical cross-entropy, whereas DeepLDA changes the final layer and objective to impose discriminative structure on latent representations.
- A supervised DNN maps inputs through nonlinear layers to softmax class probabilities and optimizes parameters with stochastic gradient descent.
- Categorical cross-entropy maximizes the likelihood of each target class for individual training examples.
- Unlike categorical cross-entropy, the standard objective does not directly constrain latent representations to be linearly separable.
- Figure 1 contrasts architectures that share DNN feature propagation but differ in their final layer and optimization target.
3 DEEP LINEAR DISCRIMINANT ANALYSIS (DEEPLDA)
DeepLDA places an LDA objective on top of a DNN, optimizing class-separating latent distributions and using the resulting LDA structure for classification.
- 3.1 LINEAR DISCRIMINANT ANALYSIS: DeepLDA maximizes between-class relative to within-class scatter, producing low within-class variance and high between-class variance in a C−1-dimensional space.
- 3.3 MODIFIED DEEPLDA OPTIMIZATION TARGET: The method regularizes within-class scatter and focuses optimization on the smallest eigenvalues to distribute discriminative variance across all latent dimensions.
- 3.3 MODIFIED DEEPLDA OPTIMIZATION TARGET: DeepLDA avoids trivial optimization that separates already-distant classes while neglecting neighboring classes by targeting eigenvalues below a variance threshold.
- 3.3 MODIFIED DEEPLDA OPTIMIZATION TARGET: The LDA-based objective can be optimized end-to-end with back-propagation and mini-batch stochastic gradient descent.
- 3.4 CLASSIFICATION BY DEEPLDA: For unseen samples, the trained network’s hidden representations are projected with an LDA matrix, converted into class probabilities, and assigned the highest-probability class.
4 EXPERIMENTS
DeepLDA is evaluated against categorical cross entropy and prior results on MNIST, CIFAR-10, and supervised STL-10 using a shared convolutional architecture and dataset-specific training setups. It achieves competitive performance on MNIST and CIFAR-10, while substantially outperforming CCE on STL-10.
- Experimental setup: Experiments used VGG-like convolutional networks with global average pooling and batch normalization, with DeepLDA batch sizes of 1,000 for MNIST and CIFAR-10 and 200 for STL-10.CCE training used a batch size of 128 for all datasets; the architecture was chosen to support covariance estimation.
- MNIST: DeepLDA achieved a 0.29% MNIST test error without data augmentation and produced comparable results with a linear SVM on its learned representation.Early stopping with best-model selection on MNIST-50k outperformed training with 10,000 additional examples in MNIST-60k.
- CIFAR-10: On CIFAR-10, DeepLDA and the CCE network both achieved state-of-the-art results without data augmentation, although DeepLDA performed slightly worse than CCE.The dataset was trained with global contrast normalization, ZCA whitening, and random horizontal flips.
- STL-10: 81.46% STL-10 test accuracy with 4,000 training images surpassed the same architecture trained with CCE by more than 3 percentage points.This comparison used a fully supervised setting without the dataset’s additional unlabeled images.
- STL-10: 66.97% average test accuracy on STL-10 with 1,000 labeled images exceeded CCE by 9.53 percentage points.End-to-end DeepLDA features also outperformed LDA applied to CCE-network features, which reached 59.48%.
5 INVESTIGATONS ON DEEPLDA AND DISCUSSIONS
The experiments examine DeepLDA’s learned representations, eigenvalue structure, and sensitivity to image size. DeepLDA performs best on larger STL-10 images and distributes discriminative variance across latent dimensions.
- DeepLDA outperforms CCE by 3 percentage points on STL-10 Method-4k.
- 5.1 DOES IMAGE SIZE AFFECT DEEPLDA?: Downscaling STL-10 reduces the performance of both CCE and DeepLDA, with DeepLDA performing best on larger images.The comparison uses original 96 × 96 images and downscaled 32 × 32 images.
- 5.2 EIGENVALUE STRUCTURE OF DEEPLDA REPRESENTATIONS: DeepLDA’s explained discriminative variance correlates naturally with the classification potential of its learned representations.
- 5.2 EIGENVALUE STRUCTURE OF DEEPLDA REPRESENTATIONS: DeepLDA increases individual eigenvalues during training and distributes discriminative variance more equally across available latent dimensions than CCE.
6 CONCLUSION
The paper presents DeepLDA as an end-to-end neural interpretation of LDA that learns linearly separable latent representations. It achieves competitive benchmark performance and surpasses CCE on supervised STL-10, especially with reasonably sized images.
- DeepLDA maximizes generalized LDA eigenvalues to learn linearly separable representations end to end.
- DeepLDA achieves competitive results on MNIST and CIFAR-10 and outperforms CCE on supervised STL-10 by more than 9% test set accuracy.
- The results suggest that DeepLDA performs best with reasonably sized images, represented here by 96 × 96-pixel inputs.
APPENDIX A: GRADIENT OF DEEPLDA-LOSS
The appendix derives gradients for the DeepLDA loss with respect to the hidden representation. Training focuses on selected generalized-LDA eigenvalues and differentiates the associated scatter-matrix structure.
- The DeepLDA loss back-propagates through the hidden representation H by differentiating its optimization target.
- The objective maximizes the k smallest generalized-LDA eigenvalues that remain below a specified threshold.
- The gradient derivation starts from the generalized LDA eigenvalue problem and the definitions of within- and between-class scatter.
- Gradients of the total and class covariance matrices are combined to obtain derivatives of the within- and between-class scatter matrices.
APPENDIX B: DEEPLDA LATENT REPRESENTATION
DeepLDA’s STL-10 latent representation uses the available feature dimensions and exhibits orthogonal internal representations. This structure favors linear decision boundaries in the projected space.
- DeepLDA uses all available feature dimensions in the STL-10 latent representation.
- Many internal representations are orthogonal to one another, an LDA implication that favors linear decision boundaries.