PCANet: A Simple Deep Learning Baseline for Image Classification?

Tsung-Han Chan; Kui Jia; Shenghua Gao; Jiwen Lu; Zinan Zeng; Yi Ma

Source-linked AI summary

PCANet: A Simple Deep Learning Baseline for Image Classification?

Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, Yi Ma

arXiv:1404.3606v2Apr 14, 2014cs.CVcs.LGcs.NE

TL;DR

Image classification needs features that handle substantial intra-class variability without requiring task-specific hand-crafted design or difficult network training. The paper introduces PCANet, which uses cascaded PCA filters, binary hashing, and block histograms, and evaluates it across diverse visual-recognition tasks. PCANet is generally competitive with or better than state-of-the-art feature approaches and serves as a simple baseline, though its extreme simplicity may be insufficient for highly variable databases.

Problem
Hand-crafted features often require new domain knowledge for new data and tasks, while training effective deep networks depends on parameter tuning and ad hoc tricks.
Method
PCANet processes images with cascaded PCA filters followed by binary hashing and block-wise histograms, requiring minimal learning and no numerical optimization solver for filter learning.
Results
Across face, handwritten-digit, texture, and object classification tasks, PCANet is generally on par with or better than prefixed, hand-crafted, and DNN-learned features.
Takeaways & Limitations
PCANet provides a simple but surprisingly competitive baseline for empirically evaluating more advanced deep-learning designs and multistage features.
Takeaways & Limitations
PCANet may be insufficient for challenging databases such as Pascal and ImageNet when images exhibit substantial variability, because its structure and learning method are extremely simple and unsupervised.

Abstract

from arXiv · show

In this work, we propose a very simple deep learning network for image classification which comprises only the very basic data processing components: cascaded principal component analysis (PCA), binary hashing, and block-wise histograms. In the proposed architecture, PCA is employed to learn multistage filter banks. It is followed by simple binary hashing and block histograms for indexing and pooling. This architecture is thus named as a PCA network (PCANet) and can be designed and learned extremely easily and efficiently. For comparison and better understanding, we also introduce and study two simple variations to the PCANet, namely the RandNet and LDANet. They share the same topology of PCANet but their cascaded filters are either selected randomly or learned from LDA. We have tested these basic networks extensively on many benchmark visual datasets for different tasks, such as LFW for face verification, MultiPIE, Extended Yale B, AR, FERET datasets for face recognition, as well as MNIST for hand-written digits recognition. Surprisingly, for all tasks, such a seemingly naive PCANet model is on par with the state of the art features, either prefixed, highly hand-crafted or carefully learned (by DNNs). Even more surprisingly, it sets new records for many classification tasks in Extended Yale B, AR, FERET datasets, and MNIST variations. Additional experiments on other public datasets also demonstrate the potential of the PCANet serving as a simple but highly competitive baseline for texture classification and object recognition.

1 INTRODUCTION

PCANet is proposed as a simple, easily trained baseline that replaces complex or hand-crafted feature learning with cascaded PCA filters, binary hashing, and block-wise histograms. Despite this simplification, it performs competitively across diverse image-classification tasks and can support empirical evaluation of more advanced architectures.

Motivation: Hand-crafted features often succeed on specific tasks but generally require new domain knowledge for new data and conditions.The motivating variability includes lighting, misalignment, deformation, occlusion, and corruption.
Motivation: Deep convolutional networks learn stacked filter, nonlinear, and pooling stages, but effective training depends on parameter tuning and ad hoc tricks.Their filters are commonly learned with optimization methods such as stochastic gradient descent.
Implications: PCANet is intended as a simple baseline for empirically assessing advanced multistage features and deep-learning architectures.The paper also argues that its linear-map, hashing, and histogram structure is amenable to mathematical analysis.
PCANet: PCANet uses PCA filters, binary hashing, and block-wise histograms to emulate convolutional-network processing with minimal learning.The PCA filters adapt to data, hashing supplies nonlinear processing, and block histograms produce the output features.
PCANet: PCANet omits nonlinear operations until its final binary-hashing and histogram stage, yet this simplification does not undermine performance on typical datasets.This design contrasts with the usual early nonlinear processing in ConvNet and ScatNet architectures.
Empirical findings: Across face, digit, texture, and object classification tasks, PCANet is generally on par with or better than prefixed, hand-crafted, and DNN-learned features.The reported results include competitive or state-of-the-art performance across several benchmark datasets.

2 CASCADED LINEAR NETWORKS

PCANet builds a cascaded network by learning PCA filters from mean-removed image patches, then applying binary hashing and block-wise histograms. Its simple two-stage design omits intermediate nonlinearities, yet achieves efficient training and competitive representations.

2.1.1 The first stage: PCA: PCANet learns PCA filters from overlapping, mean-removed image patches collected across the training images.The first-stage filters are principal eigenvectors of the patch covariance matrix and minimize reconstruction error within orthonormal filters.
2.1.2 The second stage: PCA: Stacked PCA stages produce hierarchical outputs, with the second stage generating L1L2 real-valued responses from the first-stage filter outputs.The architecture can repeat this process for deeper networks, although the described model uses two stages.
2.1.3 Output stage: hashing and histogram: Binary hashing converts each group of L2 second-stage responses into an integer-valued image with values ranging from 0 to 2^L2−1.The Heaviside function assigns one to positive responses and zero otherwise, and the resulting bits are treated as a decimal code.
2.1.3 Output stage: hashing and histogram: Block-wise histograms of the hashed images form the final PCANet features, using 2^L2 bins per block and either overlapping or non-overlapping blocks.Non-overlapping blocks were empirically suitable for faces, while overlapping blocks suited digits, textures, and objects; histograms provide some translation invariance.
2.1.4 Comparison with ConvNet and ScatNet: PCANet contains no intermediate nonlinearity, and adding absolute rectification after the first stage produced no improvement in classification results.The authors suggest that output quantization and local histograms already provide sufficient invariance and robustness.
2.3 Two Variations: RandNet and LDANet: RandNet replaces learned PCA filters with Gaussian random filters, whereas LDANet learns filters using class-label information and LDA.LDA maximizes the ratio of inter-class variability to summed intra-class variability within orthonormal filters.

3 EXPERIMENTS

The authors evaluate PCANet, RandNet, and LDANet across face recognition and verification, handwritten digit recognition, texture discrimination, and object recognition tasks.

3 EXPERIMENTS: The experiments cover face recognition, face verification, handwritten digit recognition, texture discrimination, and object recognition.The evaluated networks are PCANet and its two variations, RandNet and LDANet.

3.1 Face Recognition on Many Datasets

Across MultiPIE, Extended Yale B, AR, and FERET, PCANet uses PCA-learned cascaded filters to achieve robust and competitive face recognition under illumination, pose, deformation, and occlusion.

Training and testing: PCANet filters are learned from MultiPIE and transferred to new subjects and multiple face-recognition datasets.The experiments cover MultiPIE, Extended Yale B, AR, and FERET.
Impact of the number of filters: PCANet-2 is best across all tested first-stage filter counts, while PCANet-1 is best for L1 ≥ 4 in the one-stage comparison.The study varies L1 from 2 to 12 for one-stage networks and from 4 to 24 for two-stage networks with L2 = 8.
Impact of block size: PCANet-2 exceeds 90% accuracy under translations up to 4 pixels, in-plane rotation up to 8°, and scale variation from 0.9 to 1.075.Larger histogram blocks improve PCANet-2 robustness, although they may reduce PCANet-1 performance.
Comparisons: PCANet outperforms RandNet and LDANet on MultiPIE comparisons and exceeds P-LBP across tested occlusion levels on Extended Yale B.On MultiPIE, PCANet also handles illumination variation better than LBP while matching LBP for cross-pose and cross-expression variation.
Extended Yale B: 99.58% recognition is achieved on Extended Yale B, while accuracy remains 86.49% when 60% of each test image is occluded.The authors attribute this robustness to PCA filters emphasizing informative face patches and reducing the contribution of occluded patches.

3.2 Face Verification on LFW Dataset

On unconstrained LFW face verification, PCANet was evaluated in an unsupervised setting using aligned LFW images. Square-root PCANet-2 achieved competitive verification performance.

LFW contains 13,233 images of 5,749 individuals with substantial pose, expression, illumination, clothing, and hairstyle variation.
The experiments use the unsupervised setting to evaluate learned features without metric learning or discriminative model learning.
Square-root processing improves PCANet features, consistent with similar performance boosts reported for other features on LFW.
86.28% accuracy is achieved by square-root PCANet-2, making it competitive with state-of-the-art methods.The comparison reports single-descriptor results; the best known unsupervised result cited is 88.57% using four descriptors.
Unlike concurrent ConvNet studies, PCANet is trained on the aligned LFW-a dataset rather than an outside database, with less elaborate alignment requirements.

3.3 Digit Recognition on MNIST Datasets

PCANet was tested across MNIST and its variations, where two-stage models generally outperformed random, LDA-based, and single-stage alternatives. It achieved state-of-the-art results on four variation tasks.

The MNIST evaluation comprises nine classification tasks, with MNIST basic used to study filters and block-overlap settings.
PCANet outperforms RandNet and LDANet for almost all tested first-stage filter counts under the stated two-stage settings.The experiments vary L1 from 2 to 12 for one-stage networks and from 4 to 24 for two-stage networks with L2 = 8.
PCANet-2 performs best across the tested block-overlap conditions, with its minimum error rate at a block overlap ratio of 0.5.
PCANet-2 achieves state-of-the-art results on four of eight remaining MNIST-variation tasks: basic, bg-img, bg-img-rot, and convex.
10.95% error on bg-img improves over the 12.25% result reported for method.
PCANet-2 outperforms a single-stage PCANet-1 constructed with equivalent filter count and receptive-field size.
Learned filters include horizontal and vertical stripes for rectangle edges, while some rect-img filters become low-pass to capture background responses.

3.4 Texture Classification on CUReT Dataset

PCANet was evaluated on CUReT texture classification under pose and illumination variation, with results compared against established methods including ScatNet-2.

CUReT contains 61 texture classes, each representing one material under different pose and illumination conditions, with additional surface variations complicating classification.
The experiment uses 92 selected images per class, split into 46 training and 46 testing images, with results averaged over 10 random splits.
PCANet is trained with k1 = k2 = 5, L1 = L2 = 8, and 50×50 histogram blocks, followed by a linear SVM.
ScatNet-2 followed by a PCA-based classifier achieves the best reported result in the CUReT comparison.

3.5 Object Recognition on CIFAR10

PCANet was extended to RGB images and combined with spatial pyramid pooling for CIFAR10 object recognition. Its simple unsupervised configuration remained competitive, though it lagged the state-of-the-art method.

CIFAR10 contains 50,000 training and 10,000 test RGB images across 10 classes, with variation in object position, scale, color, and texture.
RGB PCANet learning gathers channel-specific data matrices and maps the learned vectors into multichannel PCA filters.
Spatial pyramid pooling is connected to PCANet's output layer to extract information intended to be invariant to large poses and complex backgrounds.The modification is described as particularly useful for object recognition, without significant improvement in earlier face, digit, and texture experiments.
The CIFAR10 configuration uses a linear SVM, k1 = k2 = 5, L1 = 40, L2 = 8, 8×8 blocks, half-block overlap, and 4×4, 2×2, and 1×1 SPP regions.
77.14% accuracy is achieved by PCANet-2, with a 1.5% improvement when features from different filter sizes are combined.
PCANet-2 has around 11% lower accuracy than the state-of-the-art method without data augmentation, while remaining fully unsupervised and simple.

4 CONCLUSION

PCANet is a simple unsupervised convolutional network that combines cascaded PCA, binary hashing, and block histograms, offering an alternative perspective for analyzing convolutional architectures. Across diverse image-classification tasks, it generally matches strong learned, prefixed, and hand-crafted features, while remaining limited on poorly prepared, highly variable imagery.

PCANet combines cascaded PCA, binary hashing, and block histograms in a simple unsupervised network.Once its architectural parameters are fixed, filter learning is described as extremely simple and efficient.
Its cascaded linear map and nonlinear output stage provide an alternative perspective on convolutional deep learning networks.The paper suggests this simplicity could facilitate mathematical analysis and justification of effectiveness.
Across face, digit, texture, and object tasks, PCANet outperforms RandNet and LDANet and is generally on par with ScatNet and ConvNet variants.Its performance is also often better than engineered hand-crafted features such as LBP and LQP.
PCANet performs competitively when images are roughly aligned and lack diverse scales or poses.The paper identifies challenging databases such as Pascal and ImageNet as settings where its simple unsupervised structure might be insufficient.
The paper concludes that PCANet extracts useful information for faces, digits, and textures and serves as a valuable baseline for advanced architectures.