Source-linked AI summary
Horizontal and Vertical Ensemble with Deep Representation for Classification
Jingjing Xie, Bing Xu, Zhang Chuang
TL;DR
With limited labeled data, the paper asks how deep networks can classify effectively and whether learned representations can further improve classification. It proposes voting and stacking ensembles, and reports that horizontal voting and horizontal stacked ensemble robustly improve performance in the challenge setting, while hierarchical voting may not improve accuracy as expected.
Problem
The paper asks how deep networks can achieve good classification with limited labeled data and whether learned features can further improve classification.
Source: §1 · p. 1Method
The paper proposes Vertical Voting, Horizontal Voting, and Horizontal Stacked Ensemble for deep networks trained with labeled and unlabeled data.
Source: §1 · p. 1Results
Horizontal Voting and Horizontal Stacked Ensemble robustly improve performance, while representations from different layers may not improve classification accuracy as expected.
Source: §6 · p. 5Takeaways & Limitations
In the ICML 2013 Black Box Challenge, the authors report good performance using deep networks with very limited labeled data and massive unlabeled data.
Source: §6 · p. 5The authors state that more data knowledge or labeled training examples could have enabled further investigation of hierarchical representations.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Representation learning, especially which by using deep learning, has been widely applied in classification. However, how to use limited size of labeled data to achieve good classification performance with deep neural network, and how can the learned features further improve classification remain indefinite. In this paper, we propose Horizontal Voting Vertical Voting and Horizontal Stacked Ensemble methods to improve the classification performance of deep neural networks. In the ICML 2013 Black Box Challenge, via using these methods independently, Bing Xu achieved 3rd in public leaderboard, and 7th in private leaderboard; Jingjing Xie achieved 4th in public leaderboard, and 5th in private leaderboard.
1. Introduction
With labeled data scarce but unlabeled data abundant, the paper asks how to train large deep networks reliably and whether their hierarchical representations can improve classification. It proposes three ensemble methods to improve deep-network accuracy and robustness.
1. Introduction
- Because labeled data is rare while unlabeled data is abundant, the paper motivates automatically learning representations as an alternative to hand-crafted features.The authors note that automatic representation learning can outperform human feature engineering in some fields.Source: §1 · p. 1
- Using 130,000 unlabeled examples to pre-train a network with over 16 million parameters, the authors fine-tune it with only 1,000 labeled examples.The network has five denoising auto-encoder layers and three maxout layers.Source: §1 · p. 1
- The paper proposes Vertical Voting, Horizontal Voting, and Horizontal Stacked Ensemble to improve deep-network classification accuracy and robustness.It also discusses their performance and combination strategies.Source: §1 · p. 1
2. Background
Deep networks transform inputs through increasingly abstract representations before producing class probabilities. Their training combines unsupervised pre-training on unlabeled data with supervised fine-tuning on labeled data.
2. Background
- A deep network applies successive transformations to input data to produce representations of increasing abstraction and complexity.The architecture places pre-training layers near the bottom, hidden layers above them, and a softmax output layer at the top.Source: §2 · pp. 1–2
- Pre-training layers may use RBMs or auto-encoders, while hidden layers can include sigmoid, tanh, or maxout layers.The softmax layer produces class probabilities.Source: §2 · p. 2
- Training first uses unlabeled data for layer-wise unsupervised pre-training, then fine-tunes all network parameters using labeled data.A standard classifier such as Random Forest or SVM can instead use learned representations for class prediction.Source: §2 · p. 2
3. Vertical Voting, Horizontal Voting and Horizontal Stacked Ensemble
The section presents three ensemble approaches that use representations from intermediate layers or predictions from multiple training epochs. Vertical Voting combines classifiers on intermediate representations; horizontal methods aggregate predictions across epochs, either by voting or by stacking.
3. Vertical Voting, Horizontal Voting and Horizontal Stacked Ensemble
- The section introduces Vertical Voting, Horizontal Voting, and Horizontal Stacked Ensemble as methods for improving classification performance.The accompanying algorithms describe training and collecting predictions or representations for each method.Source: §3 · p. 2
3.1. Vertical Voting
- Vertical Voting ensembles classifiers trained on intermediate-layer representations, which may retain useful features discarded when the softmax uses only the top-level representation.The proposed rationale is that these intermediate features may be diverse and yield a lower error rate.Source: §3.1 · p. 2
3.2. Horizontal Voting
- Horizontal Voting averages class-probability predictions from networks trained across a relatively stable range of epochs to reduce output instability.The motivation is that a small labeled set can make error rates oscillate, complicating the choice of a reliable epoch.Source: §3.2 · pp. 2–3
3.3. Horizontal Stacked Ensemble
- Horizontal Stacked Ensemble collects network outputs across a continuous epoch range into a new feature space for classification.Its next step is described as similar to Stacked Generalization.Source: §3.3 · p. 3
4. Models and Experiments
Experiments use the ICML 2013 Black Box dataset and compare six models, with a deep network as the benchmark for testing the proposed methods and their combinations. The models vary in pre-training and in how they combine layer- or epoch-level outputs.
4. Models and Experiments
- The experiments use 1,000 labeled examples, 135,735 unlabeled examples, and 10,000 test examples split between public and private sets.The data have 1,875 dimensions, nine classes, and are human-unreadable.Source: §4 · pp. 3–4
- Model 2 is the deep-network benchmark; the six-model comparison tests the proposed methods and explores combination strategies.All experiments share the same learning rate and momentum.Source: §4 · p. 4
4.1. Model 1: Shallow neural network (without pre-training)
- Model 1 is a shallow network without pre-training, with three maxout layers and a softmax layer.Its architecture is 1875-1500-1500-1500-9 neurons.Source: §4.1 · p. 4
4.2. Model 2: Deep neural network with unsupervised pre-training
- Model 2 adds five denoising auto-encoder layers for unsupervised pre-training to Model 1, using more than 130 thousand unlabeled data; its stated architecture has 1,875 input and 9 output neurons.The neuron sequence is 1,875-1,500-1,000-1,500-1,200-1,500-1,500-1,500-1,500-9.Source: §4.2 · p. 4
4.3. Model 3: Deep neural network with Vertical Voting
- Model 3 adds Vertical Voting to Model 2, using representations from its three maxout layers for prediction.Source: §4.3 · p. 4
4.4. Model 4: Deep neural network with Horizontal Voting
- Model 4 adds Horizontal Voting to Model 2 by averaging networks trained from epochs 651 through 850.Source: §4.4 · p. 4
4.5. Model 5: Deep neural network combined Horizontal and Vertical Voting
- Model 5 combines Vertical and Horizontal Voting, first voting over three hidden-layer predictions per epoch, then combining predictions across 200 epochs.Source: §4.5 · p. 4
4.6. Model 6: Deep neural network with Horizontal Stacked Ensemble
- Model 6 adds Horizontal Stacked Ensemble to Model 2, ensembling 200 networks trained from epochs 651 through 850.Source: §4.6 · p. 4
5. Results and Discussion
The experiments compare pre-training and three ensemble approaches for deep-network classification. Horizontal Voting and the Model 6 ensemble show gains, while Vertical Voting is inconsistent and combining the voting methods adds computation.
5.1. Model 1 and Model 2
- Pre-training layers contributed 20.19% and 19.09% to Model 2’s scores on the public and private test sets, respectively.Models 1 and 2 used a softmax classifier.Source: §5.1 · p. 4
5.2. Model 3
- Vertical Voting combined Random Forest predictions from representations at layers h5–h7, but the voted predictions did not have the highest accuracy in one network; the authors’ explanations remained unvalidated.The suggested explanations were limited feature diversity, overfitting, and the possibility that adjusting voting weights could improve results.Source: §5.2 · pp. 4–5
5.3. Model 4
- Voting predictions from 200 selected epochs reduced the risk from choosing a poorly performing epoch, despite substantial oscillation in their validation error rates.The selected epoch range was (650, 850], and Table 2 reports error-rate statistics for the 200 examples on a validation set comprising 10% of the training set.Source: §5.3 · p. 5, Table 2 · p. 5
- Horizontal voting achieved scores of 0.68220 on the public test set and 0.67240 on the private test set, the best among the authors’ experiments during the challenge.Source: §5.3 · p. 5
5.4. Model 5
- About 3.58% and 1.58% improvements over Model 3 were reported for Model 5, attributed to Horizontal Voting.Compared with Model 4, Model 5’s accuracy rose slightly on the public test set but fell on the private test set; combining the voting methods also cost more computation resources.Source: §5.4 · p. 5
5.5. Model 6
- 0.68540 on the public test set and 0.67440 on the private test set were Model 6’s accuracies; it improved 2.82% and 3.56% over Model 2 and outperformed Model 4.Similar results were observed in other networks using this method.Source: §5.5 · p. 5
6. Conclusion and Future Work
The authors report good classification performance in the ICML 2013 Black Box Challenge using very limited labeled data and massive unlabeled data. They find that horizontal ensemble methods robustly improve performance, while more data knowledge or labeled examples could support further study of hierarchical representations.
6. Conclusion and Future Work
- Using very limited labeled data and massive unlabeled data, the authors achieved good performance in the ICML 2013 Black Box Learning Challenge.They focus on classification without prior knowledge of the data.Source: §6 · p. 5
- The authors found that hierarchical representations across network layers did not necessarily improve classification accuracy, whereas linear Horizontal Voting and Horizontal Stacked Ensemble robustly improved performance.The paper proposes and evaluates Vertical Voting, Horizontal Voting, and Horizontal Stacked Ensemble methods.Source: §6 · p. 5
- More knowledge about the data or more labeled training examples could support further investigation of hierarchical representations.The authors suggest exploring this question on other datasets in future work.Source: §6 · p. 5
Related papers
This paper cites
- Maxout Networks2013