PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Forest

Zhi-Hua Zhou, Ji Feng

arXiv:1702.08835v4cs.LGstat.ML

TL;DR

The paper asks whether deep learning can be realized with non-differentiable modules instead of neural networks. It proposes gcForest, a deep decision-tree model trained without backpropagation, and finds competitive performance across many tasks, although it is inferior on some image tasks.

  • Problem

    The paper investigates whether deep learning can be realized with non-differentiable modules and without backpropagation.

  • Method

    gcForest constructs a deep forest from decision trees, using cascade processing and representation learning without relying on backpropagation.

  • Results

    gcForest achieves performance highly competitive to deep neural networks across a broad range of tasks and excellent performance across various domains, though it is inferior on some image tasks.

  • Takeaways & Limitations

    The study exhibits the possibility of constructing deep models with non-differentiable modules and without backpropagation.

  • Takeaways & Limitations

    gcForest performs inferiorly on some image tasks, though the authors believe feature design could significantly improve its performance.

Abstract

from arXiv · show

Current deep learning models are mostly build upon neural networks, i.e., multiple layers of parameterized differentiable nonlinear modules that can be trained by backpropagation. In this paper, we explore the possibility of building deep models based on non-differentiable modules. We conjecture that the mystery behind the success of deep neural networks owes much to three characteristics, i.e., layer-by-layer processing, in-model feature transformation and sufficient model complexity. We propose the gcForest approach, which generates \textit{deep forest} holding these characteristics. This is a decision tree ensemble approach, with much less hyper-parameters than deep neural networks, and its model complexity can be automatically determined in a data-dependent way. Experiments show that its performance is quite robust to hyper-parameter settings, such that in most cases, even across different data from different domains, it is able to get excellent performance by using the same default setting. This study opens the door of deep learning based on non-differentiable modules, and exhibits the possibility of constructing deep models without using backpropagation.

1 Introduction

The paper asks whether deep learning can be built from non-differentiable modules rather than neural networks and proposes gcForest, a deep decision-tree ensemble. gcForest reduces manual configuration through data-dependent complexity determination and shows robust performance across domains.

  • Deep learning models are typically neural networks composed of differentiable nonlinear layers trained by backpropagation.
  • DNNs require many design choices, substantial labeled data, and manually predetermined architecture complexity, complicating training and limiting some applications.The passages also describe neural networks as difficult to interpret and analyze theoretically.
  • The paper asks whether deep learning can be realized with non-differentiable modules and trained without backpropagation.
  • gcForest constructs a non-neural deep model as a cascade-structured decision-tree ensemble with representation learning by forests.Multi-grained scanning can further enhance representation learning and potentially provide contextual or structural awareness.
  • gcForest automatically determines cascade depth from data, uses fewer hyper-parameters than DNNs, and can operate on small-scale data while controlling training costs.
  • Experiments report robust performance across different data and domains, often achieving excellent results with the default setting.

2 Inspiration

The paper conjectures that deep learning benefits from layer-by-layer processing, in-model feature transformation, and sufficient model complexity. It uses these characteristics to motivate deep forests and draws ensemble-design guidance from accuracy, diversity, and complementarity.

  • 2.1 Inspiration from DNNs: Unlike DNNs, decision trees and boosting machines operate on original features without creating new features during learning.
  • 2.1 Inspiration from DNNs: Layer-by-layer processing is proposed as an important factor because flat networks lack this characteristic regardless of their complexity.
  • 2.1 Inspiration from DNNs: gcForest is designed to provide the conjectured characteristics within a non-neural model based on decision trees.
  • The paper conjectures that layer-by-layer processing, in-model feature transformation, and sufficient model complexity are crucial characteristics of successful deep models.
  • 2.2 Inspiration from Ensemble Learning: Combining relatively weaker learners can be preferable to combining only accurate learners when their complementarity is greater.
  • 2.2 Inspiration from Ensemble Learning: For ensemble learning, error-ambiguity decomposition indicates that lower ensemble error is associated with accurate and diverse individual classifiers.The passage presents this as guidance for ensemble construction rather than a directly optimized objective.
  • 2.2 Inspiration from Ensemble Learning: A limitation of ensemble construction is that no diversity measure has become a well-accepted formal definition, and some diversity mechanisms are not universally effective.
  • 2.2 Inspiration from Ensemble Learning: gcForest is presented as a decision-tree ensemble that uses almost all major diversity-enhancement mechanisms.

3 The gcForest Approach

gcForest builds a deep forest from cascaded decision-tree ensembles, transforming features between levels and automatically extending the cascade when validation performance improves. Multi-grained scanning adds transformed features from sliding windows before cascade processing.

  • 3.1 Cascade Forest Structure: Each cascade level ensembles decision-tree forests and passes its processing result to the next level.The design uses multiple forest types to encourage ensemble diversity.
  • 3.1 Cascade Forest Structure: Forests estimate class distributions at leaf nodes, average them across trees, and concatenate the resulting class vectors with the original features.These augmented features become input to the next cascade level.
  • 3.1 Cascade Forest Structure: Cross-validation generates class vectors for the next level and reduces overfitting risk during feature augmentation.The vectors from multiple folds are averaged before being used as augmented features.
  • 3.1 Cascade Forest Structure: The cascade terminates when adding a level yields no significant validation-performance gain, so model complexity is determined adaptively.This allows gcForest to accommodate different training-data scales rather than using a fixed depth.
  • 3.2 Multi-Grained Scanning: Multi-grained scanning uses sliding windows to generate differently grained feature vectors before training forests on the transformed representations.For spatial data, windows preserve panel structure; the resulting class vectors are concatenated as transformed features.

4 Experiments

The experiments evaluate gcForest across a broad range of tasks using a fixed cascade configuration and default settings, while deep neural networks receive task-specific architecture tuning. The comparison highlights gcForest’s easier tuning and the difficulty of choosing one DNN architecture across datasets.

  • 4 Experiments: Experiments compare gcForest with deep neural networks and other popular learning algorithms across a broad range of tasks.The stated goal is to test whether gcForest can achieve competitive performance with easier parameter setting.
  • 4.1 Configuration: gcForest uses the same cascade structure across experiments, with four completely-random forests and four random forests per level, each containing 500 trees.Three-fold cross-validation generates class vectors, and the number of cascade levels is automatically determined.
  • 4.1 Configuration: Cascade growth uses separate growing and estimating subsets, stopping when a new level does not improve estimated performance.The reported split is 80% for growing and 20% for estimating.
  • 4.1 Configuration: The same gcForest setting is used for all tasks, whereas DNNs receive task-specific tuning and architecture selection.For DNNs, architectures are evaluated on a validation set before retraining the selected network.
  • 4.1 Configuration: A DNN architecture reached 80% accuracy on ADULT but only 30% accuracy on YEAST when the same architecture was used.The authors use this contrast to show that DNN structure hyper-parameters cannot generally be fixed across tasks.

Image Categorization

On MNIST, gcForest is evaluated against LeNet-5, SVM, Random Forest, and Deep Belief Nets. Using default settings, it achieves highly competitive test performance.

  • Image Categorization: gcForest achieves highly competitive MNIST test performance while simply using the default settings.The comparison includes LeNet-5, an RBF-kernel SVM, a 2,000-tree Random Forest, and reported Deep Belief Net results.

Face Recognition

On ORL face recognition, gcForest is compared with a tuned CNN under three training-set sizes. It runs well across all three cases using the same configuration.

  • Face Recognition: The ORL dataset contains 400 grayscale facial images from 40 people, with test performance reported after selecting 5, 7, or 9 training images per person.A random guess would achieve 2.5% accuracy because there are 40 possible outcomes.
  • Face Recognition: The CNN baseline uses two convolutional layers, pooling, a 128-unit dense layer, and a 40-unit soft-max output layer.It is trained with ReLU, cross-entropy, dropout, adadelta, batch size 10, and 50 epochs.
  • Face Recognition: gcForest runs well across all three ORL training-size cases using the same configuration.The experiment compares it with a CNN and varies the number of training images per person.

Music Classification

The GTZAN experiment evaluates gcForest for music-genre classification using MFCC representations and compares it with neural and conventional baselines.

  • GTZAN contains 10 music genres with 100 clips per genre, split into 700 training and 300 testing clips.
  • Each 30-second clip is transformed into a 1,280 × 13 MFCC feature matrix.
  • The CNN baseline uses 13×8 convolution kernels with 32 feature maps, followed by pooling and fully connected layers of 1,024 and 512 units.
  • The study also compares gcForest with an MLP, Random Forest, Logistic Regression, and SVM.
  • Test accuracies for the music-classification comparison are summarized in Table 4.

Hand Movement Recognition

The sEMG experiment classifies six hand movements from multivariate time-series records and compares gcForest with MLP and LSTM baselines.

  • The sEMG dataset contains 1,800 records assigned to six hand movements: spherical, tip, palmar, lateral, cylindrical, and hook.
  • Each record is a time series with 500 sensor features per second and 3,000 associated features.
  • The baselines include an MLP with input-1,024-512-output structure and an LSTM with 128 hidden units and sequence length six.
  • Test results for the hand-movement classification experiment are summarized in Table 5.

Sentiment Classification

The experiments span sentiment, low-dimensional, image, and architectural analyses, reporting dataset-specific settings, performance comparisons, efficiency, and design effects.

  • Sentiment Classification: IMDB uses 25,000 training and 25,000 testing reviews represented by tf-idf features, so CNNs are not directly applicable.
  • Sentiment Classification: Because tf-idf features lack spatial or sequential relationships, gcForest omits multi-grained scanning for IMDB.
  • Low-Dimensional Data: MLP performance is sensitive to architecture: input-16-8-8-output with ReLU reaches 76.37% on ADULT but 33% on LETTER.
  • High-Dimensional Data: On CIFAR-10, gcForest with its default setting is inferior to state-of-the-art DNNs but is the best among non-DNN approaches.
  • Efficiency: On IMDB, gcForest takes 2,404 seconds or 40 minutes, compared with 4,650 seconds or 77.5 minutes for the CPU-trained MLP.
  • Architecture Analysis: Multi-grained scanning improves performance when spatial or sequential feature relationships exist, while concatenating multiple-grain features performs worse than the current design.

5 Related Work

Related work positions gcForest as a non-neural deep model built from decision-tree ensembles, multi-grained scanning, cascade processing, and feature re-representation.

  • gcForest is a decision-tree ensemble approach that uses multiple categories of diversity-enhancement mechanisms.
  • Unlike prior tree–neural-network studies whose final models are differentiable, gcForest develops deep models based on non-differentiable modules.
  • Its multi-grained scanning examines data with different sliding-window sizes and generates multiple instances from each training example.
  • The cascade uses multiple levels and grains, forming a cascade of cascades in which each grade is an ensemble of ensembles.
  • Ensembles within the same grade are used together for feature re-representation, and cross-validation generates inputs for the next grade.
  • The approach differs from stacking because stacking can overfit with more than two grades and cannot by itself enable a deep model.

6 Future Issues

The paper identifies future work in richer feature re-representation, computational efficiency, and broader use of forest-generated features, including unlabeled data.

  • Feature re-representation: Improving feature re-representation is a key future issue, with richer class-vector features proposed to encode more information.Suggested features include parent-node, sibling-node, and decision-path information, though additional features may not always improve generalization.
  • Feature re-representation: Decision tree forests may support richer re-representation because they can encode abundant information and serve as AutoEncoders.This suggests that AutoEncoder-like ability is not exclusive to neural networks.
  • Efficiency and scalability: Larger deep forests may improve generalization, but training them requires addressing speed, memory consumption, and limited GPU suitability.Potential directions include specialized hardware, distributed computing, feature sampling, component reuse, and model compression.
  • Efficiency and scalability: Feature sampling can reduce storage for long transformed vectors while increasing ensemble diversity.The paper relates this strategy to combining random tree forests with random subspaces and suggests smarter sampling methods such as BLB or feature hashing.
  • Efficiency and scalability: Reusing components and applying twice-learning may reduce storage and improve prediction efficiency for large learned models.The proposed reuse targets scanning, class-vector generation, and forest-training components.
  • Learning with limited labels: Completely-random forests can enhance ensemble diversity and may enable active or semi-supervised learning because tree growth does not require labels.Labels are needed only to annotate leaf nodes, potentially reducing annotation requirements.

7 Conclusion

The paper asks whether deep learning can be built from non-differentiable modules and proposes gcForest, a decision-tree-based deep model trained without backpropagation. Experiments find competitive performance across many domains with fewer hyper-parameters, while acknowledging inferior results on some image tasks and substantial room for improvement.

  • Motivation: The paper asks whether deep learning can be realized with non-differentiable modules by reproducing layer-wise processing, feature transformation, and sufficient complexity.These characteristics motivate a non-neural deep model.
  • Method: gcForest constructs a deep model from decision trees, and its training process does not rely on backpropagation.The method is designed to retain key characteristics associated with deep neural networks.
  • Results: gcForest uses much fewer hyper-parameters than deep neural networks and achieves excellent performance across various domains.The experiments support robustness across different tasks and data settings.
  • Implications: The study opens a direction for non-neural deep learning based on non-differentiable modules.The authors present deep forest as an alternative rather than a replacement for deep neural networks.
  • Limitations: gcForest performs competitively with deep neural networks across a broad range of tasks but is inferior on some image tasks.The authors attribute room for improvement partly to the current simple feature re-representation scheme and the relative maturity of deep neural networks.
Loading 1702.08835v4…