Source-linked AI summary
Learning with Pseudo-Ensembles
Philip Bachman, Ouais Alsharif, Doina Precup
TL;DR
The paper addresses limited analysis connecting dropout’s structural noise with ensemble and robust-learning methods, and develops a pseudo-ensemble framework and robustness regularizer. The regularizer matches dropout in fully supervised learning, extends naturally to semi-supervised learning, and improves a real-world sentiment-analysis model.
Problem
Limited analysis connected dropout’s model-structure perturbations with classic ensembles and other robust-learning approaches.
Method
The paper formalizes pseudo-ensembles and regularizes parent models to minimize output variation under noise applied to inputs and internal model states or structure.
Results
The regularizer matched dropout’s supervised MNIST performance, extended to semi-supervised learning with state-of-the-art results on some real-world datasets, and improved a pseudo-ensemble RNTN beyond the original model.
Takeaways & Limitations
Pseudo-ensembles provide a framework for developing robust-learning algorithms and improving already powerful models on sentiment analysis.
Takeaways & Limitations
In the sentiment-analysis setting, ℓ2-hinge-loss performed better than softmax →xent; switching losses degraded both dropout and PEA results without changing their ranking.
Abstract
from arXiv · showhide
We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.
1 Introduction
The paper formalizes pseudo-ensembles to connect dropout-like model perturbations with robust learning. It introduces a regularizer that matches dropout in supervised learning, extends to semi-supervised learning, and improves a sentiment model.
- Pseudo-ensembles connect dropout’s structural noise process with classic ensemble methods and robust-model learning.The paper addresses limited analysis relating dropout to these established approaches.
- The proposed regularizer minimizes variation in model outputs under noise applied to inputs and internal states or structure.It is developed within the pseudo-ensemble framework and related explicitly to standard dropout.
- In fully supervised learning, the regularizer reproduces dropout performance, while in semi-supervised learning it achieves state-of-the-art performance on some real-world datasets.
- A Recursive Neural Tensor Network is converted into a pseudo-ensemble using Gaussian parameter fuzzing and latent subspace sampling.Both perturbation types contribute to significant improvements beyond the original model.
2 What is a pseudo-ensemble?
A pseudo-ensemble is a collection of child models generated by perturbing a shared parent model with a tractable noise process. Learning seeks predictions robust to these perturbations by optimizing expected loss over noisy child models.
- A pseudo-ensemble consists of ξ-perturbed child models fθ(x; ξ) spawned from a parametric parent model fθ.The perturbation ξ is sampled from a noise process pξ.
- Dropout is a pseudo-ensemble in which randomly masked node activities extract subnetworks from a shared source network.The source network is the parent, sampled subnetworks are children, and node-mask sampling defines the noise process.
- The noise process may take arbitrary forms, provided that sampling ξ and imposing it on the parent model remain computationally tractable.The paper illustrates this flexibility with Gaussian perturbations of mixture means and Wishart perturbations of covariances.
- Supervised pseudo-ensemble learning optimizes expected loss for observations, labels, noise realizations, and the corresponding perturbed child-model outputs.The loss L measures prediction error relative to the true label.
- The framework supports novel noise processes beyond neural-network masking and linear-model feature noise.
3 Related work
Pseudo-ensembles differ from traditional ensembles by sharing parameters and optimizing individual child outputs, while extending robustness beyond input perturbations to model-space noise. This framework relates to stochastic programming, robust optimization, and noisy linear models.
- Unlike boosting, pseudo-ensembles optimize expected loss over individual members rather than the joint ensemble output.Shared parameters and structure also correlate child-model behavior.
- Robust-learning work has largely focused on input perturbations, including stochastic programming and robust optimization over parameter perturbations.
- Prior linear-model methods efficiently optimize expected performance under input noise such as Gaussian, zero-masking, or Poisson perturbations.
- Pseudo-ensembles move noise from input-space into model-space, which can reproduce input-space noise by adding an initial identity layer.Thus, model-space noise includes input-space noise as a special case.
4 The Pseudo-Ensemble Agreement regularizer
PEA regularization encourages agreement among pseudo-ensemble child models by penalizing variation in layer activities under perturbations. It extends naturally to semi-supervised learning because its penalties do not require labels.
- PEA regularization: PEA regularization controls distributional properties of layer activity vectors generated by perturbing earlier layers while leaving the current layer unperturbed.The output layer is treated as layer d, with the unperturbed parent and perturbed child providing the compared responses.
- PEA regularization: The regularizer applies variance penalties between parent and child-model activity distributions, with λ_i controlling each layer’s relative importance.The penalties can be computed between independently sampled child-model pairs, and several variance measures are possible.
- Feature co-adaptation: With output-layer KL divergence, PEA trades off parent-model accuracy against feature co-adaptation under perturbations to earlier-layer activity distributions.The supervised loss depends only on the parent model, while the pseudo-ensemble appears in the regularization term.
- Feature co-adaptation: PEA reproduces standard dropout’s performance benefits, supporting a role for discouraging feature co-adaptation and output robustness in dropout’s empirical success.The paper reports this conclusion from experiments in the fully supervised setting.
- Semi-supervised learning: PEA extends as-is to semi-supervised learning because its penalties require no labels, using labeled examples for the supervised objective and unlabeled examples for PEA regularization.The paper considers tanh-variance and xent-variance output penalties; xent-variance combines KL divergence with an entropy penalty.
5 Testing PEA regularization
The paper evaluates PEA regularization across supervised, semi-supervised, and transfer-learning settings. PEA matches dropout in supervised MNIST and outperforms prior methods in nearly all semi-supervised MNIST settings.
- PEA was tested on supervised MNIST, semi-supervised MNIST, and semi-supervised transfer learning.The experiments used implementations written in THEANO and evaluated all three scenarios.
- Supervised MNIST: 1.08% averaged error matched standard dropout’s 1.05% error on supervised MNIST.PEA used a penalty at the output layer while classification loss and gradients were computed only for the unperturbed parent network.
- Semi-supervised MNIST: Semi-supervised MNIST used labeled-set sizes of 100, 600, 1000, and 3000, averaging 10 splits except for 100 labels, which used 50 splits.The remaining MNIST training samples were treated as unlabeled data.
- Semi-supervised MNIST: PEA regularization without pre-training outperformed previous methods in every semi-supervised MNIST setting except 100 labeled samples.PL+ performed better than PEA in the 100-labeled-sample setting.
- Semi-supervised transfer learning: The transfer-learning experiment pre-trained a convolutional network on CIFAR-100 before training a non-convolutional target-domain model on 120 images.The target domain contained 120 images across 10 classes, with neither classes nor images appearing in the source domains.
6 Improved sentiment analysis using pseudo-ensembles
The paper adapts a compact Recursive Neural Tensor Network into a pseudo-ensemble using subspace sampling and weight fuzzing. Combining both perturbations substantially improves sentiment performance, reaching state-of-the-art results on the binary task.
- The compact RNTN was evaluated on fine-grained and binary sentiment prediction in the Stanford Sentiment Treebank.Both tasks measured root-level, or whole-phrase, prediction accuracy.
- Compact RNTN: The compact model replaces the original 2n × 2n × n tensor with an n × n × n tensor.Its transform matrix is M ∈ R^n×2n+1, and its classification matrix is C ∈ R^c×n+1.
- Perturbations: Pseudo-ensembles were generated with subspace sampling and weight fuzzing.Subspace sampling retained n squared randomly selected latent dimensions, while weight fuzzing added zero-mean Gaussian perturbations during training.
- Results: Adding weight fuzzing improved performance beyond the full RNTN, while adding subspace sampling improved it further.Using both noise types pushed the compact RNTN well past the full RNTN.
- Results: Combining weight fuzzing and subspace sampling produced state-of-the-art performance on the binary sentiment task.The model evaluated test-time outputs by averaging over 50 randomly sampled subspaces.
- Feedforward computation: The RNTN feedforward process parses the sentence into a tree, looks up leaf vectors, and recursively transforms child vectors at internal nodes.The tensor-based transform computes each internal node representation from its children.
7 Discussion
The discussion presents pseudo-ensembles as a unifying framework for dropout and feature noising, and as a basis for robust regularization and improved sentiment models. It also identifies semi-supervised learning as a promising area for further algorithms.
- Pseudo-ensembles unify methods such as dropout and feature noising in linear models.The framework captures child models generated by perturbing a shared parent model.
- The proposed regularizer performs well empirically and provides insight into mechanisms behind dropout’s success.The paper applies the framework to make an already powerful sentiment model perform better on a competitive benchmark.
- The authors anticipate that pseudo-ensembles can support additional successful algorithms, especially for semi-supervised learning.This is presented as a forward-looking conclusion within the paper’s proposed framework.