Source-linked AI summary
Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation
Zhiyong Cheng, Daniel Soudry, Zexi Mao, Zhenzhong Lan
TL;DR
The paper addresses whether binary multilayer networks trained with EBP can support multiclass image classification, beyond prior binary-classification results. It evaluates architectures and dropout on MNIST, finding good BMNN performance and improved results with dropout, while spatial configuration helps real-weight but not binary networks.
Problem
The paper investigates whether EBP-trained binary multilayer networks can extend from binary classification to multiclass image classification.
Method
The study trains binary and real-weight multilayer networks with EBP across different architectures on the MNIST digits dataset, including dropout and image spatial configuration.
Results
BMNNs achieve good MNIST classification performance with EBP, dropout significantly improves them, and image spatial configuration improves real-weight but not binary networks.
Takeaways & Limitations
EBP supports effective BMNN image classification on MNIST, although BMNN performance is not as good as that of real-weight networks.
Takeaways & Limitations
The EBP derivation relies on mean-field and large-fan-in Gaussian approximations, while the exact posterior update contains an intractable exponential summation.
Abstract
from arXiv · showhide
Compared to Multilayer Neural Networks with real weights, Binary Multilayer Neural Networks (BMNNs) can be implemented more efficiently on dedicated hardware. BMNNs have been demonstrated to be effective on binary classification tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text datasets. In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks. The performances of binary neural networks with multiple hidden layers and different numbers of hidden units are examined on MNIST. We also explore the effectiveness of image spatial filters and the dropout technique in BMNNs. Experimental results on MNIST dataset show that EBP can obtain 2.12% test error with binary weights and 1.66% test error with real weights, which is comparable to the results of standard BackPropagation algorithm on fully connected MNNs.
1 INTRODUCTION
Real-valued multilayer networks achieve strong performance but demand substantial computational resources, motivating binary-weight alternatives. Because standard gradient methods cannot directly train binary networks, the paper evaluates EBP for image classification and studies architectural and dropout effects.
- Real-valued multilayer networks can require large computational and energetic resources because effective models are often massive.The introduction cites GoogLeNet's 22 layers and tens of thousands of hidden units as an example.
- BackPropagation and similar gradient-descent methods cannot be directly used to train binary neural networks.
- EBP is evaluated for image classification with binary and real weights, while varying network depth, layer size, and dropout strategies.
2 EXPECTATION BACKPROPAGATION
Expectation BackPropagation (EBP) learns multilayer neural-network weights through Bayesian posterior updates approximated with mean-field factorization, Gaussian activations, and forward–backward propagation. The procedure supports binary weights by updating their parameterization and clipping the resulting configuration.
- Bayesian formulation: EBP seeks weights that maximize the posterior probability given labeled data, then uses the resulting configuration for MAP prediction under expected zero-one loss.The posterior is updated online as training samples arrive sequentially.
- Bayesian formulation: The exact Bayesian update is intractable because it requires storing and updating an exponential number of posterior values, so EBP uses a mean-field approximation.The approximation factorizes the posterior and permits direct marginal updates, although the resulting marginal-likelihood expression still contains an intractable exponential summation.
- Forward approximation: Large fan-in and mean-field independence let EBP approximate each layer’s normalized input as Gaussian, enabling sequential forward computation of activation distributions.The activation function is sign(um), and the forward pass obtains P(y|Wij,l) for each weight before its posterior update.
- Forward–backward updates: EBP updates weights with a forward pass followed by a backward pass that uses Taylor-expanded derivative terms to approximate the output likelihood.The backward pass propagates derivatives through layers and updates P(Wij,l) during each training epoch.
- Output computation: EBP provides deterministic and probabilistic outputs: EBP-D uses the learned configuration, whereas EBP-P computes the MAP output from the final-layer mean activation.Both outputs are derived from the learned weight configuration and forward-pass quantities.
- Binary-weight implementation: For binary-weight networks, each weight takes values in {−1,+1}; EBP updates the associated parameter h and obtains the BMNN configuration by clipping.The binary implementation computes expected weights through the parameterization and applies the update procedure summarized in Algorithm 1.
3 IMPLEMENTATION OF EBP ON IMAGE CLASSIFICATION
The paper extends EBP-based BMNN evaluation from binary text classification to multiclass image classification, including spatial image inputs and deeper or narrower architectures. Standard CNN architectures remain outside this study's scope.
- 3 IMPLEMENTATION OF EBP ON IMAGE CLASSIFICATION: EBP is evaluated on multiclass image datasets rather than only the high-dimensional text and binary-classification tasks studied previously.The experiments target image classification with multiple layers and varying hidden-unit counts.
- 3 IMPLEMENTATION OF EBP ON IMAGE CLASSIFICATION: The implementation examines EBP on architectures with multiple layers and different numbers of hidden units, including configurations with smaller fan-in.This design tests whether EBP remains effective beyond the previously evaluated setting.
- 3 IMPLEMENTATION OF EBP ON IMAGE CLASSIFICATION: Images are supplied either as flattened 1D pixel vectors or through spatial filtering that preserves image spatial configuration similarly to CNNs.The spatial-filtering approach uses local image structure when feeding inputs to the multilayer network.
- 3 IMPLEMENTATION OF EBP ON IMAGE CLASSIFICATION: Standard CNN architectures are reserved for future work, so the present implementation studies spatially informed inputs within MNNs rather than full CNNs.The paper explicitly identifies standard CNN evaluation as outside the current study.
4 EXPERIMENTS
Experiments on MNIST evaluate EBP-trained binary and real-weight multilayer networks across depth, width, dropout, and spatial-filtering configurations. Binary networks perform well, while depth and dropout help and spatial filtering benefits real but not binary weights.
- Experiment Setup: The MNIST study trains one- and two-hidden-layer networks with varied hidden-unit counts using 120 epochs, with both 1D-vector and spatial-filtering inputs.Dropout uses p = 0.8 for hidden and input units.
- Effects of Hidden Unit Number and Hidden Layer Number: Two-hidden-layer EBP-P networks outperform one-hidden-layer networks, even with 200 hidden units per layer.Increasing hidden units improves one-hidden-layer performance, whereas binary EBP-D can degrade rapidly in deeper models.
- Effects of Dropout: Dropout improves performance across all tested configurations and makes binary EBP-D performance reasonable.With dropout, one-hidden-layer performance continues improving from 800 to 1000 hidden units, unlike the no-dropout results.
- Effects of Spatial Filtering: Spatial filtering produces 3.56% error for the best binary model, worse than the 1D-input results, but improves real-weight performance.The best spatial-filtering configuration uses 256 hidden units with 13 × 13 inputs per hidden unit and 256 inputs per output unit.
- Summary: The experiments indicate that EBP works for BMNN image classification even when fan-in is only a few hundred.The reported summary also states that binary BMNNs perform below real-weight MNNs overall.
5 CONCLUSIONS
The paper evaluates EBP-trained binary multilayer networks for MNIST image classification across architectures and input-processing methods. Dropout improves BMNN performance, spatial configuration helps real-weight networks but not BMNNs, and broader validation remains necessary.
- EBP-trained BMNNs achieve good MNIST classification performance across different network architectures.
- Dropout significantly improves BMNNs trained with EBP, whereas image spatial configuration improves real-weight networks but not BMNNs.
- The conclusions are limited to MNIST; BMNN performance with EBP requires validation on other image datasets such as CIFAR10.