PaperlayerView exact revision on arXiv

Source-linked AI summary

Ternary Neural Networks for Resource-Efficient AI Applications

Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, Frédéric Pétrot

arXiv:1609.00222v2cs.LGcs.AIcs.NE

TL;DR

Deep neural networks are costly to compute and store, limiting deployment on resource-constrained devices. The paper introduces ternary neural networks trained with a teacher-student method and implemented in dedicated hardware, achieving improved resource efficiency and competitive or better accuracy.

  • Problem

    High DNN computation and storage requirements limit deployment on smartphones, wearables, drones, and other resource-constrained devices.

  • Method

    The paper trains TNNs through a two-stage teacher-student approach and develops purpose-built FPGA and ASIC hardware using ternary weights and activations.

  • Results

    TNN ASIC hardware improves area efficiency by 147× to 635×, energy efficiency by 1.4× to 3.1×, and throughput by 2.1× to 2.7× versus TrueNorth.

  • Takeaways & Limitations

    TNNs provide a hardware-oriented route to resource-efficient deep learning by eliminating multiplications and floating-point operations while maintaining competitive accuracy.

Abstract

from arXiv · show

The computation and storage requirements for Deep Neural Networks (DNNs) are usually high. This issue limits their deployability on ubiquitous computing devices such as smart phones, wearables and autonomous drones. In this paper, we propose ternary neural networks (TNNs) in order to make deep learning more resource-efficient. We train these TNNs using a teacher-student approach based on a novel, layer-wise greedy methodology. Thanks to our two-stage training procedure, the teacher network is still able to use state-of-the-art methods such as dropout and batch normalization to increase accuracy and reduce training time. Using only ternary weights and activations, the student ternary network learns to mimic the behavior of its teacher network without using any multiplication. Unlike its -1,1 binary counterparts, a ternary neural network inherently prunes the smaller weights by setting them to zero during training. This makes them sparser and thus more energy-efficient. We design a purpose-built hardware architecture for TNNs and implement it on FPGA and ASIC. We evaluate TNNs on several benchmark datasets and demonstrate up to 3.1x better energy efficiency with respect to the state of the art while also improving accuracy.

I. INTRODUCTION

The paper targets DNN deployment on resource-constrained devices by combining ternary computation with teacher-student training and specialized hardware. TNNs use weights and activations in {−1, 0, 1} to avoid multiplications while retaining compatibility with established training techniques.

  • DNN growth in layers, weights, and floating-point computation limits deployment on drones, self-driving cars, and mobile phones.
  • Existing approaches either preserve floating-point operations while increasing sparsity or discretize weights, but discretization can reduce classification accuracy for limited efficiency gains.
  • TNNs constrain weights and activations to {−1, 0, 1} and use a teacher-student approach in which the student mimics corresponding teacher neurons without multiplications.
  • The teacher network can use dropout, batch normalization, and convolutions, while the student shares its architecture and uses ternary parameters.
  • The purpose-built TNN hardware achieves up to 2.7× better throughput, 3.1× better energy efficiency, and 635× better area efficiency than the state of the art.

A. The Teacher Network

The teacher network is trained with stochastic ternary neuron outputs while retaining otherwise flexible architecture and training methods. Its behavior and weights then guide construction of the student network through neuron-wise ternarization.

  • The teacher may use any architecture and standard training algorithm, with the sole constraint that neuron outputs stochastically take values −1, 0, or 1.
  • Stochastic firing is added after activation, using tanh, hard tanh, or soft-sign to produce values in (−1, 1) before ternarization.
  • The teacher’s unconstrained weights allow compatibility with batch normalization and dropout, while prior studies associate the ternary constraint with regularization and reduced over-fitting.
  • After teacher training, each student neuron independently imitates its corresponding teacher neuron using the teacher weights as a guide for ternary weight selection.
  • Weight ternarization uses lower and higher thresholds on teacher-neuron weights, followed by a two-threshold step activation for ternary student outputs.
  • Figure 1 compares teacher weight, activation, and ternary-output distributions with the student weight distribution and output matching process.

1) Output Ternarization:

Output ternarization sets the student neuron’s two activation thresholds from transfer-output distributions grouped by the teacher’s ternary outputs. A linear discriminant separates the three resulting output clusters.

  • Output ternarization determines two step-function thresholds for each ternary neuron given its ternary weights.
  • The method forms three student transfer-output distributions according to whether the teacher output is −1, 0, or 1.
  • A simple classifier identifies boundaries between the three distributions and uses them as the student activation thresholds.
  • The boundaries are obtained with a linear discriminant applied to kernel density estimates of the three distributions.

2) Weight Ternarization:

The method ternarizes teacher-network weights using sign-preserving thresholds and selects configurations that best reproduce teacher outputs. A layer-wise greedy search reduces the cost of threshold selection, with special handling for the output layer and optional retraining.

  • Weight Ternarization: Teacher weights are ternarized by preserving their order and sign while applying lower and upper thresholds.Weights below t_lo map to −1, weights above t_hi map to 1, and intermediate weights are set to 0.
  • Threshold Selection: The score function evaluates how well a ternary student neuron mimics the teacher across input samples.Scores accumulate when student and teacher outputs match, and the maximum-scoring threshold configuration is selected.
  • Greedy Search: A greedy dichotomic search replaces fully exhaustive threshold search to reduce the search space iteratively.The search evaluates pivot points and retains the region containing the maximum score; its worst-case complexity is O(∥W∥^2).
  • Output-Layer Ternarization: The output layer uses actual training labels rather than teacher outputs during ternarization and converges after a few passes.Output neurons are updated independently in round-robin iterations until convergence.
  • Retraining: Optional staggered retraining updates only non-ternarized teacher layers before the next layer is ternarized.Early stopping is used, and a few dozen iterations suffice for this retraining step.

III. RELATED WORK

Related work reduces neural-network resource requirements through compression, sparsity, quantization, binary computation, or specialized hardware. The paper positions complete discretization as necessary to eliminate multiplications altogether.

  • Quantized Networks: Binary and ternary methods discretize weights, activations, or gradients to reduce floating-point operations during training or inference.Examples include BinaryConnect, TernaryConnect, Ternary Weight Networks, DoReFa-Net, and QNN.
  • Binary Convolution: XNOR-Nets convert convolution calculations to bitwise operations but retain real-valued scaling factors.The scaling factors are applied after bitwise computation, so floating-point operations are reduced only to some extent.
  • Complete Discretization: Complete discretization is required to eliminate multiplications, but only a few studies address this extreme setting.The paper contrasts this requirement with approaches that are only partially discretized.
  • Fully Binary Networks: Bitwise NN discretizes inputs, weights, and outputs as binary values and uses a global sparsity parameter during training.CNNs are not supported in Bitwise-NNs according to the related-work discussion.
  • Neuromorphic Hardware: TrueNorth-compatible methods use spiking architectures and modified backpropagation to satisfy chip constraints.Their ternary weights include a synaptic connection parameter determining whether a connection exists.

IV. EXPERIMENTAL ASSESSMENT OF TERNARIZATION AND CLASSIFICATION

The experiments assess both the student’s ability to reproduce its teacher and the classification performance of fully discretized ternary networks. They span MLP and CNN architectures across several benchmark image datasets.

  • Experimental Goals: The experiments target ternarization quality relative to a real-valued teacher and classification accuracy of fully discretized ternary networks.These are identified as the two main experimental goals.
  • MLP Architectures: MLP experiments vary depth across 2, 3, and 4 layers and use 250, 500, 750, or 1000 neurons per layer.The MLPs are evaluated in a permutation-invariant manner.
  • CNN Architectures: CNN experiments use VGG-like CNN-Small and CNN-Big architectures with n = 64 and n = 128, respectively.The architecture combines convolution, max-pooling, fully connected, and L2SVM output layers.
  • Datasets: The evaluation covers MNIST, CIFAR-10, CIFAR-100, SVHN, and GTSRB benchmark datasets.These datasets include grayscale digits, color-image classification, street-view digits, and German traffic signs.
  • Training Setup: For the CNN datasets, teacher networks are trained before producing student TNNs, with ternarized weights used during teacher training.The modified training procedure is intended to improve teacher accuracy and speed up student-network construction.

A. Ternarization Performance

Ternarization quality is assessed by the accuracy gap between teacher and student networks under different exhaustive-search settings. Lower search effort generally causes only marginal degradation, while ε = 0.95 provides a favorable runtime–accuracy trade-off.

  • Accuracy Gap: Ternarization performance is measured as the accuracy difference between teacher and student networks on training and test sets.Table III reports this gap for three exhaustive-search threshold values.
  • Network Depth: Deeper networks generally achieve better ternarization performance, partly because student layers can correct errors from upper layers.Using the teacher’s original output as reference prevents errors from being amplified through the network.
  • Search Strategy: Lower ε values generally reduce ternarization performance, but the decrease is marginal and can occasionally reverse after teacher weight updates.ε = 1 denotes fully exhaustive search, whereas ε = 0 denotes fully dichotomic search.
  • Runtime–Accuracy Trade-off: ε = 0.95 achieves the optimal runtime–accuracy trade-off, with exhaustive search used for only 20% of neurons and practically zero expected accuracy gaps.For the largest 1000-neuron layer, dichotomic and exhaustive search take 2 minutes and 63 minutes, respectively.

B. Classification Performance

TNN classification is evaluated against fully and partially discretized methods across benchmark datasets, with fully discretized networks requiring no floating-point operations or multiplications. TNNs trade a small amount of accuracy against partially discretized approaches for improved energy efficiency.

  • Fully discretized methods require no floating-point operations or multiplications, unlike partially discretized methods.
  • The experiments avoid data augmentation and network ensembles to reduce unfair comparison with related methods.
  • 1.67% is TNN’s error rate on MNIST using a single 3-layer MLP with 750 neurons per layer.
  • TNN exceeds TrueNorth on CIFAR10 by more than 4% when fully discretized, while performing similarly on SVHN and worse on CIFAR100.
  • TNNs lose only a small percentage of accuracy relative to partially discretized studies while providing better energy efficiency.

V. PURPOSE-BUILT HARDWARE FOR TNN

The paper designs a hardware architecture optimized for ternary weights and activations, then evaluates its latency, throughput, energy efficiency, and area efficiency.

  • The purpose-built architecture is optimized for ternary neuron weights and activation values {−1, 0, +1}.
  • The hardware is evaluated using latency, throughput, energy efficiency, and area efficiency.

A. Hardware Architecture

The TNN hardware uses a pipelined, programmable architecture with compact ternary arithmetic, implemented for FPGA and ASIC evaluation. Measurements report high throughput and competitive energy and area efficiency against prior hardware.

  • A. Hardware Architecture: The fully connected-layer design forms a pipeline matching neural-network processing steps, with layer counts and maximum dimensions fixed at synthesis time.
  • A. Hardware Architecture: Each layer stores programmable neuron weights or output ternarization thresholds, allowing reuse of an architecture across datasets.
  • A. Hardware Architecture: Ternary values use 2-bit two’s-complement encoding, reducing each neuron to one integer adder/subtractor and one register.
  • A. Hardware Architecture: The RTL design targets both reprogrammable FPGA and ASIC technologies.
  • B. Hardware Performance: 195 K images/s throughput, 3.8 W power, and 20.5 µs latency are achieved on Sakura-X at 200 MHz.
  • B. Hardware Performance: 3.63 µJ per image, 98.14% accuracy, 255 102 images/s, and 8.09 µs latency outperform TrueNorth in the reported middle operating zone.
  • B. Hardware Performance: The ASIC version compares well with TrueNorth on throughput, area efficiency, and energy efficiency, while outperforming EIE in energy and area efficiencies.
  • B. Hardware Performance: CNN hardware results are reported for both FPGA and ASIC implementations, with worst-case FPGA measurements and dataset-specific zero-activation accounting for ASIC power.

VI. DISCUSSIONS AND FUTURE WORK

The paper presents TNNs as resource-efficient networks whose ternary activations support inherent pruning, alongside hardware with strong throughput and area efficiency. Compared with TrueNorth, the reported ASIC improves area efficiency, energy efficiency, and throughput across stated ranges.

  • TNNs avoid multiplication and floating-point operations, supporting resource-efficient deep learning for embedded applications.
  • Ternary activations let each neuron choose its sparsity parameter and remove weights with little contribution, inherently pruning unnecessary connections.
  • 147× to 635× area-efficiency, 1.4× to 3.1× energy-efficiency, and 2.1× to 2.7× throughput improvements are reported against TrueNorth.
Loading 1609.00222v2…