PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperNetworks

David Ha, Andrew Dai, Quoc V. Le

arXiv:1609.09106v4cs.LG

TL;DR

The paper addresses how to generate useful weights for deep convolutional and long recurrent networks without relying solely on shared weights. It trains hypernetworks end-to-end to generate static or dynamic weights, obtaining competitive or better results across image recognition, language modelling, handwriting generation, and translation while using fewer parameters. The approach is constrained by memory requirements in a general dynamic formulation and by evaluation-comparison limits in one language-modelling setting.

  • Problem

    The work targets practical weight generation for deep convolutional and long recurrent networks, where recurrent models traditionally use shared weights across layers or time.

  • Method

    Small hypernetworks are trained end-to-end with backpropagation to generate weights for larger main networks, using static generation for convolutional models and dynamic generation for recurrent models.

  • Results

    Hypernetworks are competitive with or sometimes better than state-of-the-art models across image recognition, language modelling, handwriting generation, and neural machine translation, while using fewer parameters.

  • Takeaways & Limitations

    The results support hypernetworks as an efficient and scalable way to generate non-shared or adaptive weights for practical convolutional and recurrent architectures.

  • Takeaways & Limitations

    The general dynamic hypernetwork formulation can require Nz times the memory of a Basic RNN, limiting the hidden units usable in practical applications.

Abstract

from arXiv · show

This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as relaxed form of weight-sharing across layers. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve near state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline models while requiring fewer learnable parameters.

1 INTRODUCTION

The paper introduces hypernetworks that generate weights for main networks, using layer-structure information and optionally dynamic embeddings to support convolutional and recurrent models.

  • A small hypernetwork generates weights for a larger main network, whose input-output behavior remains that of an ordinary neural network.The hypernetwork receives inputs containing information about a layer’s weight structure and produces that layer’s weights.
  • Layer embedding vectors describe entire weight sets and can be learned end-to-end, enabling approximate weight-sharing within and across main-network layers.
  • Dynamic hypernetworks generate embeddings over time, allowing recurrent-network weights to change across timesteps and adapt to the input sequence.
  • Hypernetworks generate non-shared LSTM weights that outperform standard LSTM, while achieving near state-of-the-art results across language modelling and handwriting generation tasks.
  • On CIFAR-10 image classification, hypernetworks obtain respectable results relative to state-of-the-art models while using fewer learnable parameters.

2 MOTIVATION AND RELATED WORK

The paper positions hypernetworks as a gradient-trained alternative to evolutionary weight-generation methods, targeting practical architectures while balancing flexibility, simplicity, and computational cost.

  • Evolutionary approaches reduce large weight-search spaces by evolving smaller networks that generate structures for larger main networks, but reported results have often remained small-scale.
  • Hypernetworks differ from HyperNEAT by training the generator and main network jointly with gradient descent, making the approach more efficient.
  • Compared with Compressed Weight Search and HyperNEAT, the approach balances model flexibility with training simplicity.The paper characterizes DCT priors as potentially too simple and HyperNEAT’s joint architecture-and-weight evolution as often excessive for practical problems.
  • Prior work used networks to predict convolutional parameters but did not explore this approach for recurrent networks, which this paper identifies as a main contribution.
  • Hypernetworks use layer embedding vectors for convolutional and recurrent architectures, while coordinate inputs can also approximately recover convolutional structure in fully connected networks.

3 METHODS

Hypernetworks generate main-network weights from layer embeddings, offering relaxed weight-sharing for convolutional and recurrent architectures. Static hypernetworks factorize convolutional kernels, while dynamic HyperRNNs generate time-varying recurrent weights; a memory-efficient variant scales rows instead of constructing full matrices.

  • Overview: Hypernetworks treat convolutional and recurrent networks as endpoints of a weight-sharing spectrum, providing a relaxed alternative between shared and unshared weights.Convolutional networks avoid weight-sharing but may use redundant parameters, whereas recurrent networks share weights across layers or timesteps.
  • Static hypernetworks: For convolutional networks, each layer embedding zj is mapped by the hypernetwork to that layer’s kernel matrix Kj.The kernel contains Nin × Nout filters, each with dimensions fsize × fsize.
  • Static hypernetworks: The convolutional hypernetwork slices Kj into Nin smaller kernels, using separate projections Wi and shared output parameters Wout and Bout to construct them.The final kernel is formed by concatenating the generated slices, while shared output parameters make the two-layer linear hypernetwork more compact.
  • Dynamic hypernetworks: Dynamic hypernetworks use a recurrent network to generate weights for another recurrent network, allowing the main network’s parameters to vary across timesteps.In HyperRNN, the hypernetwork receives xt concatenated with ht−1 and jointly trains with the main RNN through backpropagation.
  • Dynamic hypernetworks: The full dynamic formulation requires Nz times the memory of a Basic RNN, limiting the hidden units usable in practical applications.The authors address this by projecting embeddings into weight-scaling vectors that scale rows of a single matrix instead of constructing a full weight matrix.

4 EXPERIMENTS

Experiments evaluate static hypernetworks for convolutional image recognition and dynamic HyperLSTM models for language, handwriting, and translation. Hypernetworks remain competitive across these settings, with especially strong sequence-modeling results and parameter-efficiency trade-offs for convolutional networks.

  • 4.1 Using Static Hypernetworks to Generate Filters for Convolutional Networks and MNIST: 99.24% test accuracy was achieved on MNIST, comparable to 99.28% for the conventional method, while representing 12,544 kernel weights with a 4-parameter embedding.The hypernetwork itself had 4,240 parameters.
  • 4.2 Static Hypernetworks for Residual Network Architecture and CIFAR-10: Hypernetworks generated filters for deep residual networks, producing classification accuracy similar to the reported baseline implementation.The generated weights were used without modification in a batch-normalization setting.
  • 4.2 Static Hypernetworks for Residual Network Architecture and CIFAR-10: ~1.25-1.5% classification accuracy was sacrificed while drastically reducing model parameters when relaxed weight sharing was imposed on the deep residual network.Each layer retained 64 degrees of freedom to distinguish its filters from other layers.
  • 4.3 HyperLSTM for Character-Level Penn Treebank Language Modelling: HyperLSTM outperformed larger or deeper LSTM variants and matched Layer Norm LSTM, while combining HyperLSTM with Layer Norm produced an additional performance gain.Increasing embedding size or stacking HyperLSTM layers also increased performance.
  • 4.4 HyperLSTM for Hutter Prize Wikipedia Language Modelling: On enwik8, HyperLSTM was competitive with Layer Norm LSTM, and Layer Norm HyperLSTM achieved respectable results; the 2048-unit version approached state-of-the-art performance.HyperLSTM also converged more quickly per training step than LSTM and Layer Norm LSTM.
  • 4.5 HyperLSTM for Handwriting Sequence Generation: HyperLSTM’s dynamic weight adjustment produced regime changes concentrated around written words and sometimes characters, differing from statistical normalization while achieving similar performance.The recurrent cell’s policy changed model parameters over time during handwriting generation and text sampling.
  • 4.6 HyperLSTM for Neural Machine Translation: The HyperLSTM cell improved the existing GNMT model and achieved state-of-the-art single-model results for the translation dataset.The experiments also demonstrated applicability to large-scale production models.

5 CONCLUSION

The paper presents hypernetworks as end-to-end trained networks that generate weights for other neural networks. Across image recognition, language modelling, and handwriting generation, the method uses fewer parameters while remaining competitive with or sometimes exceeding state-of-the-art models.

  • Hypernetworks generate weights for another neural network and are trained end-to-end with backpropagation.The paper focuses on static hypernetworks for convolutional networks and dynamic hypernetworks for recurrent networks.
  • The method works well while using fewer learnable parameters.
  • Across image recognition, language modelling, and handwriting generation, hypernetworks are competitive with or sometimes better than state-of-the-art models.

A.1 HYPERNETWORKS TO LEARN FILTERS FOR A FULLY CONNECTED NETWORKS

The fully connected MNIST experiment tests whether a hypernetwork can learn useful filters from positional inputs. Although convolutional-like filters emerge, the resulting classifier performs substantially worse than a conventional fully connected network, motivating a different embedding approach for practical tasks.

  • An 801-parameter hypernetwork predicts hidden-layer weights from input-pixel and weight coordinates in a 784-256-10 MNIST classifier.The hidden layer contains 200704 weight parameters, whereas the hypernetwork is a four-layer feedforward ReLU network.
  • The virtual coordinates-based approach used by HyperNEAT and DPPN has limitations in practical tasks such as image recognition and language modelling.

A.2 CONCEPTUAL DIAGRAMS OF STATIC AND DYNAMIC HYPERNETWORKS

The conceptual diagrams distinguish static and dynamic hypernetworks by the type of main network whose weights they generate. Static hypernetworks target feedforward networks, while dynamic hypernetworks target recurrent networks.

  • A feedforward network is paired with a static hypernetwork that generates its weights.
  • A recurrent network is paired with a dynamic hypernetwork that generates its weights.

A.2.1 FILTER VISUALIZATIONS FOR RESIDUAL NETWORKS

The appendix visualizes ordinary and hypernetwork-generated kernels in a deep residual network. The generated 32x32x3x3 kernel is formed by concatenating four basic kernels.

  • Figures 12 and 13 compare normal and hypernetwork-generated 16x16x3x3 and 32x32x3x3 kernels.
  • The generated 32x32x3x3 kernel is constructed by concatenating 4 basic kernels.

A.2.2 HYPERLSTM

HyperLSTM extends HyperRNN by generating LSTM weights and biases from embeddings produced by a smaller HyperLSTM cell. The design can use layer normalization, efficient row-wise weight scaling, and dropout in the main LSTM.

  • A.2.2 HYPERLSTM: HyperLSTM generates weights and biases as functions of embeddings produced by a smaller HyperLSTM cell.Separate embeddings are used for the input, hidden-state, and bias components of each LSTM gate.
  • A.2.2 HYPERLSTM: Layer normalization can be added to the HyperLSTM, while dropout is generally applied inside the main LSTM rather than the smaller HyperLSTM cell.For larger systems, dropout may be applied to both networks.
  • A.2.2 HYPERLSTM: The four LSTM gates use gate-specific embeddings derived from linear projections of the HyperLSTM Cell’s hidden states.The gates are i, g, f, and o.
  • A.2.2 HYPERLSTM: The efficient HyperLSTM modifies weight-matrix rows with generated scaling vectors rather than explicitly generating complete matrices.This follows the memory-efficient HyperRNN formulation.
  • A.2.2 HYPERLSTM: The main LSTM updates its cell and hidden states using gated candidate updates, dropout, and layer normalization.The formulation applies dropout to the candidate update and layer normalization to the cell state before the hidden-state output.

A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM

The implementation places a layer-normalized LSTM cell inside the HyperLSTM to generate timestep-specific embeddings and weight-scaling vectors. Initialization choices are designed to support stable training, while dropout is confined to one main-LSTM location.

  • A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM: The authors recommend exposing HyperLSTM through the same interface as a normal recurrent-network cell.They caution that the reported initialization parameters may be far from optimal depending on the task.
  • A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM: The HyperLSTM Cell receives the concatenated input signal and main-LSTM hidden units, and uses layer normalization.It is implemented as a normal LSTM cell inside the HyperLSTM interface.
  • A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM: Embedding vectors are generated at each timestep by linear projection from the HyperLSTM Cell.The first two projection weights are initialized to zero with biases initialized to one; the third uses a small normal initialization.
  • A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM: Weight-scaling vectors are generated from the embeddings, with orthogonal initialization for selected weights and zero initialization for specified biases.Scaling vectors are initialized to 0.1 rather than 1.0 to help gradient flow.
  • A.2.3 IMPLEMENTATION DETAILS AND WEIGHT INITIALIZATION FOR HYPERLSTM: Dropout is applied only at the single gate in the main-LSTM update equation.The authors describe this gate as usable like a standard feed-forward dropout gate.

A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS

The experiments cover image classification, character-level language modeling, handwriting generation, and machine translation using task-specific data splits, optimizers, regularization, and HyperLSTM configurations. The handwriting setup lacks a test set, so its quantitative results may overfit validation data.

  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: MNIST experiments use held-out validation and test sets, image padding with random crops, and optimization with Adam or Nesterov Momentum SGD.The supplied setup includes a 55000 / 5000 / 10000 split for one MNIST experiment and a 45000 / 5000 / 10000 split for another.
  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: Character Penn Treebank models train on sequences of length 100 with Adam, gradient clipping, and dropout applied to input, output, and recurrent components.Evaluation generates the entire sequence without dynamic evaluation.
  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: The enwik8 setup uses 1800-unit networks, sequence length 250, a 256-unit HyperLSTM Cell, and embedding size 64.A larger configuration uses 2048 main hidden units, a 512-unit HyperLSTM Cell, sequence length 300, and recurrent dropout in the HyperLSTM Cell.
  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: Handwriting generation follows Graves’s architecture with a Mixture Density Network and removes samples shorter than length 300 because they contain recording errors and noise.The data are normalized and use the same train/validation split as Graves (2013).
  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: The handwriting evaluation has no test set and may overfit the validation set, motivating future evaluation on larger datasets.The authors treat qualitative sample assessment as important alongside quantitative log-likelihood results.
  • A.3 EXPERIMENT SETUP DETAILS AND HYPER PARAMETERS: The translation experiment replaces GNMT WPM-32K LSTM cells with LayerNorm HyperLSTM cells while retaining the same number of hidden units.The HyperLSTM Cell uses 128 units and embedding size 32; experiments use one model without reinforcement learning or ensembles.
Loading 1609.09106v4…