PaperlayerView exact revision on arXiv

Source-linked AI summary

Modular Networks: Learning to Decompose Neural Computation

Louis Kirsch, Julius Kunze, David Barber

arXiv:1811.05249v1cs.LGcs.AIstat.ML

TL;DR

Conventional networks execute all their parts for each input, making larger models costly to train. This paper learns reusable modules and data-dependent module choices jointly, without regularization to enforce module diversity. Its language-modeling comparisons favor modular networks, while image-classification training gains do not improve generalization.

  • Problem

    Conventional networks typically execute all parts for each input, and scaling them increases training time roughly quadratically.

    Source: §1 · p. 1

  • Method

    A generalized-EM procedure treats module choices as latent variables and jointly learns input-dependent decompositions and module parameters without diversity regularization.

    Source: §1 · p. 1, §2.1 · p. 4, §2.3 · p. 4

  • Results

    In language modeling, single-module selections outperform fixed-module baselines and modular networks have lower test perplexities than REINFORCE and noisy-top-k methods; image-classification training gains do not improve generalization.

    Source: §3.2 · p. 6, §3.3 · p. 8

  • Takeaways & Limitations

    The method produces deterministic module choices and modular layers that can be incorporated into neural architectures.

    Source: §5 · p. 9

    In image classification, the controller overfits to specific features, and modularization does not improve generalization; more investigation is needed for tasks with limited data.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

1 Introduction

Scaling conventional networks can sharply increase training costs because all parts typically run for every input. The paper proposes end-to-end learning of reusable modules and input-dependent module choices, without regularization to enforce module diversity.

1 Introduction

  • Conventional networks execute all parts for each input, and scaling them leads to a roughly quadratic increase in training time.The passage links this growth to both more required iterations and higher cost per sample.Source: §1 · p. 1
  • The proposed method learns a task’s decomposition into reusable modules end-to-end, treating module choice as a latent variable.The approach maximizes a variational lower bound of the likelihood.Source: §1 · p. 1
  • The algorithm learns to use a variety of modules without regularization and yields less noisy training than baselines.This contrasts with approaches that use regularization to avoid module collapse.Source: §1 · pp. 1–2
  • A small fixed number of modules is selected for each input, and only those modules’ gradients need to be computed during backpropagation.The shared module pool allows modules to be used at multiple network locations.Source: §1 · p. 2

2 Modular Networks

Modular layers select input-conditioned modules from a shared pool, and generalized EM learns both module assignments and parameters. The section also compares training approaches and explains how the method avoids module collapse.

2 Modular Networks

  • Each modular layer selects K of M input-conditioned modules, then combines their outputs by concatenation or summation for subsequent computation.The experiments use concatenation and summation; the shared module pool can be reused across processing stages.Source: §2 · p. 2

2.1 Learning Modular Networks

  • Generalized EM alternates partial E-steps that test sampled module compositions with mini-batch M-steps updating module and controller parameters to improve a variational likelihood bound.The controller guides reassignment, while module parameters adapt to the datapoints assigned to them.Source: §2.1 · pp. 3–4

2.2 Alternative Training

  • Alternative training includes REINFORCE, which estimates expectations by sampling, and noisy top-k mixture-of-experts, which evaluates and backpropagates through only k active units.The noisy top-k approach adds noise to gating outputs before disabling all but the maximum k units.Source: §2.2 · p. 4

2.3 Avoiding Module Collapse

  • Avoiding module collapse without regularization requires many gradient updates to modules and controller between E-steps; collapse yields lower likelihood, while module paths cluster during training.The figure depicts two data subsets starting with different modules and converging on the same modules by the end of training.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 Experiments

The experiments test modular networks on toy regression, Penn Treebank language modeling, and CIFAR10 image classification. They examine whether modules specialize, how training behaves, and where modularization improves performance.

3 Experiments

  • The experiments define per-example module-selection entropy as uncertainty to minimize and batch-level entropy as a measure for whether different inputs use different modules.Low batch-level entropy corresponds to module collapse; the desired value is large.Source: §3 · p. 5

3.1 Toy Regression Problem

  • With one layer, two modules, and one selected per input, each module specializes to one Gaussian component’s linear transformation, yielding perfect training and generalization performance.Loss and per-example module-selection entropy converge to zero, while batch module-selection entropy stabilizes near log 2.Source: §3.1 · p. 6

3.2 Language Modeling

  • On Penn Treebank, training uses a modified GRU with a modular state update, and tests four K-of-M configurations against REINFORCE, noisy top-k, and fixed-module baselines.The evaluation reports test perplexity after 50,000 iterations; the dataset has 0.9 million words and a vocabulary of 10,000.Source: §3.2 · p. 6
  • For K = 1, modular networks have lower test perplexity than REINFORCE and noisy top-k across configurations, and outperform fixed-module baselines when selecting from 5 or 15 modules.With three static modules, the baseline uses three times the computation yet has worse test perplexity than one intelligently selected module.Source: §3.2 · p. 6, Table 1 · p. 7
  • Training becomes less noisy and module selection becomes deterministic, while all modules are used at the end; language-model modules specialize in grammatical and semantic contexts.Examples include numerals, sentence beginnings, and occurrences of “the” before a predicted noun or adjective.Source: §3.2 · pp. 7–8, Figure 4 · p. 7, Table 2 · p. 8

3.3 Image Classification

  • On CIFAR10, modularization raises training accuracy but does not improve generalization; the controller overfits to features, while a more constrained convolutional controller reduces overfitting.The authors state that more investigation is needed to understand modularization on tasks with limited data.Source: §3.3 · p. 8, Figure 6 · p. 8

4 Related Work

The related work contrasts modular networks with mixture-of-experts, routing, and other conditional-computation approaches. It highlights deterministic composition, regularization-free module use, and automatically learned decomposition as distinctions from prior methods.

4 Related Work

  • Mixture-of-experts models combine parameterized functions as weighted sums, whereas this method selects modules deterministically and composes them hierarchically; prior stacked mixtures had not shown clear computational benefits from summing all paths.The related work identifies stacked mixtures [10] as one prior approach to sequential and parallel composition.Source: §4 · pp. 8–9
  • Generalized Viterbi EM trains diverse module use without artificial regularization, unlike REINFORCE [3] and noisy top-k gating [25], which require regularization to avoid collapse.The authors note that regularization can force optimization toward a potentially sub-optimal log-likelihood.Source: §4 · p. 9
  • Unlike [25], which assigns probabilities to individual modules, this architecture assigns probabilities to module compositions and lets each modular recurrent layer update the state itself.The authors describe this recurrent state update as a first, to their knowledge.Source: §4 · p. 9
  • Routing networks condition module selection on tasks and/or datapoints, but a reported single agent conditioned on both used only one or two modules; applying this method to multitask learning remains future work.Source: §4 · p. 9
  • In robotics, prior methods required manual specification of module decomposition and reuse, while this approach learns both automatically; applying it to policy modules is future work.Source: §4 · p. 9
  • Activation sparsity and winner-take-all computation can be difficult to parallelize on GPUs, while structured sparsity may require non-sparse training or may not be conditional.Source: §4 · p. 9

5 Conclusion

The conclusion presents a jointly learned modular decomposition with deterministic choices, no diversity regularization, and less noisy training. The method is applied to language modeling and image classification, with larger-scale and broader-domain use left for future work.

5 Conclusion

  • The method jointly learns modules and their decomposition, producing deterministic choices without regularization for module use and less noisy training than previous work.Source: §5 · p. 9
  • Modular layers can be incorporated into neural architectures, and the modular gated recurrent unit supports minimalistic sequence-to-sequence models based on modular computation.Source: §5 · p. 9
  • The method was applied to language modeling and image classification to learn modules for those tasks.Source: §5 · p. 9
  • Future work may examine scaling to larger problems, architectures, and domains, as well as applications in transfer learning, reinforcement learning, and lifelong learning.Source: §5 · p. 9

A Modular GRU

The modular GRU uses a modular layer to produce the candidate state within a recurrent update. Its controller and modules depend on the current input and previous hidden state, with parameters shared across timesteps.

A Modular GRU

  • The modular layer produces the candidate state used in the modular GRU’s recurrent update.Source: A Modular GRU · p. 12
  • At each timestep, both the controller and modules depend on the current input and previous hidden state.Source: A Modular GRU · p. 12
  • The controller parameters and module parameters are shared across all timesteps.Source: A Modular GRU · p. 12

B Toy Regression Problem

The toy regression setup uses inputs drawn from a two-component Gaussian mixture and generates targets using component-dependent linear transformations. The transformations include a random rotation and random diagonal scaling.

B Toy Regression Problem

C Language Modeling Experiments

Language-modeling training uses sequences of 35 steps and modules with 8 units. The E-step samples paths, followed by gradient updates in partial M-steps.

D Image Classification Experiments

The image-classification model has three convolutional modular layers followed by two fully connected modular layers. It executes a subset of modules per layer and compares against a parameter-matched non-modular baseline.

Loading 1811.05249v1…