Source-linked AI summary
Learning Factored Representations in a Deep Mixture of Experts
David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever
TL;DR
The paper addresses how to scale neural networks by increasing parameters while computing only input-appropriate portions. It extends Mixture of Experts into a multilayer model with layer-specific gating, finding location- and class-factorized assignments on jittered MNIST and effective use of combinations on monophone speech.
Problem
Deep networks execute the entire network for every input, limiting network size despite the potential value of larger parameterized models.
Method
The Deep Mixture of Experts uses different gating and expert sets at each layer, dynamically assembling input-specific combinations with exponentially many effective paths.
Results
On jittered MNIST, DMoE learns location-dependent first-layer and class-specific second-layer experts; on monophone speech, all expert combinations are used.
Takeaways & Limitations
DMoE demonstrates effective use of complementary expert combinations while maintaining a modest model size and using different subsets of units for different inputs.
Takeaways & Limitations
The examined model continuously mixes expert outputs rather than restricting computation to the top few experts, so further extension is needed to use only a small part of the model per input.
Abstract
from arXiv · showhide
Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associating each input with a combination of experts at each layer, yet maintains a modest model size. On a randomly translated version of the MNIST dataset, we find that the Deep Mixture of Experts automatically learns to develop location-dependent ("where") experts at the first layer, and class-specific ("what") experts at the second layer. In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones. These demonstrate effective use of all expert combinations.
1 Introduction
Deep Mixture of Experts extends Mixture of Experts with layer-specific gating, creating exponentially many effective expert combinations while using different subsets of units for different inputs. On jittered MNIST and monophone speech, it learns complementary or distinctive expert assignments across combinations.
- Background: Mixture of Experts combines outputs from N expert networks using a learned gating network that maps inputs to expert mixtures.The model is presented as a stepping stone toward larger networks that compute only selected portions for each input.
- Contribution: Deep Mixture of Experts uses a different gating network at each layer to create exponentially many effective expert paths.Each input is associated with a combination of experts, allowing the model to be large while using different subsets of units for different inputs.
- Results: On two datasets, DMoE factors jittered-MNIST representations into location and class and uses all expert combinations for monophone speech.The MNIST assignments are location-dependent at the first layer and class-specific at the second layer.
2 Related Work
The paper situates DMoE among mixture-based models and conditional computation approaches. Unlike standard, probabilistic, product, and hierarchical mixtures, it dynamically assembles input-specific expert combinations through layer-conditioned gating.
- Mixture of Experts: Standard Mixture of Experts learns expert networks and a gating network whose weighted outputs produce the final class prediction.Each expert maps x to C class outputs, while the gating network assigns a distribution over the N experts.
- Probabilistic interpretation: The standard MoE output can be interpreted as class probabilities marginalized over the selected expert.The passage identifies p(e_i|x) with the gating probability and p(c|e_i,x) with the expert softmax output.
- Related models: Product of Experts combines log probabilities to form a product rather than mixing expert outputs.
- Related models: Hierarchical Mixture of Experts organizes gating networks in a tree, with expert outputs corresponding to leaves and mixed by node weights.
- Deep Mixture of Experts: DMoE dynamically assembles suitable expert combinations for each input by conditioning each layer on the previous layer’s output.This enables an exponentially large number of effective experts.
3 Approach
The Deep Mixture of Experts extends Mixture of Experts with separate gating and expert sets across layers, creating exponentially many expert paths while using different unit subsets for different inputs. Training constrains gating assignments to prevent early expert dominance, then removes that constraint for fine-tuning.
- Architecture: DMoE introduces separate expert and gating sets at multiple layers, plus a final linear layer.The mixtures are composed across layers to produce the final output.
- Training: SGD can produce a degenerate minimum in which initially successful experts overpower the others through feedback from repeated selection.Higher gating weights cause more frequent selection and further training of those experts.
- Training: A gating-assignment constraint promotes balanced expert usage during training by controlling assignments relative to their running mean.The constraint is later lifted and the model receives a second fine-tuning phase.
4 Experiments
The experiments evaluate DMoE on translated MNIST and monophone speech, comparing second-layer mixtures with single-expert, concatenated, and parameter-matched fully connected alternatives. The designs vary expert and hidden-unit configurations while keeping the first MNIST layer as a mixture.
- 4.1 Jittered MNIST: MNIST uses grayscale 36 × 36 images formed by random uniform translations of ±4 pixels, with ten digit classes.The test set uses one random translation per image.
- 4.1 Jittered MNIST: The second-layer mixture is compared with a single fixed expert and concatenated experts, while the first layer remains a mixture.For h hidden units, concatenation uses N · h hidden units, providing a larger comparison model.
- 4.1 Jittered MNIST: DMoE is also compared with one-layer and fully connected deep networks matched for total parameter count.The parameter-matched fully connected network expands first-layer units while retaining the same second-layer units.
- 4.2 Monophone Speech: The speech dataset contains approximately one million 440-dimensional samples with 40 possible phoneme classes.Each sample has 11 frames spaced 10ms apart and 40 frequency bins.
- 4.2 Monophone Speech: The speech model uses 4 first-layer experts, 16 second-layer experts, and 128 hidden units at both layers.Each gating network has two layers with 64 hidden units.
5 Results
DMoE achieves competitive performance while learning complementary expert assignments across layers: translation-dependent first-layer experts and class-dependent second-layer experts on jittered MNIST, with distinct expert combinations also used for speech.
- Jittered MNIST: DMoE often suffers test-set overfitting, but performs almost as well as a same-parameter fully connected DNN.Across model sizes, its training error generally falls between the single-expert and concatenated-expert baselines.
- Jittered MNIST: The first layer assigns experts by translation, whereas the second assigns experts by class, making their combinations effective.Assignments are uniform across class for the first layer and across translation for the second.
- Jittered MNIST: Expert-combination examples preserve translation across columns and class across rows, while grouping easily confused classes such as 3 and 5.First-layer experts occupy rows and second-layer experts occupy columns.
- Jittered MNIST: Table 1 compares DMoE with single-second-layer, concatenated-second-layer, and same-parameter DNN baselines.The DMoE and baselines are evaluated on randomly translated MNIST.
- Monophone Speech: On monophone speech data, each expert combination handles a distinct input portion, and second-layer choices depend little on first-layer choices.The figure shows 16 combinations using four second-layer experts for visualization.
6 Conclusion
DMoE is a promising step toward large sparse models, but its continuous mixtures do not yet restrict computation to only a small subset of experts.
- Conclusion: DMoE uses continuous mixtures of expert outputs rather than restricting computation to the top few experts.Top-expert restriction is identified as necessary to achieve the goal of using only a small part of the model per input.
- Conclusion: For jittered MNIST, DMoE factorizes representations into translation and class, while monophone speech uses distinctive expert combinations.These patterns demonstrate effective use of all expert combinations.