Parallel training of DNNs with Natural Gradient and Parameter Averaging

Daniel Povey; Xiaohui Zhang; Sanjeev Khudanpur

Source-linked AI summary

Parallel training of DNNs with Natural Gradient and Parameter Averaging

Daniel Povey, Xiaohui Zhang, Sanjeev Khudanpur

arXiv:1410.7455v8Oct 27, 2014cs.NEcs.LGstat.ML

TL;DR

The paper addresses how to train large DNNs across machines without excessive communication, using periodic parameter averaging with efficient NG-SGD. Experiments report improved convergence and enable the averaging approach, while the authors acknowledge limited theoretical explanation and evaluation scope.

Problem
Large-scale DNN training needs multiple machines without the excessive network traffic of communicating parameters after every minibatch.
Method
The framework periodically averages and redistributes parameters across independent SGD jobs, while NG-SGD uses an efficient approximate inverse-Fisher update with factored matrices.
Results
NG-SGD improves convergence over plain SGD and makes periodic parameter averaging effective for parallel training across machines.
Takeaways & Limitations
Periodic parameter averaging can support parallel training on machines lacking fast interconnections when combined with NG-SGD.
Takeaways & Limitations
The paper does not explain why parameter averaging works with non-convex DNNs or why NG-SGD is especially helpful, and reports results from only one setup.

Abstract

from arXiv · show

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.

1 INTRODUCTION

The framework trains DNNs across separate machines by infrequently averaging and redistributing parameters, reducing communication compared with minibatch-level synchronization. It combines this approach with efficient NG-SGD, which empirically improves its effectiveness.

1 INTRODUCTION: Multiple SGD processes train on separate machines with different data, averaging and redistributing model parameters only every minute or so.The framework targets large-scale speech-recognition training while limiting network traffic.
1 INTRODUCTION: The paper describes the methods and establishes empirically that they work well, without developing proofs explaining parameter averaging or NG-SGD’s effectiveness.The authors explicitly frame the contribution as practical and empirical.
1 INTRODUCTION: The framework includes two NG-SGD variants: a simple method and an online method.Technical details are provided in Appendices A and B.

2 PROBLEM SETTING

The problem setting is DNN-based speech recognition, where acoustic feature vectors are classified into clustered HMM-state labels and trained using frame-level log-probabilities.

2 PROBLEM SETTING: DNNs classify acoustic feature vectors x ∈ R^D into discrete labels y ∈ Y representing clustered HMM states of context-dependent phones.The typical label inventory has |Y| ≃5000 classes.
2 PROBLEM SETTING: Each training pair corresponds to one speech frame, with features derived from short-time spectral properties and adjacent frames spliced together.Frames are typically extracted at 100 per second with 25-millisecond duration.
2 PROBLEM SETTING: Training maximizes the sum of log p(y_i|x_i) over all training frames.The paper calls this SGD although maximization makes it gradient ascent.
2 PROBLEM SETTING: The resulting log-probabilities for all labels are used as costs in Viterbi search for the most likely word sequence.Supervision labels come from an HMM Viterbi alignment of each reference utterance.

3 SGD WITH PARAMETER AVERAGING

Parameter averaging runs independent SGD jobs that periodically synchronize by averaging parameters, reducing communication while requiring learning-rate scaling and stability controls. The implementation also addresses divergence and compares learning-rate schedules across methods.

3.1 PARAMETER-AVERAGING OVERVIEW: N machines independently process different randomized data subsets, then average and redistribute parameters after each has processed K samples.Typical settings use N = 4 and K = 400 000, repeating across a specified number of epochs.
3.1 PARAMETER-AVERAGING OVERVIEW: An outer iteration is the time required for each job to process K training examples.The number of outer iterations per epoch depends on K and the dataset size.
3.2 LEARNING RATE AND STABILITY: The effective learning rate is η_t/N, so increasing jobs requires increasing η_t proportionally to N for linear speedup.Averaging dilutes each individual job’s update N-fold.
3.2 LEARNING RATE AND STABILITY: Parameter averaging is chosen over summing updates because summation can overshoot equilibrium and destabilize training.The stated concern arises when curvature and learning rate make SGD approach equilibrium within K samples.
3.2.1 LEARNING RATE SCHEDULE: Experiments use an exponentially decreasing learning rate, typically from 0.01 to 0.001, across 4 to 20 epochs.The schedule was not separately tuned for SGD and NG-SGD in the reported large-scale experiments.
3.2.2 MAXIMUM PARAMETER CHANGE: A maximum parameter change per minibatch is enforced to prevent parameter divergence, especially early in training near output layers.Without control, parameters can become very large and the objective can approach negative infinity.

4 NATURAL GRADIENT FOR SGD

NG-SGD replaces scalar learning rates with an approximate inverse-Fisher matrix, implemented efficiently through factored blocks and estimated by simple or online procedures. The method is designed to make natural-gradient updates practical for large DNNs.

4 NATURAL GRADIENT FOR SGD: NG-SGD scales gradients by a symmetric positive definite approximation to the inverse Fisher matrix.This matrix serves as the learning-rate component of the update.
4 NATURAL GRADIENT FOR SGD: Natural Gradient here denotes an approximate inverse-Fisher method rather than exact movement along the computationally difficult Riemannian gradient path.The terminology follows prior work using the same approximation.
4.1 WE CAN REPLACE THE SCALAR LEARNING RATE IN SGD WITH A MATRIX: A scalar learning rate can be replaced by a symmetric positive definite matrix that preconditions the sampled objective gradient.The scalar schedule η_t remains separate from the matrix component E_t.
4.2 THE INVERSE FISHER MATRIX IS A SUITABLE LEARNING-RATE MATRIX: The Fisher information matrix is motivated as an expected outer product of score derivatives and can share Hessian-like transformation properties.Its equality with the Hessian requires stringent conditions, including a correct model at the true distribution.
4.3 WE NEED TO APPROXIMATE THE FISHER MATRIX IN PRACTICE: For millions of parameters, direct Fisher inversion costs O(n^3), motivating factored approximations.Prior approaches use block or low-rank structure; this work uses one block per weight matrix.
4.4 OUR FACTORIZATION OF THE FISHER MATRIX: Each weight-matrix block is represented as a Kronecker product of factors A_i and B_i, each approximated by λI+XX^T.The implementation avoids explicitly constructing Kronecker products or vectorized weight matrices.
4.5 ESTIMATING THE FACTORIZED FISHER MATRIX: The simple estimator uses held-out samples from the current minibatch, whereas the online estimator uses previous minibatches with a forgetting factor.The online method is generally preferred because it is faster on GPUs and usually learns faster with less noisy estimates.
4.6 IMPLEMENTATION: Minibatch updates use matrices X_t and Y_t, whose rows represent backpropagated derivatives and corresponding weight-matrix inputs.The update applies factor-based transformations to these quantities without dividing gradients by minibatch size.

5 COMMENTS ON THE THEORY

The paper gives limited proof claims for its factored-Fisher NG-SGD methods, relying on assumptions that do not hold in practical deep learning. It identifies convergence conditions and open analysis problems for finite minibatches and online covariance estimation.

The authors state that this is not a theoretical paper and provide no proofs, instead discussing what may or may not be provable.
Under Gaussian independence assumptions, the Fisher matrix is expected to have the proposed factored form, with factors corresponding to uncentered covariances.
The assumed conditions do not hold in practical deep learning, although the authors consider the factorization reasonable.
For factored-Fisher learning rates, the authors believe a slightly modified method could be shown to converge under conditions similar to ordinary SGD.
The current minibatch rescaling makes the Fisher matrix depend on the current sample, complicating convergence proofs and motivating alternative rescaling or asymptotic analyses.
Finite-minibatch online covariance estimation does not converge to the true covariance because of noise and finite forgetting, though another approximation objective might be analyzable.

6 EXPERIMENTS

Experiments evaluate the framework on a 1600-hour Fisher English conversational-speech task using GMM and DNN systems. NG-SGD improves convergence across job counts, while larger parallel runs show sub-linear speedups and final WER is relatively insensitive to job count.

System details and Word Error Rate performance: The Fisher English3 experiment uses 1600 hours of transcribed conversational telephone speech and a self-defined 3.3-hour held-out test subset.
System details and Word Error Rate performance: The GMM baseline uses 40-dimensional LDA+MLLT features with fMLLR adaptation, 7 880 context-dependent states, and 300 000 Gaussians.
System details and Word Error Rate performance: DNN1 is a 19.3-million-parameter p-norm network trained for 12 epochs with 8 parallel jobs using online NG-SGD.
Results: The main evaluation plots training objective against processed data and simulated time for parallel plain SGD and NG-SGD with 1, 2, 4, 8, and 16 jobs.
Results: NG-SGD curves are consistently above plain-SGD curves, while online NG-SGD is similar through 4 jobs and converges somewhat slower with 8 and 16 jobs.
Results: For small job counts, training achieves linear speedup, but speedup becomes sub-linear beyond roughly 4 jobs because more epochs are needed for equivalent improvement.
Results: The simulated-time comparison uses outer-iteration times of 88 seconds for plain SGD, 93 seconds for online NG-SGD, and 208 seconds for plain NG-SGD on a K20 GPU.

7 CONCLUSIONS

The paper presents NG-SGD as improving convergence and enabling periodic parameter averaging across separate SGD runs. The authors note that the mechanism behind this dependence is not well understood.

NG-SGD experimentally improves convergence compared with plain SGD.
Periodic parameter averaging and redistribution enables parallel training across multiple SGD runs.The method is intended to support machines without fast interconnections.
The authors do not have a good explanation for why parallel training works only with Natural Gradient.They suggest that smaller parameter steps and robustness to training-set reorderings may be relevant.

A.1 OVERVIEW OF SIMPLE NATURAL GRADIENT METHOD

The method estimates inverse-Fisher transformations for minibatch rows using information from the other rows, then returns a modified matrix for training.

The method uses other minibatch elements to estimate the factors of the Fisher matrix.
Given a minibatch matrix X, it applies inverse-Fisher multiplication to each row and returns a modified matrix ¯X.Rows represent minibatch elements, while columns match a weight-matrix row or column dimension.
For each row, the Fisher matrix is estimated from the other rows of the minibatch.
The method smooths the Fisher estimate with the identity matrix and rescales the output to match the input Frobenius norm.

A.2 DETAILS OF METHOD (NOT CONSIDERING EFFICIENCY)

The simple natural-gradient method smooths Fisher matrices, transforms each minibatch row, and rescales the result, while accepting a small minibatch-related contamination.

The simple method smooths the Fisher matrix with the identity before transforming minibatch rows.
The method defines scalar β from the input matrix norm, minibatch dimensions, and constants α = 4 and ǫ = 10^-20.
The transformed output is rescaled so ¯X has the same Frobenius norm as input X.
If the rescaling denominator is zero, γ is set to one.
Computing β and γ without holding out the current sample violates independence, but the authors expect little bias with minibatches of at least 100.

A.3 EFFICIENT COMPUTATION IN SIMPLE METHOD

The efficient implementation avoids costly full-dimensional operations through low-rank updates, adaptive computation choices, and GPU/CPU division of labor. The online variant maintains a stateful low-rank Fisher approximation and rescales each output.

The implementation computes inverse-Fisher transformations efficiently using low-rank structure and avoids explicitly storing some large factors.
It chooses column-space computation when N > D and row-space computation otherwise.
On CPUs the simple computation can take about 20% of backpropagation time, but on GPUs with larger minibatches it can take the majority of the time.
The online method estimates a rank-R Fisher approximation from minibatch data, with R smaller than the input dimension D.
The method adds identity smoothing, updates ρ and D, and floors them at ǫ to avoid exact zeros.
The output is rescaled to preserve the Frobenius norm, with γ set to one when the denominator is zero.
The online update uses a power-method-inspired procedure and symmetric eigendecomposition to update covariance directions and scaling factors.
GPU computation is combined with CPU processing for small matrix expressions and eigendecomposition, while Woodbury identities reduce inverse-Fisher multiplication.

B.5 SUMMARY OF THE ONLINE NATURAL GRADIENT METHOD

The online natural-gradient method processes minibatches through a factored Fisher-matrix update, with periodic updates and scaling designed for efficient and stable training. The broader implementation combines this with parameter-change limits, parallel averaging, and model combination.

Algorithm overview: The algorithm maps an input matrix X ∈ R^N×D to an output matrix X̄ ∈ R^N×D as part of online natural-gradient SGD.The described instance corresponds to one row or column dimension of a weight matrix.
Fisher-matrix updates: The factored Fisher matrix is updated initially and whenever J divides t; otherwise, the existing approximation is applied without updating.The configuration uses J = 4, so Fisher-factor updates normally occur every four steps after the initial iterations.
Scaling and output: The method computes η from minibatch size N and S, then applies a scaling factor γ_t to produce the main output X̄_t = γ_t X̂_t.The implementation also outputs per-row quantities needed to enforce the maximum parameter-change constraint.
Implementation branches: For N > D, the implementation uses one strategy for computing L_t and K_t, while smaller minibatches use a different computation for L_t.GPU implementations transfer the resulting symmetric matrices to the CPU for small derived computations and eigendecomposition.
Stability controls: A maximum parameter-change constraint is applied separately per layer to reduce instability and parameter divergence, although the authors do not claim the method is exceptionally good.The constraint is enforced by scaling the update, and the implementation avoids explicitly creating a temporary matrix for the norm calculation.

C.6 INPUT DATA NORMALIZATION

The input-normalization method uses within-class and between-class covariance statistics to rescale discriminative directions while avoiding unstable transforms. The implementation also uses layer-wise backpropagation and dataset-dependent initialization choices.

Covariance-based normalization: The normalization method uses within-class covariance W and between-class covariance B accumulated from class-labeled data.It is designed to be invariant to arbitrary affine transforms of the input.
Covariance rescaling: After an LDA-type transform, each direction is scaled by (b_i + 0.001)/(b_i + 1), making the resulting total covariance b_i + 0.001.Here b_i is the ratio of between-class to within-class covariance.
Numerical stability: A singular-value decomposition floors singular values, limiting large transformed values when the training covariance is close to singular.The flooring step is motivated by possible mismatched test data or roundoff effects and usually affects only a handful of singular values.
Parameter initialization: Instead of generative pre-training, the implementation uses layer-wise backpropagation that incrementally adds hidden layers and retrains briefly after each addition.The final softmax parameters are discarded when each new hidden layer is added.
Parameter initialization: For smaller datasets, initializing earlier layers from a network trained on another large dataset can improve results over layer-wise backpropagation.The authors typically use a larger network when initializing this way.
Parallel-training safeguard: Parameter averaging can degrade the objective immediately after random initialization, so those iterations select the locally best model instead of averaging.The selection uses each model’s objective on the subset of data on which it was trained.

C.8 SEQUENCE TRAINING

For sequence training, the framework applies model averaging and natural-gradient SGD to lattice-based objectives, adapting minibatch construction and learning rates to sequential data. Online decoding uses only past information through i-vector inputs and achieves performance comparable to the prior recipe.

Sequence-training objectives: Sequence training uses model averaging and NG-SGD, with state-level Minimum Bayes Risk as the usual objective and MPE and Boosted MMI also implemented.The paper’s main focus remains parallel standard cross-entropy training.
Lattice-based training: Parallel sequence training randomizes small lattice pieces, removes pieces with zero derivatives, and forms minibatches from chunks drawn across utterances.Learning rates are adjusted so layers undergo similar relative parameter changes per outer iteration.
Natural-gradient scope: The simple NG-SGD method is not applicable to sequence-training minibatches because their frames are sequential rather than independently sampled.Only the online natural-gradient method is applied in this setting.
Online decoding: Online decoding avoids backward-in-time dependencies by using i-vectors and acoustic features based only on information available through the current time.The i-vector is extracted from the speaker’s audio up to and including the current time.
Online decoding: The online setup uses an i-vector plus ±7 frames of MFCCs without cepstral mean normalization, with training arranged to support both per-speaker and per-utterance decoding.Speakers are split into artificial speakers with no more than two utterances for training.
Reported outcome: The online feature method generally matches the previous fMLLR-adapted recipe across several datasets, while offering greater convenience for applications and cross-system transfer learning.The comparison is against mean-normalized MFCCs processed with LDA and MLLT and using ±4 frames.