PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures

John R. Hershey, Jonathan Le Roux, Felix Weninger

arXiv:1409.2574v4cs.LGcs.NEstat.ML

TL;DR

Model-based methods encode problem knowledge but make inference difficult, whereas deep networks offer straightforward fixed-depth inference but generic architectures. The paper unfolds iterative inference into layers and unties parameters, yielding model-guided architectures that achieve competitive speech-enhancement accuracy with far fewer parameters.

  • Problem

    Model-based methods can incorporate problem knowledge but require difficult inference, while conventional DNNs offer straightforward inference but make knowledge incorporation and architecture design difficult.

  • Method

    The paper unfolds model-based inference iterations into network layers, unties parameters across layers, and applies the framework to MRFs and NMF-based speech enhancement.

  • Results

    The best deep NMF topology achieved an SDR of 10.20 dB versus 9.57 dB for the best DNN, with comparable parameter counts.

  • Takeaways & Limitations

    Deep unfolding provides a framework for deriving model-guided deep architectures and training methods that would otherwise be difficult to obtain.

  • Takeaways & Limitations

    Speech-enhancement extensions remain open for models with continuity constraints or factorial structure, and further work is proposed for other inference algorithms.

Abstract

from arXiv · show

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are generic and it is unclear how to incorporate knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a model-based approach and an associated inference algorithm, and \emph{unfold} the inference iterations as layers in a deep network. Rather than optimizing the original model, we \emph{untie} the model parameters across layers, in order to create a more powerful network. The resulting architecture can be trained discriminatively to perform accurate inference within a fixed network size. We show how this framework allows us to interpret conventional networks as mean-field inference in Markov random fields, and to obtain new architectures by instead using belief propagation as the inference algorithm. We then show its application to a non-negative matrix factorization model that incorporates the problem-domain knowledge that sound sources are additive. Deep unfolding of this model yields a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters.

1 Introduction

Deep unfolding combines model-based problem structure with the trainability and fixed-depth inference of deep networks. It unfolds iterative inference into layers, unties parameters across layers, and applies the framework to MRFs and speech enhancement.

  • Framework: Deep unfolding converts the iterations of a model-based inference algorithm into neural-network layers and unties parameters across layers for discriminative gradient-based training.This retains the model’s internal structure while allowing a fixed number of layers to be optimized for performance.
  • Motivation: Model-based methods incorporate problem-level knowledge but can have mathematically and computationally intensive inference.Examples of usable constraints include signal additivity, visual occlusion, geometry, sparsity, and low-rank structure.
  • Motivation: Conventional DNNs provide finite closed-form inference and strong application-specific performance, but make prior-knowledge incorporation and architecture design difficult.The paper characterizes conventional DNNs as closer to mechanisms than problem-level formulations and essentially black-box methods.
  • Applications: The framework generalizes unfolded inference across models and algorithms, including sigmoid networks from mean-field inference and alternative architectures from belief propagation.A power-mean formulation is used to unify and generalize the MRF architectures.
  • Speech enhancement: Unfolding and untying NMF iterations yields a non-negative deep network with multiplicative backpropagation-style updates that preserve non-negative parameters.The architecture incorporates the assumption that sound-source power spectra are approximately additive.
  • Speech enhancement: Speech-enhancement experiments found the unfolded NMF model competitive in accuracy with conventional sigmoid DNNs while requiring a tenth as many parameters.The contribution list identifies speech enhancement as the experimental domain for evaluating the approach.

2 General formulation of deep unfolding

The general formulation treats inference as an iterative optimization process and unfolds its updates into a neural-network-like computation graph. Untying parameters across layers expands the representable inference functions, while introducing over-fitting risk.

  • Inference formulation: Deep unfolding applies to models whose inference is an optimization problem, including variational inference, loopy belief propagation, and non-negative matrix factorization.These examples use different optimization or update procedures to estimate latent quantities, enforce marginal consistency, or optimize a non-negative basis expansion.
  • Inference formulation: The framework defines model parameters, hidden quantities, observed variables, an iterative update rule, and a discriminative objective over the inferred outputs.The discriminative objective is generally bi-level because inferred outputs depend on an optimization problem parameterized by the model.
  • Training: Gradient descent can optimize parameters through the iteration sequence, with truncation used to align training-time inference with the test-time procedure.The derivatives are computed recursively in a backpropagation-like manner.
  • Unfolding: Inference iterations become neural-network layers: intermediate variables form layer nodes, and each update determines the transformation and activation between layers.The final iterates provide the output-layer nodes.
  • Parameter untying: Untying parameters across layers can represent a more complex range of inference functions than the original model, which remains a special case.The stated cost is a possibility of over-fitting, handled similarly to over-fitting in deep neural networks.

3 Applications to Markov random fields

The paper applies deep unfolding to Markov random fields, showing that mean-field inference can yield conventional sigmoid networks while belief propagation produces alternative architectures. Untying parameters across layers and generalizing message updates expands the range of resulting deep networks.

  • 3.1 Mean field inference: Mean-field inference in binary pairwise MRFs unfolds into conventional sigmoid networks with a special structure connecting inputs to all layers.Untying parameters can modify this structure to recover the full generality of conventional sigmoid networks.
  • 3.1 Mean field inference: Conventional feed-forward sigmoid networks can be interpreted either as mean-field inference in similarly structured MRFs or as deeper unfoldings of more compact models.This interpretation permits changing the model structure or inference algorithm to generate network variants.
  • 3.2 Belief propagation: Belief propagation provides an alternative unfolding route because it computes exact posteriors on tree-structured graphs and can motivate architectures for general MRFs.The framework also supports structured base models and feed-forward update schedules.
  • 3.3 Generalized message passing: Mean-field and belief-propagation messages can be unified through a weighted power mean, encompassing both update types as special cases.For BP, the message schedule can also be optimized as part of deep unfolding rather than selected heuristically.
  • 3.3 Generalized message passing: Sigmoid-based unfolded architectures face optimization difficulties from vanishing gradients, while generalized BP activations may be even more complicated.The unnormalized log form is presented as a possible simplification.

4 Deep non-negative matrix factorization

The paper unfolds non-negative matrix factorization into a deep non-negative network for source separation while retaining the model’s additive signal assumption. Untied layer parameters and multiplicative backpropagation-style updates support discriminative training under non-negativity constraints.

  • 4 Deep non-negative matrix factorization: NMF represents non-negative spectral features using source-specific non-negative basis vectors and activation coefficients.The source bases and activations are arranged in matrices, with optional column normalization to avoid scaling indeterminacy.
  • 4 Deep non-negative matrix factorization: The NMF inference objective minimizes β-divergence under non-negativity constraints, with β = 1 giving generalized KL divergence and β = 2 giving squared error.An L1 sparsity constraint favors solutions with few active basis vectors at a time.
  • 4 Deep non-negative matrix factorization: Deep unfolding produces a more powerful architecture than the original iterative NMF method while retaining the basic assumption that sound sources are additive.After K iterations, source reconstruction uses a Wiener filtering-like approach that enforces source estimates summing to the mixture.
  • 4 Deep non-negative matrix factorization: Discriminative NMF addresses the limitation that independently trained source bases are not jointly optimized for separation performance from mixtures.Its objective combines reconstruction terms and application-dependent source weights, including emphasis on speech in speech denoising.
  • 4 Deep non-negative matrix factorization: Deep NMF unfolds the entire NMF model into a deep non-negative neural network and unties its parameters across layers as Wk.The resulting model is called deep NMF.
  • 4 Deep non-negative matrix factorization: Multiplicative backpropagation-style updates train each unfolded Wk matrix while respecting non-negativity constraints.The method recursively back-propagates positive and negative gradient parts to form multiplicative updates.

5 Experiments

The experiments evaluate DNN and deep NMF source-separation systems on the CHiME development set, using varied network topologies and training procedures. The DNN baseline estimates a mask applied to the mixture, while deep NMF preserves an additive signal model.

  • Experimental setting: The evaluation uses reverberated noisy speech mixtures from the CHiME challenge, with speaker-independent WSJ-0 data and non-stationary home-environment noise.Training, development, and test mixtures span six SNRs from -6 to 9 dB.
  • DNN baseline: DNN inputs concatenate T = 9 consecutive spectral-feature frames, with logarithmic magnitude spectra used instead of the deep NMF representation.The DNN has K −1 tanh hidden layers and a logistic output layer.
  • DNN baseline: The DNN output is trained as a time-frequency mask whose application to the mixture reconstructs clean speech rather than estimating speech directly.This objective produces 1.5 dB improvements relative to mask estimation.
  • Training and evaluation: DNN topologies are compared by SDR on the CHiME development set, using back-propagation, stochastic gradient descent with momentum, layer-wise pre-training, and regularization.Early stopping and Gaussian input noise are used for the DNN experiments.
  • Deep NMF setup: Deep NMF experiments use sparse NMF as a baseline and combine KL-divergence updates in lower layers with squared error in the discriminative top layer.Bases are initialized from SNMF models, and experiments vary layer count, context, and basis sizes.

6 Discussion

Deep NMF improves over sparse NMF and can outperform conventional DNNs with comparable or substantially fewer parameters. Increasing discriminatively trained layers generally improves performance, but deeper networks increase inference cost and leave speed–accuracy trade-offs for future study.

  • Performance: 10.20 dB SDR is achieved by the best deep NMF topology, exceeding the best DNN result of 9.57 dB with comparable parameter counts.The models use 4.8M and 5.5M parameters, respectively.
  • Performance: 9.64 dB SDR is achieved by the smallest deep NMF topology that beats the best DNN, using only 440K parameters.Only 80K parameters are discriminatively trained.
  • Topology effects: Discriminative training of the first layer produces the largest improvement, while training additional layers consistently improves performance, especially at low SNRs.Each added layer contributes only a modest number of parameters.
  • Topology effects: For the same model size, K = 25 layers provide large performance gains without increasing training time and complexity.The trade-off is increased computation cost during inference.
  • Future work: Further experiments are needed to identify intermediate topologies with the best speed–accuracy trade-off and to extend unfolding to continuity, factorial, loopy belief-propagation, and variational-inference models.These directions are presented as potential future work.
  • Conclusion: The conclusion presents deep unfolding as a general framework for using model-based problem constraints to guide the design of otherwise difficult-to-navigate deep architectures.The paper demonstrates this with MRF inference and non-negative matrix factorization.

A Derivations for parameters of binary MRFs

This appendix section explains how sigmoid-network parameters can be derived from binary MRF parameters and how the mapping can be reversed.

  • Sigmoid-network parameters are derived from binary MRFs, and an equivalent set of MRF parameters can be extracted from sigmoid-network parameters.

A.1 Binary MRF to sigmoid network parameters

The derivation specializes binary MRF parameters to binary hidden and visible variables and accounts for constants and duplicated edge potentials in the energy expression.

  • The binary MRF derivation uses hi and vl in {0, 1}, indexed over hidden and visible units, with 2 × 2 log-potential matrices.
  • Adding a constant c independent of h changes the parameter relation to a = b + c without affecting the relevant hidden-state dependence.
  • The factor 1/2 in hT Ah prevents double-counting symmetric edge potentials, while h-independent terms are omitted because they do not affect p(h|v).

A.2 Sigmoid network to binary MRF parameters

The section derives binary MRF parameters from sigmoid-network parameters and characterizes generalized message passing through power-mean formulations. Feedforward networks emerge as a special case with uniform backward messages, while limiting exponents recover standard sigmoid, mean-field, and belief-propagation behaviors.

  • A.2 Sigmoid network to binary MRF parameters: Sigmoid-network parameters A, b, and C can be converted into binary MRF parameters ψ using the same underlying model.The derivation specifies ψ values for hidden-hidden and hidden-visible edges, with bias assignments chosen without changing p(h|v).
  • A.2 Sigmoid network to binary MRF parameters: Assigning bias terms to individual edge potentials does not affect p(h|v) because the corresponding terms are independent of the neighboring visible variable.Constants of proportionality in the potentials likewise factor out of the final conditional distribution.
  • A.2 Sigmoid network to binary MRF parameters: With untied feedforward parameters, treating the backward-going message as uniform cancels that message and yields a feedforward activation update.The uniform case is expressed as µi←j = 1/2.
  • A.2 Sigmoid network to binary MRF parameters: As λ → 0, the generalized message and belief updates reduce to the standard sigmoid activation function.The construction uses power-mean limits to establish the sigmoid special case.
  • A.2 Sigmoid network to binary MRF parameters: The message-passing family interpolates between mean field and belief propagation, and between sum-product and max-product BP through λ and κ.λ = 1 gives BP, λ → 0 gives MF, κ = 1 gives sum-product BP, and κ → ∞ gives max-product BP.

D Derivations for deep NMF

The deep NMF derivations define the unfolded architecture, objective, and gradients for source reconstruction. They support forward computation followed by backward differentiation, with a final-layer discrepancy chosen to match the reconstruction target and evaluation criterion.

  • D Derivations for deep NMF: Deep NMF initializes H0 and defines an unfolded architecture whose objective function is optimized across its layers.The architecture is specified with H0 initialized, for example, randomly.
  • D Derivations for deep NMF: Source-specific basis functions and activations are stacked using index sets Il, allowing each source to have a different number Rl of basis functions.The set Il contains the indices associated with source l.
  • D Derivations for deep NMF: Gradients for the kth layer are computed directly at the final layer and by the chain rule for lower layers.The rewritten expression supports a backward pass after the forward values hk have been computed.
  • D Derivations for deep NMF: Using Euclidean distance at the final layer for magnitude spectra directly corresponds to maximizing SNR, up to the difference between noisy and oracle phases.The resulting objective is therefore an upper-bound optimization for actual SNR.
  • D Derivations for deep NMF: The single-source reconstruction setting uses β = 2 with a Wiener filter at the reconstruction layer, while the source-specific matrices and activations define the reconstructed source.The reconstruction focuses on one source, such as speech, using the L2 norm.

D.3 Intermediate (k + 1-th) layer derivative w.r.t. Hk and Wk, k < K

This section derives recursive positive and negative gradient computations for intermediate deep-NMF layers and turns them into multiplicative updates. The formulation combines KL-divergence intermediate updates with L2 reconstruction, while backpropagation and matrix operations reduce computation and memory requirements.

  • D.3 Intermediate layer derivatives: Using β = 1 in intermediate layers and β = 2 at the final layer gives better results than using the same β throughout.The authors suggest KL divergence may better suit mixture decomposition, while L2 is used for the final reconstruction objective.
  • D.3 Intermediate layer derivatives: For unnormalized Wk in optimized final layers, the update for Hk+1 determines the structure of the intermediate layers.Lower, non-optimized layers use normalized basis functions to avoid trivial rescaling solutions.
  • D.3 Intermediate layer derivatives: Multiplicative updates split each gradient into positive and negative parts and use their elementwise ratio as the update factor.The positive and negative components are computed recursively for each variable in the unfolded network.
  • D.3 Intermediate layer derivatives: The recursive gradient formulation applies to any model with non-negative parameters, despite being written using NMF variable names.This extends the update construction beyond the specific NMF notation used in the derivation.
  • D.3 Intermediate layer derivatives: After a forward pass, gradients are propagated from the last layer downward, followed by parameter gradients and multiplicative Wk updates.Only the next layer’s positive and negative H-gradient parts must remain available, limiting memory requirements.
Loading 1409.2574v4…