PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Deep Learning

Nir Shlezinger, Jay Whang, Yonina C. Eldar, Alexandros G. Dimakis

arXiv:2012.08405v3eess.SPcs.LG

TL;DR

Classical model-based methods can fail under inaccurate or overly simple assumptions, while generic DNNs demand massive data and computation and are difficult to interpret. This survey organizes hybrid model-based deep learning by inference mechanism into model-aided networks and DNN-aided inference, reviewing their designs, guidelines, and applications. Reported examples show that hybrid systems can match or improve black-box performance with substantially fewer parameters and reduce inference runtime in specific tasks.

  • Problem

    Classical models can degrade under inaccurate assumptions, while purely data-driven DNNs require massive data and computation and are difficult to interpret.

  • Method

    The survey categorizes hybrid systems into model-aided networks, which design DNN architectures from model-based methods, and DNN-aided inference, which augments model-based algorithms with deep learning.

  • Results

    Hybrid examples achieved similar or improved denoising results with 3%−10% of conventional CNN trainable parameters, while DetNet was reported faster than approximate message passing and semi-definite relaxation for a specified MIMO task.

  • Takeaways & Limitations

    The systematic framework provides design guidelines and concrete examples for combining partial domain knowledge with learning across signal processing, communications, and control.

Abstract

from arXiv · show

Signal processing, communications, and control have traditionally relied on classical statistical modeling techniques. Such model-based methods utilize mathematical formulations that represent the underlying physics, prior information and additional domain knowledge. Simple classical models are useful but sensitive to inaccuracies and may lead to poor performance when real systems display complex or dynamic behavior. On the other hand, purely data-driven approaches that are model-agnostic are becoming increasingly popular as datasets become abundant and the power of modern deep learning pipelines increases. Deep neural networks (DNNs) use generic architectures which learn to operate from data, and demonstrate excellent performance, especially for supervised problems. However, DNNs typically require massive amounts of data and immense computational resources, limiting their applicability for some signal processing scenarios. We are interested in hybrid techniques that combine principled mathematical models with data-driven systems to benefit from the advantages of both approaches. Such model-based deep learning methods exploit both partial domain knowledge, via mathematical structures designed for specific problems, as well as learning from limited data. In this article we survey the leading approaches for studying and designing model-based deep learning systems. We divide hybrid model-based/data-driven systems into categories based on their inference mechanism. We provide a comprehensive review of the leading approaches for combining model-based algorithms with deep learning in a systematic manner, along with concrete guidelines and detailed signal processing oriented examples from recent literature. Our aim is to facilitate the design and study of future systems on the intersection of signal processing and machine learning that incorporate the advantages of both domains.

I. INTRODUCTION

Model-based methods use hand-designed mathematical models but can degrade when assumptions are inaccurate, while purely data-driven DNNs require substantial data and computation and remain difficult to interpret. The article surveys hybrid model-based deep learning through two inference-centered strategies and provides design guidance and examples.

  • Model-Based Methods: Model-based methods use domain knowledge and simplifying assumptions to perform tractable, interpretable, and computationally efficient inference.Examples include statistical models based on measurements and physics, deterministic representations, Kalman filtering, and message passing.
  • Data-Driven Schemes: Purely data-driven DNNs avoid analytical approximations but typically require massive datasets, lengthy training, high computational burden, and offer limited interpretability.These drawbacks are especially relevant for hardware-limited devices that must adapt to dynamic conditions.
  • Article Scope: The survey presents systematic design guidelines, implementation approaches, literature examples, and future challenges across signal processing, communications, and control applications.It covers approaches including deep unfolding, DNN-aided inference, and neural augmentation, then concludes with qualitative comparisons and research topics.
  • Model-Based Methods: Inaccurate or overly simple model assumptions can degrade model-based performance when the true dynamics are complex or poorly known.This limits applicability when the underlying distribution is unknown, costly to estimate, or too complex to express analytically.
  • Model-Based Deep Learning: The article defines model-based deep learning as hybrid systems that combine established model-based algorithms with deep learning while occupying a middle ground between model-based and data-driven methods.The surveyed systems incorporate domain knowledge through model-based algorithms and learn from data through deep learning techniques.
  • Model-Based Deep Learning: Model-aided networks design problem-specific DNN architectures from model-based operations, whereas DNN-aided inference augments model-based algorithms with learned components.DNN-aided inference can learn to overcome partial or mismatched domain knowledge from data.

III. BASICS OF DEEP LEARNING

Deep learning combines expressive DNN architectures, task-specific loss functions, and optimizers to learn mappings from data. This section introduces these components, their training challenges, and common learning settings and tasks.

  • Core Components: Deep learning systems comprise three fundamental components: a DNN architecture, a task-specific loss function, and an optimizer.The architecture defines the function class, the loss specifies the task, and the optimizer searches for parameters within that class.
  • Neural Network Architecture: DNNs define a highly expressive function class through compositions of parametrized layers that map inputs to outputs.Layers may include affine transformations such as fully connected and convolutional layers, interleaved with nonlinear activations.
  • Optimization: Gradient-based optimizers are used because globally minimizing neural-network losses is generally intractable at large parameter scales.Mini-batch stochastic gradient descent estimates gradients from randomly selected samples, while methods such as RMSProp and Adam adapt parameter step sizes.
  • Learning Settings: The loss function determines both the learning task and the training-data requirements, distinguishing supervised, semi-supervised, and unsupervised learning.The supplied examples include supervised learning with input-label pairs and tasks such as regression, classification, and structured prediction.
  • Classification: In classification, a DNN maps each fixed-size input to a probability vector over K classes, commonly trained with cross-entropy loss.The target label is represented as a one-hot encoding, while the output provides a soft categorical decision.
  • Generative Models: GANs can generate high-resolution samples with strong visual quality, but adversarial training is unstable and lacks a probabilistic interpretation for objective evaluation.StyleGAN2 is reported to generate 1024 × 1024 images nearly indistinguishable from real photographs to human observers.

IV. MODEL-AIDED NETWORKS

Model-aided networks tailor DNN architectures to model-based algorithms, retaining domain structure while learning parameters or mappings from data. Deep unfolding instantiates this strategy by converting iterative optimization into trainable layers, with examples showing faster inference and competitive denoising using fewer parameters.

  • Design Outline:: Model-aided networks design dedicated DNN architectures from model-aware algorithms rather than using generic off-the-shelf networks.They can learn missing model parameters, including channel matrices, dictionaries, and noise covariances.
  • Design Outline:: Deep unfolding converts an iterative algorithm into a DNN by designing each layer to resemble one iteration.The workflow identifies an iterative algorithm, fixes the iteration count, makes free parameters trainable, and trains the network end-to-end.
  • Example 1: Deep Unfolded Projected Gradient Descent:: DetNet unfolds projected gradient descent for symbol detection in linear memoryless MIMO Gaussian channels.Its output after Q iterations is converted into an estimated symbol vector by taking the sign of each element.
  • Example 1: Deep Unfolded Projected Gradient Descent:: DetNet outperforms leading MIMO detection algorithms based on approximate message passing and semi-definite relaxation when provided sufficient training examples.The unfolded network also uses an order of magnitude fewer layers than the model-based optimizer requires for convergence; reported runtime is three times faster than approximate message passing and over 80 times faster than semi-definite relaxation for the stated batch and channel.
  • Example 2: Deep Unfolded Dictionary Learning:: DCEA unfolds proximal gradient iterations for Poisson image denoising and implicitly learns the dictionary from training data.It achieves similar or improved results to black-box CNNs while using 3%−10% of their overall number of trainable parameters.
  • Example 2: Deep Unfolded Dictionary Learning:: Deep unfolding relies on knowledge of the model describing the setup, although its parameters may be externally provided or learned during training.Accurate model knowledge incorporates structure into the architecture, whereas deviations between the true measurement relationship and the assumed model can degrade performance.

B. Neural Building Blocks

Neural building blocks decompose model-based algorithms into task-specific modules, replacing selected computations with dedicated DNNs. DeepSIC illustrates this approach by implementing iterative soft interference cancellation through learned classification blocks that can operate beyond the original channel-model assumptions.

  • Neural building blocks: Neural building blocks represent model-based algorithms as interconnected modules, each assigned a concrete computational task and potentially implemented by a dedicated DNN.The interconnection may be sequential or parallel, and the overall network can be trained end-to-end or block by block.
  • DeepSIC for MIMO detection: DeepSIC implements iterative soft interference cancellation using neural building blocks for interference cancellation and soft decoding.Each block estimates conditional symbol PMFs from the observation and estimates produced in the previous iteration.
  • DeepSIC for MIMO detection: Unlike model-based iterative SIC, DeepSIC learns these block computations from data and does not require accurate complete knowledge of the underlying statistical model.The model-based method assumes additive interference contributions and the channel model in (8).
  • DeepSIC for MIMO detection: Classification DNNs can compute conditional distributions in complex nonlinear setups, allowing the learned receiver to realize iterative SIC for arbitrary channel models.This extends the application scope beyond the linear Gaussian channels for which iterative SIC was originally designed.
  • Training: End-to-end DeepSIC training jointly updates K · Q building-block networks, so large K is expected to require a large labeled set.Sequential training can train each DNN individually and is expected to require a smaller training set, whereas sufficiently large training may favor end-to-end accuracy.

V. DNN-AIDED INFERENCE

DNN-aided inference combines a traditional model-based algorithm with DNN-augmented intermediate computations. It is particularly suited to settings with partial domain knowledge, including neural augmentation for robustness to inaccurate model parameters.

  • DNN-aided inference: DNN-aided inference retains a traditional model-based inference procedure while augmenting some intermediate computations with DNNs.This distinguishes it from model-aided networks, whose complete architecture imitates a model-based algorithm.
  • DNN-aided inference: The framework targets scenarios where available domain knowledge determines the algorithm but unmodeled or analytically complex components are handled by deep learning.The review divides DNN-aided inference into structure-agnostic methods, methods exploiting known structures, and neural augmentation methods.
  • Neural augmentation: Neural augmentation methods use a parallel deep learning system to compensate for errors caused by inaccurate underlying-model parameters.Inference remains based on the model-based algorithm and its available domain knowledge.

A. Structure-Agnostic DNN-Aided Inference

Structure-agnostic DNN-aided inference learns signal priors or optimization components while retaining a model-based recovery procedure. Examples include generative-prior compressed sensing and plug-and-play optimization.

  • Structure-Agnostic DNN-Aided Inference: DNN-aided inference learns signal structures or statistical properties in forms compatible with model-based optimization.This approach targets inverse problems such as denoising, sparse recovery, deconvolution, and super resolution.
  • Structure-Agnostic DNN-Aided Inference: The design replaces analytically difficult, domain-dependent optimization components with DNNs trained separately or within the complete inference system.The procedure selects an optimization method, replaces complicated components, and trains the inserted module separately or end-to-end.
  • Compressed Sensing: In compressed sensing, a pre-trained generator replaces an explicit sparsity assumption by constraining recovered signals to the generator's range.The generator maps latent vectors to signals in the target domain and can be reused across downstream tasks.
  • Compressed Sensing: Recovery minimizes L(z) = ∥HG(z) −x∥2 over the latent variable, then returns G(z) as the reconstructed signal.The loss is differentiable but highly non-convex, so gradient-based optimization has no closed-form solution or guarantee.
  • Compressed Sensing: VAE-based priors improve digit recovery at small measurement counts, while LASSO performs better as observations approach the signal dimension.The generator may omit the ground-truth signal when it does not perfectly model the data distribution.
  • Compressed Sensing: With M = 500 noisy measurements out of N = 12288 dimensions, GAN-based recovery produces more reasonable CelebA samples in the low-measurement regime.The experiment evaluates compressed sensing on noisy observations.

B. Structure-Oriented DNN-Aided Inference

Structure-oriented DNN-aided inference preserves an established algorithmic structure while learning model-specific computations. Learned factor graphs show accurate message passing, robustness to model uncertainty, and advantages over black-box detection with limited data.

  • Structure-Oriented DNN-Aided Inference: Structure-oriented inference uses DNNs to robustify established algorithms without explicitly characterizing every statistical detail.It is suited to problems with known structures such as causality, finite memory, or state-space relationships.
  • Learned Factor Graphs: The sum-product algorithm exploits factorized Markov distributions to approach MAP detection with complexity that grows linearly with sequence length t.Message passing over the factor graph computes the posterior distribution efficiently.
  • Learned Factor Graphs: Learned factor graphs retain the known graph structure while DNNs learn function-node computations from labeled data.For stationary Markovian relationships, learning can focus on the shared function node rather than the complete inference mapping.
  • Learned Factor Graphs: Combining model-agnostic DNNs with model-aware inference improves robustness to model uncertainty and outperforms inaccurate-model sum-product inference.Explicitly modeling Markov structure also improves performance over sliding bidirectional RNN detection with limited training data.
  • Discussion: Compact intermediate DNNs support computationally limited devices, scarce-data training, and online adaptation to temporal statistical variation.These properties have been used for velocity tracking and online training of coded-communication receivers.

C. Neural Augmentation

Neural augmentation keeps the complete model-based inference algorithm and uses an external DNN to correct intermediate computations. In Kalman smoothing, this improves robustness to inaccurate or nonlinear models while requiring substantially less data than a standalone GNN.

  • Neural Augmentation: Neural augmentation interleaves a complete model-based algorithm with an external DNN that corrects intermediate computations.Unlike DNN-aided inference, the DNN remains an external correction module rather than replacing algorithmic components.
  • Neural Augmentation: The design selects an iterative algorithm, identifies exchanged information, learns correction terms from the same quantities, and trains the hybrid system end-to-end.Losses may include both final estimates and intermediate iteration outputs.
  • Kalman Smoothing: Neural-augmented Kalman smoothing targets inaccurate state-space models, including linear approximations of nonlinear dynamics where extended Kalman methods may be sub-optimal.The DNN corrects inaccurate log-likelihood-gradient computations using messages from the model-based smoother.
  • Kalman Smoothing: The correction network is a graph neural network whose learned terms update the model-based log-likelihood-gradient iterations.The architecture maintains node and message variables and produces correction terms through neural mappings.
  • Results and Scope: 10−20 times fewer samples are required than for an individual GNN to achieve similar MSE results.The hybrid system is trained with a weighted ℓ2 loss over intermediate layers, with later iterations receiving greater weight.
  • Results and Scope: The method requires an approximation of the model for message computation, while its correction DNN still requires comparatively substantial training data.Its data requirement is nevertheless lower than that of a standalone DNN for similar accuracy.

VI. CONCLUSIONS AND FUTURE CHALLENGES

The article maps hybrid model-based and data-driven inference into model-aided networks and DNN-aided inference, providing systematic design approaches, guidelines, and examples. It concludes by addressing advantages, approach selection, and future research challenges.

  • VI. CONCLUSIONS AND FUTURE CHALLENGES: Hybrid systems are categorized as model-aided networks or DNN-aided inference according to how model-based algorithms and deep learning are combined.The article presents representative approaches for both strategies.
  • VI. CONCLUSIONS AND FUTURE CHALLENGES: The review provides systematic design guidelines and concrete examples for combining domain knowledge with data-driven inference.It is presented as a tutorial mapping of model-based deep learning methods.
  • VI. CONCLUSIONS AND FUTURE CHALLENGES: The concluding discussion covers advantages, selecting a design approach for an application, and future research challenges.These topics are organized in the article's final section.

A. Advantages of Model-Based Deep Learning

Model-based deep learning combines handcrafted algorithms with learned components to address complex environments, model uncertainty, and data efficiency. Its hybrid structure can extend inference beyond conventional model domains while requiring less data than generic DNN architectures.

  • Advantages of Model-Based Deep Learning: Learned components can facilitate inference when the underlying model is too complex to capture accurately in closed form.The paper cites implicit regularization for compressed sensing beyond sparse signals and neural building blocks for nonlinear setups.
  • Advantages of Model-Based Deep Learning: Hybrid inference can improve resilience to model uncertainty compared with relying solely on domain knowledge.Examples include GNN-augmented Kalman smoothing under mismatched state-space dynamics and learned factor graphs for signal-processing inference.
  • Advantages of Model-Based Deep Learning: Problem-tailored hybrid architectures require notably less data to learn an accurate mapping than conventional DNN architectures.The paper demonstrates this through comparisons involving learned factor graphs and a sliding bidirectional RNN system.

B. Choosing a Model-Based Deep Learning Strategy

The paper recommends selecting a model-based deep learning strategy by characterizing available knowledge and data, choosing a suitable model-based algorithm, and matching the hybrid approach to implementation challenges. It also identifies theoretical guarantees, interpretability, edge intelligence, and new applications as open directions.

  • Choosing a Model-Based Deep Learning Strategy: Strategy selection begins by characterizing available domain knowledge, its uncertainties, and the amount of labeled and unlabeled data.This first step distinguishes known assumptions from inaccurate approximations and assesses available samples.
  • Choosing a Model-Based Deep Learning Strategy: The model-based algorithm should be chosen from available knowledge while accounting for performance, complexity, and real-time requirements.Unknown knowledge can later be compensated for through deep learning tools.
  • Choosing a Model-Based Deep Learning Strategy: Implementation challenges determine whether to use approaches such as neural augmentation, deep unfolding, or model-aided networks.These approaches address missing knowledge, model uncertainty, and slow iterative inference in different ways.
  • Choosing a Model-Based Deep Learning Strategy: Deep unfolding and neural building blocks can learn iterative computations using fewer iteration-equivalent layers than the original algorithm requires.The stated goal is reliable inference with reduced inference time when conventional iterative methods need many iterations to converge.
  • Performance Guarantees: Theoretical performance guarantees remain an important unresolved research direction for hybrid systems.Model-based methods offer established guarantees, whereas such guarantees are difficult to characterize for conventional deep learning; preliminary results exist for specific methods.
  • Deep Learning Algorithms: Model-based deep learning is presented as a framework for incorporating human knowledge and improving interpretability in new deep learning algorithms.The paper also envisions using model-based methods to improve existing DNNs or design interpretable architectures.
  • Collaborative Model-Based Deep Learning: Limited computational and data resources, latency, and privacy constraints make hybrid strategies attractive for edge intelligence.The paper highlights smartphones, sensors, and autonomous cars as relevant edge-device settings.
  • Unexplored Applications: Signal processing, communications, and control are identified as natural application areas for robust and structurally informed hybrid systems.The paper points to emerging opportunities involving technologies such as IoT networks and autonomous systems.

APPENDIX

The appendix provides detailed formulations of iterative signal-processing procedures, including projected-gradient and sparse-coding methods, iterative soft interference cancellation, and probability-based symbol decoding.

  • Projected Gradient Descent: Projected gradient descent alternates a gradient step on the unconstrained objective with projection onto the constrained variable set.The recursion uses a step size and an initial estimate.
  • Convolutional Sparse Coding: Convolutional sparse coding formulates clean-image recovery from noisy observations using a convolutional dictionary.The dictionary variable is constrained to be block-Toeplitz, and an ℓ1 regularizer controls sparsity.
  • Alternating Optimization: Alternating optimization addresses the convolutional sparse-coding formulation through successive update equations.The appendix introduces the iteration index and a constraint linking the sparse variable to the current estimate.
  • Proximal Gradient Descent: Proximal gradient descent solves the ℓ1-regularized problem through repeated iterations controlled by a threshold parameter dictated by the regularization weight.The iterations are indexed by q = 0, 1, 2, . . . .
  • Iterative SIC: Iterative SIC generates K distribution vectors over possible symbols at each iteration and uses interference cancellation followed by soft decoding.The method estimates interference statistics from previous-iteration probability mass functions.
  • Iterative SIC: Soft decoding approximates each symbol’s conditional distribution from an interference-canceled observation and combines it with a uniform prior over possible symbols.The resulting conditional PMFs are used to form symbol estimates.
  • Iterative SIC: After the final iteration, each symbol is decoded by maximizing its estimated PMF, and the overall estimate concatenates the individual symbol decisions.This produces the vector estimate ˆs = [ˆs1, . . . , ˆsK].

D. Detailed Formulation of Sparsity-Based CS (Example 4, Section V)

The appendix details sparse recovery, ADMM optimization, and sum-product inference. These formulations replace difficult or coupled computations with convex relaxation, variable splitting, proximal updates, and recursive message passing.

  • D. Detailed Formulation of Sparsity-Based CS (Example 4, Section V): Sparse recovery represents s∗ in a dictionary B with a sparse coefficient vector and seeks a sparse representation consistent with noisy observations.The formulation introduces a noise threshold and then absorbs B into the sensing matrix for subsequent analysis.
  • D. Detailed Formulation of Sparsity-Based CS (Example 4, Section V): The NP-hard ℓ0 problem is relaxed to a convex ℓ1-regularized LASSO objective.For Gaussian A with M = Θ(l log N), the unique minimizer equals s∗ with high probability.
  • E. Detailed Formulation of ADMM (Example 5, Section V): ADMM decouples the regularizer from the likelihood term by introducing an auxiliary variable v.The resulting constrained formulation is solved through an augmented Lagrangian and alternating updates.
  • E. Detailed Formulation of ADMM (Example 5, Section V): ADMM’s update sequence combines proximal mappings for the regularizer and likelihood with a gradient-ascent step.The hyperparameter α controls the proximal updates, and one likelihood update has a closed-form linear-system expression.
  • F. Detailed Formulation of Sum-Product Method (Example 6, Section V): The sum-product method first represents a factorizable distribution as a factor graph with function nodes, edges, and half-edges.The factorization uses functions linking successive vector variables.
  • F. Detailed Formulation of Sum-Product Method (Example 6, Section V): Recursive message passing computes the joint distribution along the factor graph from forward and backward path messages.The local joint expression combines incoming messages with the intervening factor function.
  • F. Detailed Formulation of Sum-Product Method (Example 6, Section V): The resulting MAP detector has complexity that grows linearly with block size t.When t is large, message scaling prevents numerical messages from tending toward zero without changing the MAP rule.

G. Detailed Formulation of Iterative Kalman Smoother (Example 7, Section V)

The iterative Kalman smoother formulation derives message-based updates from a state-space model and its log-likelihood gradient. These updates repeat until convergence, after which the resulting s(q) is used as the estimate.

  • The state-space model establishes the joint distribution of the state and observations.
  • The formulation derives the log-likelihood gradient and expresses its entries using message-based summands.
  • The iterative procedure repeats until convergence, then uses the resulting s(q) as the estimate.
Loading 2012.08405v3…