PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning in Deterministic Computational Mechanics

Leon Herrmann, Stefan Kollmannsberger

arXiv:2309.15421v1cs.LG

TL;DR

Deep learning research in deterministic computational mechanics is extensive, while existing reviews often focus on specific applications rather than general methods. This review organizes methods into five categories and concludes that simulation enhancement is currently especially useful, whereas gradient-based methods remain preferred when differentiable solvers are available.

  • Problem

    Deep-learning literature in deterministic computational mechanics is extensive and diverse, while existing reviews often address specific applications rather than general methods.

    Source: §1.2 · p. 3

  • Method

    The review surveys general methods in deterministic computational mechanics through five categories: simulation substitution, simulation enhancement, neural-network discretizations, generative approaches, and deep reinforcement learning.

    Source: §1.2 · p. 3

  • Results

    The review identifies simulation enhancement as currently one of the most useful approaches, while gradient-based methods remain preferred when differentiable solvers are available.

    Source: deep reinforcement learning · p. 37

  • Takeaways & Limitations

    The method-focused overview aims to clarify the state of the art and highlight similarities between methods across scientific communities.

    Source: §1.2 · p. 3

    Few approaches are competitive with classical methods, and most current research remains at an early stage focused on demonstrating possibilities rather than accuracy and efficiency.

    Source: deep reinforcement learning · p. 37

Abstract

from arXiv · show

The rapid growth of deep learning research, including within the field of computational mechanics, has resulted in an extensive and diverse body of literature. To help researchers identify key concepts and promising methodologies within this field, we provide an overview of deep learning in deterministic computational mechanics. Five main categories are identified and explored: simulation substitution, simulation enhancement, discretizations as neural networks, generative approaches, and deep reinforcement learning. This review focuses on deep learning methods rather than applications for computational mechanics, thereby enabling researchers to explore this field more effectively. As such, the review is not necessarily aimed at researchers with extensive knowledge of deep learning -- instead, the primary audience is researchers at the verge of entering this field or those who attempt to gain an overview of deep learning in computational mechanics. The discussed concepts are, therefore, explained as simple as possible.

1. Introduction

The introduction situates deep learning in computational mechanics within rapidly growing research and presents a method-focused overview organized around five categories. It briefly introduces neural networks and the notation used throughout the review.

1.1. Motivation

  • Publication activity in machine learning and computational mechanics rose sharply around 2017, after earlier trends showed limited computational-mechanics interest.Figure 1 counts publications across artificial-intelligence topics and subtopics since 1999.Source: §1.1 · p. 3, Figure 1 · p. 3

1.2. Taxonomy of Deep Learning Techniques in Computational Mechanics

  • The review focuses on general deep-learning methods rather than application-specific surveys, aiming to clarify the state of the art and highlight similarities across communities.It structures the overview as simulation substitution, simulation enhancement, discretizations as neural networks, generative approaches, and deep reinforcement learning.Source: §1.2 · p. 3
  • The five categories distinguish replacing a full simulation, replacing selected components, building discretizations from neural-network operations, generating similar data, and learning reward-maximizing interactions.In computational mechanics, reinforcement-learning environments are modeled by governing physical equations; this approach can be useful when gradient information is unavailable.Source: §1.2 · p. 4

1.3. Deep Learning

  • Neural networks approximate functions through nested linear transformations and nonlinear activations, with parameters updated by gradient-based optimization to minimize a cost function.Supervised learning uses labeled data to define the cost function; unsupervised learning does not rely on labeled data.Source: §1.3 · p. 4
  • The review writes x and y unbolded even when vector-valued, reserves boldface for degrees of freedom, and introduces abbreviations for common network architectures.These include fully connected, convolutional, recurrent, and graph neural networks.Source: §1.3 · p. 5

2. Simulation Substitution

Simulation substitution uses neural networks to replace a simulation with learned mappings for forward and inverse problems. The section surveys data-driven space-time and time-stepping methods, their encodings and architectures, and challenges in training reliable surrogates.

2. Simulation Substitution

  • Simulation substitution replaces the simulation with a neural-network surrogate for forward or inverse PDE and ODE problems.The section distinguishes forward solution computation from inverse identification of PDE coefficients or operators and ODE system functions.Source: §2 · p. 5, §2.1.1 · p. 6

2.1. Data-Driven Modeling

  • Data-driven modeling learns from labeled input-output pairs, while FC-NNs approximate space-time operators and CNN image mappings exploit spatial relationships on uniform grids.Image-to-image mappings can cover full domains, subdomains, or individual pixels; subdomain and pixel approaches require interface treatment, and the simplest FC-NN approach is limited to simple problems because it requires abundant data.Source: §2.1 · p. 5, The simplest procedure is to… · p. 6, §2.1.1.2 · p. 6

2.1.1.3. Model Order Reduction Encoding

  • Model order reduction encodings combined with neural networks currently achieve the review’s most impressive data-driven surrogate modeling results.Encodings can use PCA, proper orthogonal decomposition, or reduced manifold learning, and can be combined with most other methodologies discussed.Source: §2.1.1.3 · p. 7

DeepONet

  • DeepONets predict an operator by combining coordinate-dependent basis functions from a TrunkNet with problem-specific coefficients from a BranchNet.Their dot product yields the solution prediction.Source: DeepONet · pp. 7–8

Fourier Neural Operators

  • Fourier neural operators learn mappings between functions on uniform grids using non-local Fourier-space transformations within each layer.Each layer combines a transformed linear operation with a linear operation outside Fourier space before applying an activation; the transformations are learnable.Source: Fourier Neural Operators · p. 8

2.1.1.5. Neural Network Approximation Power

  • Although neural networks can approximate solutions of various PDEs, training data and optimization errors impede finding sufficiently good optima; active learning and transfer learning address these challenges.Active learning requests classical simulations near predictions that fail an error indicator, while transfer learning starts from a similar task to accelerate training.Source: §2.1.1.5 · p. 9, §2.1.1.6 · p. 9

2.1.2.1. Recurrent Neural Networks

  • Recurrent neural networks use hidden states to retain information across time steps, but can struggle with long-term dependencies because of vanishing or exploding gradients.LSTMs, GRUs, and transformers are cited as architectures addressing this shortcoming.Source: §2.1.2.1 · p. 10

2.1.2.2. Dynamic Mode Decomposition

  • Dynamic mode decomposition fits a linear operator to successive snapshots and propagates states forward, while Koopman-based and autoencoder variants extend the framework toward nonlinear dynamics.The basic framework is valid only for linear dynamics; Koopman theory represents nonlinear systems through an infinite-dimensional linear operator, and autoencoders provide augmented-state encodings and a map back to the original space.Source: §2.1.2.2 · pp. 10–11

2.2. Physics-Informed Learning

Physics-informed learning incorporates governing equations and constraints into neural-network training, either through loss terms or by construction. The section surveys methods for solving and identifying PDEs, handling boundaries and domains, and accelerating training, while noting important scope and performance limits.

PINNs [203] approximate either the solution u(x), the coefficients λ(x), or both with FC-NNs.

  • PINNs approximate the solution, PDE coefficients, or both, and train by minimizing PDE residuals at collocation points alongside boundary and optional data losses.Automatic differentiation supplies prediction gradients used in the residual and boundary terms.Source: PINNs [203] approximate either the… · p. 12, NBN X · p. 12

Variational Physics-Informed Neural Networks

Boundary Conditions

  • Penalty-based boundary enforcement can unbalance competing losses, whereas output modifications can satisfy Dirichlet conditions a priori and separate derivative networks can enforce Neumann conditions strongly.The derivative-network approach requires an additional constraint matching its predictions to derivatives of the solution.Source: Boundary Conditions · p. 13

Ansatz

  • Ansatz choices include CNNs, classical bases, and NN-based discretizations; prior features or progressively added layers can guide learning from coarse to refined solutions.Figure 5 depicts each added layer as a projection from the coarser level plus a learned correction.Source: Ansatz · pp. 14–15, Figure 5 · p. 15

Domain Decomposition

  • Domain-decomposition methods use patchwise integration or separate subnetworks, with interface conditions supporting continuity, parallelization, and adaptive network depth.hp-variational PINNs use patchwise test functions with a global solution network; conservative PINNs penalize solution and flux discrepancies at interfaces.Source: Domain Decomposition · p. 15

Acceleration Methods

  • Acceleration proposals include residual-based sampling, time-dependent normalization, residual-derivative losses, numerical differentiation, and specialized network architectures.A cost reduction need not improve solution quality when the cost does not correspond to the error norm; the H−1 norm is proposed as an alternative cost.Source: Acceleration Methods · pp. 15–16

Applications To Forward Problems

  • For forward problems, PINNs do not outperform classical solvers in speed at engineering-relevant accuracy, while incorporating extensive solution measurements can make them viable in some cases.Reported speed gains often omit training effort; data-driven methods are reported to outperform PINNs, which are framed as a regularization technique for reducing generalization error.Source: Applications To Forward Problems · p. 16

Applications To Inverse Problems

  • PINNs are more applicable to inverse problems with full-domain knowledge than to partial observations, and time-stepping variants include parareal corrections and discrete-time networks predicting Runge–Kutta stages.Partial-knowledge inversions often focus on simpler tasks, while discrete-time PINNs learn stages and the next solution value from spatial inputs.Source: Applications To Inverse Problems · p. 16, §2.2.2.1 · p. 17, The q + 1 predictions ˆun · p. 18

The q + 1 predictions ˆun

The review covers methods for identifying dynamics and enforcing physical structure. These range from interpretable equation discovery to learned reduced models and constructions that constrain the network to obey physical laws.

PDE-Net

  • PDE-Net learns system dynamics and an underlying differential equation by estimating spatial derivatives with learnable convolutional filters and combining them into an interpretable operator.The filters adapt their approximation order to fit measurements, while the gradient type is predefined.Source: PDE-Net · p. 18

SINDy

  • SINDy identifies dynamics through sparse regression over candidate functions, and autoencoder-based latent coordinates can increase its expressivity.The latent-space model supports full-space reconstruction but is no longer interpretable in the full space.Source: SINDy · pp. 18–19, eNN · p. 19

Multistep Methods

  • A neural network can approximate the right-hand side of an ODE, with a residual formulated from a linear multistep scheme.The approach is closely linked to discrete-time PINNs, whose Runge–Kutta reformulation supplies a cost for learning the forward solution.Source: Multistep Methods · p. 19

2.2.3. Enforcement Of Physics By Construction

  • Physics can be enforced by construction by restricting the learnable space, including through stream functions, symmetry or invariance constraints, and learned Lagrangians or Hamiltonians.Learning through a Lagrangian or Hamiltonian guarantees energy conservation in the described dynamical-system approaches.Source: §2.2.3 · p. 20

3. Simulation Enhancement

Simulation enhancement uses deep learning to improve individual components of a classical simulation while leaving other parts to classical methods. The review covers data preparation, physical modeling, numerical methods, optimization, and post-processing.

3.1. Pre-processing

  • Pre-processing spans supervised data preparation, neural-network initial guesses or adaptable ansatzes for classical solvers, and learned mesh densities or direct mesh generation.Data preparation examples include crack detection, scan-artifact removal, signal denoising, and low-frequency extrapolation.Source: §3.1 · p. 20, §3.1.1 · p. 20, §3.1.2 · p. 20

3.2. Physical Modeling

  • Physical models can be substituted by neural networks, represented by predicted parameters of an assumed model, or identified as parsimonious mathematical expressions.Model substitution can violate physical laws; architectures that enforce physical requirements by construction are reported to behave better on unseen data, especially in extrapolation.Source: §3.2 · p. 21, §3.2.3 · p. 22, §3.2.1 · p. 22

3.2.3. Model Identification

  • Model identification seeks interpretable constitutive laws by selecting relevant terms from candidate functions, while model substitution and parameter identification cover other physical-modeling choices.EUCLID uses unsupervised displacement and reaction-force data, derives stresses from strain energy, and applies Lp regularization to promote sparse model weights.Source: §3.2.3 · pp. 22–23

3.3.1. Algorithm Enhancement

  • Neural networks can enhance numerical algorithms by correcting numerical errors, estimating tunable parameters, or replacing selected solver components.Examples include quadrature and strain-displacement corrections for distorted elements, improved gradients on coarse grids, and learned time steps or finite-element test functions.Source: §3.3.1 · p. 24

3.3.2. Multiscale Methods

  • Multiscale methods use learned superelements or alternating global and local analyses to represent behavior across scales.Neural networks may predict internal displacements, stresses, reaction forces, or local-model boundary conditions; recurrent networks can incorporate history-dependent behavior.Source: §3.3.2 · p. 24

3.3.3. Optimization

  • Optimization methods learn forward operators, sensitivities, or updated states, and can also parameterize design variables with neural networks.A learned forward operator can speed optimization iterations and simplify gradient computation, although automatic differentiation through time-stepping procedures may cost more than alternatives.Source: §3.3.3 · pp. 25–26
  • Neural corrections to sensitivities can be learned in several ways, but an accurate learned forward operator may still yield inaccurate sensitivities.One approach adds a loss term using adjoint-state labels; others correct coarse-grid sensitivity estimates or map coarse solutions to fine-grid sensitivities.Source: §3.3.3 · p. 25

3.4. Post-Processing

  • Post-processing can refine coarse solutions, but dynamical models accumulate error across time steps, making solver-in-the-loop corrections more effective than purely supervised trajectory refinement.Corrections are propagated through a differentiable solver before the next coarse step; more correction steps can improve accuracy at higher computational effort.Source: §3.4 · p. 26

4. Discretizations As Neural Networks

Discretizations can be built from neural-network components to bring deep-learning techniques into classical simulation and motivate architectures tailored to physical problems. The finite-element example expresses the discretized equations using a stiffness matrix, degrees of freedom, and body loads.

4. Discretizations As Neural Networks

  • Neural-network building blocks can construct PDE discretizations, enabling deep-learning techniques in classical frameworks and motivating architectures tailored to mechanics problems.The stated techniques include automatic differentiation, gradient-based optimization, and GPU-based parallelization.Source: §4 · p. 27

4.1. Finite Element Method

  • A finite-element discretization is written as Kijuj − bi = 0, relating the stiffness matrix and degrees of freedom to the body load.Finite-element neural networks use this system as their starting point.Source: §4.1 · p. 27

Inserting the assembly into the system of equations from equation (62) yields

  • With uniform elements and constant material properties per element, local stiffness can be pre-integrated to separate material coefficients in assembling the global stiffness matrix.The assembly is then inserted into the finite-element system of equations.Source: §4.1 · p. 27, Inserting the assembly into the system… · p. 27

The nested summation has a similar structure of a FC-NN, a(l)

The section recasts several mechanics computations as neural-network operations, including finite-element assembly, mesh interpolation, convolutional stencils, material homogenization, and differential-equation solvers. These representations connect familiar numerical procedures to neural-network structures and, in some cases, offer specific computational or modeling advantages.

The nested summation has a similar structure of a FC-NN, a(l)

  • Finite-element equations resemble a fully connected network, while HiDeNNs represent shape functions as networks whose weights depend on nodal positions.Shared nodal displacements form elemental interpolations that are superposed across element domains; learnable node positions enable r-refinement.Source: The nested summation has a similar… · pp. 27–28

They are shared, as the nodal displacements ue

  • HiDeNN shape functions can be extended with convolutions to smooth the finite-element space, while the cited speedup of several orders of magnitude lacks independent confirmation.The convolutional implementation is described as suitable for GPU parallelization.Source: They are shared, as the nodal… · p. 28
  • FEA-Net replaces global stiffness-matrix multiplication and assembly with convolution, and can outperform classical finite elements for nonlinear problems on uniform grids using the same iterative solver.The convolution supports residual minimization without constructing or storing the global stiffness matrix.Source: They are shared, as the nodal… · pp. 28–29

4.2. Finite Difference Method

  • Finite-difference stencils can be implemented as convolutional filters and repeatedly applied as an RNN; automatic differentiation across time steps can be costly for full waveform inversion.A discretized wave equation can also act as an analog RNN whose learned material distribution classifies encoded input signals from measurements.Source: §4.2 · p. 29

4.3. Material Discretizations

  • Deep material networks connect analytical homogenization building blocks that combine and rotate effective compliance tensors using learnable volume fractions and rotation angles.The networks are applied to representative-volume-element homogenization with phase materials varied during evaluation.Source: §4.3 · p. 29, Figure 12 · p. 30

4.4. Neural Differential Equations

  • Neural ordinary differential equations view recurrent networks as continuous-time dynamics, with the initial condition as input and the solution at time T as output.The adjoint-state method computes sensitivities without backpropagating through every solver step, reducing the stated memory burden.Source: §4.4 · p. 30

Note that the residual connection, i.e., u(k)

Residual connections let information pass beyond a recurrent unit in an iterative correction scheme. A related approach applies an autoencoder repeatedly to a solution until convergence.

Note that the residual connection, i.e., u(k)

5. Generative Approaches

Generative approaches model a dataset’s probability distribution to produce new data resembling the training data. The review identifies microstructure generation and generative design as two prominent computational-mechanics applications.

5. Generative Approaches

Autoencoders facilitate data generation by mapping high-dimensional training data {xi}N

The section surveys generative model architectures and their computational-mechanics uses, from data synthesis and design exploration to conditional generation and anomaly detection.

Autoencoders facilitate data generation by mapping high-dimensional training data {xi}N

  • Autoencoders encode high-dimensional samples into a reduced latent vector and decode it back; variational autoencoders can sample that latent distribution to generate similar data.Source: Autoencoders facilitate data… · p. 31

5.2. Generative Adversarial Networks

  • GANs train a generator and discriminator in a minimax game, but can suffer mode collapse or stalled learning when the discriminator gets too strong.A Wasserstein-distance cost is one proposed remedy for vanishing gradients.Source: §5.2 · pp. 31–32
  • Conditional GANs take an additional input, while InfoGANs use a latent code and mutual-information term to make generated data vary meaningfully with that code.GANs typically produce higher-quality data than variational autoencoders, whereas autoencoders provide a structured latent space for smooth interpolation.Source: §5.2 · p. 32

5.3. Diffusion Models

  • Diffusion models learn a neural-network reverse denoising process that transforms Gaussian noise into samples matching the training distribution; normalizing flows instead compose invertible transformations.Source: §5.3 · p. 32

5.4.1. Data Generation

  • Generative models create data for microstructures, velocity models, and optimized structures, and those datasets can train surrogate models or inverse operators.Specialized sequence models and latent-vector architectures can construct three-dimensional microstructures from two-dimensional images.Source: §5.4.1 · p. 33

5.4.2. Generative Design & Design Optimization

  • Generative design reparameterizes the design space through latent variables or generator inputs, while diversity terms and novelty-oriented objectives encourage broader exploration.Variational autoencoder GANs combine high-quality designs with well-behaving gradients for microstructure optimization [550].Source: §5.4.2 · pp. 33–34

5.4.3. Conditional Generation

  • Conditional generators can produce multiple outputs for one input, supporting design and data-generation tasks; CycleGANs use cycle-consistency losses to preserve essential features across transformations.Examples include topology optimization, physical-property-conditioned design, and seismogram-to-velocity prediction.Source: §5.4.3 · p. 34

5.4.4. Anomaly Detection

  • Anomaly detection trains generative models on undamaged structures and uses reconstruction mismatch to localize defects, while discriminator scores can flag flawed samples or surrogate inputs outside the training domain.The discriminator does not guarantee an accurate assessment of surrogate-model validity.Source: §5.4.4 · pp. 34–35

6. Deep Reinforcement Learning

Deep reinforcement learning models an agent with neural networks and learns through interaction with an environment, which may represent governing physical equations. The review describes applications including control, topology optimization, PDE solving, and adaptive mesh refinement, alongside strategies to reduce the cost of repeated environment interactions.

6. Deep Reinforcement Learning

  • An agent takes actions and receives updated states and rewards, aiming to maximize cumulative reward; deep reinforcement learning uses neural networks to model the agent.Treating the environment as a black box can be useful when differentiable physics is infeasible.Source: §6 · p. 35, Figure 13 · p. 35

6.1. Applications

  • Deep reinforcement learning is mainly used for inverse problems with a nondifferentiable PDE solver, including fluid-flow control, unmanned-vehicle control, and identifying constitutive laws.The described control schemes are learned in simulations and subsequently applied in experiments.Source: §6.1 · pp. 35–36
  • Topology-optimization agents can add or remove material elements to form binary structures, minimize truss volume under stress constraints, or promote design diversity.Source: §6.1 · p. 36
  • High-dimensional PDEs can be recast as stochastic control problems and solved with reinforcement learning.Source: §6.1 · p. 36
  • For adaptive mesh refinement, an agent chooses elements to refine from the mesh and solution state, with reward based on error reduction against a ground-truth solution.The trained agent can be applied to previously unseen simulations.Source: §6.1 · p. 36

6.1.1. Extensions

  • Repeatedly solving the differential equation makes reinforcement learning expensive; parallel environments or agents and learned environment surrogates are proposed to accelerate learning.The surrogate-based procedure cycles through model learning, behavior learning, and environment interaction.Source: §6.1.1 · p. 36

7. Conclusion

The conclusion synthesizes deep-learning methods for computational mechanics and evaluates their potential and limitations. It identifies performance comparison with classical methods as an important need, while emphasizing different strengths and conditions across method categories.

7. Conclusion

  • The review structures the state of the art around deep-learning methods and outlines their potentials and limitations.Source: §7 · p. 36, deep reinforcement learning · p. 37
  • Few approaches are competitive with classical methods, and the field remains early-stage; future work should assess accuracy, efficiency, generalization, reliability, and training-inclusive computation times through fair comparisons.The conclusion aligns this call with a review of topology optimization.Source: deep reinforcement learning · p. 37

deep reinforcement learning.

  • Simulation substitution suits parameterized models evaluated repeatedly, but data demands currently limit it to small parameter spaces unless model-order reduction lowers dimensionality.Physics-informed learning can reduce data needs and improve generalization, although penalty-based enforcement adds computation and does not guarantee physical satisfaction; construction-based enforcement guarantees the imposed physics.Source: deep reinforcement learning · p. 37
  • Simulation enhancement is presented as especially useful where classical methods struggle, including data-driven constitutive-law formulation, and enables direct comparisons while learning smaller simulation components.The components are ideally expensive and have limited accuracy requirements.Source: deep reinforcement learning · p. 37
  • Treating discretizations as neural networks is a research direction for mechanics-tailored networks that can exploit deep-learning libraries and GPU hardware.Source: deep reinforcement learning · p. 37
  • Generative approaches have been used to create realistic data for other models, incorporate data-based vague constraints in optimization, and detect anomalies.Their versatility is noted in settings where accuracy of a specific instance is less central.Source: deep reinforcement learning · p. 37
  • Deep reinforcement learning has shown encouraging results in complex-physics vehicle control and is mainly applicable where efficient differentiable solvers are unavailable.Where differentiable solvers exist, gradient-based methods remain the state of the art and are preferred.Source: deep reinforcement learning · p. 37

A. Deep Reinforcement Learning

Deep reinforcement learning seeks policies that maximize discounted cumulative reward in environments modeled as Markov Decision Processes. Its methods differ by whether they model the environment and whether they optimize a policy or learn values.

A. Deep Reinforcement Learning

  • An MDP specifies states, actions, rewards, and transition probabilities, with future states depending only on the current state and action.The transition model allows environments to be nondeterministic.Source: A. Deep Reinforcement Learning · p. 38
  • A reinforcement learning algorithm seeks a policy that maximizes cumulative reward, with discounting that gives greater importance to immediate rewards.Policy quality can be assessed by state-value or action-value functions, which represent expected future reward under the policy.Source: A. Deep Reinforcement Learning · p. 38
  • Model-based methods use an environment model to sample transitions or, when it is differentiable, directly use reward gradients to update the policy.Model-free methods can also enhance optimization; the review connects differentiable models to optimization through differentiable physics solvers.Source: A. Deep Reinforcement Learning · p. 38
  • Policy-based methods directly optimize policies, value-based methods learn value functions to select policies, and actor-critic methods combine an acting policy with a critic.Each approach can be modeled with neural networks.Source: A. Deep Reinforcement Learning · pp. 38–39

A.1. Deep Policy Networks

Deep policy networks represent policies as neural networks mapping states to actions. Their quality is assessed using expected cumulative reward, and gradient ascent can learn the policy.

A.1. Deep Policy Networks

  • A deep policy network models the mapping from states to actions as a neural network.The policy is written as â = π(s; θ).Source: §A.1 · p. 39
  • The network’s quality is assessed by expected cumulative reward formulated in terms of the action-value function Q(s, a).Source: §A.1 · p. 39
  • The policy gradient can be applied in a gradient-ascent scheme to learn an optimal policy.Source: §A.1 · p. 39

A.2. Deep Q-Learning

Deep Q-learning learns an action-value function using the Bellman optimality criterion and temporal-difference updates. A neural network is trained on TD error, then used to select the action with the highest value.

A.2. Deep Q-Learning

  • Deep Q-learning uses the Bellman optimality criterion to recursively define the optimal value function, with the next state denoted s′.The recursion can be formulated analogously for the action-value function.Source: §A.2 · p. 39
  • Temporal-difference learning updates the current action-value estimate using the reward and a next-state TD target, with the difference serving as the TD error.The update uses a learning rate α.Source: §A.2 · p. 39
  • TD(0) looks one step ahead, TD(N) generalizes this approach, and the infinite-step limit is equivalent to Monte Carlo learning.Source: §A.2 · p. 39
  • Deep Q-learning trains a neural network for the action-value function using a loss composed of mean squared TD error, then extracts the policy that maximizes that function.Source: §A.2 · p. 39

Related papers

Loading 2309.15421v1…