Topology Optimization via Machine Learning and Deep Learning: A Review

• TO determines optimal material distributions under prescribed loads, boundary conditions, and design constraints, enabling design without a meaningful initial configuration.The review identifies density-based SIMP, ESO, level-set, and MMC as major TO methodologies.Source: §1 · p. 2
• FEA-driven iterations make TO computationally costly: increasing mesh size 125-fold increased required time 4,137-fold, while fine-resolution TO can take hours to days.Three-dimensional TO incurs still higher costs as iteration demands increase across SIMP, BESO, and level-set methods.Source: §1 · p. 2
• Although approximation, GPU, and parallel-computing methods reduce runtime, TO costs remain high, motivating ML-based approaches to accelerate optimization.ML-based TO is presented as a response to the persistent computational burden of conventional iterative design.Source: §1 · p. 2
• Data-based TO methods use AI, ML, or DL to learn from data and rapidly optimize initial topologies, with this review emphasizing DL methodologies.ML is defined as learning patterns from data, while DL uses multilayer hierarchical neural networks trained from data.Source: §1 · p. 3
• The review organizes MLTO studies by why ML is applied from a TO perspective and how TO becomes an ML problem from an ML perspective.It also analyzes current limitations and desirable future directions; Figure 2 presents the review framework.Source: §1 · p. 3, Figure 2 · p. 3

2. TO Perspective: Why Use ML

• MLTO serves seven purposes: accelerating iteration, non-iterative optimization, meta-modeling, design-space reduction, optimizer improvement, generative design, and postprocessing.Figure 3 visualizes these purposes, and Table 1 classifies the corresponding studies.Source: §2 · p. 4, Figure 3 · p. 4, Table 1 · p. 5

2.1. Acceleration of Iteration

• Iterative-acceleration methods perform conventional TO until an intermediate stage, then use ML predictions to bypass the slow, costly convergence stage.Inputs include intermediate topologies, density gradients, loading and boundary conditions, or predicted gradients; some methods refine ML predictions with subsequent SIMP updates.Source: §2.1 · p. 5, (Sosnovik & Oseledets, 2019) · p. 6

2.2. Non-Iterative Optimization

• Non-iterative MLTO directly predicts designs from loading, boundary, material, volume-fraction, or objective conditions, while multistage networks improve resolution or engineering performance.Single-stage models include ResUnet, cGAN, WGAN, and physics-guided networks using FEA-derived strain or stress information; multistage methods upscale low-resolution predictions, model multiresolution designs, or sequentially predict topology evolution.Source: (Abueidda et al., 2020) · p. 6, §2.2.1 · p. 7, §2.2.2 · p. 7

2.3. Meta-Modeling

• Meta-modeling replaces or supplements computationally expensive FEA calculations for objective values, sensitivities, constraints, or hybrid screening during TO.ML surrogates predict compliance, material properties, stiffness, stresses, sensitivities, reliability, manufacturing performance, or motor objectives; hybrid approaches use predictions to decide when accurate FE calculations are necessary.Source: §2.3 · p. 8, §2.3.1 · p. 8, §2.3.2 · p. 9

2.4. Dimensionality Reduction of Design Space

• ML reduces TO design-space dimensionality by learning PCA-, logistic-PCA-, or VAE-based latent representations and optimizing within those lower-dimensional spaces.These representations support topology generation, surrogate-based optimization, and global search; in Guo et al. (2018), MOGA performed best among the attempted optimization methods.Source: §2.4 · pp. 10–11

2.5.1. Optimization Parameter Tuning

• ML improves optimization by tuning algorithm parameters, screening feasible designs, recommending parameters from similar problems, and predicting topology variations.Methods include PCA with SVR or KNN, extratree classification, graph-based parameterization, neural ranking, meta-learning, and Bayesian optimization.Source: §2.5.1 · pp. 11–12

2.5.2. Reparameterization

• Neural reparameterization represents density fields with network parameters and updates them by backpropagating physics-based objective losses, reducing explicit design variables and avoiding preconstructed datasets in some frameworks.The approach extends to 2D and 3D TO, nonlinear and multimaterial problems, and can impose manufacturing or length-scale constraints through projection operators.Source: §2.5.2 · p. 12

2.6. Generative Design

• Generative MLTO expands design diversity through naïve generators and engineering-performance guidance from simulations, predictors, dimensionality reduction, clustering, or conventional TO.Engineering information can enter the loss, training data, latent space, clustering process, or iterative SIMP workflow; simulation-based methods often avoid training data.Source: §2.6 · p. 13, §2.6.2 · pp. 13–15

2.6.3. Detailed Features

• Generative studies target detailed features, stable diversity, manufacturability, aesthetics, and reinforcement-learning-based diversity across increasingly specialized design constraints.PGGAN and self-attention learn detailed or global features; convex relaxing CGAN addresses convergence and mode collapse; other methods incorporate manufacturing distributions, SIMP-based aesthetics, or RL rewards for diversity.Source: §2.6.3 · p. 16, §2.6.4 · p. 16, §2.6.5 · p. 16

2.7. Postprocessing

• ML postprocessing addresses the high cost of high-resolution TO and unclear boundaries by upscaling low-resolution results or producing smooth, explicit structural boundaries.ANN, CNN, and SRCNN support high-resolution reconstruction, including 2D and 3D problems with varied boundaries, domains, and loadings; SVM and MLP methods process boundaries and hole shapes.Source: §2.7 · p. 17, §2.7.1 · pp. 17–18, §2.7.2 · p. 18

2.8. Multiscale, Multimaterial, and Other Processes

• Other MLTO applications reduce multiscale or multimaterial complexity and address specialized fluid-structure and graph-based structural optimization problems.Clustering can reduce MTO cost while improving microstructural connectivity, and multimaterial optimization combines density clustering with kriging and global optimization; DNN, Monte Carlo search, graph embedding, and RL support other problems.Source: §2.8 · p. 18, §2.8.1 · pp. 18–19, §2.8.2 · p. 19

2.5.2. Reparameterization

• Reparameterization uses neural networks and physics-based losses to optimize continuous, mesh-free density and displacement fields, reducing dataset needs while retaining connectivity, though repeated FEA can remain costly.Zehnder et al. (2021) used implicit neural representations for 2D and 3D mesh-free TO, but required more computational time than conventional FEM.Source: §2.5.2 · p. 13, §2.9 · p. 20

2.9. Relations between MLTO Purposes

• Across MLTO purposes, methods either replace expensive iterative stages for one-shot prediction or accelerate them while preserving accuracy, with tradeoffs involving pixelwise accuracy, data collection, and residual FEA cost.Reparameterization reduces dataset requirements and can ensure connectivity, whereas iterative objective evaluations and supervised-data collection can still impose substantial computational burdens.Source: §2.9 · p. 20

3. ML Perspective: How to Use ML

• ML-based topology optimization is organized by learning algorithm into supervised, unsupervised, semi-supervised, and reinforcement learning categories, alongside classifications by data, loss, physics, and applications.The learning-algorithm taxonomy is presented as one of several perspectives for analyzing ML applications to topology optimization.Source: §3.1 · p. 20, §3 · p. 20

3.1.1. Supervised Learning

• Supervised learning most often uses NNs and CNNs to predict design features or performance, optimize designs, replace objective or sensitivity calculations, and process topology boundaries.LSTMs model topology sequences, while SVMs, SVRs, SVM-based postprocessing, Gaussian processes, and other regressors support prediction and optimization.Source: §3.1.1 · pp. 21–23

3.1. Learning Algorithm

• Across learning algorithms, ML is applied to reduce TO computation by predicting designs or performance, replacing FEA-related calculations, approximating high-dimensional shapes, and guiding iterative optimization.The review includes examples replacing objective, sensitivity, or elasticity calculations; predicting subsequent densities or sequences; and using surrogates or learned gradients.Source: §3.1.1 · pp. 21–22, §3.1.2 · p. 24

3.1.2. Unsupervised Learning

• Unsupervised learning primarily uses generative models for design exploration, including GANs and autoencoders, while PCA and clustering support reduced-space prediction, optimization, and lower-cost iterative TO.GAN studies generate, condition, refine, upscale, or evaluate designs; autoencoders generate feasible designs, and clustering can use surrogate feasibility models.Source: §3.1.2 · pp. 23–24

3. ML Perspective: How to Use ML

• The reviewed methods also integrate physical information through design constraints, loads, boundary conditions, FEA outputs, performance evaluations, or physics-informed loss functions.Such integration appears in conditional GANs, performance-reflective training, forward networks, and models that combine learned predictions with FEA or SIMP refinement.Source: §3.1.1 · p. 22, §3.1.2 · p. 23, §3.1.3 · pp. 24–25

3.1.3. Semi-Supervised Learning

• Semi-supervised approaches combine labeled and unlabeled learning, often through active learning or hybrid generative and predictive networks, to generate physically informed designs with fewer labeled data or lower computational budgets.Examples incorporate physical properties or performance into latent spaces, labels, forward networks, loss functions, or combinations with SIMP.Source: §3.1.3 · pp. 24–25

3.1.4. Reinforcement Learning

• Reinforcement learning has high potential but remains insufficiently studied in TO, using value-based methods for design optimization and policy-based methods for design exploration.Q-learning and double-deep Q-learning optimize graph-based or sequential topologies, while e-greedy, UCB, TS, IDS, and PPO guide search or maximize design diversity.Source: §3.1.4 · p. 25

3.2. Data

• Overall, DLTO input data vary according to model purpose, including design conditions, intermediate results, final designs, FEA, coordinates, and combined information.The review classifies these frameworks by the form of DL input.Source: §3.2 · p. 25

3.2.1. Design Parameter (Condition)

• Design parameters encode loads, boundary conditions, constraints, or coordinates and support performance prediction, direct optimization, and generation or exploration of new geometries.Prediction can replace FEA for material or microscale properties; optimization maps conditions to geometries; exploration uses generative models, GANs, or Q-Learning.Source: §3.2.1 · pp. 25–26

3.2.2. Intermediate Optimization Result

• Intermediate optimization results are used for prediction and optimization, replacing objective, sensitivity, feasibility, or stress calculations and accelerating or restarting topology optimization.Inputs include intermediate densities, gradients, density fluctuations, data-point features, or clustered material distributions.Source: §3.2.2 · p. 26

3.2.3. Final Optimization Result

• Final optimization shapes support performance prediction, design exploration, and postprocessing such as boundary processing, resolution enhancement, hole-shape prediction, and new-shape generation.Generative approaches include VAEs, GANs, autoencoders, and reinforcement learning, with some models targeting high-performance or original designs.Source: §3.2.3 · pp. 26–27

3.2.4. Structural Information (FEA Information)

• FEA information primarily enables prediction and design optimization by replacing sensitivity or gradient calculations, assessing damage, and generating shapes from structural responses.Models use density, displacement, stress, gradients, filtered variables, or clustered FEA information, sometimes together with design parameters.Source: §3.2.4 · p. 27

3.2.5. Coordinates in Design Space

• Coordinate inputs enable density prediction and optimization through loss functions incorporating FEA or system energy, while reduced design variables can accelerate optimization.Coordinate-based methods derive target density fields through sensitivity filtering or predict voxel-representing design variables.Source: §3.2.5 · p. 27

3.2.6. Multimodal Input

• Multimodal models combine design parameters with intermediate states or FEA information to predict optimal designs, guide reinforcement-learning actions, or replace compliance and sensitivity calculations.Examples combine loads and boundary conditions with stress, shapes from prior iterations, intermediate FEA, or other structural information.Source: §3.2.6 · p. 28

3.3. Loss Function

• DLTO loss functions minimize shape or performance discrepancies, incorporate FEA-derived objectives and constraints, and combine distinct losses when multiple models are used.Shape losses include quadratic norm, MSE, and BCE; performance losses include MSE, MAPE, and quadratic forms, while additional terms address regularization, style, mode collapse, and volume constraints.Source: §3.3 · p. 28, §3.3.1 · p. 28, §3.3.2 · p. 28

3.4.1. FEA as a loss function term in the training process

• In Method 2, FEA-derived objective and constraint information enters the loss function, directing backpropagation toward higher-performance designs.This approach is reported for multiple DLTO studies, including Hoyer et al. (2019), Chandrasekhar & Suresh (2021c), Chandrasekhar & Suresh (2021a), Chandrasekhar & Suresh (2021b), Zhang et al. (2021a), Zehnder et al. (2021), Jiang & Fan (2019a), and Jiang & Fan (2019b).Source: §3.3.3 · p. 28, §3.4.2 · p. 30

3.5. Applications and Generalization

• DLTO applications span beams, bridges, trusses, nanophotonics, metasurfaces, periodic structures, wheels, motors, heat conduction, fluid structures, and both 2D and 3D domains.The reviewed literature is dominated by 2D shape optimization, but 3D shape optimization is also being pursued.Source: §3.5 · p. 30, §3.5.1 · p. 31, Table 3 · p. 31

a1) Diversification of Load Conditions

• Diversifying load conditions enables generalization within fixed design domains, including brackets, trusses, automobile wheels, conductive heat-transfer systems, metamaterials, and periodic structures.Several studies vary load position, direction, ratio, or source locations, while others learn from distributions of problems or latent spaces trained on multiple load conditions.Source: §3.5.2 · pp. 31–32, a1) Diversification of Load Conditions · p. 32

a2) Diversification of Load and Boundary Conditions

• Models generalize across load and boundary conditions by receiving them as inputs, updating condition-dependent losses, repeatedly remeshing, or training on diversified low- and high-resolution shapes.Examples include Zhang et al. (2019), Nie et al. (2021), Yu et al. (2019), Malviya (2020), Halle et al. (2021), Hoyer et al. (2019), Zhang et al. (2021a), and Wang et al. (2021).Source: a2) Diversification of Load and… · p. 32
• DLTO still lacks a broad 3D expansion mechanism, although studies extend generalized optimization to 3D beams, objects, robot grippers, periodic materials, and multiscale structures.Rawat and Shen (2019a) optimized 3D beams without varying loads and boundary conditions, whereas other studies combined such variation with 3D extensions.Source: a2) Diversification of Load and… · p. 32
• Flexible design-domain generalization is pursued through graph-based representations, reinforcement learning, online gradient updates, clustering, and networks or hybrid SIMP-DL procedures applicable to varied domains and conditions.Reported approaches include Hayashi and Ohsaki (2020), Brown et al. (2022), Bi et al. (2020), Kallioras et al. (2020), Kallioras & Lagaros (2020), Kallioras & Lagaros (2021a), Kallioras & Lagaros (2021b), Zheng et al. (2021), and Xue et al. (2021).Source: a2) Diversification of Load and… · pp. 32–33

4.1. Computational Complexity

• MLTO can instantly infer topologies after training, but supervised methods require costly optimized datasets, especially for 3D problems (Kallioras & Lagaros, 2020; Rade et al., 2020; Yu et al., 2019).Examples include datasets of 80,000, 100,000, and 13,500 optimized structures for neural-network training.Source: §4.1.1 · p. 33, §4.1 · p. 33

4.1.2. Computational Cost of FEA

• MLTO faces an accuracy–cost tradeoff: FEA or iterative TO improves engineering performance but can consume substantial computation, including approximately 50% of cost for a 60×30 mesh.Parallel FEM computation has been proposed, but further work is needed to use FEA while minimizing computational cost, particularly for large 3D problems.Source: §4.1.2 · p. 33

4.1.3. Physics vs. Data

• Physics-informed neural networks can reduce dependence on large labeled datasets, prevent disconnected designs, support extrapolation, and lower data-collection burdens, but MLTO PINN results remain benchmark-level (Zhang et al., 2019).PINNs incorporate governing PDEs into loss functions, motivating future MLTO studies for data-limited and 3D topology optimization.Source: §4.1.3 · pp. 33–34

4.2. Scalability

• MLTO scalability is limited because many methods target one fixed design, cannot handle new boundary or loading conditions, and remain difficult to extend to 3D and complex optimization problems.Industrial applicability requires adaptive methods that generalize across domains and conditions; neural-network density reparameterization can extend to 3D without advance data, although FEA cost remains.Source: §4.2 · p. 35, §4.2.2 · p. 35, §4.2.3 · p. 35

4.3.1. Topology Resolution and Connectivity

• MLTO often produces low-resolution, blurry, disconnected, or checkerboard-patterned designs, while manufacturing constraints are rarely included despite the need for manufacturable AM and injection-molding results.Connectivity, detailed features, and manufacturing feasibility remain necessary for product development (Banga et al., 2018; Zhang et al., 2019).Source: §4.3.1 · pp. 35–36, §4.3.2 · p. 36

4.3.3. Robust Learning Performance

• MLTO networks require stronger validation because instability, cherry-picking, mode collapse, and inadequate evaluation metrics undermine reliability and design diversity (Gillhofer et al., 2019; Li et al., 2021; Rawat & Shen, 2019a).GANs, cGANs, and generative-design approaches have been explored to improve accuracy, condition outputs, and address diversity limitations.Source: §4.3.3 · p. 36

• ML-based TO studies use supervised, unsupervised, and reinforcement learning for acceleration, non-iterative optimization, meta-modeling, design-space reduction, optimizer improvement, generative design, and postprocessing.These approaches speed optimization and reduce computational cost.Source: §5 · p. 36
• Current ML-based TO remains limited by scalability across design domains, data-collection burden, low topology resolution, manufacturing infeasibility, weak ML performance, and expensive FEA and computation.Source: §5 · p. 36
• Future work should target effective ML-based TO for 3D domains while using physics information to obtain optimal results and explain engineering phenomena.Source: §5 · p. 36

Appendix A. Background of TO

• TO minimizes an objective F subject to a volume constraint and possibly additional constraints, with material density ρ(x) constrained to void or solid values.The state field u satisfies a linear or nonlinear state equation.Source: Appendix A · p. 44
• Appendix A introduces density-based, evolutionary structural optimization (ESO), level set, and moving morphable component (MMC) methods as representative TO methodologies.Figure 24 presents MBB half-beam optimization results obtained with the four methods.Source: Appendix A · p. 44, Figure 24 · p. 44

A.1. Density-Based Method

• SIMP derives material distribution for specified loads, boundary conditions, and design spaces using element densities bounded between x_min and 1 rather than binary values.The positive x_min prevents singular finite-element matrices and avoids the inability to re-densify zero-density elements.Source: §A.1 · p. 45
• The compliance-minimization formulation uses element densities as design variables, with displacement, stiffness, material volume, design-domain volume, and volume fraction defining the optimization problem.The method represents density-dependent material behavior through element-level stiffness contributions and global compliance.Source: §A.1 · p. 45
• Power-law interpolation penalizes intermediate densities by scaling Young’s modulus between void stiffness E_min and solid-material stiffness E_0, with numerical experiments indicating p = 3 as suitable.Continuous relative density produces continuously varying Young’s modulus, while the penalty reduces gray elements’ influence on total stiffness.Source: §A.1 · p. 45
• Gradient-based optimization requires sensitivity analysis at each iteration, and the optimality-criteria method commonly updates density-based design variables using move limits and numerical damping.Sensitivity evaluates how density changes affect the objective and constraints, while OC supplies the design-variable update rule.Source: §A.1 · p. 45
• Sensitivity and density filters use weighted neighboring-element averages within a user-defined mesh-independent radius to mitigate checkerboards and grayscale issues.The sensitivity filter averages sensitivities, whereas the density filter averages physical densities; filtered densities then modify sensitivity analysis for the physical density.Source: §A.1 · p. 46

Evolution and basic formulation

• ESO, introduced by Xie and Steven (1993, 1997), removes unnecessary material, while BESO, introduced by Yang et al. (1999), removes unnecessary and adds necessary material.The basic ESO formulation minimizes compliance.Source: §A.2 · p. 46

Sensitivity-based element removal

• ESO removes elements whose sensitivity numbers fall below a prescribed criterion.Sensitivity numbers are calculated to determine which elements are removed.Source: §A.2 · p. 46

Complementary BESO

• Complementary BESO uses nodal sensitivities averaged from connected elements to address mesh dependency and checkerboard patterns without requiring convergence.Huang and Xie (2007) introduced this complementary method.Source: §A.2 · p. 46

Sensitivity filtering

• BESO identifies neighboring nodes within a minimum filter radius and combines their contributions using distance-based linear weights to improve element sensitivities.The filtered sensitivity calculation uses the total nodes in the filter domain and distance-dependent weighting.Source: §A.2 · p. 47

Bidirectional updating

• BESO evaluates sensitivities for void elements as well as solid elements, removing low-sensitivity solid elements and adding elements whose sensitivities exceed the addition threshold.The supplied passage specifies removal using α_del and addition using α_add.Source: §A.2 · p. 47

A.3. Level Set Method

• The method captures free boundaries in linear elastic structures, producing crisp designs that require less postprocessing while allowing void formation, extinction, and merging.Source: §A.3 · p. 47
• Boundaries are implicitly represented by the zero-level curve of a scalar function 𝛷, whose evolution regulates boundary movement and merging in three-dimensional space.The evolving boundary is expressed as S(t) = {x(t): 𝛷(x(t), t) = k}.Source: §A.3 · p. 47
• A Hamilton–Jacobi-type PDE determines boundary evolution, with its speed vector depending on the optimization objective and selected through sensitivity-analysis descent directions.Source: §A.3 · p. 47

A.4. MMC Method

• MMC determines optimal structures through component shape, length, thickness, direction, and layout, contrasting with pixel-based and node point-based methods.Source: §A.4 · p. 48
• MMC can encode complex and diverse topologies with few components while allowing components to overlap without structural effects in overlapping regions.Source: §A.4 · p. 48
• A level-set function represents each component’s occupied area, distinguishing its interior, boundary, and exterior within the design domain.Source: §A.4 · p. 48
• Component movement, dilation or shrinkage, and rotation are controlled by x0, y0, L, t, and θ, with m=6 used in the study.Source: §A.4 · p. 48
• The MMC optimization minimizes I(d) subject to constraint functionals g_i(d)≤0, with design variables d_i=(x0_i,y0_i,L_i,t_i,θ_i) over admissible sets.Source: §A.4 · p. 48

Appendix B. Background of ML

• Deep learning comprises four types: supervised learning uses labeled data, unsupervised learning discovers patterns in unlabeled data, semi-supervised learning combines both, and reinforcement learning learns reward-increasing action strategies.Semi-supervised learning may train larger unlabeled datasets with small amounts of labeled data, whereas reinforcement learning does not require separately provided data or labels.Source: Appendix B · p. 49

B.1. Supervised Learning

• Labeled datasets support predictions for new data through classification or regression, with classification covering binary and multiclass categories and regression predicting continuous values.Representative algorithms include KNN, SVM, decision trees, random forests, and neural networks.Source: §B.1 · p. 49
• KNN classifies based on proximity among similar instances, while SVM supports classification and regression by maximizing class margins and separating hyperplanes.KNN is attributed to Cover & Hart (1967), and SVM to Burges (1998).Source: §B.1 · p. 49
• Decision trees classify instances by feature-based paths from root nodes, whereas random forests combine multiple decision trees for classification and regression.Decision trees are attributed to Murthy (1998), and random forests to Breiman (2001).Source: §B.1 · p. 49
• Artificial neural networks mathematically model brain-neuron structures and developed into deep neural networks with deeper hidden layers; CNNs are prominent for image-data applications.CNNs comprise convolutional, nonlinearity, pooling, and fully connected layers.Source: §B.1 · p. 49

B.2. Unsupervised Learning

• Unsupervised learning uses unlabeled inputs to discover data characteristics and patterns through similarity-based clustering.The model learns from input data without receiving labels.Source: §B.2 · p. 49
• Its main applications are clustering, dimensionality reduction, and generative modeling.Clustering groups similar data; dimensionality reduction compresses high-dimensional data while preserving meaningful features; generative models create data resembling existing examples.Source: §B.2 · p. 49
• K-means groups data into k clusters by cluster centers and average distances, but generally does not converge to the optimal partition.It is an iterative hill-climbing clustering algorithm designed to find an optimal partition.Source: §B.2 · p. 49
• PCA reduces dimensionality by finding a maximum-variance basis while preserving most data variation.Reduced samples with similar features are likely to gather in a low-dimensional latent space.Source: §B.2 · p. 49
• GANs generate data through generator–discriminator competition, while cGANs add conditioning information and VAEs generate samples from latent-space features (Mirza & Osindero, 2014).A VAE trains an encoder to support decoder-based generation.Source: §B.2 · p. 49

B.3. RL

• RL learns decisions through agent–environment interaction, commonly formulated as a Markov decision process without requiring explicit models.The agent receives states and rewards from the environment after taking actions.Source: §B.3 · p. 50
• Trial-and-error training updates actions toward maximizing discounted cumulative reward, R_t = ∑ γ^k·R_t+k, with γ ∈ (0,1) discounting future rewards.At time t, action a_t produces state s_t and reward R_t from the environment.Source: §B.3 · p. 50
• RL uses value-based methods such as DQN and DDQN, policy-gradient methods such as advantage actor–critic and asynchronous advantage actor–critic, and hybrid actor–critic methods including TRPO and PPO.Value methods estimate expected returns, whereas policy methods update parameters in the direction of increasing reward.Source: §B.3 · p. 50