Source-linked AI summary
Direct shape optimization through deep reinforcement learning
Jonathan Viquerat, Jean Rabault, Alexander Kuhnle, Hassan Ghraieb, Aurélien Larcher, Elie Hachem
TL;DR
Direct shape optimization lacks a clear demonstration using deep reinforcement learning, despite the method’s potential for nonlinear, high-dimensional problems. The paper combines PPO, an artificial neural network, Bézier-curve shape generation, and CFD-based rewards to optimize airfoils. With an adequate lift-to-drag reward, the agent generates wing-like shapes without prior aerodynamic knowledge, while reward shaping can accelerate learning and impose constraints.
Problem
Direct shape optimization with deep reinforcement learning had not been clearly demonstrated, although airfoil optimization is a nonlinear, high-dimensional, active problem.
Method
The paper uses PPO with an artificial neural network to generate 2D Bézier-curve shapes evaluated by CFD simulations, in a single-step DRL optimization setup.
Results
The agent generated wing-like airfoil shapes without prior aerodynamic knowledge, and shaped rewards reached a learning plateau after approximately 1500 shapes versus 3000 with the baseline reward.
Takeaways & Limitations
The method provides a generic shape-optimization strategy whose optimization process is agnostic to the application domain when a relevant reward function can be defined.
Takeaways & Limitations
The CFD setup typically uses Re = 200, keeping training computationally limited for a proof of concept and future benchmarking.
Abstract
from arXiv · showhide
Deep Reinforcement Learning (DRL) has recently spread into a range of domains within physics and engineering, with multiple remarkable achievements. Still, much remains to be explored before the capabilities of these methods are well understood. In this paper, we present the first application of DRL to direct shape optimization. We show that, given adequate reward, an artificial neural network trained through DRL is able to generate optimal shapes on its own, without any prior knowledge and in a constrained time. While we choose here to apply this methodology to aerodynamics, the optimization process itself is agnostic to details of the use case, and thus our work paves the way to new generic shape optimization strategies both in fluid mechanics, and more generally in any domain where a relevant reward function can be defined.
1 Introduction
Airfoil shape optimization is an active, challenging problem because it combines industrial relevance with nonlinear and high-dimensional design. Existing methods have important trade-offs, while direct shape optimization with deep reinforcement learning remained insufficiently demonstrated; this paper introduces such an approach.
- Motivation: Airfoil optimization remains active because its industrial relevance combines with nonlinearity and high dimensionality.The paper situates airfoil optimization within longstanding applications across academic and industrial domains.
- Existing approaches: Gradient-based methods are computationally efficient in large spaces but can become trapped in local optima and struggle with discontinuous or strongly nonlinear objectives.Their sensitivity to starting points is especially pronounced for strongly nonlinear systems.
- Existing approaches: Gradient-free methods address these difficulties and genetic algorithms can find global optima, but gradient-free implementation and application are more complex.Genetic algorithms are also described as less sensitive to computational noise than gradient-based methods.
- Neural-network methods: Supervised neural-network methods have substantial shape-optimization literature, but they require labeled input-output datasets.The paper notes that supervised approaches approximate mappings between input and output spaces from labeled examples.
- Research gap: Before this work, only two cited reinforcement-learning studies addressed related morphing-airfoil optimization, leaving direct shape optimization with DRL unclear.Those studies optimized morphing dynamics with two or four parameters using Q-learning methods.
- Contribution: The paper applies PPO with an artificial neural network to generate 2D Bézier-curve shapes and evaluates their surrounding flow through 2D numerical simulation.The authors present this as the first application of DRL to direct shape optimization and plan to release the code open source.
2 Methodology
The methodology couples CFD simulations with PPO-based deep reinforcement learning to generate constrained Bézier shapes and evaluate their aerodynamic forces. A single-step episode lets the agent optimize shapes directly from reward feedback.
- 2.1 CFD environment: The CFD environment solves incompressible Navier–Stokes flow around each shape and computes drag and lift coefficients from the resulting forces.The setup uses a finite-element projection solver with BDF2 time marching, typically at Re = 200.
- 2.1 CFD environment: The maximum simulation time is selected so monitored quantities reach stabilized average values before reward evaluation.The rule uses tmax = 2(xmax − xmin).
- 2.2 Deep reinforcement learning: PPO directly optimizes the expected reward of its policy, updating network parameters from the temporally discounted rewards collected across episodes.Unlike action-value methods, policy-gradient training improves the decision policy through gradient descent on episode rewards.
- 2.2 Deep reinforcement learning: The network emits 3n bounded values that are transformed into point positions and local curvatures, which are joined deterministically into a closed Bézier shape.The point positions are restricted by user-defined inner and outer radii, limiting reachable geometries and meshing issues.
- 2.3 Degenerate DRL: Each generated shape is evaluated as a one-step episode, allowing DRL to act as a direct nonlinear optimizer using only an indirect reward signal.The correct optimal response is not known beforehand, so the setup does not rely on supervised target shapes.
3 Results
The agent generated aerodynamic shapes by maximizing lift-to-drag reward, while reward shaping accelerated learning and enabled area constraints with a performance trade-off.
- Baseline results: The agent optimized shapes described by four points, with experiments allowing one, three, or four free points.Each free point contributes three degrees of freedom: position and local curvature.
- Baseline results: The best-performing shapes developed airfoil-like geometry and a similar trailing edge as more points were freed.With four free points, the airfoil extended across the available domain, while the trailing-edge angle remained close to 23° in the three- and four-free-point cases.
- Shaping for faster convergence: 1500 shapes were needed to reach the learning plateau with shaped reward, compared with 3000 using the baseline reward.The average plateau reward was also slightly higher with shaped reward.
- Shaping to add constraints: The constraint was introduced by adding an area-penalization term to the reward, causing the agent to trade off lift-to-drag performance against area matching.After approximately 2000 episodes, the constrained agent began generating shapes that minimized the penalization term.
- Shaping to add constraints: The area-penalized optimum had area 3.176 versus 2.733 without penalization, while its lift-to-drag ratio was approximately 30% lower.The target area was that of the unit cylinder, αcyl = π.
4 Conclusions
The paper concludes that DRL can directly optimize shapes without prior aerodynamic knowledge and that the approach may extend beyond aerodynamics. It also identifies several method refinements for future work.
- The agent generated wing-like optimal shapes from a lift-to-drag reward without prior aerodynamic knowledge.
- The degenerate DRL formulation uses single-step episodes to apply DRL algorithms as general-purpose direct nonlinear optimizers.
- Further work should investigate observations, shape deformation management, and data efficiency.
- The method is agnostic to the application use case and is extendable to other domains of computational mechanics.
A Open source code
The project releases its implementation as open-source code, including CFD, reinforcement-learning, and Bézier-shape-generation components.
- The repository contains code using FEniCS for CFD, Tensorforce for reinforcement learning, and homemade Bézier-curve shape-generation code.
B DRL, policy gradient and PPO algorithm
Reinforcement learning trains an agent to make decisions by interacting with an environment and maximizing discounted rewards. Policy-gradient methods directly optimize stochastic policies, while PPO stabilizes updates by clipping policy changes and adding entropy regularization.
- Reinforcement learning: At each timestep, the agent observes a state, selects an action, and receives a reward while generating a policy-dependent trajectory.The objective is to maximize discounted cumulative reward, with γ controlling the priority given to immediate versus distant rewards.
- Reinforcement learning: Q-learning uses estimated action values in a discrete, finite action space, whereas policy-gradient methods can handle discrete and continuous actions.In deep reinforcement learning, Q-functions can be represented by deep neural networks and optimized through the Bellman equation.
- Policy gradient: Policy-gradient methods directly optimize a stochastic policy to maximize expected discounted reward rather than learning an auxiliary value function.They can handle high-dimensional action spaces and often provide smoother convergence, but may converge to local minima.
- PPO algorithm: PPO uses a surrogate advantage objective to compare the updated policy πθ with the previous policy πθk.The advantage function measures whether an action performs better or worse than the average action in a state.
- PPO algorithm: PPO clips policy-ratio updates to prevent steps that move too far from the previous policy and could damage performance.The clipping threshold is controlled by the small user-defined parameter ε, with the clipping direction determined by the sign of the advantage.
- PPO algorithm: PPO estimates advantages with GAE, uses subsampled mini-batch updates, and adds entropy regularization to discourage over-confident policies.Entropy regularization keeps the policy distribution closer to uniform unless there is a strong signal favoring otherwise.
C Shape generation using B´ezier curves
The shape generator converts points supplied by the agent into a boundary made of successive cubic Bézier curves. Sorting, angle averaging, control-point construction, and final sampling produce a broad range of attainable shapes.
- Shape generation using B´ezier curves: The generator begins with ns points supplied by the agent, sorts them by ascending trigonometric angle, and computes angles between consecutive points.It then computes an average angle around each point before constructing the curve.
- Shape generation using B´ezier curves: Each consecutive point pair is joined by a cubic Bézier curve whose endpoints lie on the curve and whose interior control points define endpoint tangents.The tangent directions at the endpoints are controlled by the corresponding θ* values.
- Shape generation using B´ezier curves: Sampling the successive Bézier curves produces the shape boundary, allowing a wide variety of shapes to be attained.Figure 11 presents the consecutive stages from point sorting through final boundary sampling.