PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control

Jannis Becktepe, Aleksandra Franz, Nils Thuerey, Sebastian Peitz

arXiv:2601.15015v2cs.LG

TL;DR

RL-based active flow control is difficult to compare because studies use heterogeneous setups and evaluation protocols, while existing benchmarks have solver, differentiability, 3D, and multi-agent limitations. FluidGym introduces a standalone, fully differentiable PyTorch benchmark with standardized environments and protocols. Its results show DPC can substantially accelerate learning in one environment and remain competitive in a more turbulent setting, while the authors identify limited seeds and GPU-only execution as important scope boundaries.

  • Problem

    Heterogeneous AFC setups and evaluation protocols, together with benchmark dependence on external solvers and limited differentiability, 3D, and multi-agent support, impede systematic comparison.

  • Method

    FluidGym combines a GPU-accelerated CFD solver with a standalone PyTorch RL interface providing standardized environments, protocols, and differentiable control.

  • Results

    DPC outperforms PPO and SAC by approximately one and two orders of magnitude, respectively, on CylinderJet2D-easy-v0, and is competitive with TD-MPC and SAC on RBC2D-hard-v0.

  • Takeaways & Limitations

    FluidGym provides a unified platform for comparing classical RL, multi-agent RL, and gradient-based control methods in diverse 2D and 3D AFC environments.

  • Takeaways & Limitations

    Results use a limited number of random seeds, fast simulation requires a CUDA-enabled GPU, and DPC is demonstrated only on a subset of environments.

Abstract

from arXiv · show

Reinforcement learning (RL) has shown promising results in active flow control (AFC), yet progress in the field remains difficult to assess as existing studies rely on heterogeneous observation and actuation schemes, numerical setups, and evaluation protocols. Current AFC benchmarks attempt to address these issues but heavily rely on external computational fluid dynamics (CFD) solvers, are not fully differentiable, and provide limited 3D and multi-agent support. To overcome these limitations, we introduce FluidGym, the first standalone, fully differentiable benchmark suite for RL in AFC. Built entirely in PyTorch on top of the GPU-accelerated PICT solver, FluidGym runs in a single Python stack, requires no external CFD software, and provides standardized evaluation protocols. We present baseline results with PPO, SAC, DPC, and TD-MPC, and release all environments, datasets, and trained models as public resources. FluidGym enables systematic comparison of control methods, establishes a scalable foundation for future research in learning-based flow control, and is available at github.com/safe-autonomous-systems/fluidgym.

1. Introduction

RL offers promise for active flow control, but heterogeneous setups and weak evaluation practices make progress difficult to assess. FluidGym addresses these issues with a standalone, differentiable, standardized benchmark.

  • Motivation: Heterogeneous actuators, sensor placements, physical settings, algorithms, and hyperparameters hinder direct comparisons across RL flow-control studies.Insufficiently rigorous evaluation and few random seeds also increase statistical variance.
  • Limitations of Existing Benchmarks: Existing AFC benchmarks commonly depend on external CFD solvers, creating brittle software stacks and requiring additional coupling interfaces.These dependencies demand CFD expertise and complicate integration with Python RL code.
  • Limitations of Existing Benchmarks: Limited differentiability in existing benchmarks prevents end-to-end use of DPC and recent differentiable RL methods.The limitation applies because differentiability is absent or available only in a small subset of scenarios.
  • FluidGym: FluidGym is a standalone, fully differentiable AFC benchmark built entirely on PyTorch without external solver dependencies.It integrates with common RL interfaces and algorithm frameworks through one Python package.
  • FluidGym: FluidGym standardizes task definitions across diverse environments while supporting SARL, MARL, 2D and 3D tasks, and transfer-learning studies.The benchmark includes standardized configurations and an extensive publicly available experimental study.

2. Background and Related Work

AFC research uses RL to address complex flow-control problems, but existing studies and benchmarks remain difficult to compare reproducibly. The literature is constrained by inconsistent evaluation, external solver coupling, limited differentiability, and incomplete 3D and multi-agent coverage.

  • Flow Control and RL: Fluid flows require CFD because Navier–Stokes equations exhibit complex nonlinear behavior across spatial and temporal scales.CFD approximates these equations through spatial and temporal discretization, but such simulations are computationally expensive.
  • Flow Control and RL: Classical AFC methods can require simplified models, full-state information, or expensive online optimization, limiting scalability to complex high-dimensional flows.RL has consequently been explored for drag reduction, turbulent channel-flow control, and heat-transfer enhancement.
  • Multi-Agent Control: Distributed actuation has motivated MARL and convolutional RL approaches for wall turbulence, heat transfer, and spatially distributed control.These applications include arrays of jets or heaters and other spatially distributed actuators.
  • Evaluation Challenges: Evaluation practices vary in initial conditions, baselines, test reuse, random seeds, and statistical reporting, making generalization and robustness difficult to assess.Many studies compare only with uncontrolled baselines, reuse training initial conditions, or report a single run without seeds.
  • Benchmark Gaps: Existing AFC benchmarks typically rely on external CFD solvers, lack broad differentiability, provide limited MARL support, and are predominantly 2D.These gaps leave important parts of the AFC landscape uncovered, including high-fidelity 3D flow physics.

3. FluidGym: Overview

FluidGym unifies differentiable CFD simulation and RL interaction in a PyTorch-based interface supporting multiple control modes. Its benchmark spans standardized protocols, agent configurations, difficulty levels, and diverse flow-control environments.

  • Architecture: FluidGym is designed as a standalone Python RL–CFD interface that is end-to-end differentiable, supports multi-agent control, and includes high-fidelity 3D tasks.Its design principles directly target accessibility, differentiability, distributed control, and three-dimensional flow coverage.
  • Architecture: The framework integrates the GPU-accelerated PICT solver with a modular PyTorch interaction layer, allowing environment stepping and backpropagation through shared autograd mechanisms.This removes the need for external CFD software or coupling code.
  • Interaction Modes: In the 2D RBC example, 12 lower-boundary heaters are controlled either by one agent producing the full vector or by agents controlling individual actuators locally.Local actions are aggregated and mapped to boundary actuation values, while observations derive from virtual sensor measurements.
  • Interaction Modes: FluidGym supports SARL, MARL, and gradient-based interaction modes through standardized observation, action, and reward interfaces.MARL agents receive local actions and observations, while gradient-based methods use differentiability with respect to applied actions.
  • Training and Evaluation: FluidGym standardizes training and evaluation with train, validation, and test splits containing ten randomly generated initial domains each.The environments also expose standardized interfaces and support parallel execution across multiple GPUs.
  • Benchmark Environments: The suite contains 13 environments based on four flow scenarios, each offered at three difficulty levels with distinct challenges for learning control policies.The scenarios include cylinder flow, RBC, airfoil flow, and turbulent channel flow.

4. Experiments

Experiments evaluate PPO, SAC, DPC, and TD-MPC across FluidGym’s environments, difficulty levels, three-dimensional and multi-agent settings, and transfer tasks. SAC generally leads standard RL performance, DPC benefits from differentiability, and learned policies transfer across dimensionality and domain size.

  • Experimental setup: Experiments use PPO, SAC, DPC, and TD-MPC across FluidGym environments, with five random seeds and ten test episodes.Metrics are reported per step rather than cumulatively, and hard 3D airfoil results are limited to MA-PPO.
  • Overall benchmark performance: SAC achieves the highest normalized test-set relative improvement across difficulty levels, while MA-PPO performs slightly better on TCF environments.The comparison uses performance profiles and interquartile means over environment categories and difficulty levels.
  • Overall benchmark performance: SAC reliably outperforms PPO across difficulty levels, whereas multi-agent variants are more comparable.The authors associate the multi-agent pattern with increased sample counts reducing SAC’s usual sample-efficiency advantage.
  • Gradient-based learning: DPC outperforms PPO and SAC by approximately one and two orders of magnitude in training speed on CylinderJet2D-easy-v0, while matching TD-MPC and SAC on RBC2D-hard-v0.DPC training degraded over time in the reported experiments and was stopped after 104 steps.
  • Multi-agent control: MA-PPO coordinates bottom-wall heating in RBC3D-easy-v0 to form two stable convection rolls.The observed spatial heating patterns suggest coordinated behavior among agents.
  • Policy transfer across dimensionality: The transferred 2D→3D policy outperforms 3D-trained baselines on easy and hard cylinder tasks and is on par with PPO and MA-SAC at medium difficulty.These results indicate robust direct transfer despite the added complexity of three-dimensional flow.
  • Policy transfer across domain size: Policies trained in small TCF domains perform comparably to opposition control and substantially outperform policies trained directly in large domains.The finding supports spatially transferable control strategies learned in simplified domains.
  • Computational cost: TD-MPC has the lowest training runtimes among standard RL algorithms, while DPC requires 1.5–2× higher training times than standard RL.PPO and SAC have similar training times except in TCF environments.

5. Limitations and Future Work

FluidGym’s evaluation remains constrained by computational cost, hardware requirements, limited DPC coverage, and standard rather than optimized baseline hyperparameters. Future work targets stronger statistical evaluation, broader differentiable-control studies, expanded environments, and new physical regimes.

  • Limitations: Limited random seeds reduce the statistical robustness of algorithm comparisons, and fast simulation currently requires a CUDA-enabled GPU.CPU-only execution is not supported because the solver depends on custom CUDA kernels.
  • Limitations: DPC is demonstrated on only a subset of environments, and systematic comparisons with other differentiable control approaches are not included.The paper specifically identifies broader 3D DPC experiments and differentiable RL as future directions.
  • Limitations: Baseline algorithms use standard off-the-shelf hyperparameters, which improves comparability but may not reflect optimal performance.This limits how strongly benchmark differences can be interpreted as algorithmic maxima.
  • Future work: Future work will expand random-seed counts, differentiable-control evaluations, geometries, physical regimes, and progressively challenging environments.Planned extensions include magnetohydrodynamic flows for electrically conducting fluids.

6. Conclusion

FluidGym provides a unified, differentiable benchmark for RL in active flow control, combining diverse environments with consistent interfaces and evaluation protocols. Its released environments and models are intended to improve reproducibility and comparability.

  • Conclusion: FluidGym combines GPU-accelerated CFD with standardized interfaces across diverse 2D and 3D single- and multi-agent environments.The suite also supports gradient-based methods within the same benchmark framework.
  • Conclusion: Releasing environments and trained models lowers the barrier to entry and supports reproducibility and comparability.The conclusion presents these resources as part of FluidGym’s practical contribution to AFC research.

Impact Statement

FluidGym is presented as a research tool for advancing control methods for fluid systems, with potential applications including energy efficiency, emission reduction, and energy harvesting. The paper also emphasizes that its formal RL and differentiable simulation foundations support systematic study while important deployment and computational limitations remain.

  • FluidGym is intended to advance control methods for fluid systems, with potential impacts including more energy-efficient transport and industrial processes, emission reduction, and energy harvesting.
  • Deploying learning-based controllers in safety-critical settings requires rigorous validation because FluidGym environments are idealized and omit the full complexity, uncertainties, and constraints of real systems.
  • Training RL algorithms on high-fidelity simulations can be computationally expensive and energy-consuming, motivating future work on more sample-efficient algorithms.
  • In AFC, RL agents select actions from sensor observations in partially observable settings, while multi-agent formulations assign individual observation and reward functions to agents.
  • PICT simulates incompressible flows with PISO, finite-volume discretization, implicit Euler time advancement, and differentiable simulation gradients.

B.3. Validation

FluidGym environments are numerically validated against prior results and refinement studies across cylinder, convection, airfoil, and turbulent-channel configurations. The reported comparisons show close agreement while documenting task-specific deviations and computational scope boundaries.

  • The PICT solver was previously numerically validated, and FluidGym additionally provides numerical evidence for the correctness of its environments.
  • Flow Past a Cylinder: 3.84% relative deviation separates FluidGym’s uncontrolled cylinder drag coefficient of 3.328 from the approximately 3.205 value reported by Rabault et al.
  • Rayleigh–Bénard Convection: 2.298% relative deviation is reported for the 96-resolution Rayleigh–Bénard grid, with learning behavior consistent with previous studies.
  • Flow Past an Airfoil: The airfoil environment produces mean drag 0.278 and mean lift 0.993, compared with prior averages of 0.324 drag and 1.003 lift.
  • Turbulent Channel Flow: The turbulent-channel opposition-control case yields 20% drag reduction, while the RL-controlled case reaches 30%, both agreeing closely with prior work.
  • Medium and hard 3D Airfoil cases are excluded from this work because of computational limitations.

C.1. Numerical Setup

The benchmark defines numerical domains, actuation, observations, difficulty levels, and task-specific rewards across cylinder, RBC, and airfoil environments.

  • Cylinder: Cylinder domains dynamically increase cell counts with difficulty and extrude spanwise for 3D configurations.The 2D cylinder grid is parameterized by N_angular; 3D uses the same angular resolution spanwise.
  • Reward functions: Task rewards target drag reduction for cylinders, reduced heat transfer for RBC, and improved lift-to-drag efficiency for airfoils.Cylinder rewards penalize lift, RBC rewards use the uncontrolled Nusselt-number reference, and airfoil rewards average lift and drag over the actuation interval.
  • Cylinder: Cylinder control uses parabolic jets or rotation, with temporally smoothed actions and velocity observations extended by spanwise components in 3D.The 3D jet setup contains spanwise information, while sensor planes and velocity components can be matched to 2D observations.
  • Cylinder: Cylinder difficulty increases from Re = 100 to Re = 500, with higher Reynolds numbers producing greater turbulence and unsteadiness.Medium and hard settings introduce three-dimensional flow interactions.
  • Rayleigh–Bénard convection: RBC environments use bottom-boundary heaters, temperature observations, and Rayleigh-number difficulty levels spanning distinct 2D and 3D ranges.Heater temperatures are normalized, clipped to a maximum of 1.75, and spatially smoothed; 3D observations include a 3 × 3 heater window.
  • Airfoil: Airfoil control uses surface-mounted synthetic jets with zero net mass flux, exponential action smoothing, and Reynolds-number-based difficulty levels.The NACA 0012 setup uses a 20° angle of attack, while difficulty ranges from Re = 10^3 to Re = 5 · 10^3.

D. Experimental Setup

Experiments use shared compute resources, fixed algorithm configurations, and environment-specific adjustments for multi-agent, differentiable-control, and large-scale channel-flow settings.

  • Hardware and software: The shared experimental setup uses 32 GB RAM and 32 CPU cores of an AMD EPYC 7742 processor.Most experiments use this hardware unless otherwise stated.
  • Hardware and software: CylinderJet3D-hard-v0 experiments instead use 8 AMD EPYC 7763 CPU cores.The differing hardware configuration is reported specifically for this environment and SARL experiments.
  • Algorithms: PPO, SAC, and TD-MPC use the hyperparameters listed in Tables 5–7, with SAC TCF experiments limited to one gradient step per update.The TCF adjustment avoids excessive updates caused by its large number of pseudo multi-agent environments.
  • Algorithms: For MA-PPO and MA-SAC, each individual actor corresponds to one pseudo-environment.This defines the multi-agent experimental organization used in the benchmark.
  • Algorithms: DPC rolls out a differentiable policy for H = 40 steps and backpropagates the discounted reward through time, with gradient clipping threshold ω = 0.5.CylinderJet2D-easy-v0 DPC training stopped after 10k steps because training degraded, instead of the planned 50k steps.
  • Evaluation: Runtime measurements use random-action rollouts on a single NVIDIA A100 GPU, with 80 steps generally and 8 steps for medium and hard 3D airfoil cases.Table 8 reports wall-clock time, step counts, seeds, and total GPU hours.

E.2. Quantitative Training Results

Training and test results are reported across cylinder, RBC, airfoil, and turbulent-channel environments using mean rewards, confidence intervals, and standardized test metrics.

  • Evaluation metrics: Cylinder test metrics measure drag reduction relative to the mean drag of ten uncontrolled training episodes.Values are interquartile means over test episodes and random seeds.
  • Evaluation metrics: RBC test metrics measure heat-transfer improvement relative to the uncontrolled mean instantaneous Nusselt number.The reference uses ten uncontrolled training episodes, and results are summarized by interquartile mean.
  • Evaluation metrics: Airfoil test metrics measure aerodynamic-efficiency improvement relative to the uncontrolled mean efficiency.The reported values are interquartile means over test episodes and random seeds.
  • Evaluation metrics: TCF test metrics measure drag reduction relative to the uncontrolled mean wall stress.The reference is computed from ten uncontrolled training episodes, with interquartile means reported across episodes and seeds.

E.4. Qualitative Test Results

Qualitative test visualizations compare uncontrolled and final controlled flow fields across benchmark environments, using environment-specific scalar fields, isosurfaces, and normalization ranges.

  • Evaluation scope: Qualitative visualizations compare uncontrolled and final controlled fields for every environment and algorithm using seed 0.This establishes the scope of the qualitative comparison but does not provide quantitative performance values.
  • Cylinder environments: CylinderJet2D and CylinderRot2D visualizations show the vorticity field for qualitative test results.The figures provide qualitative flow-field views for the two 2D cylinder control configurations.
  • Cylinder environments: CylinderJet3D visualizations use vorticity-magnitude isosurfaces at difficulty-specific levels and velocity-magnitude surface coloring.Velocity is normalized separately for each difficulty level relative to the uncontrolled-flow mean velocity.
  • RBC environments: RBC2D and RBC3D visualizations encode temperature values from 0 to 1.75.Both qualitative figures use temperature coloring to represent the convection fields.
  • Airfoil environments: Airfoil3D visualizations use difficulty-specific vorticity-magnitude isosurfaces and separately normalized velocity-magnitude coloring.The isosurface levels are 2.0, 3.5, and 4.5 for easy, medium, and hard cases.
  • Turbulent channel flow: TCFSmall3D-both and TCFLarge3D-both visualizations show Q-criterion isosurfaces for the controlled flow fields.The figures present qualitative results for the two large-channel configurations.
Loading 2601.15015v2…