PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learning

Ruimin Dai, Guodong Chen, Randy Harsuko, Kunpeng Liu, Nori Nakata

arXiv:2608.28791v1cs.AI

TL;DR

Long-horizon EGS well-control optimization is costly because it combines high-dimensional decisions with repeated high-fidelity hydrothermal simulations. The paper constructs a diffusion-surrogate environment for PPO using reservoir temperature and pressure states, injection-rate actions, and surrogate economic rewards. On a fractured EGS benchmark, the surrogate-assisted policy achieves competitive control performance while reducing dependence on expensive simulations.

  • Problem

    Long-horizon EGS optimization involves high-dimensional control spaces and repeated expensive hydrothermal simulations, while fixed control sequences do not adapt explicitly to evolving reservoir states.

    Source: §I · p. 2

  • Method

    Conditional diffusion models predict reservoir-state transitions and a separate reward surrogate estimates economic returns for PPO policy training.

    Source: §I · p. 2, B. Diffusion-Based Surrogate Model · p. 5

  • Results

    The PPO-Surrogate method achieves competitive well-control performance compared with direct COMSOL-trained PPO and existing optimization methods while reducing dependence on high-fidelity simulations.

    Source: F. Ablation and Cost Analysis · p. 8, §VI · p. 10

  • Takeaways & Limitations

    Diffusion-based surrogate environments can support long-horizon reinforcement learning for geothermal well-control optimization with less reliance on expensive high-fidelity simulations.

    Source: §VI · p. 10

Abstract

from arXiv · show

Real-time decision-making for enhanced geothermal systems (EGS) is challenging because long-term production periods involve high-dimensional control spaces and a large number of time-consuming high-fidelity hydrothermal simulations. Reinforcement learning provides a natural framework for state-dependent sequential control, but direct policy training with numerical simulators is computationally expensive. To address this issue, we propose a diffusion-surrogate guided reinforcement learning framework for long-horizon EGS well-control optimization. The reservoir temperature and pressure fields are used as system states, while injection rates are selected as control actions. A learned surrogate environment is constructed using conditional diffusion models to predict the evolution of reservoir temperature and pressure fields and a separate reward model to estimate the corresponding economic return. The surrogate environment is then integrated with Proximal Policy Optimization (PPO) for efficient policy training. Experiments on a fractured EGS benchmark show that the diffusion surrogate can accurately reproduce reservoir-state evolution over multiple control stages. The resulting surrogate-assisted PPO policy achieves competitive well-control performance compared with direct simulator-based PPO and existing optimization methods, while substantially reducing the dependence on expensive high-fidelity simulations. These results demonstrate the potential of diffusion-based surrogate environments for efficient reinforcement learning in geothermal well-control optimization.

I. INTRODUCTION

EGS well-control optimization must manage long-horizon economic trade-offs while adapting to evolving reservoir conditions. The proposed framework uses diffusion surrogates with PPO to reduce costly high-fidelity simulation during state-dependent policy learning.

Motivation

  • EGS extracts heat by circulating injected fluid through fractured hot-rock formations, with well operation governing pressure, flow, and heat transfer.Cold fluid enters through injection wells, absorbs heat, and returns through production wells for surface use.Source: §I · p. 1
  • Long-horizon well-control decisions must balance cumulative thermal returns against injection, production, and operational costs.Net present value captures this trade-off across the full production horizon.Source: §I · p. 2

Challenges

  • Existing optimization faces high-dimensional control spaces and repeated hydrothermal simulations, making computation a major bottleneck.The difficulty increases when multiple wells and control stages create many nonlinear decision variables.Source: §I · p. 2

Proposed framework

  • The formulation learns a policy mapping evolving reservoir temperature and pressure fields to well-control actions instead of optimizing a fixed open-loop sequence.This explicitly incorporates changing reservoir states into later decisions.Source: §I · p. 2
  • Conditional diffusion models approximate state transitions, while a separate reward surrogate estimates economic return for PPO training.The surrogate environment reduces the number of high-fidelity simulations required during policy learning.Source: §I · p. 2

Evaluation

  • On a fractured EGS well-control problem, the learned policy achieves competitive performance while requiring substantially fewer high-fidelity simulation evaluations.The evaluation examines surrogate accuracy, control performance, and computational cost.Source: §I · p. 2

II. PROBLEM STATEMENT

The problem statement models fractured EGS production through coupled fluid-flow and heat-transfer dynamics. These dynamics link injection operations to evolving reservoir pressure and temperature fields that can be used for control optimization.

Reservoir system

  • The benchmark EGS contains a low-permeability rock matrix intersected by highly permeable fractures that provide preferential circulation pathways.The surrounding hot rock matrix supplies thermal energy to the circulating fluid.Source: A. Hydrothermal Dynamics of Fractured… · pp. 2–3

Framework overview

  • The proposed workflow combines high-fidelity data collection, surrogate modeling, reinforcement-learning training, and policy evaluation.Figure 2 presents these four stages as the framework overview.Source: Figure 2 · p. 3

Hydrothermal dynamics

  • Fluid transport is governed by mass conservation and Darcy flow, with well operation represented through mass sources or sinks.The formulation includes fluid density, porosity, Darcy velocity, matrix-fracture exchange, permeability, viscosity, pressure, gravity, and elevation.Source: A. Hydrothermal Dynamics of Fractured… · p. 3
  • Reservoir temperature evolves through an energy balance that accounts for advective and conductive heat transfer and matrix-fracture heat exchange.The temperature equation includes effective heat capacity, fluid heat capacity, thermal conductivity, and thermal sources or sinks.Source: A. Hydrothermal Dynamics of Fractured… · p. 3

Control dependence

  • Changing injection rates modifies pressure gradients and circulation pathways, thereby altering heat transport and the spatial temperature distribution.The coupled hydrothermal equations are solved with the high-fidelity COMSOL simulator to characterize reservoir states.Source: A. Hydrothermal Dynamics of Fractured… · pp. 3–4

B. Reinforcement Learning Formulation

EGS well control is formulated as a finite-horizon Markov decision process whose state is the reservoir’s evolving thermal and hydraulic condition. A feedback policy selects injection rates to maximize discounted economic return over production.

State and action

  • At each control step, the state consists of reservoir temperature and pressure fields representing the coupled thermal and hydraulic conditions.These fields summarize the reservoir response from fluid-flow and heat-transfer processes.Source: B. Reinforcement Learning Formulation · p. 4
  • A parameterized policy maps the current reservoir state to injection-rate actions for the injection wells.Applying the action changes pressure distribution and fluid circulation, producing the next reservoir state.Source: B. Reinforcement Learning Formulation · p. 4

Objective

  • The reward uses net present value to balance thermal energy production against fluid injection and production costs.The objective is cumulative economic return over all control stages with discount factor γ.Source: B. Reinforcement Learning Formulation · p. 4

Feedback control

  • The formulation learns a feedback policy that continuously responds to evolving reservoir temperature and pressure instead of following a predefined full-horizon sequence.This represents well control as a controlled dynamical process.Source: B. Reinforcement Learning Formulation · p. 4

III. METHOD

The framework builds a surrogate environment that predicts reservoir temperature and pressure transitions from the current state and well-control action. Conditional diffusion models generate next-state fields, enabling multi-step rollouts without repeatedly invoking the high-fidelity simulator.

  • High-fidelity COMSOL simulations provide state transitions and economic returns used to construct the surrogate environment.Source: A. Method Overview · p. 4

Diffusion-Based Surrogate Model

  • Conditional diffusion models predict the next reservoir temperature and pressure fields from the current state and well-control action.Two independent diffusion models are trained for temperature and pressure prediction.Source: B. Diffusion-Based Surrogate Model · pp. 4–5

Conditional DDPM

2D Conditional U-Net

  • A conditional 2D U-Net denoises spatial reservoir fields while preserving localized temperature and pressure structures through multi-scale features and skip connections.Source: B. Diffusion-Based Surrogate Model · p. 5

Surrogate Rollouts

C. Reward Surrogate Model

The reward surrogate complements diffusion-based state prediction by estimating the stage-wise economic return required for reinforcement-learning training. It uses current state, action, and next state to provide rewards during surrogate-environment rollouts.

D. PPO-Based Policy Optimization

PPO optimizes a state-dependent well-control policy through repeated rollouts in the learned surrogate environment. The policy receives predicted reservoir states and surrogate rewards, then is evaluated with the original COMSOL simulator.

  • The policy outputs continuous well-control actions from evolving reservoir states, while diffusion and reward surrogates provide predicted next states and rewards.Source: D. PPO-Based Policy Optimization · p. 5

PPO Policy Update

Clipped Objective

Value Estimation

Training Procedure

Evaluation

A. Experimental Setup

The study uses a fractured EGS benchmark to compare surrogate-assisted PPO with direct simulator-based PPO and other well-control methods. Evaluation covers cumulative NPV, production responses, and computational performance.

  • The benchmark contains four injection wells and five production wells for evaluating fractured EGS well-control strategies.Source: A. Experimental Setup · p. 6
  • The 12,000-day production period is divided into 20 control stages, with four injection rates as actions and 64 × 64 temperature and pressure fields as states.Source: A. Experimental Setup · p. 6
  • 512 COMSOL trajectories provide 10,240 transitions, split into 8,700 training and 1,540 validation transitions for surrogate construction.Source: A. Experimental Setup · p. 6
  • PPO-Surrogate is compared with DE, GPEME, SACOSO, SHPSO, MFSKT, and PPO-COMSOL, with final policies evaluated using the original COMSOL simulator.Source: A. Experimental Setup · p. 6
  • PPO-COMSOL achieves the best cumulative NPV, while PPO-Surrogate ranks second and consistently outperforms evolutionary and surrogate-assisted baselines.Source: B. Overall Performance · p. 6, Table III · p. 7
  • PPO-based strategies produce higher initial thermal energy rates but faster average-temperature declines than baseline methods, with similar responses for PPO-COMSOL and PPO-Surrogate.Source: B. Overall Performance · pp. 6–7, Figure 4 · p. 7

C. Performance of Surrogate Model

The diffusion surrogate reproduces the main temperature and pressure structures generated by COMSOL across multiple control stages. PPO exploration also yields structured, state-dependent injection strategies rather than fixed or uniformly distributed actions.

E. Model Capacity and Data Size Analysis

Model capacity improves surrogate prediction more consistently than training-set size within the tested range, while combined temperature-pressure states improve control and reward prediction. Larger models, however, increase computational cost.

V. RELATED WORK

Prior work uses expensive optimization or surrogate objective models for geothermal well control, while reinforcement learning learns state-dependent policies. This paper instead constructs a diffusion-based surrogate environment for PPO and reports competitive performance on a fractured geothermal benchmark.

Conventional and surrogate optimization

  • Conventional well-control optimization often requires many expensive simulator evaluations, motivating surrogate-based methods.Derivative-free optimization avoids simulator gradients but can demand large numbers of costly simulations; data-driven methods approximate objectives to improve search efficiency.Source: §V · p. 9

Limitations of prior methods

  • Existing surrogate-assisted methods primarily optimize predefined control sequences rather than learning state-dependent control policies.This distinction separates objective approximation for fixed schedules from reinforcement learning based on current reservoir states.Source: §V · p. 9

Reinforcement learning

  • Reinforcement learning formulates well control as sequential decision-making in which current actions affect future reservoir states and cumulative returns.Prior work applied RL to reservoir production and waterflooding control, including continuous policies based on reservoir pressure and saturation states.Source: §V · p. 9

Diffusion-surrogate framework

  • The proposed approach constructs a complete surrogate environment: conditional diffusion models predict temperature and pressure evolution, while a separate surrogate estimates economic rewards for PPO.This differs from methods that only approximate the optimization objective.Source: §V · pp. 9–10, §VI · p. 10

Evaluation

  • On a fractured geothermal benchmark, PPO-Surrogate achieves competitive well-control performance against existing optimization methods and remains close to PPO trained directly with COMSOL.The reported results support diffusion-based surrogate environments for long-horizon geothermal well-control optimization with reduced dependence on expensive high-fidelity simulations.Source: §VI · p. 10
Loading 2608.28791v1…