PaperlayerView exact revision on arXiv

Source-linked AI summary

Strategic Investment Decision Making for Value Creation in Energy Transition: A Reinforcement Learning Approach

Yasaman Cheraghi, Reidar B. Bratvold, Aojie Hong, Ressi B. Muhammad, Sergey Alyaev

arXiv:2610.10768v1cs.CEcs.LGcs.SYeess.SY

TL;DR

Energy companies must balance the risks of delaying the transition against the financial risks of investing rapidly in immature renewable technologies. The paper models investment as a multi-objective sequential decision problem and uses RL to learn policies in a simulated energy environment. Its RL policy outperforms benchmark strategies, while its modeled path to net-zero entails a substantial cash-flow decline during the transition.

  • Problem

    Energy companies face uncertain timing and scale choices because delaying the transition risks future losses, while rapid renewable investment may jeopardize shareholder value.

    Source: Introduction · p. 2

  • Method

    The paper develops a multi-criteria sequential decision framework and conceptual simulation environment through 2050, applying RL to learn investment strategies across profit, emissions, and renewable competitive advantage.

    Source: Introduction · p. 4, Simulation Environment · p. 5, Conclusions and Future Work · p. 34

  • Results

    The RL policy outperforms benchmark strategies with the highest combined reward and smooth investment-allocation changes, reaches net-zero emissions by 2050, and sees cash flow fall through the 2030s before recovering.

    Source: Conclusions and Future Work · p. 34

  • Takeaways & Limitations

    In this simulated setting, the transition to net-zero requires planning for cash-flow declines before renewable capacity matures and cash flow recovers.

    Source: Conclusions and Future Work · p. 34

    The simulation produces an investment policy but does not involve real decision makers or define concrete implementation steps for an energy company.

    Source: Decision Quality of the Proposed… · p. 33

Abstract

from arXiv · show

The global challenge of climate change has driven significant steps to reduce CO2 emissions, guided by international agreements like the Paris Agreement of 2015. Acting too slowly could result in future losses and reputational damage, while moving too quickly could jeopardize shareholder value due to the marginal profitability or potential losses due to technology immaturity of many renewable projects. To navigate this complex transition, energy companies must adopt Sequential Decision Making (SDM) strategies to maximize value creation from decision flexibility under uncertainties. To support this, we developed a custom simulation environment to model the dynamic energy landscape up to 2050. Building on this, we designed a multi-criteria SDM framework that explores various decision strategies related to different portfolios for allocating funds across three sectors: oil & gas, renewables, and CO2 reduction. It aims to maximize value during the transition while accounting for uncertainties in productions, energy prices, and costs. This framework has three objectives: maximizing profit, minimizing CO2 social costs, and enhancing competitive advantage in the renewable energy sector. This research evaluates the use of Reinforcement Learning (RL) to identify optimal investment policies within the defined SDM framework. The agent's sequential decisions shape a virtual dynamic environment by influencing key variables such as oil and gas production, renewable energy output, CO2 emissions, and revenues. Through repeated interaction, the RL algorithm explores the state space and learns an optimal policy under uncertainty. We benchmark the RL strategy against a set of manually defined baseline policies and find it consistently outperforms them in adaptability and long-term value creation.

Introduction

The energy transition creates a timing and investment-scale dilemma: delaying risks future losses, while moving quickly can jeopardize shareholder value. The paper proposes a multi-criteria sequential decision framework and simulation environment, then uses RL to learn investment policies that outperform the stated benchmark strategies in long-term value.

Introduction

  • The framework treats energy-transition portfolio management as sequential decisions under uncertainty, balancing profit, CO₂ social costs, and renewable-sector competitive advantage.Companies decide how and when to allocate funds across projects through 2050.Source: Introduction · p. 4

Simulation Environment

  • The study applies RL within a custom simulation environment to learn an investment strategy through repeated interaction with uncertain and deterministic variables.The environment models a conceptual energy company and key transition dynamics, rather than data from a specific company.Source: Introduction · p. 4, Simulation Environment · p. 5

Introduction

  • The RL policy shifts investment gradually from oil and gas toward renewables and CO₂ reduction, reaches near-zero emissions by 2050, and has higher long-term value than benchmark strategies.Annual cash flow declines through the 2030s to a minimum in the early 2040s before partially recovering as renewable capacity matures.Source: Introduction · pp. 4–5

Decisions

  • The model spans 2025–2050, with 25 annual investment decisions from 2025 to 2049 and resulting outcomes realized from 2026 to 2050.Each allocation affects the following year's state.Source: Decisions · pp. 5–6

Environment

The environment represents a conceptual energy company with decision-sensitive and exogenous variables evolving over the transition. Its models connect investment allocations to production, emissions, costs, budget, prices, and renewable technology risk.

Environment

  • The simulation is a simplified, literature-grounded representation of a conceptual energy company, with decision-influenced variables distinguished from exogenous stochastic variables.The environment is publicly available to support reproducibility, benchmarking, and extension.Source: Environment · p. 7

Oil and Gas Production

  • Oil and gas production depends on investment, reserve change, and production growth, with initial production of 100 million BOE per year and reserves of 10,000 million BOE.Thirty percent of oil-and-gas investment is allocated to reserve expansion; the remainder supports production-capacity growth.Source: Oil and Gas Production · p. 7

Renewable Energy Production

  • Renewable investment adds installed capacity and electricity generation, while the model also incorporates efficiency improvement, technology immaturity risk, and infrastructure degradation.The modeled mix includes wind, solar, and hydropower, with wind assumed to dominate.Source: Renewable Energy Production · p. 8

CO₂ Emission Volumes

  • Net annual CO₂ emissions equal emissions from hydrocarbon production minus captured and stored CO₂, floored at zero.The emissions-per-production estimate is based on Equinor data; minor emissions from renewable development are excluded.Source: CO₂ Emission Volumes · p. 9

Operating Costs

  • Oil-and-gas, renewable-electricity, and CCS operating costs start at different levels and follow distinct annual drift rates.The stated rates are 5% for oil and gas, 1% for CCS, and -1% for renewables.Source: Operating Costs · p. 9

Available Budget

  • The investment budget combines a base investment level with reinvestment from smoothed operating cash flow and adjusts gradually toward its target.The passage gives a $1 billion base level and a 25% reinvestment fraction; the adjustment closes part of the gap each year.Source: Available Budget · p. 10

Oil & Electricity Price Model

  • Oil and electricity prices evolve independently of decision variables under geometric Brownian motion, with equal assumed volatility but different drift rates.The model uses 3% volatility, with drift rates of -0.3% for oil and 3% for electricity.Source: Oil & Electricity Price Model · p. 10

Technology Immaturity Risk Model

  • Renewable output faces an annual Bernoulli underperformance risk that is more likely early and declines to zero late in the horizon.An event reduces that year's renewable electricity output by a severity factor of 80%.Source: Technology Immaturity Risk Model · pp. 10–11

Objectives and Constraints

The framework converts monetary, environmental, and strategic objectives into a common monetary value for joint optimization. It measures profit through NPV, carbon damages through SCC, and renewable competitive advantage through a market-position proxy with specified timing and investment rules.

Objectives and Constraints

  • The framework combines profit, carbon social costs, and renewable competitive advantage by converting their objective values into a common monetary value.The objectives represent monetary, environmental, and strategic dimensions.Source: Objectives and Constraints · p. 11

Maximizing Profit

  • Profit is measured by net present value of net cash flows, using a 5% risk-free discount rate.The cash-flow measure accounts for revenues, operating costs, and discounting.Source: Maximizing Profit · p. 11

Reducing Social Costs of CO2 (SCC)

  • The SCC objective estimates economic damages per ton of CO₂, starting at $70 per ton and growing 2% annually with a 3% discount rate.The estimate draws on the DICE model and values reported in the integrated-assessment literature.Source: Reducing Social Costs of CO2 (SCC) · p. 12

Enhancing Competitive Advantage (CA)

  • Competitive advantage is represented by renewable-market position, with a maximum base share of 0.1% secured through early investment exceeding 15% of budget during a five-year window.The base share is adjusted proportionally if the investment threshold is met for fewer than five years.Source: Enhancing Competitive Advantage (CA) · pp. 12–13
  • After the initial five years, competitive advantage grows or declines according to year-to-year changes in renewable investment share, while its market-share proxy simplifies strategic position.An increase of at least 3 percentage points corresponds to 5% CA growth; maintaining or increasing by less than 3 points corresponds to 1% growth.Source: Enhancing Competitive Advantage (CA) · p. 13

Net-Zero Emission Regulations

The model imposes CO₂ reduction milestones aligned with a net-zero pathway and penalizes failure to meet them.

Net-Zero Emission Regulations

  • The model sets CO₂ reduction targets of 10% by 2035, 45% by 2040, 75% by 2045, and 95% by 2050, with reward penalties for missed targets.The passage notes that targets vary across countries and companies.Source: Net-Zero Emission Regulations · p. 14

Smooth Transition Constraint

The framework limits abrupt year-to-year investment changes to reflect the time needed to scale investment and decommission assets. It also frames investment decisions as sequential decision-making solved with reinforcement learning.

Smooth Transition Constraint

  • A change of 0.15 or more in any sector’s investment share within one year triggers a penalty, reflecting the time needed to scale investment or decommission assets.The text also notes that scaling down involves workforce reallocation.Source: Smooth Transition Constraint · p. 14

Theoretical Background

  • The section formulates investment decisions as a sequential decision-making process and presents reinforcement learning as the solution framework.It introduces MDPs, then RL, followed by DQN and evaluation and implementation details.Source: Theoretical Background · p. 14

Markov Decision Processes

Markov decision processes model stochastic decisions as repeated interactions in which actions produce state transitions and rewards. Bellman relationships express long-term value recursively, while dynamic programming requires known dynamics and can become computationally infeasible as the state space grows.

Markov Decision Processes

  • MDPs model sequential decisions as an agent choosing actions from current states, receiving immediate rewards, and transitioning to new states.The objective is to maximize cumulative long-term rewards.Source: Markov Decision Processes · p. 14
  • The Markov property makes the next state and reward depend only on the current state and action; Bellman equations express recursive state values, with γ weighting future rewards (Bellman, 1966).Under an optimal policy, the action-value function selects actions that maximize expected future rewards.Source: Markov Decision Processes · pp. 14–15
  • Dynamic programming finds optimal MDP policies when dynamics are fully known, but that assumption can be impractical and its computations grow exponentially with the number of state variables (Sutton & Barto, 1998).The resulting curse of dimensionality can make large-scale problems infeasible.Source: Markov Decision Processes · p. 15

Reinforcement Learning

Reinforcement learning learns decision policies through interaction and reward feedback rather than requiring a complete environment model. The section develops this approach through Q-learning and DQN, whose neural networks and stabilizing techniques address high-dimensional state-action spaces and unstable learning.

Reinforcement Learning

  • Reinforcement learning learns actions from environmental interaction and sampled experience rather than requiring a predefined model or full knowledge of dynamics.Unlike dynamic programming’s expected updates over possible future states and rewards, RL updates Q-values incrementally from interactions.Source: Reinforcement Learning · pp. 15–16
  • At each time step, the agent observes a state, chooses an available action, and receives a new state and numerical reward, refining its policy through repeated feedback.The agent’s actions can affect future states and rewards, including long-term outcomes.Source: Reinforcement Learning · p. 16

Deep Q Network (DQN)

  • Temporal Difference learning updates value estimates from actual experience while using estimated values of subsequent states, without waiting until an episode ends..Source: Deep Q Network (DQN) · p. 16
  • Q-learning separates the behavior policy from the target policy and uses ε-greedy action selection to balance random exploration with choosing the best-known action.Its action-value update uses the Bellman optimality equation.Source: Deep Q Network (DQN) · p. 17
  • DQN combines Q-learning with a deep neural network that approximates Q-values across high-dimensional state-action spaces, where tabular RL faces computational, storage, and coverage limits.The network takes a state as input and predicts Q-values for possible actions.Source: Deep Q Network (DQN) · p. 17
  • Experience replay randomizes batches of stored transitions, while a separate target network reduces correlation between predicted and target Q-values to stabilize learning.In this implementation, target-network parameters are updated gradually rather than copied at regular intervals.Source: Deep Q Network (DQN) · pp. 17–18

Formulating the Problem in a RL Framework

The investment problem is represented as sequential decisions in an RL framework, with states and discrete portfolio allocations feeding a reward that balances financial, carbon-cost, and competitive objectives alongside transition constraints.

State Space

  • The state includes the year in a 25-year horizon, annual social cost of carbon, previous sector investment shares, and the available investment budget.State variables are min–max normalized to [0,1] using fixed bounds.Source: State Space · pp. 18–19

Action Space

  • The action space has 231 allocations across oil and gas, renewables, and CCS, with investment shares in 0.05 increments summing to 1..Source: Action Space · p. 19
  • The discrete allocation choices are intended to reflect coarse strategic investment increments and make value-based DQN applicable with interpretable Q-values..Source: Action Space · p. 19

Reward Structure

  • Swing weighting assigns the three objectives weights of 0.55 for profit, 0.2 for reducing social cost of carbon, and 0.25 for competitive advantage.The objectives are combined because each is measured in monetary terms.Source: Reward Structure · p. 19
  • Each objective is min–max normalized to [0,1] using fixed extreme-value bounds, with lower social costs mapped to higher values so all three terms are maximized..Source: Reward Structure · pp. 19–20
  • The per-step reward adds net-zero and smoothness penalties, set at 0.25 per violated milestone and per sector whose allocation change reaches 0.15.The penalty values were chosen through trial and error so neither they nor the other reward components dominate.Source: Reward Structure · p. 20

DQN Architecture and Integrated Diagram

The DQN network maps the current state to action-specific Q-values, while the training loop uses agent–environment interactions to update its policy.

DQN Architecture and Integrated Diagram

  • The DQN takes the current state as input and outputs 231 Q-values, one for each possible action, through three ReLU hidden layers of 256, 128, and 64 units.The network is implemented in PyTorch.Source: DQN Architecture and Integrated Diagram · p. 20
  • At each step, an ε-greedy choice sets investment across oil and gas, renewables, and CO₂ reduction, with exploration gradually giving way to exploitation.ε starts at 0.999, decays by 0.99985 per episode, and has a 0.01 minimum.Source: DQN Architecture and Integrated Diagram · p. 21
  • The environment returns a next state and reward; sampled replay experiences update the main Q-network, while a soft-updated target network supports the learning loop.The loop is illustrated as interaction between the agent and environment.Source: DQN Architecture and Integrated Diagram · pp. 20–21, Figure 4 · p. 22

Implementation Details and Training

The DQN training converges to a stable reward level, and the results section evaluates the learned policy against benchmarks and examines objective trade-offs.

Implementation Details and Training

Results and Discussion

  • The results section compares the RL policy with benchmark strategies and uses Pareto plots to show trade-offs between objectives.The supplied passage introduces the evaluation but does not report the comparison results.Source: Results and Discussion · p. 23

RL Optimal Policy

The learned policy shifts investment from oil and gas toward renewables and CO₂ reduction over time, while producing changing sector outputs, emissions, and financial outcomes under uncertainty.

RL Optimal Policy

  • Renewables become the dominant later-year allocation, exceeding 60% of the annual budget, while oil and gas investment declines and CCS stabilizes around 30%.CCS rises slightly toward the horizon to meet final emissions targets and avoid penalties.Source: RL Optimal Policy · p. 24
  • The model allocates the entire available annual budget across the three sectors, although the authors note that not investing it all may be optimal in practice.Source: RL Optimal Policy · p. 25
  • The 1,000-episode P10–P90 investment-share ranges reflect different stochastic state trajectories; the policy is deterministic for a given state.The authors note that more detailed, realistic models could better characterize uncertainty.Source: RL Optimal Policy · p. 25, Figure 8 · p. 25
  • Oil production peaks at about 109 million barrels of oil equivalent around 2033 and falls below 26 million by 2050, while renewable output rises to about 27,000 GWh.Renewable production variability reduces noticeably after 2045, when modeled underperformance risk is assumed to be zero.Source: RL Optimal Policy · p. 26
  • Emissions peak at 30 million tons around 2031, fall to about 13 million tons by 2040, and decline toward zero by 2050 as the policy meets imposed targets.CO₂ intensity declines throughout the transition period.Source: RL Optimal Policy · p. 26
  • Annual cash flow averages about 5.2 billion USD through 2033, reaches roughly 1.8 billion USD in 2043, then recovers to about 3.5 billion USD by 2050.The available budget peaks at about 2.5 billion USD around 2038 and ends at about 2.2 billion USD in 2050.Source: RL Optimal Policy · pp. 26–27

Comparison with Benchmark Decision Policies

Across the evaluated benchmarks, RL achieves the highest combined reward; single-focus policies lead on their targeted objectives, while heuristic strategies perform between those extremes.

Comparison with Benchmark Decision Policies

Multi-Objective Trade-offs and Pareto Front Analysis

The Pareto analysis shows that investment trade-offs depend on how all three objectives interact, while the decision-quality review identifies boundaries of the illustrative framework. The authors conclude that multi-objective sequential decision-making under uncertainty can guide long-term energy-transition investment, with RL outperforming benchmark strategies in their simulation.

Multi-Objective Trade-offs and Pareto Front Analysis

  • A one-million-ton CO₂ reduction costs about 57 million USD in NPV on average, ranging from 44 million at high emissions to 110 million when emissions are already low.The cheapest reductions come first, making the final steps toward net-zero the most expensive.Source: Figure 10 · p. 30, Multi-Objective Trade-offs and Pareto… · p. 31
  • With low profit weight, adding CA weight raises competitive advantage to about 0.19% with modest emissions increases, but CA plateaus near 0.20%; emphasizing profit after that raises NPV at CA's expense.In Figure 10c, emissions reduction and CA can rise together until CA reaches its maximum.Source: Figure 10 · p. 30, Multi-Objective Trade-offs and Pareto… · pp. 30–31
  • Pairwise trade-offs can depend on the third objective, so examining only two metrics may miss important interactions.The CO₂-emissions–CA relationship changes with the weight assigned to profit.Source: Multi-Objective Trade-offs and Pareto… · p. 31

Decision Quality of the Proposed Framework

  • The frame was author-defined, and alternatives are limited to allocating the full annual budget among the same three sectors.The action space excludes withholding investment, divesting assets, or allocating funds to research and development.Source: Decision Quality of the Proposed… · p. 32
  • The illustrative model simplifies uncertainty and omits regulatory changes, energy demand, and competitive dynamics; its policy stops short of real-world implementation.The authors state that real decision makers would need richer, context-specific information and an implementation roadmap.Source: Decision Quality of the Proposed… · p. 33
  • Clear Values scores 85%: the three objectives are converted to monetary units, and assigned weights make their trade-offs explicit for Pareto analysis.The relevance of these values depends on how well they represent an actual company's priorities.Source: Decision Quality of the Proposed… · p. 33

Conclusions and Future Work

  • The RL policy outperformed benchmark strategies, achieving the highest combined reward with investment shifting gradually from oil and gas toward renewables.The shift accelerated after 2033; net-zero emissions were achieved by 2050, while annual cash flow declined through the 2030s before recovering as renewable capacity matured.Source: Conclusions and Future Work · p. 34
  • Future work could add energy demand and other sectors, use historical data for context-specific uncertainty models, and explore multi-objective RL for Pareto-optimal policies.The current reward combines multiple objectives into a single scalar function for tractability.Source: Conclusions and Future Work · pp. 34–35
Loading 2610.10768v1…