Source-linked AI summary
Learning Stability of Replay-Based Co-Optimization for Transmission Expansion under Strategic Bidding
Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani
TL;DR
Transmission planning and strategic bidding interact, but their learning-based co-optimization can be disrupted when transient bidding-policy degradation produces inconsistent cost–capacity samples. The paper introduces replay-based capacity updates with nearest-neighbor filtering and examines how replay-memory size affects learning. IEEE 30-bus experiments show improved robustness alongside a memory-size and learning-rate trade-off: larger memories can introduce lag and oscillations unless the learning rate is reduced.
Problem
Strategic bidding and transmission expansion are coupled, and transiently degraded bidding policies can generate misleading cost–capacity samples that destabilize capacity updates and hinder convergence.
Source: §I · p. 1, B. Instability in the Conventional… · p. 4Method
The paper studies replay-based transmission-capacity updates that reuse historical samples and apply nearest-neighbor filtering to exclude inconsistent data.
Source: §I · p. 1, B. Co-optimization of Transmission… · p. 3Results
On the IEEE 30-bus system, replay-based filtered updates improve robustness to transient policy degradation, while larger replay memories can introduce temporal lag and oscillations unless the learning rate is reduced.
Source: C. Replay-Based Capacity Update with… · p. 4, §IV · p. 6Takeaways & Limitations
Replay-memory size and capacity-update learning rate should be tuned together because larger memories improve robustness but require smaller learning rates to maintain stability.
Source: §IV · p. 6
Abstract
from arXiv · showhide
This paper investigates the behavior of learning-based co-optimization for transmission expansion under strategic bidding in electricity markets. In this framework, transmission capacities are updated while market participants simultaneously learn their bidding strategies through deep reinforcement learning, resulting in coupled and non-stationary learning dynamics. We show that transient policy degradation of bidding agents can generate inconsistent cost-capacity samples, which bias the transmission-capacity update and prevent the co-optimization process from converging to the desired solution. To mitigate this, replay-based capacity updates with nearest-neighbor filtering are introduced to exclude inconsistent samples from the update data. We then analyze a new oscillatory behavior that appears when the replay memory is enlarged. Numerical results on the IEEE 30-bus system demonstrate that enlarged replay memories improve robustness against transient policy degradation but can introduce a temporal lag, leading to oscillations unless the capacity-update learning rate is appropriately reduced. These results reveal the trade-off between replay-memory size and learning rate and provide practical guidelines for stable co-optimization.
I. INTRODUCTION
Transmission expansion planning and strategic bidding are tightly coupled, but research on market simulation and RL-based expansion has largely treated them separately. This paper studies replay-based co-optimization, where transient bidding-policy degradation can corrupt capacity updates and larger replay memories trade greater robustness for increased oscillation risk.
I. INTRODUCTION
- Market-simulation studies generally assume fixed networks, while RL-based expansion-planning studies often neglect strategic market adaptation.The paper positions co-optimization as a way to bridge these separately developed research directions.Source: §I · p. 1
- Transiently degraded bidding policies can create misleading cost–capacity relationships, making insufficient transmission capacity appear economically preferable and destabilizing capacity updates.Using inconsistent samples can prevent convergence to the desired solution.Source: §I · p. 1
- Replay-based capacity updates with nearest-neighbor filtering exclude inconsistent samples to improve robustness against transient policy degradation.Longer simulations also reveal that larger memories preserve broader historical information but introduce temporal lag, potentially causing overshoot and oscillations unless the learning rate is reduced.Source: §I · p. 1
- The paper analyzes how replay-memory size and learning rate shape the trade-off between robustness and oscillatory behavior, using numerical experiments on the IEEE 30-bus system.The experiments provide qualitative guidelines for stable co-optimization.Source: §I · pp. 1–2
- The paper is organized around the market and co-optimization framework, transient-policy instability and replay filtering, replay-memory oscillations, and a concluding section.This organization is described in the introduction.Source: §I · p. 2
II. LEARNING-BASED CO-OPTIMIZATION
The section reviews a framework that learns GenCo bidding policies and transmission-expansion decisions together. It separates market simulation from transmission-capacity updating.
II. LEARNING-BASED CO-OPTIMIZATION
- The reviewed framework jointly learns GenCo strategic bidding policies and transmission-expansion decisions.It is presented as a unified learning process.Source: §II · p. 2
- The framework’s upper component is a multi-agent reinforcement-learning environment for market simulation.The section directs readers to Sec. II-A for its description.Source: §II · p. 2
- The lower component updates transmission capacity, with its process described in Sec. II-B.Source: §II · p. 2
A. Market Simulation Using Multi-agent RL
The market model treats GenCos as learning agents that submit strategic bids into a day-ahead market cleared by DC-OPF. MADDPG trains decentralized bidding actors using centralized critics and replay-buffer experience.
A. Market Simulation Using Multi-agent RL
- Multiple GenCos strategically bid in a day-ahead market cleared by DC-OPF subject to power-balance and transmission-capacity constraints.The market is modeled as a partially observable multi-agent environment.Source: A. Market Simulation Using Multi-agent… · p. 2
- Each GenCo acts autonomously and determines its bid price through reinforcement learning.The transmission-capacity parameters remain fixed during each episode.Source: A. Market Simulation Using Multi-agent… · p. 2
- MADDPG uses decentralized actor networks for bidding and a centralized critic that evaluates agents’ joint actions during training.Experience tuples from market interactions are stored in replay buffers for off-policy actor and critic updates.Source: A. Market Simulation Using Multi-agent… · p. 2
B. Co-optimization of Transmission Capacity
Transmission capacities are learnable design variables sampled at the start of each episode, with a policy-gradient update based on market outcomes. This study reuses accumulated historical samples to estimate that gradient rather than using only recent episodes.
B. Co-optimization of Transmission Capacity
- Each transmission design variable is a line-capacity increment, sampled at the start of each episode from a Gaussian policy with fixed exploration variance.The capacity vector is updated together with market policies.Source: B. Co-optimization of Transmission… · p. 2
- The planning objective minimizes operational and transmission-investment costs, and sampled market outcomes drive a policy-gradient update of the design parameter.Operational cost is accumulated over T time steps; expansion cost is annualized using Wanu.Source: B. Co-optimization of Transmission… · pp. 2–3
- Historical samples are reused to estimate the transmission-capacity gradient, contrasting with the prior approach that used only recent episodes.A moving-average baseline reduces gradient-estimation variance.Source: B. Co-optimization of Transmission… · p. 3
III. REPLAY-BASED STABILIZATION OF CO-OPTIMIZATION
The section uses the IEEE 30-bus market model to demonstrate how replay-based capacity updates address transient policy degradation in bidding agents.
III. REPLAY-BASED STABILIZATION OF CO-OPTIMIZATION
- The IEEE 30-bus market model is used to demonstrate replay-based capacity updates against transient policy degradation of bidding agents.The section evaluates the replay-based stabilization method introduced for the co-optimization framework.Source: §III · p. 3
A. Simulation Settings
The simulation uses the IEEE 30-bus system, with strategic GenCos bidding alongside generators assumed to bid at marginal cost. Capacity optimization targets lines 4–12 and 27–28 over 48 hourly time steps.
A. Simulation Settings
- Line 4–12 and line 27–28 each start at 10 MW and have an expansion cost of $100,000/MW/year.Both lines are targets for capacity optimization in the IEEE 30-bus system.Source: A. Simulation Settings · p. 3
- Gen1, Gen2, and Gen3 are strategic bidders; all other generators bid truthfully at their marginal costs.Strategic agents have a $50/MWh marginal cost and may bid up to twice that amount; other generators have a $55/MWh marginal cost.Source: A. Simulation Settings · p. 3
- Market simulations use hourly resolution over 48 time steps, and Gen1’s bidding-policy learning is examined in relation to line 4–12 capacity updates.Gen1’s profit-maximizing bid remains around $90/MWh after capacity expansion in preliminary simulations.Source: A. Simulation Settings · p. 3
- The IEEE 30-bus diagram depicts the system configuration used for the simulations.Gen1 has significant market power, and increasing line 4–12 capacity is particularly effective in reducing total system cost.Source: Figure 2 · p. 3, A. Simulation Settings · p. 3
B. Instability in the Conventional Approach
Transient degradation of a bidding policy can drive capacity updates away from the desired solution. Low-cost samples from temporarily suboptimal bids distort the observed cost-capacity relationship and bias the capacity gradient.
B. Instability in the Conventional Approach
- In the failure case, temporarily suboptimal low-price bids produce below-trend electricity-cost samples even at insufficient capacity, biasing the estimated capacity gradient.The expected relationship is monotonically decreasing cost as transmission capacity increases.Source: B. Instability in the Conventional… · pp. 3–4
- After about 5,000 episodes, degraded Gen1 bids in the failure trajectory accompany falling line 4–12 capacity and an underestimated expansion level.The successful trajectory maintains a near-optimal bid around $90/MWh and a stable capacity update; the conventional method uses the most recent 50 episodes.Source: B. Instability in the Conventional… · p. 3, Figure 3 · p. 4
- Temporary bidding-agent learning failures can propagate to transmission planning and prevent convergence to the desired solution.The conventional update relies only on recent samples, making it vulnerable to inconsistent data and inappropriate capacity reductions.Source: B. Instability in the Conventional… · pp. 3–4
C. Replay-Based Capacity Update with Sample Filtering
Replay memory with nearest-neighbor filtering excludes inconsistent samples before transmission-capacity updates. IEEE 30-bus results show the update resists transient bidding-policy degradation, with low computational overhead.
C. Replay-Based Capacity Update with Sample Filtering
- The method reuses past co-optimization experiences, filters capacity-space neighbors with a KD-tree by retaining the highest-cost sample, then samples update minibatches from the filtered data.This preferentially removes samples associated with transient policy degradation while preserving samples that better represent the underlying cost–capacity relationship.Source: C. Replay-Based Capacity Update with… · p. 4
- With a 2,000-episode replay memory, k=5 neighbors, and 50-sample minibatches, capacity stayed appropriate through Gen1’s 11,000–13,000-episode bid-price dip (Fig. 5).The filtered samples largely followed the expected cost–capacity relationship, avoiding inappropriate capacity reductions.Source: C. Replay-Based Capacity Update with… · p. 4, Figure 5 · p. 5
- At W=7000, the capacity update’s average computational cost is less than 0.4% of MADDPG learner-update time.Across W=2000–7000, the replay-based update requires 7.2–30.4 ms.Source: C. Replay-Based Capacity Update with… · pp. 4–5
IV. OSCILLATORY DYNAMICS AND STABILITY TRADE-OFF
Replay-based updates can continue oscillating even after capacity underestimation from transient policy degradation is eliminated. Larger replay memories amplify temporal lag and oscillation, while smaller learning rates reduce oscillations and overshoot.
IV. OSCILLATORY DYNAMICS AND STABILITY TRADE-OFF
- Capacity repeatedly overshoots and undershoots around equilibrium even during stable bidding-policy learning, so the oscillation is not attributed to transient policy degradation.The replay-based update eliminates the capacity underestimation caused by transient policy degradation.Source: §IV · p. 5
- Older replay samples can mismatch current operating conditions, creating temporal lag that keeps capacity rising past its economically appropriate level before corrective reductions.Repeated overshoot-and-correction cycles appear as oscillatory dynamics.Source: §IV · p. 5
- Increasing replay-memory size generally amplifies oscillation and overshoot, whereas reducing the learning rate suppresses oscillation across all replay-memory settings (Fig. 6).Smaller learning rates produce smaller oscillations and reduced overshoot by making capacity updates more conservative.Source: Figure 6 · p. 5, §IV · p. 6
- Larger replay memories improve robustness against transient policy degradation but require smaller learning rates for stability, slowing capacity-update convergence.The paper presents this as a trade-off for jointly tuning replay memory and capacity-update dynamics.Source: §IV · p. 6
V. CONCLUSION
The IEEE 30-bus studies show that transient bidding-policy degradation can produce inconsistent cost–capacity samples and prevent convergence, while replay-based filtering improves robustness. The conclusion highlights a replay-memory and learning-rate trade-off and identifies broader transmission-expansion settings as future work.
V. CONCLUSION
- On the IEEE 30-bus system, transient bidding-policy degradation generated inconsistent cost–capacity samples that could prevent convergence, while replay-based filtering improved robustness.The update mechanism used KD-tree-based sample filtering.Source: §V · p. 6
- Larger replay memories improve robustness but require more conservative capacity updates to maintain stable learning.The conclusion identifies this as a trade-off between replay-memory size and learning rate.Source: §V · p. 6
- Future work will extend the framework to discrete investment decisions, multiple candidate projects, and larger-scale electricity-market environments.The stated aim is co-optimizing transmission planning and strategic bidding under more realistic operating conditions.Source: §V · p. 6