Source-linked AI summary
Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning
Daniel J. B. Harrold, Jun Cao, Zhong Fan
TL;DR
Intermittent renewable generation and dynamic prices complicate microgrid control, while multi-agent reinforcement learning also faces coordination and reward-assignment challenges. The paper uses multi-agent deep reinforcement learning to control hybrid storage and energy trading. The reported multi-agent approach outperforms a single global agent, and trading with external microgrids is more profitable than selling only to the utility grid.
Problem
Microgrids must manage intermittent renewable generation and dynamic prices, while multi-agent reinforcement learning presents challenges including non-stationarity, communication, and reward assignment.
Source: §1 · pp. 1–2Method
The paper uses centralized learning with decentralized execution to train multi-agent reinforcement-learning controllers for a hybrid energy storage system and energy trading.
Source: §6 · p. 16Results
Multi-agent approaches with separate agent rewards performed better than a single global agent, and selling energy to external microgrids was more profitable than selling back to the utility grid at a fixed feed-in tariff.
Source: §6 · p. 16Takeaways & Limitations
Trading benefited the primary microgrid through increased revenue and allowed external microgrids to reduce their energy bills.
Source: §6 · p. 16The external microgrids lack their own storage, renewable generation, and aggregator agents, limiting their role in trading.
Source: §5.3 · p. 15
Abstract
from arXiv · showhide
In this paper, multi-agent reinforcement learning is used to control a hybrid energy storage system working collaboratively to reduce the energy costs of a microgrid through maximising the value of renewable energy and trading. The agents must learn to control three different types of energy storage system suited for short, medium, and long-term storage under fluctuating demand, dynamic wholesale energy prices, and unpredictable renewable energy generation. Two case studies are considered: the first looking at how the energy storage systems can better integrate renewable energy generation under dynamic pricing, and the second with how those same agents can be used alongside an aggregator agent to sell energy to self-interested external microgrids looking to reduce their own energy bills. This work found that the centralised learning with decentralised execution of the multi-agent deep deterministic policy gradient and its state-of-the-art variants allowed the multi-agent methods to perform significantly better than the control from a single global agent. It was also found that using separate reward functions in the multi-agent approach performed much better than using a single control agent. Being able to trade with the other microgrids, rather than just selling back to the utility grid, also was found to greatly increase the grid's savings.
1. Introduction
The paper frames microgrid control as a choice between centralized and distributed agents managing hybrid storage under intermittent renewables and dynamic prices. It applies continuous-control deep reinforcement learning to renewable integration and energy trading, addressing gaps in prior multi-agent work.
1. Introduction
- Microgrid control must account for fluctuating renewable generation, volatile energy prices, and multi-agent challenges such as non-stationarity, communication, and reward assignment.The paper investigates whether one global controller or multiple distributed agents should manage the system.Source: §1 · pp. 1–2
1.1. Related Literature
- Prior multi-agent reinforcement-learning work leaves continuous control for microgrid agents and single-agent versus multi-agent comparisons for renewable integration or energy trading unexamined.Earlier HESS studies often used single agents, while some multi-agent approaches used basic discrete-action algorithms.Source: §1.1 · p. 2
1.2. Contributions
- The paper applies DDPG, D3PG, and TD3 and their multi-agent equivalents to continuous control of energy-storage and trading agents.The HESS combines a supercapacitor for short-term, a lithium-ion battery for medium-term, and a vanadium redox battery for long-term storage.Source: §1.2 · pp. 2–3
- The two case studies examine renewable-energy value under dynamic pricing and sales to neighboring microgrids.The paper compares centralized global control with distributed multi-agent control.Source: §1.2 · p. 2
2. Microgrid Description
The paper describes a primary microgrid and models it as a Markov decision process for reinforcement learning, then applies it to renewable integration and neighboring-microgrid energy sales.
2. Microgrid Description
- The two cases use the hybrid energy storage system to maximize renewable-energy utilization and reduce bills, then add an aggregator selling energy to neighboring microgrids competing to lower their costs.The environment description frames the microgrid as a Markov decision process for reinforcement learning.Source: §2 · p. 3
2.1. Environment Description
The simulated primary microgrid combines renewable generation, hybrid storage, and connections to the utility grid and neighboring microgrids. Its operation includes dynamic pricing with fixed-price bounds, storage characteristics, and modeled conversion efficiencies.
2.1. Environment Description
- The primary microgrid connects PV and wind generation, a hybrid energy storage system, and five external microgrids to the utility grid through AC/DC lines, inverters, and transformers.The external microgrids connect to the utility grid but not to the wholesale market.Source: §2.1 · p. 3
2.1.1. Demand and Pricing
- The simulation uses dynamic market prices capped at £144/MWh for purchases and assumes a fixed £16/MWh feed-in tariff for energy sold to the grid.The fixed tariff is selected to represent specified UK renewable tariffs.Source: §2.1.1 · pp. 3–4
2.1.2. Renewable Energy Sources
- Wind output is modeled from wind speed for two 1 MW turbines, while solar output uses hourly radiation scaled to a 5 MW solar farm.Wind generation is AC and PV generation is DC.Source: §2.1.2 · p. 4
2.1.3. ESS
- The HESS comprises a lithium-ion battery, vanadium redox battery, and supercapacitor, with distinct efficiency, self-discharge, lifecycle, and cost properties shaping their storage roles.The paper describes the battery as suited to medium-term storage, the supercapacitor to short-term storage, and the vanadium redox battery to medium- or long-term storage; cycling cost is incorporated into the reward.Source: §2.1.3 · p. 4
2.1.4. Transformers and Inverters
- Transformer and inverter efficiencies vary nonlinearly with load factor and are modeled by polynomial estimates, with minimum efficiency capped at 10%.Figure 2 presents the inverter and transformer efficiency profiles.Source: §2.1.4 · p. 4, Figure 2 · p. 5
2.2. Microgrid Aggregator (MGA)
In the trading case, the microgrid aggregator chooses how much energy to offer and a reserve price. It allocates available energy to qualifying external-microgrid bids from highest to lowest, with unsold energy returned to the utility grid.
2.2. Microgrid Aggregator (MGA)
- The aggregator agent selects the next-step sale volume and a reserve price between Pmin and Pmax for external microgrids to bid against.The reserve price gives the aggregator some control over the external-microgrid market.Source: §2.2 · p. 5
- The aggregator sells available energy to the highest bidder up to its requested volume when the bid exceeds the reserve price, then proceeds to lower bids; unsold energy goes to the utility grid at Pmin.The process continues until the available sale energy is allocated or no qualifying bids remain.Source: §2.2 · p. 5
2.3. External Microgrids (xMG)
The external microgrids (xMGs) have their own demand but no storage or renewable generation. They bid to the aggregator for energy and use the utility grid to cover or dispose of any mismatch.
2.3. External Microgrids (xMG)
- Each xMG has demand equal to one-twentieth of the primary microgrid’s demand plus a small noise term.The noise term is N(0, 0.01).Source: §2.3 · p. 5
- The xMGs have their own demand but no energy storage system or renewable energy sources.Source: §2.3 · p. 5
- An xMG bids for an energy volume and price, with its unmet demand bought from the utility grid at Pmax.Energy received beyond its demand is returned to the utility grid at Pmin.Source: §2.3 · p. 5
2.4. Markov Decision Process
The control problem is represented as a Markov decision process with states, actions, and rewards. Agents observe energy-system conditions, control storage and trading decisions, and receive rewards tied to grid savings and operating constraints.
2.4. Markov Decision Process
- The control process is modelled as a Markov decision process defined by state and action spaces, a reward function, and state transitions.Source: §2.4 · p. 5
2.4.1. States
- ESS agents observe storage charge, demand, wholesale price, renewable generation, time, and forecasts; in the trading case, agents also observe aggregator terms and xMGs observe their own demand.Microgrid demands, primary-microgrid ESS charge, and renewable generation are not shared between agents for privacy.Source: §2.4.1 · p. 5
2.4.2. Actions
- ESS agents choose charge or discharge power, while the aggregator sets sell volume and reserve price and each xMG selects its buying volume and bid price.The global controller outputs three actions in the first case study and five in the trading case; decentralized agents choose actions for their own roles.Source: §2.4.2 · pp. 5–6
2.4.3. Rewards
- Rewards also account for ESS operating cost, self-discharge, and penalties when an action would push charge beyond capacity limits.The reward terms are combined and scaled by 0.01 and the number of agents to keep reward magnitude typically around -1 to 1 and improve learning stability.Source: §2.4.3 · p. 6
- Raw energy cost is normalized against an idle baseline, while multi-agent rewards can use each agent’s marginal contribution to grid savings.A shared grid-savings reward may cause credit-assignment problems, while individual-savings rewards can ignore other ESS policies; the marginal-contribution approach uses an individual-idle baseline.Source: §2.4.3 · p. 6
3. Reinforcement Learning Methodology
The paper introduces value-based, policy-gradient, and actor-critic reinforcement learning, then applies DDPG and its D3PG and TD3 variants to multi-agent energy control. The multi-agent approach uses centralized learning with decentralized execution to address challenges of learning when agents affect one another.
3.1. Fundamentals
- Reinforcement learning learns a policy through state observations, actions, rewards, and transitions, with the objective of maximizing discounted future reward.The paper distinguishes value-function, policy-gradient, and actor-critic methods, and uses DDPG and its variants alongside a value-function benchmark.Source: §3.1 · pp. 6–7, §3.2 · p. 7
3.2.1. Deep Deterministic Policy Gradient (DDPG)
- DDPG uses an actor to select actions and a critic to evaluate performance, with each agent’s actor observing its own relevant state and critic also receiving all agents’ actions.Source: §3.2.1 · p. 7
- DDPG explores by injecting learned parameter noise, trains off-policy from replayed transitions, and uses target actor and critic networks to stabilize learning.The critic is updated by gradient descent on mean-squared error, while the actor is updated by deterministic policy gradient ascent.Source: §3.2.1 · p. 7
3.2.2. Distributional DDPG (D3PG)
- D3PG addresses over-generalization under nondeterministic rewards by estimating a return-value distribution instead of only an expectation.It projects probability masses onto equally spaced return supports, whose masses sum to Q(s, a), and uses KL divergence between predicted and projected target distributions as the loss.Source: §3.2.2 · p. 8
3.2.3. Twin Delayed DDPG (TD3)
- TD3 addresses DDPG value overestimation with two critic and target-critic pairs, using the minimum target estimate and updating actor and target networks less often.Source: §3.2.3 · pp. 8–9
3.3. Multi-Agent Reinforcement Learning
- In multi-agent settings, changing behavior by other agents makes the environment appear non-stationary and can complicate learning or shared-reward credit assignment.Source: §3.3 · p. 9
- MADDPG uses centralized learning with decentralized execution: agents act from their own observations but evaluate performance using critics that observe all agents’ actions.Each agent has its own critic and reward function, supporting competitive, collaborative, or mixed settings.Source: §3.3 · p. 9
4. Case Studies
The case studies evaluate energy storage control and xMG trading using multi-year hourly simulations, forecasts, and algorithmic benchmarks. The benchmark set includes multi-agent DQN variants and a rule-based storage controller, with different comparisons used for the trading case.
4.1. Simulation Setup
- The simulations use four years of hourly demand, price, and weather data, with weekly episodes and 100 training episodes followed by 100 evaluation episodes during continued online learning.Episodes contain 168 time-steps; forecasts for demand, price, and renewable generation are supplied as state observations.Source: §4.1 · p. 9
4.2. Benchmarks
- The HESS case study uses two alternative multi-agent RL algorithms and a model-based benchmark, while nonlinear pricing and energy conditions rule out deterministic optimization methods such as linear programming.Source: §4.2 · p. 9
- For xMG trading, benchmarks are limited to DDPG algorithms because competition precludes model-based methods and alternative multi-agent RL methods use discrete action spaces.Source: §4.2 · p. 9
4.2.1. Multi-Agent Deep Q-Networks (MADQN)
- MADQN agents act in succession, passing each action to the next agent as an observation rather than using centralized learning.MADQN selects among five actions per agent, while MARainbow selects among nine.Source: §4.2.1 · p. 10
4.2.2. Rule-Based Model (RBM)
- The rule-based model charges when renewable generation exceeds demand and discharges when demand exceeds generation, prioritizing the supercapacitor, then LIB, then VRB.Its rule targets renewable utilization but does not account for dynamic wholesale energy prices.Source: §4.2.2 · p. 10
5. Results and Discussion
The section compares algorithm performance and agent behavior across renewable-energy integration and external-microgrid trading, then identifies improvements to the modeled system.
5.1. Case Study 1: HESS
- In Case Study 1, the agents achieved significant energy-cost savings over two years, even with a low fixed selling tariff, by maximizing the value of generated renewable energy.Figure 6 presents smoothed episodic rewards and adjusted savings over time.Source: §5.1 · p. 10, Figure 6 · p. 11
5.1.1. Algorithm Performance
- Marginal-contribution rewards improved all algorithms, while shared-reward results were less consistent; MADQN was limited by its discrete action space and MARainbow lagged the DDPG-based multi-agent methods.The reported results attribute the multi-agent benefit in this setting to reward-function design rather than distributed control alone.Source: §5.1.1 · p. 11, Table 3 · p. 11
- MAD3PG with marginal-contribution rewards performed 55.4% better than DDPG and outperformed single-agent controllers and shared-reward multi-agent methods.The results also report lower ESS loss and higher savings before ESS operating costs for the marginal-contribution multi-agent approaches.Source: Table 2 · p. 11, §5.1.1 · p. 11
5.1.2. Agent Behaviour
- Agents generally stored peak-noon PV for later high-price periods, favored the LIB, and showed clearer ESS-specific behavior under multi-agent control; cycling costs discouraged maximum-power operation.MAD3PG used the supercapacitor most aggressively, while some agents charged at low morning prices even when renewable generation did not exceed demand.Source: §5.1.2 · pp. 12–13
5.2. Case Study 2: xMG Trading
- Selling energy to external microgrids on the primary grid’s terms greatly increased savings over renewable-utilization improvements alone, while the multi-agent advantage over single-agent control narrowed.Figure 8 plots rewards and savings for Case Study 2.Source: §5.2 · p. 13, Figure 8 · p. 13
5.2.1. Algorithm Performance
- MATD3 performed best, improving 27.00% over DDPG; D3PG made 98.49% of its savings through trading, compared with 77.78% for MATD3.MATD3 with shared rewards still improved over the single-agent methods, and TD3 also outperformed MADDPG.Source: §5.2.1 · pp. 13–14, Table 5 · p. 13
5.2.2. Agent Behaviour
- In trading, the SC generally served short-term storage and the LIB medium-term storage, while some single-agent methods neglected ESSs; after about 100 episodes, agents reached an equilibrium in which the MGA typically restricted sales to make xMGs compete.Separate ESS rewards supported policies suited to each storage type and helped prevent fixation on high aggregator rewards; trading also generated revenue for the primary grid, reduced xMG bills, and minimized utility-grid load.Source: §5.2.2 · pp. 14–15
5.3. Improvements and Further Work
- Proposed improvements include giving xMGs their own storage, renewable generation, and aggregators; modeling day-ahead imports and variable feed-in tariffs; providing longer forecasts; and exploring other communication methods and agent types.Longer forecasts are proposed to help exploit VRB long-term storage, which agents often avoid because of its poor cycling efficiency.Source: §5.3 · pp. 15–16
6. Conclusion
The paper applies multi-agent reinforcement learning to control a microgrid hybrid energy storage system and trade energy with external microgrids. Its multi-agent approaches outperform a single global agent, while aggregator-mediated sales are more profitable than selling energy back to the utility grid.
6. Conclusion
- The paper applies multi-agent reinforcement learning to control a hybrid energy storage system for renewable energy use, lower bills, and trading with external microgrids.It explores centralized learning with decentralized execution using MADDPG and the D3PG and TD3 variants.Source: §6 · p. 16
- Multi-agent approaches with a separate reward function for each agent performed better than a single global agent controlling the network.Separate rewards helped agents evaluate their individual contributions to the shared goal.Source: §6 · p. 16
- Selling energy to external microgrids through an aggregator was more profitable than maximizing renewable energy use or selling to the utility at a fixed feed-in tariff.Trading increased primary-microgrid revenue, helped external microgrids reduce bills, and reduced load on the main utility grid.Source: §6 · p. 16