PaperlayerView exact revision on arXiv

Source-linked AI summary

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Zhanhua Pan, Xin Qin, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu

arXiv:2609.37321v1cs.MAcs.AIcs.SYeess.SY

TL;DR

Power-market MARL benchmarks often focus on one market, simplify clearing rules, or rely on CPU solvers, limiting systematic study across diverse settings. PowerMarketJax provides a five-market JAX benchmark that preserves market-specific rules and combines GPU market simulation with policy training. Across the markets, learned bidding behavior depends strongly on market design; the implementation achieves up to 33× higher throughput than CPU-based implementations.

  • Problem

    Existing power-market MARL environments often focus on one market, rely on CPU clearing, or simplify market rules, limiting systematic study across diverse settings.

    Source: §1 · p. 2

  • Method

    PowerMarketJax provides five MARL power-market environments with distinct market rules and implements market clearing and policy computation in JAX.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Across five markets, learned bidding behavior depends strongly on market design, and PowerMarketJax achieves up to 33× higher throughput than CPU-based implementations.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Market mechanisms shape the learning landscape by setting agent incentives and feeding learners’ actions back into prices, dispatch, and procurement.

    Source: §6 · p. 9

    The benchmark is not deployment-ready, simplifies real market operations, and its experiments do not establish real participant behavior or equilibrium strategies.

    Source: §6 · p. 9

Abstract

from arXiv · show

Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1,024 X 1,200 parallelisms across both environments and market participants, achieving up to 33X speedup over CPU-based baselines. Our open-source benchmark is available at: https://github.com/powermarketjax/PowerMarketJax.

1 INTRODUCTION

Power-market bidding is increasingly difficult to understand as participants, products, and market mechanisms diversify. PowerMarketJax addresses gaps in existing MARL environments with a five-market benchmark and GPU-based JAX implementation, and its experiments show that market design shapes what independent learners discover.

1 INTRODUCTION

  • Existing environments often cover one market, rely on costly CPU clearing, or simplify pricing and settlement rules, limiting systematic MARL study.The simplifications can remove market features that shape bidding incentives.Source: §1 · p. 2
  • 33× higher throughput than CPU-based implementations is achieved by compiling JAX market clearing and policy computation into a GPU computation graph.The benchmark runs 1,024 parallel 1,200-participant market environments at over 27,000 environment steps per second.Source: §1 · p. 2
  • PowerMarketJax unifies five markets while preserving each market’s clearing, pricing, and settlement rules under a common interface.The markets are day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer energy trading, and local flexibility.Source: §1 · p. 2
  • Market design shapes what independent learners discover: profitable outcomes may require coordinated changes, lie beyond lower-profit regions, or disappear as adoption increases.The participants’ profits depend on their own bids and others’ actions through market clearing.Source: §1 · pp. 1–2

2 RELATED WORK

Existing power-market RL environments demonstrate learning under varied mechanisms and physical constraints, but typically address a single market. PowerMarketJax spans five markets and combines JAX-based clearing with policy computation to support systematic study while retaining market rules and constraints.

2 RELATED WORK

  • Existing power-market RL environments study settings including network-constrained real-time markets, European markets, distribution networks, and local energy markets.These works demonstrate the potential of (MA)RL for bidding under different mechanisms and physical constraints.Source: §2 · p. 2
  • PowerMarketJax covers five representative markets across transmission and distribution levels, preserving their clearing, pricing, settlement, and physical constraints.It also provides a unified framework for agent–market interaction.Source: §2 · p. 2
  • PowerMarketJax runs market clearing and agent policy computation in JAX, avoiding the repeated CPU–GPU transfers common to separated simulation and learning.Existing environments commonly separate CPU-based market simulation from GPU-based policy learning, making market clearing a training bottleneck.Source: §2 · p. 2
  • Unlike rule-based order-matching environments such as EconoJax and JAX-LOB, PowerMarketJax targets optimization-based clearing constrained by power-grid engineering requirements.Power markets couple strategic economic decisions to a physical grid.Source: §2 · p. 3

3 BACKGROUND

The benchmark represents five energy, reserve, and flexibility markets that operate across timescales and grid levels. Participants submit bids under market-specific products, constraints, and clearing and settlement rules.

3 BACKGROUND

  • The five representative settings span day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer trading, and local flexibility.They cover scheduling against forecasts, adjustment to imbalances, reserve procurement, local energy exchange, and distribution support.Source: §3 · p. 3, Figure 1 · p. 3
  • Day-ahead wholesale clears forecasts before delivery, while real-time balancing redispatches resources as actual demand and renewable output deviate from forecasts.The day-ahead schedule establishes the baseline for real-time balancing; real-time deviations settle at potentially more volatile prices.Source: §3 · p. 3
  • Ancillary services procure reserves and rapid-response capabilities because energy scheduling alone cannot handle disturbances occurring faster than dispatch intervals.Providers are paid for reserved capacity, so revenue can depend on maintaining availability rather than energy production alone.Source: §3 · p. 3
  • Peer-to-peer markets match local buy bids and sell offers, with unmatched demand and supply settled through the grid at retail and feed-in prices.This difference in residual transaction prices creates an incentive for local trading.Source: §3 · pp. 3–4
  • Local flexibility procurement pays resources to address distribution congestion or voltage limits, with accepted providers paid their submitted bids.The DSO may request changes in generation or consumption at specified locations, times, and durations.Source: §3 · p. 4

4 POWERMARKETJAX: POWER MARKET ENVIRONMENTS IN JAX

PowerMarketJax models five distinct power markets through a common MARL interface while preserving their market-specific rules. Its accelerator-based execution compiles market clearing and policy learning together, with fixed-shape operations enabling GPU execution.

4 POWERMARKETJAX: POWER MARKET ENVIRONMENTS IN JAX

  • Five JAX MARL environments preserve market-specific clearing, pricing, settlement, and physical constraints behind a common agent–market interface.The suite covers day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer energy trading, and local flexibility.Source: §4 · p. 4, Figure 2 · p. 4

4.1 FIVE MARKET ENVIRONMENTS AS POSGS

  • Each market is formulated as a general-sum partially observable stochastic game, with its step, observations, actions, clearing, reward, and state transition specified.The reward is tied to each participant’s profit under the corresponding settlement rule.Source: §4.1 · p. 4, §4 · p. 4
  • Day-ahead agents bid across segments and delivery periods for joint 24-hour clearing; real-time agents bid each interval and are rewarded under two-settlement accounting.Real-time clearing redispatches committed generators subject to network and ramping constraints.Source: §4.1 · pp. 4–5
  • Ancillary-services clearing co-optimizes energy dispatch and reserve procurement, with rewards including energy settlements and reserve capacity payments.The clearing respects network, ramping, response-time, and reserve-capacity constraints.Source: §4.1 · p. 5
  • Peer-to-peer agents submit price bids or asks for fixed trading quantities, while local-flexibility aggregators submit price–quantity offers to the DSO.The P2P market matches bids and settles residual grid transactions; local flexibility uses pay-as-bid settlement and may partially accept offers.Source: §4.1 · p. 5

4.2 CLEARING AND LEARNING IN A SINGLE COMPUTATION GRAPH

  • Market simulation and policy learning run within a single compiled computation graph, with reset and step implemented as pure functions.The graph uses jit, vmap, and lax.scan for compiled steps, parallel environments, rollout horizons, and policy updates.Source: §4.2 · p. 5
  • Fixed-shape clearing uses a primal–dual interior-point solver for linear programs; day-ahead commitment uses relaxation, rounding, and continuous re-solving.P2P sorting, matching, and settlement use JAX primitives without an optimization solver.Source: §4.2 · p. 6
  • Clearing is validated against a NumPy reference, HiGHS dual prices, and exact mixed-integer HiGHS solutions; agents and environments are vectorized for GPU execution.Within each environment, agents remain coupled through market clearing even as policy evaluation is batched across agents and environments.Source: §4.2 · p. 6

5 RESULTS

The experiments evaluate four independent IPPO/SAC learners, with and without parameter sharing, across five power markets using held-out data. Outcomes vary by market, including gains from coordinated bidding, limits to unilateral learning, and strategy profitability that changes as adoption grows.

5 RESULTS

  • The evaluation compares four independent IPPO/SAC learners, with parameter sharing or agent-specific parameters, against truthful bidding and market-specific strategies on held-out data.The learners use each agent’s own available information and maximize its own market profit, without a centralized critic or inter-agent communication.Source: §5 · p. 6

5.1 DAY-AHEAD WHOLESALE MARKET

  • On M1, IPPO-PS and IPPO-NoPS raise α_i to 1.50 and 1.56, increasing total generator profits by £643 and £917 million over 36 evaluation days.Independent learners can miss the joint gain because individual signals do not reveal it and exploration rarely reaches the higher-bid region.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 REAL-TIME BALANCING AND ANCILLARY SERVICES MARKETS

  • In M3, joint reserve bids at £150/MWh raise total profit by £9.05 million per day, while learned policies earn only £0.56–£3.69 million per day above truthful bidding.In M2, raising a bid alone is rarely profitable, and learned bids move toward cost; in M3, a single-generator increase yields at most £0.08 million per day.Source: §5.2 · pp. 7–8, Figure 4 · p. 7

5.3 PEER-TO-PEER ENERGY MARKET

  • A fixed arbitrage schedule earns C1.13 per household daily alone, but becomes unprofitable around 210–240 adopters as market prices erode the arbitrage spread.IPPO-NoPS leads only a subset of households to arbitrage, while IPPO-PS leads none to do so.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 LOCAL FLEXIBILITY MARKET

  • A single aggregator can earn more by raising its offer, but small increases reduce profit before the more profitable region around 3.2 times the floor.Gradient-based learners remain near the floor; the softplus price mapping further weakens exploration there.Source: §5.4 · p. 8, Figure 6 · p. 9
  • Under SAC-PS, procurement periods rise from 5.0% to 18.7% with 24 aggregators and from 4.9% to 15.4% with 34 aggregators.A baseline monitor that excludes learner-induced charging largely removes the effect, sharply reducing procurement and learner returns.Source: §5.4 · p. 9, Figure 6 · p. 9

6 CONCLUSION

The section presents PowerMarketJax’s formulations for real-time bidding, ancillary services, peer-to-peer trading, and local flexibility, alongside findings on how market design shapes learning and GPU execution increases throughput. It frames the benchmark as a research tool rather than a deployment-ready simulator or predictor of real bidding behavior.

6 CONCLUSION

  • PowerMarketJax integrates clearing, settlement, policy evaluation, and learning in a GPU-accelerated JAX pipeline while retaining market mechanisms and physical constraints.Its five markets share a POSG framework for evaluating bidding behavior and market outcomes across market designs.Source: §6 · p. 9
  • Across five markets, learners can miss profitable joint outcomes, abandon strategies when individual deviations lose money, lose arbitrage gains as strategies spread, or fail to cross lower-profit regions.Market mechanisms shape incentives and feed collective actions back into prices, dispatch, and procurement, creating challenges in coordinated exploration and endogenous responses.Source: §6 · p. 9
  • PowerMarketJax is a research benchmark, not a deployment-ready simulator or bidding system, and its simplified environments and limited experiments do not predict real behavior or establish equilibrium strategies.Experiments cover only the considered systems and independent PPO and SAC variants; future work could broaden market designs, participant models, uncertainty, and MARL algorithms.Source: §6 · p. 9

A.1.1 MARKET MODEL

  • The day-ahead market jointly clears all hourly delivery periods through a security-constrained unit commitment, using relax-round-resolve and a final dispatch solve to obtain schedules and prices.Commitment is first relaxed and rounded; dispatch is then re-solved with commitment fixed, and LMPs come from the final solve's dual variables.Source: §A.1 · p. 13, §A.1.1 · p. 14

A.1.2 PRICING AND SETTLEMENT

  • Day-ahead generators settle cleared quantities at their bus LMP, while true variable costs determine evaluated profit but do not enter clearing; no uplift means committed units can earn negative profit.LMPs include energy and congestion components, with a load-curtailment term possible when its upper bound binds; transmission losses are not modeled.Source: §A.1.2 · p. 15

A.1.3 PARTIALLY OBSERVABLE STOCHASTIC GAME

  • Day-ahead bidding is a partially observable general-sum POSG: generators choose markups without seeing rivals' private costs or offers, and each receives its realized market profit as reward.The resulting payoff depends on joint bids and market outcomes, while commitment and dispatch carry into subsequent market steps.Source: §A.1.3 · pp. 15–16

A.2.1 MARKET MODEL

  • Real-time balancing fixes day-ahead commitment and redispatches resources against realized demand in 30-minute intervals, enforcing nodal balance, transmission limits, and inter-interval ramping.Under two-settlement, day-ahead schedules settle at day-ahead LMPs and only deviations settle at real-time LMPs; a 24-hour day has 48 sequential clearing steps.Source: §A.2 · p. 17, §A.2.1 · p. 17, §A.2.2 · p. 18

A.2.3 PARTIALLY OBSERVABLE STOCHASTIC GAME

  • Generators observe their own and public conditions but not competitors’ private information or bids, submit real-time markup offers, and earn policy-dependent profits through repeated interaction.Each generator seeks to maximize expected discounted profit over the real-time horizon.Source: §A.2.3 · pp. 19–20

A.3 M3: ANCILLARY SERVICES MARKET

  • The ancillary-services market procures reserve availability, which may never be activated, and can co-optimize reserve with real-time energy under shared operating constraints.The section presents its clearing model, pricing and settlement rules, and corresponding partially observable stochastic game.Source: §A.3 · p. 20

A.3.1 MARKET MODEL

  • The market has fast 10-minute and slow 30-minute reserve products, and shared generator capacity creates a trade-off between energy output and reserve awards.A longer response time allows a generator to provide more reserve for a given ramp rate.Source: §A.3.1 · pp. 20–21

A.3.2 PRICING AND SETTLEMENT

  • Reserve prices reflect both the submitted reserve offer and the opportunity cost of capacity withheld from energy, while reserve capacity is paid at the product clearing price.No additional generation cost is assigned to reserve availability unless it is activated.Source: §A.3.2 · pp. 21–22

A.3.3 PARTIALLY OBSERVABLE STOCHASTIC GAME

  • Generators jointly offer energy and reserve while observing public conditions but not competitors’ private costs or current offers, and learn how to allocate capacity through repeated interactions.Energy dispatch, reserve awards, and both energy and reserve prices depend jointly on submitted offers.Source: §A.3.3 · pp. 23–24

A.4 M4: PEER-TO-PEER LOCAL ENERGY MARKET

  • The peer-to-peer market lets end-use prosumers trade locally through periodic double auctions, settling matched quantities at a common price and routing unmatched deficits or surpluses to the grid.Participants’ local net positions determine whether they offer to sell or submit demand.Source: §A.4 · p. 24

A.4.1 MARKET MODEL

  • Before each auction, a participant’s net energy position determines whether it is a seller or buyer; the battery is the only local device with a physical state carried between intervals.The battery state of charge evolves between trading intervals.Source: §A.4.1 · pp. 24–25
  • The P2P double auction matches lowest-priced seller asks with highest-priced buyer bids while bids meet or exceed asks, assigning awards in sorted order.The common clearing price is the midpoint of the admissible price interval; unmatched quantities settle with the upstream grid.Source: §A.4.1 · pp. 25–26, §A.4.2 · p. 26

A.4.2 PRICING AND SETTLEMENT

  • P2P participants receive or pay a common local price, with unmatched energy settled at grid prices; matched buyer and seller payments are budget balanced within the local market.The settlement excludes upstream-grid transactions and any network charges not modeled.Source: §A.4.2 · p. 26

A.4.3 PARTIALLY OBSERVABLE STOCHASTIC GAME

  • The P2P market is a partially observable stochastic game: prosumers choose battery power and price, but cannot observe others’ contemporaneous bids or private states.Each participant’s return depends on all participants’ policies because joint bids and asks determine matching, cleared volume, and price.Source: §A.4.3 · pp. 27–28

A.5 M5: LOCAL FLEXIBILITY MARKET

  • The local flexibility market addresses distribution-network congestion and voltage violations by procuring temporary changes in power injection or consumption from connected resources.Its product is deviation from a predefined baseline, and accepted aggregators are paid their own submitted offer prices rather than a common clearing price.Source: §A.5 · p. 28

A.5.1 MARKET MODEL

  • The DSO clears single price–quantity offers by minimizing procurement cost subject to distribution-network operating limits, with emergency curtailment as a feasibility backstop.The radial-network clearing formulation is a linear program and may partially accept an offer.Source: §A.5.1 · pp. 29–30

A.5.2 PAYMENT AND SETTLEMENT

  • Under pay-as-bid settlement, accepted aggregators earn their submitted prices, while the value of flexibility depends on its location and ability to relieve binding network constraints.The curtailment backstop creates an implicit upper value tied to the value of lost load; injections downstream of a constrained line can relieve its flow, while other locations may not.Source: §A.5.2 · p. 30

A.5.3 PARTIALLY OBSERVABLE STOCHASTIC GAME

  • Local flexibility is modeled as a partially observable game in which aggregators submit price–quantity offers and planned charging for network-constrained DSO clearing.Each aggregator observes its own operating state and public network needs, but not rivals’ private states or contemporaneous offers; delivering flexibility can reduce battery energy available in later intervals.Source: §A.5.3 · pp. 31–32

B COMPUTATION PERFORMANCE

  • The GPU benchmark compares three execution layouts across five markets, measuring environment steps per second with market clearing and one policy forward pass per step.Measurements exclude compilation time and average three runs.Source: B COMPUTATION PERFORMANCE · p. 33

B.1 THREE LAYOUTS ON THE PEER-TO-PEER MARKET

  • In the 1,200-household peer-to-peer market, all-on-GPU execution reaches 27 929 environment steps per second—22 times host-CPU and 33 times hybrid-layout throughput.Throughput remains unchanged when parallel environments increase from 64 to 1,024 or when host-thread allocation changes.Source: Figure 7 · p. 33, §B.1 · p. 33

B.2 ACROSS THE FIVE MARKETS

  • Across all five markets, GPU execution delivers 2.3 to 22 times host-CPU throughput and 2.3 to 33 times hybrid-layout throughput; scaling from 1 to 128 environments multiplies throughput by 2.9 to 12.The throughput comparison uses 64 parallel environments; the scaling comparison measures all-on-GPU execution.Source: §B.2 · p. 33, B COMPUTATION PERFORMANCE · p. 33, Figure 8 · p. 34

B.3 AGAINST OPEN-SOURCE TOOLS

  • For real-time balancing, all-on-GPU throughput with 64 parallel environments is 71 times PyPSA’s and 8.8 times cvxpy’s.For the same number of training environment steps, GPU PowerMarketJax trains PPO 3.8 times faster than Stable-Baselines3 with default settings.Source: Figure 9 · p. 34, §B.3 · p. 34

B.4 IMPLEMENTATION VALIDATION

The section validates market clearing against reference implementations and an independent solver, then measures the day-ahead procedure’s gap from exact commitment solutions. It also documents how to install, run, configure, and use the five-market interface.

B.4 IMPLEMENTATION VALIDATION

  • Clearing and settlement are compared with NumPy references, JAX linear-program dual prices with HiGHS, and three-stage M1 clearing with exact HiGHS solutions.The exact comparison uses binary commitment decisions on 36 evaluation days per test system under truthful offers.Source: §B.4 · pp. 34–35
  • Median excess production costs were 2.04% on case29gb, 0 on case73rts, and 1.10% on case813nem; maximum gaps were 3.78%, 1.79%, and 1.95%, respectively.On 23 of 36 case73rts evaluation days, the three-stage commitment matched the exact optimum.Source: §B.4 · p. 35, Figure 10 · p. 36

C QUICKSTART AND API

  • The quickstart section covers installation, a minimal rollout, the shared interface, market construction, configuration, and command-line experiments.Its scope is the software interface; market models are described in Appendix A.Source: C QUICKSTART AND API · p. 37

C.1 INSTALLATION

  • The package supports Python 3.10–3.12; CUDA 12 is opt-in and Linux-only, while Windows GPU use runs through WSL2.The reported results and test runs were obtained on Linux.Source: §C.1 · p. 37

C.2 MINIMAL ROLLOUT

  • A minimal day-ahead rollout clears three consecutive market days, with each transition, including clearing, inside jit.The example uses case29gb and bids at twice cost.Source: §C.2 · p. 37

C.3 THE ENVIRONMENT INTERFACE

  • The shared interface returns agent-aligned observations, rewards, costs, and termination status; constraint costs remain separate from profit rewards.Four markets use truncation boundaries, while P2P is terminal because its final reward settles remaining battery energy.Source: §C.3 · p. 39

C.4 CONSTRUCTING THE FIVE MARKETS

  • Market constructors take scenario-specific inputs, and changing clearing-problem size requires rebuilding and recompiling the environment.Parameters such as demand series and episode length are passed at each call.Source: §C.4 · pp. 39–40

C.5 SCENARIO AND LEARNER CONFIGURATION

  • Experiments use Python configuration and market-specific scripts for learner settings, training, evaluation, and non-learning baselines.Learning scripts train on training days and evaluate the policy’s mean action on held-out days, excluding exploration noise.Source: §C.5 · p. 40, §C.6 · p. 40

C.7 CASES AND DATA

The implementation batches market rollout and policy learning into one compiled GPU iteration, while the training setup uses four independent learners with shared hyperparameters across markets.

C.7 CASES AND DATA

  • The repository includes built-in network cases and parquet demand, generation, price, and smart-meter series, so listed examples run without a separate data download.The case registry contains 18 network cases: 10 transmission and 8 distribution.Source: §C.7 · p. 41

C.8 CLEARING AND LEARNING IN A SINGLE COMPUTATION GRAPH

  • One compiled GPU iteration rolls out N environments for T steps, then updates on their N × T experiences using epoch and minibatch scans.The policy acts inside the same function as market clearing; SAC also keeps and samples a fixed-size replay buffer inside that function.Source: §C.8 · pp. 41, 43, Figure 11 · p. 42
  • vmap batches clearing, settlement, observation construction, and policy inference across environments instead of making sequential environment calls.lax.scan repeats the rollout for a fixed horizon, resetting ended episodes in place so each step retains the same shape.Source: §C.8 · pp. 41–42

D.1 FOUR LEARNERS

  • The suite provides IPPO and SAC with parameter sharing or agent-specific parameters; all four are independent learners optimizing individual profit without a centralized value network.Under parameter sharing, agents share networks and pool samples; without it, each agent has its own networks and samples.Source: §D.1 · p. 44

D.2 HYPERPARAMETERS

  • The five markets use one untuned set of hyperparameters based on CleanRL defaults, with SAC reward scaling and fixed observation-normalization constants.SAC divides rewards by their truthful-offer standard deviation before Q regression; M4 SAC and M5 IPPO have specified market-specific differences.Source: §D.2 · p. 44, Table 3 · p. 44

D.3 TRAINING SCALE

The section specifies evaluation and the three day-ahead test systems, including their data windows and load conditions. It also explains why unilateral nuclear markups may not move prices in the British system.

D.3 TRAINING SCALE

  • M5 evaluates checkpoints every 10 iterations and stops when the last five validation returns exceed the preceding five by 0 to 5%.M1–M4 evaluate at the final iteration; M5 selects the checkpoint with the highest validation return.Source: §D.3 · p. 45, §D.4 · p. 45

D.4 EVALUATION PROTOCOL

  • M1–M3 and M4 evaluate mean actions on held-out days or episodes, while M5 evaluates sampled actions on test days; training returns include exploration noise.The untrained network is evaluated using the same method, and truthful offers provide the reference policy.Source: §D.4 · p. 45

E.1.1 MARKET SETTING

  • Each test system uses one year of data, with 36 evaluation days; three example days represent the 10th, 50th, and 90th percentiles of daily demand.The British system has 66 units, with nuclear units among the cheapest; demand exceeds the combined capacity of the 16 cheapest units in almost every hour.Source: §E.1.1 · p. 46, Table 5 · p. 46, Figure 12 · p. 47
  • In case29gb, generation is concentrated north and load south, and congestion occurs on central and southern corridors under truthful offers.Because demand is usually beyond the capacity of the cheapest units, one nuclear unit’s unilateral markup does not move price, whereas a joint markup raises it.Source: §E.1.1 · p. 46, Figure 12 · p. 47
  • The RTS-GMLC examples have zero, two, and seven congested lines, while the Australian examples have three, four, and four.On case73rts, net demand reaches a 2,500 MW floor in more than half the hours; case813nem’s 11,500 MW floor is reached only a few hours.Source: §E.1.1 · p. 47

E.1.2 AGENT TRAINING

On the British test system, three learners finish near their highest training returns while IPPO-PS falls back. On the other two systems, both trained IPPO variants’ returns decline as price-setting units move closer to cost.

E.1.2 AGENT TRAINING

  • IPPO-NoPS, SAC-PS, and SAC-NoPS finish near their highest British-system training returns, while IPPO-PS stays lower and rises before falling.IPPO-PS’s sampled markup rises to about 1.7 and then returns to about 1.5.Source: §E.1.2 · p. 47
  • On the other two test systems, both IPPO variants’ training returns fall as price-setting units move closer to cost and high-offer units lose output to rivals.The account attributes the decline to initial joint markups giving way to competition, which lowers prices without lowering production cost.Source: §E.1.2 · p. 47
  • In case73rts, gas makes up most of the supply curve; congestion ranges from none on the low-load example day to seven lines on the high-load day.The figure describes the RTS-GMLC system with 73 units.Source: Figure 13 · p. 48

E.1.3 DISCUSSION OF RESULTS

Across the transmission test systems, learned offers raise prices and generator profits while production costs change little. Results also vary by market: real-time offers face limited unilateral gains, and learners show different training trajectories and markup patterns.

E.1.3 DISCUSSION OF RESULTS

  • Mean LMP rises on all 36 evaluation days across the three systems, while total production cost differs by less than 1% and total unit profit rises.The mean LMP reaches 1.50 and 1.56 times truthful levels for IPPO-PS and IPPO-NoPS on case29gb, respectively; the increase is smallest on case73rts.Source: §E.1.3 · p. 48

E.1.4 DETAILS OF SUPPLY CURVES

  • At peak demand on case29gb, IPPO-PS raises the supply curve while demand stays fixed; the 14 nuclear units remain far left of demand under both offer profiles.The price rises with the learned curve.Source: §E.1.4 · p. 48, Figure 18 · p. 53

E.1.5 MARKUPS OF DIFFERENT LEARNERS

  • All four learners average near a 1.5 markup on case29gb, but IPPO-PS has the narrowest spread and IPPO-NoPS the widest.All learners mark up the 14 nuclear units less than their own system-wide mean; IPPO-NoPS keeps their markups near truthful offers.Source: §E.1.5 · p. 48, Figure 19 · p. 53

E.2.1 MARKET SETTING

  • In real-time balancing, units offer a generation-cost markup from 1 to a cap of 2 once per half-hour, on case29gb and case73rts.Most output is scheduled day ahead; median absolute half-hourly demand deviation is 214 MW, under 1% of mean demand, limiting gains from raising an offer alone.Source: §E.2.1 · p. 54

E.2.2 AGENT TRAINING

  • On case29gb, IPPO-PS training return falls throughout and ends lowest among four learners; IPPO-NoPS and SAC-PS also fall less, while SAC-NoPS rises then stays level.Mean markup of 30 committed units starts between 1.48 and 1.61 and ends at 1.04 (IPPO-PS), 1.24 (IPPO-NoPS), 1.40 (SAC-PS), and 1.39 (SAC-NoPS).Source: §E.2.2 · p. 54

E.2.3 DISCUSSION OF RESULTS

In real-time balancing, independent learners can miss joint gains because unilateral actions provide weak or misleading profit signals. The ancillary-services setup co-clears energy and reserve, with training results reported on three transmission systems.

E.2.3 DISCUSSION OF RESULTS

  • On case29gb, IPPO-PS returns to truthful offers, while the other three learners finish with higher total profit and lower system cost.The untrained networks already assign different markups; training moves them toward truthful offers rather than discovering the higher-profit position.Source: §E.2.3 · p. 54, Figure 22 · p. 56

E.2.4 UNILATERAL AND JOINT MARKUPS

  • For 19 of 30 committed units, truthful offers maximize unilateral profit; no unit gains more than $0.23 million per day, and 27 lose at markup 2.00.A unit’s lost output and profit are largely taken by rivals, leaving total profit almost unchanged.Source: §E.2.4 · p. 55, Figure 23 · p. 56
  • Joint real-time markups raise total profit and lower system cost only when units choose different markups; a common markup leaves both nearly unchanged.Under the two-settlement rule, half-hour output deviations roughly offset across an hour, so higher real-time prices transfer money between units without increasing the total.Source: §E.2.4 · p. 55, Figure 24 · p. 57

E.3.1 MARKET SETTING

  • The ancillary-services setting jointly clears energy and reserve offers, with two reserve products requiring response within 10 and 30 minutes and requirements set at 5% of forecast demand.Reserve awarded to a unit cannot exceed its rampable output; the market uses case29gb, case73rts, and case813nem.Source: §E.3.1 · p. 58

E.3.2 AGENT TRAINING

  • On case29gb, IPPO-PS training return falls from −106 k£ per generator per day in the first five iterations to −126 in the last 20; the other learners rise slightly.Both IPPO learners’ returns rise on case73rts and case813nem.Source: §E.3.2 · p. 58

E.3.3 DISCUSSION OF RESULTS

In ancillary services, training does not improve learners’ profitability relative to their untrained networks, despite gains over truthful offers. A much larger coordinated reserve-offer gain is available, while production cost remains close to its truthful level.

E.3.3 DISCUSSION OF RESULTS

  • The untrained networks earn £2 million to £4.5 million per day more than truthful offers; after training, no learner gains more than its own untrained network.IPPO-PS gives back most of its gain, ending £3.95 million below its untrained network; the other three remain £2.71 million to £3.69 million above truthful offers.Source: §E.3.3 · p. 58
  • Raising every unit’s reserve offer to £150/MWh increases total profit by £9.05 million per day, a gain no learner finds.Only £0.51 million to £1.63 million per day of learner profit comes from reserve payments; the rest comes from energy.Source: §E.3.3 · p. 58
  • On the other two test systems, profit changes are within a few tens of thousands of dollars per day; after training, IPPO-PS earns the same or less, while IPPO-NoPS earns slightly more.Production cost stays within 0.6% of its truthful level.Source: §E.3.3 · p. 58

E.3.4 RESERVE OFFER ANALYSIS

In ancillary services, unilateral reserve-offer increases yield little, whereas coordinated increases can raise total profit substantially. The following peer-to-peer market section describes its household community and reports that learner returns change little after early training progress.

E.3.4 RESERVE OFFER ANALYSIS

  • When one of 29 committed units raises its reserve offer alone to £150/MWh, the median profit change is zero and the largest gain is £0.08 million per day.When all 66 units raise offers together, total profit increases by £9.05 million per day, with production cost changing by less than 0.001%.Source: §E.3.4 · pp. 58–59, Figure 27 · p. 60

E.4.1 MARKET SETTING

  • The peer-to-peer community contains 1,200 Belgian households with PV and batteries; each household is an agent with quarter-hour metering and a 0.011 MWh battery.The market uses a community double auction, with unmatched energy settled with the grid.Source: §E.4.1 · p. 61, Figure 28 · p. 61

E.4.2 AGENT TRAINING

  • Both IPPO learners’ training returns rise fastest in the first 50 iterations, while both SAC learners reach their level within a few dozen iterations.From iterations 100–149 to 351–400, each IPPO learner’s mean return rises by only 0.0009 to 0.0027 EUR per household per quarter-hour.Source: §E.4.2 · p. 62, Figure 30 · p. 62

E.4.3 DISCUSSION OF RESULTS

P2P learners differ in episode outcomes and bidding behavior, while arbitrage gains shrink as participation grows. The local-flexibility findings concern a Swiss feeder where procurement is driven by a congested line.

E.4.3 DISCUSSION OF RESULTS

  • On all 64 episodes, IPPO-NoPS lowered price and raised profit; IPPO-PS had the highest mean profit, −0.769 to −0.685 EUR versus −1.201 under truthful offers.The three SAC-PS seeds disagreed on clearing prices; two averaged 270.3 and 272.9 EUR/MWh, versus 238.1 EUR/MWh under truthful offers.Source: §E.4.3 · p. 62, Figure 31 · p. 63

E.4.4 ARBITRAGE GAIN AND PARTICIPATION

  • As participation grows, charging raises noon prices and discharging lowers evening prices, eroding the spread until it cannot cover round-trip losses and degradation costs.The saturation is market-driven: more agents adopting the same strategy change the prices they face.Source: §E.4.4 · p. 63, Figure 33 · p. 64

E.4.3 DISCUSSION OF RESULTS

  • NoPS learners charged during PV-surplus hours and discharged in the evening, while PS learners stayed near the lowest state of charge; their learned bid prices also differed.IPPO-NoPS used a large charge-discharge cycle and IPPO-PS buyers leaned toward the retail tariff; SAC-PS seeds learned widely differing prices.Source: §E.4.3 · p. 63, Figure 32 · p. 64

E.4.4 ARBITRAGE GAIN AND PARTICIPATION

  • Arbitrage gains fell from 1.129 EUR per household episode with one participant to −1.248 EUR with all 1,200, turning negative at 210–240 participants.With 600 arbitraging households, each idle household gained 0.573 EUR per episode.Source: §E.4.4 · p. 63, Figure 33 · p. 64

E.5.1 MARKET SETTING

  • The Swiss local-flexibility test uses feeder 459 0, with 129 buses and 128 lines; the operator clears procurement hourly and pays each winner its own offer.The price floor is the replacement cost, 149.88 CHF/MWh.Source: §E.5.1 · p. 65, Figure 34 · p. 65
  • The flexibility requirement is based on how much line flow would exceed 98% of its rating without procurement, including load and aggregators’ planned charging.Voltage stayed within limits all year in the Swiss test system, so the congested line was the only reason for procurement.Source: §E.5.1 · p. 65
  • Only batteries downstream of the congested line can relieve it; in the 2040 placement, 12 such batteries can supply 2.51 MW against a maximum 0.43 MW requirement.With batteries idle, procurement occurred on 9 of the 36 test days.Source: §E.5.1 · pp. 65–66, Figure 35 · p. 66

E.5.3 ANALYSIS OF THE PROCUREMENT REQUIREMENT

Learned flexibility-market policies procure more often than truthful offers, and procurement rises further when aggregators’ planned charging increases congested-line flow. Training returns plateau quickly.

E.5.3 ANALYSIS OF THE PROCUREMENT REQUIREMENT

  • Planned charging raises baseline flow on the congested line, prompting procurement; more than 80% of daily procurement under learners exceeded what truthful offers would buy.Under truthful offers, procurement occurred in 43 of 864 test hours; cleared prices were within 1% of the floor for all IPPO seeds and 75% of SAC seeds.Source: §E.5.3 · p. 66
  • All four learners’ training returns reached a plateau within about 100 iterations.The plotted lines show geometric-mean return across seeds, with bootstrap 95% confidence intervals.Source: Figure 36 · p. 67
  • Procurement-period shares rose from 5.0% to 18.7% with 24 aggregators and from 4.9% to 15.4% with 34 when truthful offers were replaced by SAC-PS learners.With the monitor on, all six paired runs in the 34-aggregator configuration showed daily procurement falling from about 3.2 to 0.15 MWh and daily learner return from about 118 to 1 CHF.Source: §E.5.3 · p. 67

E.5.4 ANALYSIS OF A UNILATERAL MARKUP

A unilateral markup reduces an aggregator’s awarded volume near the price floor, but higher offers can raise its return. The offer mapping becomes nearly flat near the floor, weakening the return gradient to the price action.

E.5.4 ANALYSIS OF A UNILATERAL MARKUP

  • Aggregator 7’s award fell from 8.86 to 2.53 MWh after a small markup and stayed at 2.53 MWh at 9,900 CHF/MWh; its return rose again above about 481–489 CHF/MWh in the two 3.9 MW configurations.In the 34-aggregator, 6.3 MW configuration, return exceeded its floor value again only at about 4,590 CHF/MWh.Source: §E.5.4 · p. 67
  • Near the floor, the offer mapping is nearly flat, so the return gradient reaching the price action weakens and helps explain why learners stay near the floor.The figure reports that all IPPO seeds cleared within 1% of the floor.Source: §E.5.4 · p. 67, Figure 38 · p. 69
  • The left panel plots offer premium against price action on a log scale; the right compares aggregator 7’s hourly award at three offers, with others at the floor.Near the floor, the offer premium gap shrinks by about a factor of e for each unit decrease in action.Source: Figure 38 · p. 69

Related papers

Loading 2609.37321v1…