Strategic Arbitrage in Electricity Markets: A Multi-Agent Policy Optimization Framework

I. INTRODUCTION

• Consolidated storage may manipulate prices by withholding arbitrage potential, which ISOs cannot test as directly as conventional-generator withholding because storage costs depend on future arbitrage potential.The passage also notes that load aggregators may raise a similar challenge, with coordination across millions of devices adding operational complexity.Source: §I · p. 1
• The framework treats arbitrage-based withholding as a multi-agent policy-optimization game in which operators learn closed-loop policies in a stochastic real-time market.The authors describe this as a first framing of arbitrage-based withholding as a multi-agent policy optimization game.Source: §I · p. 1
• On the modified IEEE RTS-GMLC benchmark, stochastic gradient play converges reliably to equilibria where greater storage consolidation raises storage revenue at the expense of social welfare.The case study is presented as a tool for characterizing the economic implications of consolidated storage and load aggregators.Source: §I · p. 1

A. Continuous games

• In smooth continuous games, a local Nash equilibrium leaves no player able to improve through a local deviation, while an LASE additionally requires a stable game-Jacobian.Every Nash equilibrium is either an LASE or a strict saddle point; general-sum gradient dynamics can cycle or diverge, and LASEs need not be Nash equilibria.Source: A. Continuous games · pp. 1–2
• Potential games restore convergence guarantees under mild assumptions because gradient play becomes cooperative ascent on a shared potential function.Every stationary point is then a Nash equilibrium.Source: A. Continuous games · p. 2

B. Stochastic games

• A stochastic game specifies agents, a state space, state-dependent feasible actions, reward functions, transitions, initial-state and noise distributions, and a discount factor.The transition function maps state, joint action, and noise to the next state.Source: B. Stochastic games · p. 2
• Each agent selects actions through a stationary policy parameterized by its own strategy parameters and constrained to that agent’s feasible action set.The policy maps the current state and parameter values to an action.Source: B. Stochastic games · p. 2
• Combining the stochastic game with its policy parameterizations induces a continuous game whose payoffs are expected returns under the resulting state and action trajectories.The trajectories follow the parameterized dynamics under sampled initial states and noise.Source: B. Stochastic games · p. 2
• A Markov potential game is a smooth parameterized stochastic game whose stage rewards admit a potential that also serves as a potential for the induced continuous game.The passage gives this as the defining relationship between the stagewise function and the induced potential.Source: B. Stochastic games · p. 2

C. Learning procedure: stochastic gradient play

• Stochastic gradient play (SGP) approximates gradient-flow updates using sampled estimates of agents’ returns and gradients in a parameterized stochastic game.Per-sample smoothness makes SGP applicable, while rollouts are sampled from the initial-state, noise, and geometric distributions to yield an unbiased estimator.Source: C. Learning procedure: stochastic… · p. 2
• Rollout backpropagation computes SGP directly, offering simpler, lower-variance gradient estimates than model-free policy-gradient methods and supporting deterministic policies.The authors compute equilibria rather than simulate the learning process itself.Source: C. Learning procedure: stochastic… · p. 2

III. FORMULATION

• The market model assigns a small number of strategic agents to manage disjoint fleets of flexible devices, each with private state, while a public market state captures exogenous grid conditions.Device state can represent battery state of charge or feasible indoor temperatures for aggregator-managed HVAC systems.Source: §III · pp. 2–3
• Agents self-schedule consumption or production, are paid or charged at the market-clearing price, and are assumed to operate in an unconstrained pooled single-node market.The model excludes forward-contract strategy and congestion-based locational market power, so its equilibria represent system-wide withholding.Source: §III · pp. 2–3
• The system cost function maps demand to total cost, and its derivative determines the market price; renewable generation enters as negative net demand.The cost function is stationary rather than explicitly time-varying.Source: §III · p. 3
• Under the stated smoothness assumption, the formulated parameterized stochastic game is per-sample smooth, permitting stochastic gradient play.The assumption requires smooth cost, baseline-demand, transition, and device-policy functions.Source: C. Learning procedure: stochastic… · p. 2, §III · p. 3

A. Potential game structure

• Testing whether the game is a potential game therefore reduces to checking whether stagewise payoffs are potential in agent demands.The theorem specializes an earlier result to smooth parameterized stochastic games and provides a directly verifiable proof route.Source: A. Potential game structure · p. 3
• Theorem 1 shows that the formulated game is a Markov potential game if a stagewise potential exists over the market state and device demands with matching payoff derivatives.The induced potential is the corresponding stagewise function evaluated along the game state and actions.Source: A. Potential game structure · p. 3
• The proof relies on other agents’ actions and the public market-state trajectory being independent of a given agent’s policy parameters.That independence follows from the construction of the transition and policy functions.Source: A. Potential game structure · p. 3

B. Cost functions

• The dispatch solution recovers merit-order operation, dispatching generators in increasing order of fuel cost to meet net demand.Negative net demand from excess VRE is handled through zero-cost curtailment.Source: B. Cost functions · p. 4
• The exact cost function is nonsmooth, so its discontinuous price function prevents applying first-order methods such as stochastic gradient play.The section therefore proposes two smooth proxy cost functions.Source: B. Cost functions · p. 4
• CLSE can approximate the exact cost arbitrarily closely as α → 0, but it does not generally induce a potential game or guarantee convergence under SGP.Larger α improves gradient conditioning at the cost of fidelity.Source: B. Cost functions · p. 4
• CQ is convex and non-decreasing for positive demand and induces a potential game, unlike CLSE.Theorem 2 establishes the potential-game property.Source: B. Cost functions · p. 4
• The analysis attributes any unstable gradient-flow dynamics under CLSE to the nonlinearity of its marginal-cost function.The benchmark experiment compares SGP convergence under CLSE and CQ.Source: B. Cost functions · p. 4

V. CASE STUDY

• The IEEE RTS-GMLC benchmark supplies a year of five-minute demand and VRE-output data for experiments across ownership structures and two cost proxies.The benchmark is a modern transmission system with high VRE penetration.Source: §V · p. 4

A. Baseline demand model

• The baseline net-demand model combines annual, weekly, and daily seasonality with autoregressive noise to represent persistent, often weather-correlated shocks.Net demand aggregates consumer load and negative VRE contributions.Source: A. Baseline demand model · p. 4
• The modeled system omits its single concentrated solar plant, treats impoundment hydropower inflow as immediate generation, and assumes the nuclear plant runs at constant full capacity.These choices simplify construction of the baseline net-demand profile.Source: A. Baseline demand model · p. 5
• A single Cochrane–Orcutt estimation iteration fits the model parameters and produces synthetic demand nearly indistinguishable from the training trajectory at annual and monthly resolutions.The fit uses M0 = 5; Figure 1 compares a simulated profile with RTS-GMLC data.Source: A. Baseline demand model · p. 5
• In winter, excess VRE frequently drives aggregate net demand below zero, a behavior linked to large price swings and high arbitrage potential.This motivates attention to the boundary between negative and positive net demand.Source: A. Baseline demand model · p. 5

B. Generator supply curve

• Each fossil generator is represented by its maximum output and a linear cost coefficient derived from its fuel price and capacity-weighted average heat rate.Minimum generation limits are set to zero because unit commitment is neglected.Source: B. Generator supply curve · p. 5
• The CQ parameters are fitted by least squares to the exact cost function over the aggregated RTS-GMLC net-demand trajectory.The LSE proxy uses smoothing parameter α = 1.Source: B. Generator supply curve · p. 5
• The exact and LSE price functions jump steeply at zero demand, from zero in the VRE-curtailment region to roughly $20k/GWh–$35k/GWh at positive demand.Because net demand frequently crosses zero, this jump is both an arbitrage opportunity and the dominant nonlinearity that may destabilize CLSE gradient dynamics.Source: B. Generator supply curve · p. 5, Figure 2 · p. 5

C. Storage model

• The case study includes grid-scale batteries, although the framework also supports load aggregators and other flexible self-scheduling participants.RTS-GMLC lacks native distributed energy resources and aggregations, so the experiment considers batteries only.Source: C. Storage model · p. 5
• Each battery’s scalar state is its state of charge, and its feasible charging or discharging action depends on timestep length, maximum rate, and maximum capacity.The state dynamics assume 100% roundtrip efficiency.Source: C. Storage model · p. 6
• A smooth actuator approximates hard clamping while preserving a locally linear region around zero control action.The actuator is shown at 10% and 90% state of charge, with β = 0.01.Source: C. Storage model · p. 6
• The linear policy combines a demand-related Fourier basis, residual, and state of charge, allowing basic arbitrage and withholding behavior while remaining interpretable and easier to train than deep networks.Its Fourier features provide an internal representation of baseline demand.Source: C. Storage model · p. 6
• The experiment represents larger storage penetration with 48 copies of the benchmark’s three-hour battery, totaling 2.4 GW and 7.2 GWh.Each device’s policy has 128 parameters with Mi = 2.Source: C. Storage model · p. 6

D. Training procedure and hyperparameters

• For each cost variant, social-planner policies are trained from zero initialization under SGP with Ri = −C to minimize discounted fuel cost serving baseline load.The authors use Adam for reliable training; this cooperative-game case is equivalent to stochastic gradient ascent on a single payoff function.Source: D. Training procedure and… · p. 6
• The experiment divides 48 identical batteries among ten symmetric ownership structures, from one owner of all batteries to 48 owners of one each.Market share is the fraction of the fleet owned by each agent; policies are initialized from the social planner and remain symmetric during SGP training.Source: D. Training procedure and… · p. 6

E. Results

• Policies converge stably under SGP for all C = CQ ownership configurations and social-planner configurations, while C = CLSE training trajectories approach a neighborhood of an LASE.For C = CLSE, agents take larger return-increasing steps as they approach the LASE, then smaller, noisier steps as the expected gradient vanishes.Source: E. Results · pp. 6–7, Figure 3 · p. 7
• High-market-share policies appear to be differential Nash equilibria, but low-market-share cases remain inconclusive in the empirical Hessian analysis.Small positive eigenvalues also appear for C = CQ, where LASEs are known to be differential Nash equilibria; the authors therefore consider noise a likely explanation.Source: Figure 4 · p. 7, E. Results · p. 7
• As per-agent market share falls, storage revenue and arbitrage surplus approach their social-planner values; the monopolist favors storage revenue at social arbitrage surplus’s expense.The trends are nearly identical for C = CLSE and C = CQ.Source: Figure 5 · p. 7, E. Results · p. 7
• In the selected rollout, the social planner dispatches aggressively to flatten net demand, while the monopolist withholds flexibility, maintaining higher price variance at higher fuel cost.Intermediate configurations show progressively lower charge and discharge rates as market share increases.Source: E. Results · pp. 7–8, Figure 6 · p. 8

• Standard gradient play converges to similar equilibria under both cost proxies on a benchmark market, and those equilibria characterize arbitrage withholding and associated inefficiencies.Source: §VI · p. 8
• The generalized multi-agent policy optimization model supports natural gradient-based equilibrium-seeking updates for strategic storage and load aggregators.Source: §VI · p. 8
• CQ is stylized and induces a Markov potential game, whereas high-fidelity CLSE has no convergence guarantees.Source: §VI · p. 8
• Adding transmission constraints would preclude the closed-form solution, though a smooth proxy using log barriers and differentiable convex optimization is possible.Source: §VI · p. 8
• The claim that differential Nash equilibria under CLSE with small α approximate local Nash equilibria under C = C* is informal, and formalizing it remains future work.Source: §VI · p. 8