PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Supra-Competitive Outcomes: Collusive Behaviour in Deep Reinforcement Learning for Optimal Execution Games

Christos Spyridon Koulouris, Carlo Campajola

arXiv:2610.00619v1q-fin.TRcs.AI

TL;DR

This paper asks what behaviours allow self-interested learning agents to sustain supra-competitive outcomes. It tests learned policies in a two-player liquidation game by imposing a unilateral first-trade deviation, finding that the opponent’s punitive response offsets the deviator’s gain without materially changing the punisher’s average payoff.

  • Problem

    Prior work observed that self-interested learning agents can sustain supra-competitive outcomes, raising the question of what behaviours allow this.

    Source: §1 · p. 1

  • Method

    The paper imposes a deviation’s first trade on one learned agent and tests the other agent’s response while the deviator resumes its learned policy.

    Source: §4.2 · p. 4, §4.1 · p. 4

  • Results

    The learned response accelerates liquidation and offsets the deviator’s gain while leaving the punisher’s average payoff materially unchanged under the same deviation.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The incentive comparison and execution-specific total-variation bound both support a collusive interpretation of the tested interaction.

    Source: §5 · p. 8

    Policies receive within-episode histories that reset each episode; opponent inventories, actions, rewards, and network parameters are not supplied.

    Source: §2.4 · p. 3

Abstract

from arXiv · show

In this paper, we extend earlier findings of supra-competitive outcomes in optimal-execution games by identifying a learned punitive mechanism that deters deviations and provides behavioural evidence of collusion. We investigate this mechanism in a two-player, finite-horizon Almgren-Chriss liquidation game. Independent proximal policy optimisation agents with access to within-episode price and action histories achieve costs below the Nash benchmark. We identify a profitable deviation by training against the mean learned liquidation schedule, then impose its first trade on one of the original agents. The opponent responds by accelerating liquidation. This response more than offsets the deviator's gain in every run and both player roles, while leaving the punisher's average payoff materially unchanged relative to not punishing under the same deviation. The punisher imposes greater losses on the deviator while preserving its own average payoff, despite the availability of more profitable, less punitive liquidation plans. Matching deviations and subsequent additional selling rise and later decline during training, while final policies retain an effective punitive response. We formalise two checks: whether punishment outweighs the gain from deviating, and whether the change in trading behaviour is large enough to account for the loss imposed. Both checks hold for the tested deviation. Together, these findings provide behavioural and economic evidence supporting a collusive interpretation of the learned supra-competitive outcomes.

1 Introduction

This paper asks what behaviours allow self-interested learning agents to sustain supra-competitive outcomes, using optimal execution to study coordination and responses to unilateral deviations. It examines whether punitive responses provide a behavioural basis for interpreting learned outcomes as collusive.

1 Introduction

  • Deviation-response experiments have been used to examine whether retaliation deters departures from supra-competitive pricing.Related work also studies contingent punishment in prisoner’s dilemma and electricity-market settings, and directly targets retaliation with CURB in pricing and quantity competition [24] [25].Source: §1 · p. 1
  • In optimal execution, each trader’s schedule affects its own costs and its competitor’s, making the setting suitable for studying coordination and punitive responses.Moderating liquidation can reduce competition for liquidity, while accelerated trading can worsen the other agent’s remaining opportunities.Source: §1 · p. 2
  • Prior execution studies report supra-competitive costs and connect history-aware learning to possible punishment, motivating direct tests of responses to deviations [15].Koulouris and Campajola [15] report more frequent and persistent supra-competitive outcomes with within-episode histories; stronger current-price conditioning alone did not reproduce the pattern.Source: §1 · p. 2
  • The paper identifies punitive responses to unilateral deviations as a mechanism that can help sustain supra-competitive outcomes and support a collusive interpretation.The paper connects these learned responses to reward–punishment accounts of collusion, in which cooperation is rewarded and departures are penalised.Source: §1 · pp. 1–2

2 Execution Environment and Learning Agents

The paper studies a two-player Almgren–Chriss liquidation game in which agents minimise their own execution costs and learn continuous trade policies from within-episode histories. It compares outcomes with Nash and TWAP benchmarks and trains independent PPO agents.

2.1 The liquidation game

  • Two agents simultaneously trade the same asset over a finite horizon, with execution prices reflecting combined orders and price impact.The game uses the Almgren–Chriss liquidation setting, and agents must fully liquidate their inventories by the horizon’s end.Source: §2.1 · p. 2

2.2 Execution cost and reward

  • Each agent minimises its own expected implementation shortfall under a risk-neutral objective, with no variance penalty or shared reward.Reported IS/N is a fixed rescaling of total shortfall; lower values indicate better execution.Source: §2.2 · p. 2

2.3 Equilibrium benchmarks

  • The competitive reference is the finite-grid closed-loop Nash equilibrium, while joint TWAP serves as the cooperative reference.For equal starting inventories, joint TWAP trades q0/N in each period.Source: §2.3 · p. 2

2.4 Information and history encoding

  • Each policy observes current price and its own inventory alongside within-episode price and executed-action histories, but not opponent actions or inventories.Histories reset each episode, and future prices and trades are excluded from the sequence encoder.Source: §2.4 · pp. 2–3

2.5 Continuous-action PPO

  • The agents use independent continuous-action PPO policies and are trained for 40,000 episodes, with evaluation over 500 test episodes per run.PPO updates follow 200-episode rollouts; the study trains ten independent agent pairs.Source: §2.5 · p. 3

3 Supra-Competitive Outcomes

Across all ten runs, both players achieve execution costs below the closed-loop Nash benchmark but remain above joint TWAP. Their average liquidation paths shift more trading towards later periods, and training checks indicate that exploration extends beyond the final supra-competitive region.

3 Supra-Competitive Outcomes

  • On average, agents sell less than Nash early and retain more inventory through the episode’s interior, shifting liquidation later.Figure 2’s pooled paths represent similar realised trajectories rather than concealing large differences among them.Source: §3 · p. 3
  • Training checks show that each player’s actions cover both sides of the Nash trade quantity at every discretionary step, beyond the final supra-competitive region.Joint training costs repeatedly approach Nash across all ten runs.Source: §3 · pp. 3–4
  • Both players’ mean testing costs are below closed-loop Nash and above joint TWAP in all ten runs.Figure 1 reports each run’s pair of mean testing costs; the experiments use negligible exogenous price fluctuations relative to market impact.Source: §3 · p. 3, Figure 1 · p. 4
  • These outcomes establish supra-competitive execution and motivate testing whether contingent punitive responses support them.Figure 1’s circles average 500 test episodes from one trained pair, and its stars mark the Nash and joint TWAP benchmarks.Source: §3 · p. 4, Figure 1 · p. 4

4 Punishing Profitable Deviations

The tested deviation is profitable against the fixed mean liquidation schedule, but the learned response accelerates the opponent’s sales and removes that gain. Both incentive and behavioural-distance checks pass for the tested deviation.

4.1 Learning a Profitable Deviation

  • The learned deviation reduces mean testing cost against the fixed mean schedule in every run, so that symmetric fixed-schedule profile is not a Nash equilibrium.The averaged deviation trajectory also remains profitable under expected execution cost.Source: §4.1 · p. 4

4.2 Identifying Learned Punitive Behaviours

  • After the deviation becomes observable at timestep 2, the punisher sells more in timesteps 2–5 and less later, exposing the deviator to earlier price impact while it holds inventory.All ten run means show additional selling in timesteps 2–5.Source: §4.2 · p. 4, Figure 3 · p. 5
  • The deviator incurs higher execution cost in every trained run, while the punisher’s average payoff remains materially unchanged under the same deviation.The punisher preserves, on average, the payoff from its supra-competitive path.Source: §4.2 · pp. 4–5, Figure 4 · p. 6
  • Sampled alternative liquidation schedules could give the punisher a higher payoff while imposing a smaller loss, yet its learned policy produces stronger punishment.The alternatives vary the punisher’s trades after the first trade and enforce full liquidation at the final timestep.Source: §4.2 · p. 5
  • Matches to the tested deviation and subsequent additional selling rise, peak around episode 18,000, and later decline during training.This pattern is consistent with the supra-competitive outcome stabilising as deviations become less frequent.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3 Incentive and Behavioural Checks for Punishment

  • Punishment exceeds the deviator’s gain in every run and paired test case, while the measured change in liquidation timing exceeds its required boundary.Both incentive and behavioural-magnitude checks pass for the tested one-step deviation.Source: §4.3 · pp. 6–7, Figure 6 · p. 7, Figure 7 · p. 7

5 Conclusions

In a finite-horizon Almgren–Chriss game, imposed first-step deviations trigger punitive liquidation responses that remove the deviation’s gain without materially reducing the punisher’s payoff. Incentive and trading-change checks support interpreting this learned interaction as collusive, while training traces show the tested pattern waning as final policies retain an effective response.

5 Conclusions

  • Independent PPO agents with within-episode price and action histories achieve below-Nash execution costs across all ten trained pairs without communication or shared rewards.The game is a two-player, finite-horizon Almgren–Chriss liquidation game.Source: §5 · p. 8
  • The first-step deviation triggers accelerated liquidation that removes its gain without materially sacrificing the punisher’s payoff, supporting a collusive interpretation.Nearby liquidation plans offer the punisher higher payoffs while imposing a smaller loss on the deviator.Source: §5 · p. 8
  • Matches to the tested deviation and subsequent additional selling rise and then decline during training, while final policies retain an effective punitive response.The authors describe this pattern as consistent with deterrence becoming established.Source: §5 · p. 8
  • The incentive comparison and execution-specific total-variation bound both hold for the tested first-step deviation across all runs and both player identities.The authors identify stronger price noise, larger agent populations, and a broader range of deviations as future tests.Source: §5 · p. 8

Related papers

Loading 2610.00619v1…