When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game

Siu Tung Wong; Carlo Campajola

Source-linked AI summary

When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game

Siu Tung Wong, Carlo Campajola

arXiv:2610.03598v1Oct 2, 2026q-fin.TRcs.AI

TL;DR

The paper asks whether RL can recover a known broker strategy and use it after environmental changes, a question motivated by the difficulty of obtaining analytical strategies for richer dynamics. It tests PPO against an analytical broker policy and derives a finite-step reward; PPO’s accuracy varies by setting, while a frozen analytical policy can be adjusted after a cost change.

Problem
The paper asks whether RL can recover and reuse an analytically known broker strategy, noting that such strategies can be difficult to obtain for richer market dynamics.
Source: §1 · p. 1
Method
The study evaluates PPO against an analytical broker strategy, derives a finite-step reward, and trains PPO to adjust the frozen policy after an execution-cost change.
Source: §3.3 · p. 3, §6 · p. 8
Results
PPO–FFNN approaches the reference action with ξ=0 but tested FFNN/LSTM remain inaccurate with ξ=100; partial-information PPO trails certainty-equivalent control, while residual PPO improves payoff after cost halving.
Source: §6 · p. 8
Takeaways & Limitations
The analytical solution serves as a benchmark for diagnosing learned policies and a useful starting policy for subsequent adaptation.
Source: §6 · p. 8
Under partial information, future work includes testing a filter in the actor and evaluating the model in practical settings with a ξ+η flow.
Source: §6 · p. 8

Abstract

from arXiv · show

Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker--trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker's continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validation-selected PPO--FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO--FFNN and PPO--LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker's observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes \(2.22\%\) of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation.

1 Introduction

The paper asks whether reinforcement learning can recover and reuse a known broker strategy, addressing a gap in tests of state-dependent strategy recovery in broker–trader games. It evaluates full- and partial-information control and adaptation after an execution-cost change.

1 Introduction

RL is attractive for financial optimal control when richer market dynamics make analytical strategies difficult to obtain.The motivating control trade-off balances execution costs against inventory exposure and adverse price movements.Source: §1 · p. 1
The analytical broker–trader equilibrium provides an action at every state, enabling same-state action comparisons alongside realised payoffs and inventory trajectories.The game models a broker, an informed trader, and an uninformed trader.Source: §1 · p. 1
Prior work has used RL to adjust analytical schedules or incorporate closed-form policy and value structure, but not tested recovery of the known state-dependent strategy in this broker–trader game.The paper separates discretisation, policy learning, and hidden-state estimation.Source: §1 · p. 1
The paper asks whether PPO–FFNN and PPO–LSTM recover the analytical strategy with full information, approach a certainty-equivalent benchmark with hidden variables, and improve a frozen policy after cost changes.Source: §1 · p. 1

2 Related Work

Related work covers analytical execution and broker–trader games, RL for optimal execution, analytical guidance and reward shaping, and RL under partial information. The paper positions its study at the intersection of these strands.

Analytical execution and broker games

Analytical execution models relate actions to inventory, impact, order flow, and signals, while broker–trader games add strategic interaction between hedging, prices, client orders, inventory, and information [4], [25].Source: §2 · p. 2

RL for optimal execution

Prior RL work learns liquidation schedules or adjusts analytical schedules, whereas the broker–trader equilibrium enables same-state action comparisons; PPO can remain sensitive to implementation and evaluation choices [12],, [1] [7].Source: §2 · p. 2

Analytical guidance and reward shaping

Analytical guidance has included HJB losses and closed-form structures, while residual RL adjusts an existing controller and potential-based shaping preserves policy ordering [18], [11], [23].Source: §2 · p. 2

RL under partial information

For hidden states, prior approaches summarize observations recurrently or estimate hidden variables, motivating comparison of recurrent and feedforward PPO with a history-based certainty-equivalent controller [10] [13].Source: §2 · p. 2

3 Environment Design

The environment places PPO in a broker–trader game with an analytical reference policy, discretizes the dynamics, and derives a finite-step reward from the broker’s payoff. PPO actions are clipped before execution.

3.1 Broker Execution Game

PPO chooses the broker’s trading rate while the informed trader follows an analytical feedback rule and uninformed flow remains exogenous; the analytical game supplies state-dependent controls.The informed trader continues using its feedback rule even when PPO acts suboptimally, so the resulting interaction is not a subgame Nash equilibrium.Source: §3.1 · p. 2

3.2 Discrete PPO Environment

The finite-grid environment uses left-endpoint Euler–Maruyama transitions, gives PPO full- or partial-information observations, and lets the broker reconstruct past informed flow for the next decision.Network inputs are rescaled before entering either policy network.Source: §3.2 · p. 2

3.3 Finite-step reward

The finite-step reward retains discrete product-rule terms, omits the unpredictable price-innovation term with zero conditional expectation, and has expected accumulated return equal to the cash-and-inventory objective up to fixed initial value.The omitted innovation means the accumulated quantities can differ on an individual realised path.Source: §3.3 · p. 3

3.4 Environment Step

PPO evaluates the likelihood of its raw sampled action during training, while the environment clips the executed trading rate to [−100, 100] before the informed trader’s response and state update.Source: §3.4 · pp. 3–4

4 Experimental Results

The experiments check finite-grid alignment, test whether full-information PPO recovers the analytical broker action, and compare partial-information PPO with a certainty-equivalent controller. PPO approaches the reference in the zero-flow setting but remains inaccurate with stochastic uninformed flow; the partial-information controller stays close to the reference while PPO has larger errors and lower payoffs.

4 Experimental Results

The experiments use five seeds and four million transitions per seed, select checkpoints by payoff on 50 validation paths, and evaluate on 250 held-out test paths.The same test paths are used for every trained policy, and reported 95% Student-t intervals are across five independently trained policies.Source: §4 · p. 4

4.1 Block A: Continuous and finite-grid alignment

At the training grid Δt=2 × 10^-4, action RMSE is 0.003389 and LQ loss is 1.383 × 10^-8; both decrease as the grid is refined.The zero-flow reference action bound is 1.07×10−5, with no crossing on 500 additional simulated paths; reward identity checks are at floating-point precision.Source: §4.1 · p. 4

4.2 Block B: Full-information PPO recovery

With stochastic uninformed flow, the tested PPO–FFNN and PPO–LSTM remain inaccurate, while the selected zero-flow FFNN approaches the reference action.The zero-flow FFNN reports an LQ loss of 0.045 ± 0.031 and action RMSE of 5.494 ± 2.163 across five roots.Source: §4.2 · p. 4

4.3 Block C: Actor capacity and learning diagnostics

Supervised learning shows both network architectures can approximate the analytical action, but the supervised LSTM varies considerably across simulator-control seeds.The supervised FFNN remains close to the reference when it controls the simulator.Source: §4.3 · p. 4
Multiplying rewards by 100 lowers payoff for every seed; versus the corrected baseline, paired changes (95% intervals) were 1.48[−7.34, 10.31] without Δt², −29.24[−33.93, −24.54] after scaling, and −1.09[−8.56, 6.37] without S_k.Source: §4.3 · pp. 5–6
Critics distinguish states with generally higher or lower expected returns but cannot reliably rank nearby actions from the same state.In the stochastic-flow setting, this local action-ranking weakness is identified as a plausible source of weak PPO performance.Source: §4.3 · p. 6
Potential-based shaping produces no reliable improvement: mean payoff rises by 1.57 and action RMSE falls by 1.76, with favourable changes in only three of five seeds.Shaping also increases per-step reward standard deviation by approximately 75%.Source: §4.3 · p. 6

4.4 Block D: Partial-information control

The causal certainty-equivalent controller remains close to the reference under partial information, while both PPO agents have substantially lower payoffs and larger action errors.The CE controller uses the broker’s causal history while informed-trader inventory, signal, and current trading rate are hidden before action.Source: §4.4 · pp. 6–7

5 Adapting the Analytical Policy with PPO

The section tests PPO as a residual correction to a frozen analytical broker policy when execution cost is misspecified. Halving the cost yields small, repeatable test-payoff gains, but only in the lower-cost case tested.

5 Adapting the Analytical Policy with PPO

On 100 original-cost test paths, replaying the frozen analytical policy gives trading-speed RMSE 6.36×10−7 and inventory RMSE 3.29 × 10−9.The frozen and replayed analytical trajectories closely match.Source: §5 · p. 7
PPO learns an additive correction to the analytical action, which remains frozen from the original-cost environment.The correction is a zero-initialized, two-layer width-64 Tanh FFNN of the full-information observation.Source: §5 · p. 7
The tests use full information and σξ = 100, with separate adjustments for execution costs of 0.0006, 0.0012, and 0.0018.Policies are selected on validation paths and evaluated on 100 common test paths.Source: §5 · p. 7
0.00519 mean test-payoff increase; all five PPO adjustments improved payoff and closed 2.22% of the gap to the changed-cost reference.The improvement was repeatable only in the lower-cost case tested.Source: §5 · pp. 7–8, Table 4 · p. 8

6 Discussion and Conclusion

PPO approaches the analytical reference in the zero-flow full-information case, but remains inaccurate with stochastic uninformed flow and under partial information. The analytical policy also provides a useful starting point for adaptation after execution costs change.

6 Discussion and Conclusion

With full information and ξ=0, PPO–FFNN approaches the reference action, but policies remain inaccurate when ξ=100.Numerical checks and supervised learning indicate the errors are not primarily due to discretisation, reward implementation, or insufficient FFNN capacity.Source: §6 · p. 8
With the informed trader’s inventory and signal hidden, the certainty-equivalent controller remains close to the reference, while PPO–FFNN and PPO–LSTM use history less effectively.The result indicates that the available history supports accurate control in this environment.Source: §6 · p. 8
After the execution cost is halved, residual PPO yields a small, repeatable payoff improvement from the frozen analytical policy.The analytical policy serves as both a benchmark for diagnosing learned policies and a starting point for adaptation.Source: §6 · p. 8
The evidence points to exploration interacting with reward-based value estimation, with the critic showing weak local action ranking under stochastic uninformed flow.Wider initial exploration helps when ξ=0 but does not resolve poor performance after stochastic flow is introduced.Source: §6 · p. 8
Future work could use analytical value or action-value information to improve the local learning signal and develop structured residual policies for nonlinear costs or uncertain market impact.Alternative action and policy parameterisations are also proposed.Source: §6 · p. 8
For partial information, future work could test a filter in the actor; practical applications would include ξ+η flow.Source: §6 · p. 8