When Is Inaction a Mistake? Continuation-Aware Auditing of PPO Trading Policies

1. INTRODUCTION

• Proportional transaction costs create an optimal no-trade region because weak signals may not justify turnover.Return and turnover alone do not distinguish appropriate cost-sensitive inaction from uninformative observations or a poor decision rule.Source: §1 · p. 1
• A full-belief dynamic-programming reference can use histories unavailable to a current-observation actor, and its action values assume future decisions follow the reference policy.Matching current information does not resolve whether a recommended trade retains its advantage when subsequent decisions return to the deployed policy.Source: §1 · p. 1
• The audit examines disagreement at frozen PPO deployment occupancy, projects reference values onto current observations, tests isolated deviations under incumbent continuation, and evaluates repeated deployment of observation-based alternatives.It distinguishes whether an oracle disagrees, an isolated action change helps, and replacing the decision rule improves performance.Source: §1 · p. 1
• Repeated projected-rule deployment improves all 50 policies, while historical BTCUSDT replay gains arise primarily from lower turnover costs.Information matching reduces apparent disagreement, while incumbent continuation reverses most audited trade advantages; these are distinct intervention comparisons.Source: §1 · p. 1

2. RELATED WORK

• Classical proportional-cost control derives optimal no-trade regions and predictive-signal thresholds, while financial RL studies trading and hedging under frictions.Related work also addresses costly-action RL and partially observable trading systems with temporal representations.Source: §2 · p. 1
• Restricted-information control work studies aggregation and reactive POMDPs, memory traces, optimal reactive policies under hard aggregation, and privileged training information.This paper’s projection separates hidden-information effects from within-observation disagreement using full-belief action values, but does not solve for the optimal restricted-information policy.Source: §2 · p. 1
• The paper uses an exact Kalman posterior in its linear-Gaussian setting, a lambda discrepancy for non-Markov observations, and oracle-assisted tests to localize RL bottlenecks.Known perturbations check diagnostic behavior, while deployment evaluation assesses candidate rules.Source: §2 · p. 1

3. AUDITING ORACLE-FLAGGED INACTION

• The four-stage audit evaluates the reference on visited states, removes hidden-belief distinctions unavailable in current observations, tests a projected trade under incumbent continuation, then evaluates observation-measurable rules at every step.The final stage distinguishes policy replacement from an isolated action change.Source: §3 · p. 2

3.1. Controlled market and partial observation

• The controlled market uses defaults ρ = 0.9, σz = σ = 1, λ = 0, and γ = 0.99, with normalized cost C = c/(βE|z|) = 1.5 in confirmatory cohorts.The signal-to-noise setting is S = βτ/σ = 1.Source: §3.1 · p. 2
• The study compares fully observed, raw current-signal, filtered-posterior, and fixed eight-observation-history actors under matched architecture, costs, budget, and paired seeds.The fully observed reference is obtained by value iteration on 101 signal bins; the partial-observation reference uses an augmented state including the posterior mean, current signal, and previous position.Source: §3.1 · p. 2

1. Full-belief audit

• The full-belief reference is computed by value iteration on finite grids that preserve both prior and observation coordinates, allowing projection onto actors’ observed cells.The reference and actors share feasible actions, but the reference has more information; value iteration is numerical on the induced finite models, not an exact continuous-state solution.Source: §1 · p. 2

3.2. Deployment and information projection

• The diagnostics compare frozen decision rules with a fixed feasible-action DP reference, not with a restricted-information optimum.The audit asks where disagreement occurs, what information supports it, and which future decisions make the alternative valuable.Source: §3.2 · p. 2
• The hard false-inactive label uses probability and WFI cutoffs of 0.9 and 0.1, respectively, and its count falls from 11 to 1 when the WFI cutoff rises to 0.2.The probability cutoff is invariant in the fixed-measure sensitivity check; neither cutoff is theoretically privileged.Source: §3.2 · p. 2
• For each current-observation actor, full-belief action values are projected onto its observed cells using that actor’s own deployment occupancy.The observed cell includes the current observation and endogenous action-state memory, but no explicit observation history.Source: §3.2 · p. 2
• The projected decomposition separates hidden belief distinctions from disagreement with the action preferred by projected values, while retaining oracle continuation through Q*.The two terms sum exactly to one-step OAR, not to a value gap against the optimal observation-measurable controller.Source: §3.2 · p. 2

3.3. Isolated deviations and policy replacement

• The audit tests projected trades under incumbent continuation, then evaluates three observation-based rules under repeated deployment; these distinct endpoints do not establish optimal restricted control.Replacement rules act throughout the observation space, unlike the isolated test on projected false-inactive cells; the deployment comparison uses paired reward gain, with J measured as stationary net reward in simulation and undiscounted daily net reward in BTCUSDT replay.Source: §3.3 · p. 3

4. EXPERIMENTAL PROTOCOL

• PPO uses separate two-layer, 64-unit actor and critic networks, with 2,048 continuing steps per update, four epochs, and minibatches of 256.The protocol also sets clipping to 0.2, value weight to 0.5, and normalized GAE advantages with λ_GAE = 0.95; rewards and critic targets are not normalized.Source: §4 · p. 3
• Evaluation combines same-path burn-in and scoring with exact stationary finite-model evaluation for the original arms, using seed-level bootstrap intervals for continuous metrics and paired contrasts.Same-path evaluation uses 1,000 burn-in and 1,000 scored steps; hard-incidence proportions use Wilson intervals.Source: §4 · p. 3
• The primary signal contrast is prespecified at unit observation noise; robustness checks cover entropy 0.03, 256,000 training steps, and a paired signal-by-cost factorial.A frozen follow-up adds an eight-observation arm, and the long replay reproduces its saved 61,440-step prefix exactly.Source: §4 · p. 3
• Analysis-only corruption checks separately hide prior belief, force no-trade at oracle-trade states, or retain the oracle action before following a deterministic Q*-worst rule.No policy is trained for these checks, and each uses a fixed common measure on both grids.Source: §4 · p. 3

5.1. From disagreement to deployment performance

• At unit noise, incumbent continuation reverses 99.3% of projected-hard missed-advantage mass, yet repeated projected-rule deployment improves all 50 policies (mean gain 0.245 [0.205,0.284]).Table 1 separates action diagnostics from the final policy-replacement reward-gain row.Source: Table 1 · p. 3, §5.1 · p. 3
• Mean full-belief OAR 0.293 decomposes into projection and within-cell gaps of 0.099 and 0.193; incumbent-greedy improves all 50 policies by 0.024 [0.020,0.029].A common-projection rule improves mean reward by 0.200 [0.160,0.242] but only 39/50 policies.Source: §5.1 · p. 3
• The fully observed anchor finds 11/50 hard false-inactive policies; finite-MDP deployment retains all 11, while continuous neural-actor execution preserves 11/50 labels and continuation reverses 91.9% of missed advantage.Finite-MDP stationary per-step regret is 0.356 versus 0.030.Source: §5.1 · p. 3

5.2. Information representation and diagnostic severity

• At unit noise, History-8 lowers hard incidence but not AMA or OAR, whereas Kalman reduces all three; raw-minus-history AMA is −0.009 [−0.061, 0.045].The learned-occupancy raw-minus-filtered AMA reduction is 0.117 [0.078,0.160], with descriptive rule and occupancy components of 0.064 [0.044,0.085] and 0.053 [0.033,0.073].Source: §5.2 · p. 4, Table 2 · p. 4
• Filtered-minus-raw gains are 0.133 [0.097,0.170] in stationary net reward and 13.43 [9.92,17.09] in common-measure discounted value, despite higher turnover (0.076 versus 0.052).At entropy 0.03, filtering reduces deployment soft missed advantage by 0.126 [0.092,0.160] and raises reward by 0.144 [0.112,0.177]; continuation reverses 97.2% of residual filtered missed advantage.Source: §5.2 · p. 4

5.3. Diagnostic checks with known perturbations

• Deliberate changes to information, actions, or continuation recover all three prespecified diagnostic signatures; grid refinement preserves the raw (49), filtered (23), and projected (38) hard sets.The checks support the finite approximations, not the continuous POMDP, and do not identify training causes.Source: §5.3 · p. 4, Table 3 · p. 4

5.4. A frozen intervention on BTCUSDT replay

• 135.03 bp daily net gain in 2025, with 46/50 policies improving under the frozen BTCUSDT intervention.The hand-specified rule was selected using 2024 data and frozen for 2025; 2025 gross reward fell by 3.71 bp/day while costs fell by 138.74 bp/day.Source: §5.4 · p. 4

• The audit clarifies what each comparison establishes for deployment, with controlled experiments and historical BTCUSDT replay illustrating distinct aspects of the deployment question.Source: §6 · p. 4

• The study uses numerical simulations and public historical market data and involves no human or animal subjects; ethical approval was not required.Source: §8 · p. 5