PaperlayerView exact revision on arXiv

Source-linked AI summary

AlphaOpsBench: Benchmarking End-to-End Alpha Strategy Operationalization in Prediction Markets

Huaiyu Jia, Mingxuan Zhao, Jincheng Gao, Zifan Peng, Wentao Zhang, Siguang Li, Shuo Sun

arXiv:2609.31390v1cs.CE

TL;DR

Prediction-market strategies are underspecified economic hypotheses that existing standardized benchmarks do not evaluate end to end. AlphaOpsBench connects source-grounded strategy descriptions to executable policies and historical evidence; strict canonical validity remains rare, and replay completion does not itself establish faithful operationalization.

  • Problem

    Existing benchmarks typically assume predefined assets, schemas, or numerical strategy representations, while prediction-market ideas leave event, information, decision, and execution details open.

    Source: §1 · p. 2

  • Method

    AlphaOpsBench preserves source-grounded requirements and model-owned choices while evaluating strategy design, executable policies, and complementary historical replay settings.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Direct achieves 35 canonical passes among 180 attempts and Staged achieves 20/180 on controlled tasks, while none of the 24 real-world strategies obtains a confirmed canonical pass.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Separating source requirements from model choices helps identify failures such as altered economic mechanisms and inconsistent parameter completion rather than aggregating them into opaque financial metrics.

    Source: §5 · p. 9

    AlphaOpsBench measures operationalization under controlled conditions rather than live trading, and historical replay cannot fully reproduce liquidity, queue priority, or adaptive counterparty behavior.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models increasingly generate quantitative trading strategies, yet existing benchmarks assume standardized assets, numerical features, or directly compilable strategy representations---assumptions that prediction-market strategies violate, since a coarse idea may leave the traded outcome, causal information source, signal definition, threshold, sizing, order policy, exit, and settlement behavior unspecified. We introduce \textsc{AlphaOpsBench}, which evaluates end-to-end operationalization from source-grounded economic hypotheses to auditable executable programs over 581 source-preserving strategy records and a lifecycle-scale Polymarket dataset with 1.28 million binary markets, 183.6 million cleaned executions, settlement evidence, and limit-order-book history, comparing Direct generation against a Staged design-then-code protocol. In a corrected independent-generation study over 36 controlled tasks and 24 preregistered real strategies, strict end-to-end validity remains rare: Direct and Staged obtain 35/180 and 20/180 canonical passes on the controlled cohort and no confirmed pass on the real cohort, and repeated generations vary substantially in model-owned economic choices. By contrast, 775,725 of 783,655 scheduled historical replays complete, showing that replayability is a far weaker property than source-faithful operationalization. Financial outcomes depend on the declared execution model and available historical evidence, and fee and liquidity experiments show that execution costs alter subsequent trading paths rather than acting only as ex-post deductions. \textsc{AlphaOpsBench} thus separates strategy fidelity, behavioral validity, historical executability, and financial performance in an evidence-aware benchmark for LLM-based quantitative research in prediction markets.

1 INTRODUCTION

Prediction-market strategies are economic hypotheses, not standardized rules: generating executable code requires resolving event, information, trading, and lifecycle decisions. AlphaOpsBench evaluates this end-to-end translation and links generated strategies to historical execution evidence.

1 INTRODUCTION

  • Prediction-market strategy descriptions often leave outcomes, information timing, thresholds, sizing, order placement, and exits unspecified, so executable code can diverge semantically or causally from the intended hypothesis.Unlike compact equity-style formulas, they require event-specific operational choices.Source: §1 · p. 2
  • AlphaOpsBench evaluates whether an LLM can turn a real-world prediction-market idea into an executable, testable program while preserving source-grounded requirements and recording model-owned choices.The benchmark spans strategy understanding, decision formalization, and executable policy generation.Source: §1 · p. 2
  • Two complementary backtesting frameworks connect generated strategies to historical execution: broad market replay and execution-aware assessment incorporating order-book state, transaction dynamics, and market-specific constraints.The pairing separates strategy generation from execution feasibility and financial outcome.Source: §1 · p. 2

2 RELATED WORK

Financial and trading benchmarks increasingly cover retrieval, quantitative reasoning, and executable strategies, but generally assume standardized numerical hypotheses. Prediction-market trading adds event-specific contract semantics and lifecycle actions, motivating evaluation of faithful operationalization.

2.1 FINANCIAL KNOWLEDGE AND TRADING BENCHMARKS

  • Prior financial benchmarks extend from report reasoning to retrieval, crypto analysis, time-series reasoning, alpha discovery, and executable strategy evaluation, but typically assume compact numerical relations.They do not assess underspecified economic decisions, temporal alignment, and execution semantics as an end-to-end task.Source: §2.1 · pp. 2–3

2.2 PREDICTION MARKETS AND BLOCKCHAIN-BASED TRADING

  • Prediction-market contracts require event-specific evidence, contract semantics, and lifecycle actions such as collateral splitting, position merging, and redemption, which direct code generation can silently alter.Existing benchmarks and adjacent on-chain studies do not evaluate end-to-end operationalization of incomplete prediction-market strategies.Source: §2.2 · p. 3

3 ALPHAOPS IN THE PREDICTION MARKET

AlphaOpsBench builds a source-preserving strategy and market dataset, then operationalizes ideas through design, code validation, and historical replay. Its evaluation distinguishes strategy fidelity and behavior from execution evidence and financial outcomes.

3.1 DATASET CONSTRUCTION

  • The Polymarket dataset covers 1,276,684 binary markets and 183,625,793 execution records, integrated with settlement-oracle and limit-order-book data for June 1–September 1, 2026.The strategy collection preserves 581 records, separates explicit requirements from open decisions, and grades task completeness across L1–L3 specification levels.Source: §3.1 · pp. 3–4

3.2 END-TO-END PIPELINE

  • Prediction-market strategies must specify outcome tokens, complementary claims, per-leg pricing and sizing, and settlement timing because orders, partial fills, and resolution are part of the trading lifecycle.Coarse source descriptions are converted into tasks preserving explicit requirements and source references, then standardized prompts specify data definitions and trading capabilities.Source: §3.2 · p. 4
  • Detailed design makes target markets, outcomes, observations, conditions, order parameters, and entry, exit, and no-trade rules explicit before code generation.Outcome identity is an economic decision: the extreme-probability example requires choosing between selling YES above 0.98 and buying NO below 0.02.Source: §3.2 · p. 4
  • Validation separately checks source adherence and design–implementation agreement, using static contracts and behavioral tests for outcome bindings, boundaries, action sequences, missing inputs, and position-dependent behavior.A program can faithfully implement an altered strategy, so executable code alone does not establish source fidelity.Source: §3.2 · p. 4
  • Backtests use active, liquid historical markets and complementary engines: FILL-ONLY V3 simulates finite liquidity from on-chain trade ticks, while PML2 replays reconstructed order-book depth and order events.Both distinguish submitted orders from fills; zero-fill runs count toward coverage rather than return comparisons.Source: §3.2 · pp. 4–5

3.3 EVALUATION METHODOLOGY

  • Evaluation covers generation and behavioral correctness, prediction-market reasoning, historical execution, and repeated-generation stability, reporting unavailable evidence and failure states separately from zero performance.Metrics include returns, risk exposure, execution characteristics, and cost sensitivity when sufficient execution evidence exists.Source: §3.3 · p. 5

4 EXPERIMENTS

Across experiments, strict source-faithful operationalization is rare despite extensive historical replay; reproducibility, financial outcomes, and execution-cost effects depend on the decisions and evidence available. The evaluations distinguish runnable artifacts from faithful strategies and executable, financially evaluable outcomes.

4.1 EXPERIMENTAL SETTINGS

  • The main study compares Direct and Staged across 477 source-derived tasks, while replay uses separate candidate subsets and execution-sensitivity tests vary fees, capital, and liquidity conditions.Generation, execution, and financial outcomes use separate metrics and denominators.Source: §4.1 · pp. 5–6

4.2 OPERATIONALIZATION ACCURACY AND STABILITY

  • 35/180 Direct and 20/180 Staged attempts pass canonical validation on controlled tasks, while neither method confirms a pass on the 24 real-world strategies.Only 2/36 Direct tasks and 1/36 Staged tasks pass in all five generations.Source: §4.2 · p. 6
  • Higher-level choices are generally more reproducible than thresholds, sizing, and several data-dependent decisions, while exact-action agreement varies across specification levels.Agreement is interpreted jointly with observable-pair coverage; Figure 3 also reports common-state behavior over all evaluable and reference-active states.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 HISTORICAL EXECUTION AND FINANCIAL OUTCOMES

  • 775,725 of 783,655 scheduled replays complete, but completion does not imply financial evaluability because zero-fill and insufficient-valuation episodes remain outside return aggregation.Figure 4 reports coverage, task-level Net ROI distributions, and execution evidence with metric-specific denominators.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Mean window-end Net ROI is −3.476% for PML2–Direct and −3.440% for PML2–Staged, versus +0.836% and +0.646% in the distinct V3 research setting.PML2 task-equal MDD is 3.520% for Direct and 3.481% for Staged; its order-fill rates are approximately 26%, unlike V3's modeled-liquidity results.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • Archived input-coverage and markout statistics are unavailable rather than zero, and metric availability varies across the replay population.Net ROI is recoverable for more than 95% of the main V3 population, while quantity-fill evidence is available for about one quarter of planned units.Source: §4.3 · p. 8

4.4 TRANSACTION FEES AND SLIPPAGE IN THE POLYMARKET

  • Fees can alter subsequent orders and fills under tight cash constraints; frozen-path fee subtraction is adequate only when account constraints leave the execution path unchanged.In the frozen-fill V3 analysis, median ROI is already negative at 100 bps despite a positive mean.Source: §4.4 · p. 9
  • Larger orders can consume multiple price levels, so ladder-cost measures require an explicit reference price and interpretation alongside realized fills and account-level PnL.For market 2033169, the diagnostic rises from 0.935 to 19.685 USDC while total filled quantity, notional, and final PnL remain unchanged.Source: §4.4 · p. 9

5 DISCUSSION

The discussion frames AlphaOpsBench as an evaluation of translating economic hypotheses into executable policies while preserving intent. It emphasizes prediction-market-specific reasoning and cautions that controlled historical replay is not live trading.

5 DISCUSSION

  • Separating source requirements from model-owned choices helps identify failures such as altered economic mechanisms, inconsistent parameter completion, and unevaluable logic instead of aggregating them into opaque financial metrics.The framework separates reasoning from deterministic execution to assess whether generated policies maintain economic intent.Source: §5 · p. 9
  • Prediction-market operationalization requires event semantics, information constraints, and settlement mechanics, but the benchmark measures controlled operationalization rather than live trading performance.Historical replay cannot fully replicate liquidity dynamics, queue priority, or adaptive counterparty behavior, and the benchmark is constrained by available data and execution interfaces.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents AlphaOpsBench as an end-to-end test of whether generated strategies preserve economic intent through design, code, behavior, and historical evidence. Its motivation is the gap between incomplete prediction-market ideas and executable, faithful programs, alongside contract-specific data and lifecycle requirements.

6 CONCLUSION

  • AlphaOpsBench evaluates preservation of source-specified requirements, records model-owned decisions, validates policies at design, code, and behavioral levels, and links executable artifacts to market, order-book, and settlement evidence.The conclusion reports sparse canonical acceptance on controlled tasks, none on the 24 preregistered real strategies, and variability in thresholds, sizing, and data choices across generations.Source: §6 · p. 9

B MOTIVATION

  • Prior financial benchmarks cover reasoning, alpha discovery, and trading code, but standardized assets and numerical rules do not capture prediction-market event scope, outcomes, information timing, and lifecycle policies.Executable code can still diverge from the intended hypothesis through implicit decisions about thresholds, directions, sizing, or exits.Source: B MOTIVATION · p. 13
  • Related prediction-market work evaluates trading agents, forecasting, or transaction understanding, whereas the paper targets construction of executable strategies from incomplete descriptions.Polymarket strategies may also require splitting collateral, merging positions, and redeeming resolved claims under market-specific rules.Source: B MOTIVATION · p. 13
  • The paper motivates end-to-end operationalization because incomplete strategy descriptions leave outcome, signal, threshold, sizing, order, and exit choices unresolved, while event alignment and settlement semantics also constrain valid behavior.A runnable program may therefore be semantically invalid or implement behavior that diverges from the intended strategy.Source: B MOTIVATION · pp. 13–14

C DETAILED CONSTRUCTION OF OUR DATA

  • Polymarket combines off-chain order matching with on-chain settlement and token operations; public market and resolution data plus archived CLOB depth support lifecycle reconstruction.Complete limit-order-book snapshots are not natively stored on-chain, so the paper archives depth from the CLOB data stream.Source: C DETAILED CONSTRUCTION OF OUR DATA · p. 14
  • The primary study restricts its window to June 1 through September 1, 2026, after the V2 migration, to reduce non-stationarity from changing contract, oracle, and fee regimes.Prediction-market contracts also vary in outcome space, resolution source, horizon, settlement rules, valid signals, and information timing.Source: C DETAILED CONSTRUCTION OF OUR DATA · p. 14

C.1 POLYMARKET DATASET

This section describes an auditable Polymarket data and replay pipeline linking source-preserving strategies with historical evidence while retaining data-quality and execution constraints. Its backtesting controls preserve strategy-trace, account-state, and settlement-timing requirements, and distinguish unavailable order-book evidence from genuine lack of liquidity.

C.1 POLYMARKET DATASET

  • The dataset reconstructs an aggressor-side tape from deduplicated OrderFilled events, while archived full-depth L2 books cover 2,093 of 2,208 hours (94.79%).Sports (47.07%) and crypto (25.62%) dominate market composition; 175,732,742 fills (95.70%) lack a usable intra-block index and use deterministic degraded ordering without queue-priority claims.Source: §C.1 · p. 15, Table 4 · p. 16
  • At the 2026-09-12 census, 1,323,315 of 1,348,555 traded markets (98.13%) carried consumable payout evidence, while 1,310,720 settled before September 1.A stricter population requires an in-window fill before settlement; payout evidence includes fractional vectors and does not establish on-chain redemption or profit.Source: §C.1 · p. 15, Table 3 · p. 16

D.1 REAL-WORLD STRATEGY SOURCES

  • The real-world track preserves provenance for 581 coarse strategy hypotheses, combining curated guides, independent LLM catalogs, and academic literature rather than assuming records are benchmark-ready.Acquisition requires direct, verifiable sources and a concrete edge, signal, traded outcome, position or order construction, entry, exit or settlement rule, and principal risk; candidates are deduplicated by economic mechanism.Source: §D.1 · pp. 16–17

D.2 RULES IN POLYMARKET

  • Polymarket actions include token-specific orders and non-CLOB split, merge, and redemption operations, while negative-risk conversion and multi-leg strategies require verified outcome mappings.The action vocabulary preserves grouped order legs and protocol operations rather than reducing them to generic BUY or SELL actions.Source: §D.2 · pp. 17–18
  • 581 source-preserving records were distilled into 106 canonical strategy families and 27 shared execution components spanning directional trades, quote management, grouped orders, and protocol operations.The source corpus combines hand-curated guides, LLM-generated catalogs, and academic literature; mechanism-level deduplication uses payoff source, primary signal, and state transitions.Source: §D.2 · p. 18

D.3 DETAILED STRATEGY GENERATION

  • Direct resolves open economic choices and returns design plus code in one invocation, whereas Staged validates a design before a second invocation implements it.Both arms share the task envelope and policy form; source-stated requirements are binding, while open parameters may be chosen by the model.Source: §D.3 · p. 18

D.4 CODE GENERATION

  • Candidates implement strategy logic in a restricted Python interface, and deterministic parsing, static checks, and behavioral tests retain semantic failures and unverified requirements as distinct outcomes.Direct code is not repaired by another model; Staged coders must reproduce the validated design without redesigning it.Source: §D.4 · pp. 18–19

D.5 BACKTESTING FRAMEWORK

  • Market eligibility is distinct from realized fill status, and point-in-time inputs exclude future trades, book states, resolution labels, and post-decision settlement information.V3 replays causal same-market, same-token trade prints after modeled latency; PML2 handles depth, spread, maker liquidity, queue position, and stateful order lifecycles using validated book evidence.Source: §D.5 · pp. 19–20
  • Unsupported position, protocol, or kill-switch actions fail closed, while cross-decision order changes require the target order to originate in the same hash-bound strategy trace.The trace adapter also tracks latency, expiration, partial states, reservations, and order-group outcomes.Source: §D.5 · p. 21
  • PML2 maintains separate candidate accounts, reserves required cash or inventory, journals matches, and processes settlement receipts and portfolio marks chronologically after fills.Stale, missing, crossed, or gapped books receive data-readiness or rejection states rather than ordinary no-fill observations.Source: §D.5 · p. 21

D.6 CONTROLLED DIAGNOSTIC TASKS

The controlled diagnostic set spans six prediction-market mechanisms and three specification-completeness levels, separating faithful implementation from tasks that leave economic choices open. Evaluator probes also reveal behavioral-checker blind spots, so it is treated as one evidence source rather than a complete semantic oracle.

D.6 CONTROLLED DIAGNOSTIC TASKS

  • 36 controlled diagnostic tasks span 12 base tasks across six mechanisms, each instantiated at L1–L3 specification levels.The mechanisms include complete-set trading, directional probability changes, inventory-constrained quoting, trade-flow following, external-signal residuals, and settlement or lifecycle actions.Source: §D.6 · p. 21
  • L1 fixes execution-relevant decisions, L2 leaves selected choices open, and L3 delegates most economic design while specifying the objective and interfaces.The levels measure specification completeness; the authors do not assume L3 is empirically harder than L1.Source: §D.6 · p. 21
  • External-signal tests use defined mappings rather than calibrated forecasts, while lifecycle tasks may remain inactive without source-required inventory.For L2 and L3, a reference implementation is one admissible completion, not a unique solution or profitability upper bound; constructed common states do not authorize inventory injection into historical replay accounts.Source: §D.6 · p. 21
  • The evaluator probes reveal blind spots on target outcomes and multi-step actions, plus overly restrictive behavior at two inventory boundaries.The local behavioral checker is therefore treated as one source of evidence, with canonical source-aware evaluation kept separate.Source: §D.6 · pp. 21–22

E.1 WORKED EXAMPLE: OPERATIONALIZING A VOLATILITY-ADAPTIVE MARKET-MAKING STRATEGY

The worked example traces a volatility-adaptive market-making idea through model-chosen operational rules and historical replay. It shows that executable code can still drift from source intent, while replay outcomes depend on the execution protocol and heterogeneous market episodes.

E.1 WORKED EXAMPLE: OPERATIONALIZING A VOLATILITY-ADAPTIVE MARKET-MAKING STRATEGY

  • The example is diagnostic rather than a demonstration of profitable market making: executable strategy code alone does not establish successful operationalization.The source itself leaves the volatility definition, thresholds, quote offsets, position size, and detailed exit policy unspecified.Source: §E.1 · pp. 22–23
  • The model operationalizes the source idea with an absolute one-step probability change, YES-only trading, and high- and low-volatility regimes defined by volatility and spread thresholds.The generated policy also specifies explicit sizing, quote, and order rules absent from the source description.Source: §E.1 · pp. 22–23
  • The generated policy departs from source intent by using one-step probability changes instead of an estimated volatility measure, keeping GTC orders in both regimes, and choosing quote prices that may be immediately marketable.The source instead motivates shorter quote lifetimes in volatile periods and maker-oriented post-only execution.Source: §E.1 · p. 23
  • The historical protocol evaluates two-hour market episodes with decisions once per minute, independent $1,000 accounts, and a market sample selected for data availability and coverage rather than profitability.The sample includes 77 market episodes, and the relaxed execution profile does not establish outcomes under true post-only queue dynamics.Source: §E.1 · pp. 23–24
  • Among 77 market episodes, 52 with at least one historical fill were financially evaluable under the replay contract; their mean net ROI was −3.59%, with 6 positive episodes and 1,116 fills total.These outcomes are a sensitivity evaluation of the generated decision rule, not a faithful measurement of passive spread capture.Source: §E.1 · p. 24
  • Q044 generated 81,164 simulated fills but ended at −377.19 USDC, illustrating that a common trade-flow rule encounters distinct contract and information dynamics across markets.The examples span a Federal Reserve decision, a football match, and an S&P 500 directional contract.Source: §E.1 · p. 24, Figure 5 · p. 25

F.1 EVALUATION METRICS

The benchmark evaluates both financial performance and prediction-market-specific executability. Its metrics distinguish economic outcomes from the availability of causal inputs, execution evidence, and settlement outcomes.

F.1 EVALUATION METRICS

  • Prediction-market executability is measured through Fill Ratio, Input Coverage, Execution Observability, Markout, and Settlement Coverage.These measures separate signal quality from data availability, execution feasibility, and final contract resolution.Source: §F.1 · p. 25
  • The two metric groups distinguish strategies that appear profitable from those that are both financially effective and historically executable.Financial performance is assessed under a common account and execution environment, while executability concerns support from available historical data and execution evidence.Source: §F.1 · p. 25
  • Financial metrics include Net ROI, Settled ROI, maximum drawdown, Sharpe Ratio, and Cost Drag, with short-horizon episode returns not annualized.Daily Sharpe is reported only with a complete daily NAV series; otherwise it is unavailable or labeled as a research equity-path statistic.Source: §F.1 · p. 25, Table 7 · p. 26

F.2 EXPERIMENTAL COHORTS AND AGGREGATION

The paper keeps distinct experimental populations and aggregation rules for generation, replay, and financial evaluation. Independent generations show limited canonical validity, while historical return paths and reproducibility diagnostics answer separate questions.

G.2 INDEPENDENT-GENERATION STABILITY

  • The latest independent-generation campaign covers 36 controlled tasks and 24 preregistered real-strategy tasks, with five repetitions per method and 600 full-chain attempts.Each Staged repetition generates a new Designer output before coding; invalid designs that block coding remain observed failures rather than being resampled.Source: §F.2 · p. 26, §G.2 · p. 27

F.2 EXPERIMENTAL COHORTS AND AGGREGATION

  • Financial outcomes are aggregated by first averaging eligible executions within each task and then weighting tasks equally; distinct missing, failed, and unevaluated states are retained.Metric-specific eligibility denominators and task-block bootstrap intervals preserve task-level aggregation.Source: §F.2 · pp. 26–27

G.1 HISTORICAL STRATEGY PATHS BY SPECIFICATION LEVEL

  • In the descriptive temperature-0.0 replay, episode returns clustered near zero with a downside tail, and neither arm changed monotonically across specification levels.The six cohorts are unpaired and unequal, and within-panel dispersion exceeds between-arm and between-level differences, so the figure does not support causal method comparisons.Source: §G.1 · p. 27, Figure 6 · p. 28

G.2 INDEPENDENT-GENERATION STABILITY

  • Controlled Direct and Staged generation produced 35/180 and 20/180 canonical passes, respectively, while the Real24 cohort had no confirmed canonical pass under either method.The authors caution that differing evidence requirements and evaluability across specification levels prevent interpreting the level-specific counts as a difficulty ranking.Source: §G.2 · p. 27, Table 9 · p. 29
  • Repeated designs are compared across nine economic slots, while common-state behavioral agreement is measured separately and does not establish source fidelity.Observed-pair coverage accompanies conditional agreement, and a HOLD–HOLD match alone is not evidence of faithful implementation.Source: §G.2 · p. 29
  • The raw-policy replay campaign retained 960 planned slots as distinct states, including 495 INPUT-UNAVAILABLE and 437 POLICY-NOT-RUNNABLE outcomes.Legacy research-overlay financial outcomes were not substituted for missing raw-policy evidence.Source: §G.2 · p. 29

G.3 MAIN-COHORT COVERAGE AND HISTORICAL PERFORMANCE

Historical replay coverage is high, but completion is distinct from financial eligibility and canonical generation acceptance. Financial outcomes differ across declared replay settings, so the paper reports coverage and task-equal returns without treating the settings as a paired performance comparison.

G.3 MAIN-COHORT COVERAGE AND HISTORICAL PERFORMANCE

  • Across four engine–method cohorts, 775,725 of 783,655 scheduled replay slots completed; completion did not by itself imply financial evaluability.The cohorts distinguish planned slots from unique executions, and completed replays without executable exposure or sufficient terminal valuation evidence are not assigned synthetic zero returns.Source: §G.3 · pp. 30–31, Table 11 · p. 31
  • PML2 task-equal mean Net ROI was −3.476% for Direct and −3.440% for Staged; V3 task-equal means were +0.836% for Direct and +0.646% for Staged.The V3 and PML2 distributions describe different routed candidate populations and execution assumptions, not a paired comparison of engine performance.Source: Table 12 · p. 31, §G.3 · pp. 31–32, Figure 9 · p. 32
  • Financial results use unique financially evaluable executions, averaged within each task before task-equal aggregation; missing outcomes and zero-fill episodes remain in their coverage states.Task-level mean returns, rather than pooled episodes, determine the reported distribution and positive-task rate.Source: §G.3 · p. 31
  • The financial evaluation establishes replay scale and coverage rather than a profitability claim; outcomes depend on the evidence and execution contract used for evaluation.Replay coverage, financial eligibility, and task-equal returns are reported jointly to keep evaluability distinct from performance.Source: §G.3 · pp. 29, 32
  • In the original main-cohort evaluation, neither method recorded a canonical CANDIDATEPASS, although 381 Direct and 75 Staged candidates were available for research replay.Research executability measures replay availability, not acceptance of the original generated implementation.Source: §G.3 · p. 32, Figure 10 · p. 33

G.4 METRIC AVAILABILITY AND EVIDENCE COVERAGE

Archived evidence supports broad coverage for main-cohort financial metrics but is more selective for microstructure statistics. Fee and execution analyses show that costs and depth affect measured outcomes, while the evidence and reference-price construction constrain interpretation.

G.4 METRIC AVAILABILITY AND EVIDENCE COVERAGE

  • Main-cohort financial metrics have broad evidence coverage, but quantity-fill ratios are available for only about one quarter of the same population.V3 supports Net ROI on more than 95% of its main replay slots; input-coverage and markout statistics cannot be reconstructed for the main archived cohorts.Source: §G.4 · p. 33

H ADDITIONAL FINDINGS

  • Fee diagnostics compare broad counterfactuals on recorded execution paths with paired replays that let fees and execution interventions alter later account decisions.The former provides broad sensitivity estimates; the latter tests when frozen-path accounting remains valid.Source: H ADDITIONAL FINDINGS · p. 34

H.1 TRANSACTION FEES IN POLYMARKET

  • Mean ROI decreases by 0.091 percentage points in PML2 and 0.719 percentage points in V3 under the category-policy fee counterfactual.Only 5/16,214 initially positive PML2 episodes cross zero, versus 7,799/82,200 (9.49%) in V3; these are frozen-path sensitivities, not reconstructed historical fee bills.Source: §H.1 · p. 34

H.2 EXECUTION-PRICE DEVIATION AND DEPTH

  • PML2 has a mean adverse execution deviation of 4.40% of initial capital and a median of 1.22%, while V3’s mean is close to zero.This does not establish superior V3 execution: the backends use different reference-price constructions, and the PML2 measure may combine spread crossing, depth consumption, latency, and stale references.Source: §H.2 · pp. 34–35
  • A 10-share Q055 order filled 0.63 shares at 25 cents and 9.37 at 75 cents, yielding a 71.85-cent VWAP and a 4.685-USDC ladder-cost diagnostic.The example illustrates why a quoted probability is not necessarily executable at an arbitrary trade size.Source: §H.2 · p. 35

H.3 MICROSTRUCTURE CORRELATES

  • Activity measures are strongly correlated (ρ ≈0.91–1.00), while median price jump and median impact correlate at ρ = 0.94.Effective depth is negatively associated with tail price jumps and event rates; these relationships are descriptive, not causal.Source: §H.3 · p. 35

H.4 PAIRED FEE-AWARE REPLAY AND EXECUTION SENSITIVITY

Paired fee-aware replays compare frozen-path accounting with native account and execution logic. They agree under ample capital, but fees and execution constraints can change subsequent trading paths; the evidence is limited to controlled sensitivity analyses.

H.4 PAIRED FEE-AWARE REPLAY AND EXECUTION SENSITIVITY

  • The paired study covers 529 completed configurations and three deterministic replication checks across 15 tasks and 19 task–market windows; fees, size, capital, latency, and capacity are controlled assumptions.These sensitivity analyses do not reconstruct every contract’s historical fee schedule, validate maker rebates, calibrate live millisecond latency, cover all 477 benchmark tasks, or use empirical V3 order-book quantiles.Source: §H.4 · pp. 35, 36, 39
  • With 1,000 USDC initial capital, a flat 100-bps fee reduces mean V3 PnL by 0.559 USDC, while the PML2 nonlinear schedule with r = 0.07 reduces mean PnL by 1.513 USDC.The two values are not directly comparable because the task cohorts and fee functions differ.Source: §H.4 · p. 36
  • Eight fee configurations across six task–market windows change the economic execution path when fees reduce cash available for subsequent trades.For TASK-C087-V01, 20 USDC initial capital and a 100-bps fee shift PnL from a frozen-path estimate of −0.20 USDC to −8.99 USDC under native replay; another case becomes less negative because some later losing fills are prevented.Source: §H.4 · p. 36, Figure 16 · p. 37
  • In V3, larger submitted size raises mean PnL over tested windows while a 100-bps fee reduces it at every size; PML2 mean PnL is negative throughout and declines as fee rate r increases.In PML2, increasing size from 0.25× to 1× materially changes outcomes, but 1× and 4× rows are nearly identical; the panels use different cohorts, execution models, and fee parameterizations.Source: §H.4 · pp. 36–37, Figure 17 · p. 38
  • In the O055 example, raising submitted size from 0.25× to 1× and 4× increases ladder-cost diagnostics from 0.935 to 4.685 and 19.685 USDC, while filled shares and final PnL remain unchanged.Each configuration acquires 43.5 shares and ends at −32.31 USDC; the diagnostic is not an additional strategy loss because its reference changes as liquidity is consumed. Latency effects were smaller in the tested PML2 sample and do not calibrate millisecond live trading.Source: §H.4 · pp. 37–38, Figure 18 · p. 38
  • Among 162 filled PML2 orders, median decision-book age was about 394 seconds, and only 30 had a non-crossed two-sided book no older than 120 seconds.Even for those observations, fill-to-midpoint difference may combine spread, depth consumption, residual liquidity, and repricing; only three orders had nonzero same-execution ladder cost, totaling about 4.77 USDC, so this is not a market-wide depth-cost estimate.Source: §H.4 · pp. 38–39, Figure 19 · p. 39
  • Under ample capital, population-scale frozen-fill accounting and native paired replay agree closely, whereas tighter capital constraints can change the execution path.The study includes 529 paired replays; its conclusion is that fees and execution frictions can affect strategies through direct cost and endogenous changes in feasible execution.Source: §H.4 · p. 39

I COMPUTATIONAL DETAILS; J ETHICAL STATEMENT

Related papers

Loading 2609.31390v1…