Source-linked AI summary
CrashGym: Measuring the Decision Value of Experimentation for LLM Agents in Counterfactual Market Crashes
Haocheng Lu, Haoxuan Cheng, Mingyang Chen, Minjun Zhu
TL;DR
Benchmarks rarely measure whether experiments improve the decisions they inform. CrashGym uses a replayable market simulator to compare decisions with and without laboratory access, finding lower regret for Luna with access on newly sampled worlds. It treats experimentation’s value as avoided decision loss while noting that the evidence is bounded to one synthetic crash construction and two interventions.
Problem
Many evaluations score experiment processes or products, while measuring decision benefit requires comparing the same decision-maker with and without access and observing outcomes for untaken actions.
Source: §1 · p. 1Method
Results
5.092 utility units: laboratory access lowered Luna’s mean regret on 24 newly sampled worlds, meeting the prespecified criterion.
Source: §1 · p. 2Takeaways & Limitations
The study treats experimentation’s value as avoided intervention loss and tests simulator detail by whether it improves downstream choices.
Source: §7 · p. 10, §8 · p. 11Evidence covers one synthetic crash construction and two interventions, and supports transfer within that construction rather than to other constructions or historical markets.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Language-model agents are increasingly asked to run experiments, yet benchmarks rarely measure whether those experiments improve the decisions they are meant to inform. We introduce CrashGym, a deterministic, replayable limit-order-book simulator in which an agent must choose between a deleveraging intervention and a liquidity backstop for a market about to suffer a shock-triggered crash. Before committing, the agent may query intervention experiments in a laboratory realization of the market. Its choice is then scored on an independently seeded target realization in which the simulator executes every feasible intervention, so regret relative to the best intervention is computed by directly simulating every alternative on the same target repeats, with no human or model-based judging. The simulator implements each amplification channel---persistent price impact, liquidity withdrawal, margin-driven liquidation, and an evolving strategy mix---as an explicit mechanism that can be switched on in stages. In a prospectively specified study on 24 newly sampled worlds, laboratory access reduces the mean regret of the one deployment with a complete record (Luna) from 8.878 to 3.785, a paired reduction of 5.092 utility units (nominal 95% world-bootstrap interval: 1.028--9.507), or 57.4% of the OBSERVE-ONLY mean. Prospective comparisons for two further deployments are incomplete; in a separate exploratory study, mean regret is lower with laboratory access for all four deployments evaluated. A laboratory whose strategy mix evolves endogenously yields no established advantage over one with a fixed mix. CrashGym thus measures the value of experimentation as avoided decision loss and lets simulator detail be judged by the decisions it supports.
1 Introduction
CrashGym asks whether laboratory experiments help an agent choose between deleveraging and a liquidity backstop when market amplification channels determine which intervention is better. It measures that benefit through exact counterfactual regret on an independently seeded target.
1 Introduction
- Benchmarks often score experiment processes or products, but measuring whether experiments improve decisions requires observing the same decision-maker with and without access and knowing outcomes for untaken actions.The latter counterfactual is unavailable in historical data.Source: §1 · p. 1
- CrashGym provides a deterministic market laboratory with replayable exchange mechanics and staged amplification channels, where agents choose between deleveraging and a liquidity backstop.Those interventions are defensible a priori; which is better depends on the market’s dominant amplification channel.Source: §1 · p. 1
- The task scores an intervention on an independent target where every feasible action is executed, yielding exact counterfactual regret for each decision.Laboratory experiments and target outcomes use disjoint seed cohorts; the target outcomes remain hidden from the agent.Source: §1 · p. 2, Figure 1 · p. 2
- 5.092 utility units: laboratory access lowered Luna’s mean regret on 24 newly sampled worlds, meeting the prespecified criterion.An evolving laboratory strategy mix yielded no established additional benefit.Source: §1 · p. 2
2 Related work
Related work evaluates language-model experimentation, financial decisions, and market simulation, while CrashGym focuses on how experiments improve a subsequent decision. Its synthetic market supplies counterfactual intervention outcomes unavailable in historical data.
2 Related work
- CrashGym measures how much experiments improve a subsequent independently scored decision, rather than whether agents recover latent mechanisms or complete research tasks.Related evaluations include ScienceWorld and DiscoveryWorld [16], MLAgentBench [15], CausaLab and CausalGame [5] [32], and end-to-end paper-generation systems [22].Source: §2 · p. 2
- Financial-agent evaluations typically use historical data, where outcomes of untaken actions are unavailable; CrashGym supplies those counterfactuals using synthetic rather than historical markets.Prior work includes domain-adapted financial models and agents [30] [31] [34] and InvestorBench [20].Source: §2 · p. 2
- Unlike approaches that maximize realism, CrashGym uses a simulator to supply exact counterfactual intervention outcomes and switch amplification channels on or off.The crash construction draws on leverage-cycle and fire-sale research.Source: §2 · p. 2
- CrashGym empirically compares laboratory designs but does not test Bellman-operator equivalence, and its setting also relates to sim-to-real transfer [27].Source: §2 · pp. 2–3
3 The CrashGym simulator
CrashGym is a discrete-time, single-asset limit-order-book market built to model shock-triggered crashes through explicit, switchable mechanisms. Its fixed episode schedule and deterministic replay support controlled comparisons across staged market configurations.
3 The CrashGym simulator
- CrashGym models a single-asset market with a central limit order book and makes crash amplification channels explicit and switchable in stages.The stages add persistent impact, liquidity withdrawal, margin-driven liquidation, and an evolving strategy mix cumulatively.Source: §3 · p. 3, §3.6 · p. 5
3.1 Exchange core
- Price–time priority governs limit-order matching, while integer accounting, risk checks, and audited invariants constrain exchange operations.Marketable orders are limit orders priced through the opposite best quote; unfilled quantities are cancelled immediately.Source: §3.1 · p. 3
3.2 Participants
- The market maker, noise traders, trend and fundamental traders, leveraged trader, and adaptive participant shape order flow; maker depth falls at the shock.Maker depth is 400 lots during steps 1–7 and returns to world-specific depth from the shock onward.Source: §3.2 · p. 3
3.3 Episode schedule
- A 96-step episode builds leveraged exposure, applies the intervention at step 4, and triggers the market shock at step 8 before recording the objective over time.Within each step, operations follow a fixed order, from maker re-quoting through end-of-step price and objective recording.Source: §3.3 · p. 3, Figure 2 · p. 4
3.4 Amplification mechanisms
- The shock can amplify into a crash through persistent price impact, stress-driven liquidity withdrawal, and margin-triggered forced selling.Forced sales lower prices and equity, so liquidation can repeat across steps; the withdrawal and liquidation strengths define the sampled world strata.Source: §3.4 · pp. 4–5
3.5 Reflexive strategy ecology
- An adaptive participant chooses between fundamental and trend strategies, and an evolving mix updates strategy shares using one-step counterfactual profits.Shadow trades run each candidate on a copy of the exchange; interventions can therefore affect later order flow through the strategy mix.Source: §3.5 · p. 5
3.6 Switchable mechanism stages
- The experiments compare a fixed-mix laboratory at the leverage stage with an adaptive laboratory at the full evolution stage, which is also the target configuration.The first three cumulative mechanism stages are implemented but not used in the reported experiments.Source: §3.6 · p. 5
3.7 Determinism, replay, and seed hygiene
- Address-based counter-based randomness makes draws independent of call order, while event logs support state reconstruction and seed commitments prevent reuse.Runs sharing a seed use identical random inputs; the prospective study used 384 unique seeds.Source: §3.7 · p. 5
4 The decision task
The decision task asks whether laboratory experiments help an agent choose between deleveraging and a liquidity backstop. It evaluates that choice using target outcomes for every feasible intervention, so regret is grounded in the realized target repeats.
4.1 World population
- Worlds vary shock and market parameters and are stratified by whether liquidation, withdrawal, both, or neither amplification mechanisms are active.Candidate worlds are rejected only for schema violations or duplicate constructors, not based on outcomes.Source: §4.1 · p. 6
4.2 Interventions
- The agent chooses between a one-time position-halving deleveraging sale and a backstop that posts a resting bid from step 4 onward.Both interventions take effect four steps before the shock; regret is defined only for feasible chosen actions.Source: §4.2 · pp. 6–7
4.3 Utility and regret
- Regret is the target-cohort utility gap between the chosen action and the best feasible action, computed by executing alternatives on the same target repeats.It is exact counterfactual ground truth for the eight realized repeats, but a finite-sample estimate of a world's expected regret; utility is in simulator units.Source: §4.3 · p. 7
4.4 Experiments and observations
- Paired laboratory experiments compare each intervention with a control using the same seeds, while NULL compares the control with itself.Packets aggregate public statistics and paired utility differences over eight laboratory repeats, without revealing target-cohort outcomes.Source: §4.4 · p. 7
5 Experimental protocol
The protocol compares observation-only decisions with laboratory access under several information conditions and evaluates each deployment separately. It prospectively samples new worlds for confirmation, alongside a separate exploratory study on previously examined worlds.
5.1 Information conditions
- All conditions are scored on adaptive targets, with observation-only, adaptive-lab, fixed-mix-lab, and placebo-tool information conditions.The placebo returns a fixed packet from another world; adaptive versus fixed-mix laboratory comparisons vary the evolving mix and stated laboratory configuration.Source: §5.1 · pp. 7–8
5.2 Agents and interaction
- Three served variants of ChatGPT 5.6.4 are treated as separate deployments, with GPT-5.5 included only in the exploratory study.Agents have a bounded interaction and experiment budget, and target outcomes and decision correctness are not revealed to them.Source: §5.2 · p. 8
5.3 Studies and statistical analysis
- The prospective study planned 576 decisions across 24 newly sampled worlds, two transfer directions, four conditions, and three deployments.The prespecified criterion required a negative paired mean, a nominal 95% whole-world bootstrap interval below zero, and all 48 pairs; deployments were analyzed separately without pooled or multiplicity-adjusted claims.Source: §5.3 · p. 8
- A separate exploratory study used 24 previously examined worlds and four deployments, with intervals specified after outcomes and treated as descriptive.It comprised 576 decisions across two directions and three conditions.Source: §5.3 · p. 8
6 Results
In the prospective study, Luna’s laboratory access reduced mean regret relative to observation alone, while the prospective comparisons for Sol and Terra remained unresolved. Controls and exploratory analyses qualified that result: the fixed-mix laboratory also beat observation alone, but placebo and laboratory-design contrasts were inconclusive; exploratory results favored laboratory access across deployments.
6.1 Prospective confirmation on new worlds
- 8.878 to 3.785 mean regret under OBSERVE-ONLY and ADAPTIVE-LAB, respectively, gives a paired reduction of 5.092 units on 24 new worlds.This 57.4% reduction met the prespecified primary criterion; Figure 4 presents mean regret by condition and paired differences.Source: §6.1 · p. 8, Figure 4 · p. 9, Table 2 · p. 9
- Sol and Terra’s planned primary comparisons remain unresolved, with only 20 and 16 complete primary pairs, respectively, out of 48.Their scored decisions were 88 and 82 of 192, and the study did not substitute complete-case estimates.Source: §6.1 · p. 8
6.2 What the controls establish
- FIXED-MIX-LAB also improved on OBSERVE-ONLY, but contrasts against PLACEBO-TOOLS and between the two laboratories were inconclusive.The fixed-mix versus observation contrast was −6.218; the adaptive-versus-placebo and fixed-versus-placebo intervals included zero, leaving the source of benefit uncertain.Source: §6.2 · pp. 8–9, Figure 4 · p. 9
- The evolving strategy mix showed no established advantage over the fixed mix, and its interval admitted both benefit and cost.The prospective ADAPT−FIXED difference was 1.126 with interval [−1.983, 3.756]; exploratory ADAPT−FIXED intervals also included zero.Source: §6.2 · p. 9
- The primary evidence concerns avoided utility loss rather than accuracy: optimal-action counts rose from 30 to 37 of 48, but binary tests missed their directional criterion.The regret measure weights mistaken choices by their target utility gap.Source: §6.2 · p. 9
- Luna used similar numbers of calls across laboratory and placebo conditions, and no episode exhausted the eight-call budget.The complete experiment menu was observable in every condition, so the reported improvement did not come from more calls.Source: §6.2 · p. 9
6.3 Exploratory evidence across deployments
- Exploratory mean regret was lower with laboratory access for all four deployments, but the previously examined worlds make these results motivating rather than confirmatory.Post hoc ADAPT−OBS intervals excluded zero for Sol, Terra, and GPT-5.5, but included zero for Luna.Source: §6.3 · pp. 9–10, Figure 5 · p. 10
6.4 How much decision information does each laboratory retain?
- Both laboratories preserved the target-optimal choice in nearly all 48 exploratory units, but this diagnostic gave selectors full laboratory utilities unlike the LLM agents.It measures decision information retained by each laboratory, not a comparison under equal information.Source: §6.4 · p. 10
7 Discussion
The discussion defines experimentation’s gross value as avoided intervention loss and uses decision outcomes to assess which simulator details matter. The evidence is scoped to the tested construction, with further measurement needed to distinguish experiment selection from benefit from informative packets.
7 Discussion
- 5.092 utility units is Luna’s loss avoided relative to observation alone with the adaptive laboratory, reported as V8.The proposed resource-efficiency analysis would weigh this value against measured acquisition and inference costs.Source: §7 · p. 10
- The prospective ADAPT−FIXED comparison does not establish an advantage for an evolving strategy mix; its interval admits either sign.Complete-utility selectors from both laboratories recover nearly all target-optimal actions.Source: §7 · p. 10
- The evidence covers one synthetic crash construction and two interventions with fixed implementations, and regret is not calibrated to monetary outcomes.Transfer is supported within this construction, not to other constructions or historical markets.Source: §7 · p. 11
- A strong fixed acquisition schedule with the same observations and decision rule would test experiment selection beyond benefiting from informative packets.Source: §7 · p. 11
8 Conclusion
CrashGym links experimentation to intervention choices using independent target realizations and staged market mechanisms. Prospective results support a laboratory-access benefit for Luna, while the exploratory study reports the same descriptive direction across four deployments.
8 Conclusion
- CrashGym connects acquired evidence to intervention choices by scoring regret on independent target realizations.Source: §8 · p. 11
- Its staged amplification mechanisms make simulator detail testable by its contribution to a downstream choice.Source: §8 · p. 11
- Prospective results support a laboratory-access benefit for Luna; a separate exploratory study reports lower mean regret with access for all four deployments.Source: §8 · p. 11
B Protocol and evidence details
The protocol appendix details deployment records, sampling and outcome handling, replay, and limits on interpreting the measured results.
B Protocol and evidence details
- Exploratory Terra results combine 143 decisions under one recorded configuration with one OBSERVE-ONLY decision under another; checkpoint equivalence was not independently verified.Source: B Protocol and evidence details · p. 12
- Episodes can terminate without an action after an unparseable response or output-token limit; provider-side reasoning is not returned in later turns.Source: B Protocol and evidence details · p. 12
- The prospective sample has six worlds in each of four strata, and its intervals are nominal, deployment-specific, and not joint or multiplicity-adjusted.Exploratory intervals are post hoc and descriptive.Source: B Protocol and evidence details · pp. 12–13
- The prospective census includes 214 missing outcomes; condition-dependent missingness means complete pairs are not a random subsample, and full-denominator comparisons for Sol and Terra are not replaced by complete-case estimates.The construction fixes the backstop size at 40 lots.Source: B Protocol and evidence details · p. 13, Table 5 · p. 13, Table 6 · p. 14
- Independent replay reproduces all 362 available prospective scores while preserving 214 missing outcomes; a public artifact package has not yet been released.Source: B Protocol and evidence details · p. 13
- Neither prespecified direction-specific optimal-action test meets its criterion, despite the supported primary regret reduction; successful packet use does not prove latent causal-structure recovery.Source: B Protocol and evidence details · p. 13