Source-linked AI summary
Agent Policy-Value Audit: Separating Transition Composition from Event Selection in Financial LLM Agents
Mingyang, Chen, Yida, Xu, Huiwen, Chen, Yiming Lu, Wei Jin
TL;DR
Financial-agent evaluations measure whether deployment changes realized returns but generally do not isolate event-selection skill from transition composition. The Agent Policy-Value Audit compares deployment value with a benchmark from random reassignment of the observed transitions; its tests reduce false attribution, while its retrospective application finds negative selection value.
Problem
Existing evaluations measure deployment performance but generally do not isolate whether an agent’s action changes were assigned to unusually favorable events.
Source: §1 · p. 1Method
The audit holds fixed the observed action-change counts and types, randomly reassigns them across eligible events, and separates deployment value into composition benchmark and selection value.
Source: §1 · p. 1Results
11.6% false attribution under a zero-centered paired test falls to 5.3% under the matched audit; retrospectively, +15.2 bps/event decomposes into +25.8 composition and −10.6 selection value.
Source: §1 · p. 2Takeaways & Limitations
Financial-agent evaluations should report deployment value separately from performance against the transition-composition benchmark.
Source: §7 · p. 7The audit requires policies scorable on the same events, observable payoffs for candidate actions, negligible market impact, and credible reassignment pools; it does not directly cover sequential policies that alter later states.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Financial LLM agents are often evaluated by comparing their end-to-end returns with those of a baseline and testing the paired difference against zero. This measures whether deploying the agent changes realized performance, but it does not isolate event-selection skill. An agent that frequently changes positions from flat to long can earn a positive paired return from an upward-drifting event pool even when it selects events at random. We propose the Agent Policy-Value Audit, which holds fixed the observed count of each ordered action-change type and randomly reassigns them across eligible events. The average payoff from these reassignments is the composition benchmark; the difference between observed deployment value and this benchmark is selection value. In semi-synthetic benchmarks based on real earnings-event returns, a zero-centered paired test falsely attributes passive exposure to selection skill in $11.6\%$ of no-skill replications, while the transition-matched audit reduces this rate to $5.3\%$. Applied retrospectively to 723 earnings events at 44 U.S. consumer-facing firms, the audit decomposes the agent's gross deployment value of $+15.2$ bps/event into a $+25.8$ composition benchmark and a $-10.6$ selection value. The agent does not detectably outperform matched random assignments. Financial-agent evaluations should report deployment value separately from event-selection value.
1 Introduction
Existing financial-agent evaluations measure deployment performance but generally do not isolate whether action changes were assigned to unusually favorable events. The Agent Policy-Value Audit holds transition composition fixed to separate composition from selection value.
1 Introduction
- Most existing evaluations ask whether deploying the assembled system improves realized performance, while generally leaving event-selection skill unisolated.End-to-end returns, deflated metrics, multiple-testing corrections, and walk-forward designs address other evaluation questions.Source: §1 · p. 1
- A zero-centered paired test can mistake an agent’s transition composition for selection skill when eligible event pools have nonzero average returns.For example, randomly assigned flat-to-long changes can earn positive expected payoff in an upward-drifting event pool.Source: §1 · p. 1
- The audit holds fixed observed action-change counts and types, randomly reassigns them across eligible events, and defines the reassignment average as the composition benchmark.Deployment value splits into composition benchmark plus selection value; only selection value reflects which events received the changes.Source: §1 · p. 1
- +15.2 bps/event of gross deployment value decomposes into a +25.8 composition benchmark and a −10.6 selection value; the matched audit reduces false attribution from 11.6% to 5.3%.The retrospective application covers 723 earnings events at 44 U.S. consumer-facing firms; the selection estimate is not detectably different from matched random assignments.Source: §1 · p. 2
2 Related Work
Prior financial-agent research evaluates assembled systems, return predictions, agent-as-judge benchmarks, behavioral biases, and structured workflows. The paper distinguishes its audit from policy evaluation and randomization inference by conditioning the reference on realized action transitions.
2 Related Work
- Financial-agent evaluations commonly backtest assembled systems or assess standalone return prediction, while recent work also studies agent judgments, behavioral biases, and structured workflows.These approaches do not condition attribution on the realized composition and timing of action transitions.Source: §2 · p. 2
- Unlike off-policy evaluation, the audit scores both policies on the same events when candidate-action returns are observable, but its reassignment inference depends on exchangeability.The paper frames its contribution as an evaluation target and audit protocol, not a new permutation algorithm.Source: §2 · p. 2
- The composition/selection split echoes characteristic-matched performance attribution, while the paper adds exposure attribution and decision-time-overlap checks to backtest-integrity tools.The cited related work also situates the empirical design in event-study, earnings-predictor, and investor-attention research.Source: §2 · p. 2
3 The Agent Policy-Value Audit
The audit separates deployment value from selection value by comparing observed performance with a reassignment benchmark that preserves the agent’s transition composition. Its inference depends on within-pool exchangeability, while its decomposition remains a descriptive benchmark when that assumption is contested.
3.1 Evaluation protocol
- The audit fixes each transition type’s count within eligible pools and randomly reassigns which events receive those transitions.The resulting composition benchmark is what the observed transition mix earns without event selection; selection value is the observed value beyond that benchmark.Source: §3.1 · p. 2
- The audit compares observed deployment value with transition-matched reassignment; firm-clustered uncertainty and future-only scoring under decision-time overlap are safeguards.The composition benchmark holds transition counts fixed, so intervention frequency and transition-type choice accrue to it rather than selection value.Source: Figure 1 · p. 3, §3.2 · p. 3
3.2 Inference and scope
- A zero-centered test is appropriate only when count-weighted pool averages cancel; under no selection skill, reassignment instead centers deployment value at the composition benchmark.Selection value is centered at zero under the no-skill reassignment distribution, even though deployment value generally is not.Source: §3.1 · p. 3, §3.2 · p. 3
- 11.6% versus 5.3%: the zero-centered paired test’s false-attribution rate exceeds the transition-matched audit’s rate in the reported no-skill replications.The passage reports these rates as .116 and .053 at this sample size.Source: §3.2 · p. 3
- Formal randomization inference assumes within-pool exchangeability, no market impact, and randomized cells containing both changed and unchanged eligible events.Exchangeability is the substantive assumption and cannot be established by the audit itself; the decomposition remains a descriptive benchmark when it is contested.Source: §3.2 · p. 3
4 Empirical Setting and Frozen Experiment
The empirical analysis applies a frozen experiment retrospectively to earnings events from a prespecified cohort of U.S. consumer-facing firms. It compares a multi-role LLM agent with a frozen-rule baseline, alongside registered variants and cost sensitivities.
4.1 Design status and timeline
- The policy-value decomposition and transition-matched audit were developed after traces were frozen, so the empirical analysis is retrospective rather than confirmatory.The policy arms, action mappings, scoring rules, and confirmatory comparisons were preregistered before execution; some remaining analyses are exploratory or post-registration.Source: §4.1 · p. 3
4.2 Data and policies
- The committed main cohort contains 44 consumer-facing firms, with earnings-event ground truth measured as a three-day Carhart four-factor CAR; events failing estimation or return-window requirements are excluded.Only 166 calendar-2024 events postdate the in-house backbone’s December 2023 knowledge cutoff.Source: §4.2 · pp. 3–4
- Social inputs are timestamp-filtered to a point-in-time window, while retrospective Google Trends features motivate a prespecified no-Trends rerun.X and Reddit inputs end at t−2; X is capped at the 1,500 most-recent matching posts per event.Source: §4.2 · p. 4
- The full agent uses Scout, Strategist, and Reviewer roles over a deterministic forecaster, and its headline comparison is against the registered frozen-rule baseline.A separate factorial crosses deterministic versus LLM policy class with structured-only versus five prespecified social aggregates.Source: §4.2 · p. 4, Table 2 · p. 4
- Net value charges 40 bps per nonzero event position, while gross deployment value is primary and 20/40/60 bps overlays are reported as sensitivities.The stated overlay values under the switching-cost convention are +6.6/−2.1/−10.7 bps/event.Source: §4.2 · p. 4
5 Evaluator Validation
The evaluator validation tests whether the audit recovers injected selection skill, prevents exposure and timing misattribution, and remains reliable under stress tests and out-of-period replication.
5 Evaluator Validation
- The validation evaluates skill recovery, exposure attribution, timing overlap, and exchangeability through benchmark construction, evaluator-component tests, and stress tests.The pre-2020 replication is an evaluator replication, not a held-out model test.Source: §5 · p. 4, §5.4 · p. 6
5.1 Benchmark construction
- The semi-synthetic benchmark preserves real returns, eligible baseline-action pools, and realized transition counts, varying only transition-to-event assignment across four scenarios.The scenarios include a centered null, exposure confound, known selection, and timing overlap.Source: §5.1 · p. 4
5.2 Recovery of known selection skill
- The audit recovers injected selection value along the identity line, with zero crossing at the pool’s base accuracy of 0.523 rather than q=0.5.At q=0.60, matched-test power reaches .783 within Monte Carlo error.Source: §5.2 · p. 5, Figure 2 · p. 5
5.3 Evaluator-component benchmark
- Conditioning on realized transition composition reduces exposure false attribution from 11.6% to 5.3%, while future-only scoring removes the timing artifact (.648 → .000).Correct centering costs power (.783 vs. .846–.917 for over-rejecting procedures).Source: §5.3 · p. 5
5.4 Stress tests and out-of-period replication
- As premium scale, sample fraction, and intervention rate grow, zero-centered false attribution reaches .88, while the audit stays near nominal at .04–.07.The out-of-period replication finds false attribution of 8.4% for the zero-centered test versus 4.9% for the matched audit; power at q=0.60 is .65 on the smaller table.Source: §5.4 · pp. 5–6
6 Empirical Agent Audit
The audit finds that positive gross deployment value is explained by transition composition rather than a detectable event-selection advantage. Robustness analyses preserve that attribution while showing the limits of the estimates and the external stress test.
6.2 Selection attribution
- The design reaches 80% power only at approximately 142 bps/intervention, so the null result is informative against large selection skill but not small effects.The 2024 post-cutoff subset’s interval remains too wide to exclude meaningful differences.Source: §6.2 · p. 6, §6.3 · p. 7
6.1 Deployment value
- +15.2 bps/event gross deployment value has a firm-clustered 95% CI of [−45.2, +69.2], while net value is −1.2 bps/event.The one-sided test that net value is at most zero does not reject (p=0.52).Source: Table 5 · p. 6, §6.1 · p. 6
6.2 Selection attribution
- +15.2 bps/event decomposes into a +25.8 composition benchmark and −10.6 selection value; the agent does not outperform matched assignments detectably.The observed payoff is below the matched mean, with limited power to detect small effects.Source: §6.2 · p. 6, Figure 4 · p. 7
6.3 Integrity and robustness
- Across reported robustness checks, no in-house pair shows a detectable event-selection advantage, and all 44 leave-one-firm-out reruns keep selection at or below the matched-null median.The gross intervention payoff is concentrated: 13 interventions in five excluded cells have an included-support mean of 8.1 versus 35.5 bps/intervention.Source: §6.3 · pp. 6–7, Table 8 · p. 7
6.4 External stress test
- TradingAgents’ future-only social contrast is −18.5 bps/event ([−100.1, +56.1]); announcement-aligned rescoring reduces the conventional contrast from +16.8 to +2.0.The post-cutoff 2024 subset agrees with the imprecise future-only contrast.Source: §6.4 · p. 7
7 Conclusion
Paired returns measure whether deployment changes realized outcomes, not whether the agent selected the right events. The audit separates deployment value from event-selection value, supporting explicit reporting of both for event-aligned evaluations.
7 Conclusion
- Paired returns do not establish that an agent selected the right events; report deployment value separately from performance against the transition-composition benchmark.Controlled benchmarks reduce false attribution from 11.6% to 5.3%, while the empirical application finds negative selection value statistically indistinguishable from zero.Source: §7 · p. 7
- The transition-composition benchmark makes the event-selection attribution target explicit for event-aligned financial-agent evaluations.The benchmark preserves observed transition composition when constructing the no-selection reference.Source: §7 · p. 7
- The empirical audit separates a positive gross deployment estimate from a negative selection estimate that is statistically indistinguishable from zero.Source: §7 · p. 7
8 Discussion and Limitations
The audit’s interpretation depends on its setting and assumptions. The paper cautions that the selection result is conditional on the benchmark and conditioning set, and that return-based procedures do not address leakage.
8 Discussion and Limitations
- The audit requires same-event scoring, observable payoffs for candidate actions, negligible effects of audited actions on outcomes, and credible reassignment pools.It does not directly cover settings where rewards for unchosen actions are unobserved or sequential actions alter later states.Source: §8 · p. 8
- Selection attribution assumes exchangeability rather than verifying it, and the simulations probe that assumption only on this event table.The application is retrospective, covers one event class at 44 firms, and lacks a held-out 2025 confirmation.Source: §8 · p. 8
- The results neither establish zero agent value nor validate exchangeability; selection value remains conditional on the benchmark and conditioning set.Leakage diagnostics remain necessary because a leaked feature invalidates return-based procedures, including the audit.Source: §8 · p. 8
Reproducibility, Ethics, and Data
The authors state that reported tables and figures are reproducible from de-identified intermediate data, with release of specified artifacts upon acceptance subject to data licenses.
Reproducibility, Ethics, and Data
- Reported tables and figures are reproducible from de-identified intermediate data.Source: Reproducibility, Ethics, and Data · p. 8
- Upon acceptance, the authors will release the benchmark generator, analysis code, frozen prompt hashes, preregistration record, deviation log, and per-decision traces, subject to data licenses.WRDS inputs will not be redistributed; released artifacts aggregate public posts per event and exclude user identifiers and post-level text.Source: Reproducibility, Ethics, and Data · p. 8
- The paper describes itself as a scientific evaluation, not investment advice.Source: Reproducibility, Ethics, and Data · p. 8