PaperlayerView exact revision on arXiv

Source-linked AI summary

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

Weicheng Xue

arXiv:2609.32379v1cs.LGcs.AIcs.CEcs.SE

TL;DR

The paper asks what controls are needed to interpret execution performance and audit scores in LLM agent evaluations. It uses fixed-response execution replay and matched multi-defect audit tasks, finding that the resulting scores support narrower claims than independent-run comparisons or target recall alone might suggest.

  • Problem

    Execution outcomes and audit findings are scored, but neither score by itself specifies the scope of claims it supports.

    Source: §1 · p. 1

  • Method

    Study A replays stored response tapes across execution destinations, while Study B evaluates matched zero-, one-, and two-defect tasks with audit-specificity measures.

    Source: §1 · p. 2, §5.2 · p. 4, §6.2 · p. 7

  • Results

    −0.0170 in total return is the fixed-response execution contrast, while the auditor including both targets returns the exact dual-defect set in only 21/100 cases.

    Source: §9 · pp. 11–12, §7 · p. 11

  • Takeaways & Limitations

    A performance score supports only claims that its controls identify and its scoring covers; activity and ranking uncertainty should accompany execution contrasts, and specificity should accompany recall.

    Source: §9 · p. 12

    The studies use synthetic settings and a limited construction: Study A shares one upward market phase and Study B tests two API snapshots and one coupled violation-plus-edit design.

    Source: §8 · p. 11

Abstract

from arXiv · show

What controls are needed to interpret execution performance and audit scores in LLM agent evaluations? We study two limits on these interpretations in a financial agent harness. In Study~A, comparing independent runs under idealized and stressed execution on three synthetic settings that share one 24-day upward phase mixes the execution rule with fresh model responses and portfolio feedback: the parsed decision paths agree in only $19.8\%$ of $450$ pairs. Replaying each stored response tape through both execution destinations gives a narrower result. Conditional on those responses, stressed execution changes total return by $-0.0170$ (95\% interval $[-0.0230,-0.0117]$), or $10.4\%$ of the idealized baseline, and ten seed clusters do not resolve the model ranking. Study~B corrects an incomplete answer key and replaces legacy tasks with matched zero-, one-, and two-defect tasks under an explicit multi-label prompt. The drop in target violation recall from one to two defects is positive in five of six combinations of auditor and source (median $0.267$), with three surviving Holm correction. Yet the auditor that includes both target labels most often has micro-precision $0.149$, emits findings on $98/100$ zero-defect tasks, and returns the exact dual-defect set in only $21/100$ cases. Target recall by itself therefore gives a poor account of audit quality on this construction. The studies address different limits: what an execution comparison estimates, and what target recall captures. Together, they show how fixed conditions and diagnostic controls bound the claims a score can support.

1 Introduction

The paper asks what controls let execution and audit scores support appropriately bounded claims. It distinguishes a fixed-response execution contrast from matched multi-defect auditing, against related work on agent evaluation and measurement fragility.

1 Introduction

  • Study B replaces an incomplete answer key and single-defect tasks with matched zero-, one-, and two-defect tasks, finding recall loss alongside poor specificity.The auditor including both target labels most often returns the exact dual-defect set in only 21/100 cases.Source: §1 · pp. 1–2
  • The two studies distinguish whether an execution comparison estimates its intended effect and whether a scored audit success captures audit quality.The paper frames both as limits on claims drawn from agent evaluation scores.Source: §1 · pp. 1–2
  • Parsed decision paths agree in 19.8% of 450 independent-run pairs, motivating fixed-response replay to isolate the execution mechanism.The replay estimates a conditional mechanical contrast and quantifies its uncertainty.Source: §1 · pp. 1–2

2 Related Work

  • Prior financial-agent work studies agent designs and trading performance, while execution research models trading costs and backtest research addresses strategy-search effects.This paper treats execution destination as an experimental factor in a replay that holds realized responses fixed.Source: §2 · p. 2
  • LLM-judge research documents evaluation biases, while auditor benchmarks and mutation-testing work motivate testing co-occurring defects and reporting specificity with target recall.The paper identifies single-defect corpora as unable to estimate behavior on co-occurring defects.Source: §2 · pp. 2–3
  • Evaluation-fragility research reports score and ranking changes under prompt, extraction, and repeated-trial variation; this report examines structural execution and audit conventions.Recorded-decision replay is established rather than presented as a contribution of this report.Source: §2 · p. 3

3 Two Boundaries Between Scores and Claims

The section separates two boundaries between evaluation scores and the claims they can support. It proposes identifying what each comparison or score measures, then adding controls for what remains open.

3 Two Boundaries Between Scores and Claims

  • Both studies identify a claim, state what the original comparison or score measures, add controls, and limit conclusions to the resulting evidence.The studies share this reasoning, not a single source of error.Source: §3 · p. 3
  • A fixed-response replay estimates a narrower mechanical contrast than an independently sampled closed-loop comparison, which also includes response and portfolio-state changes.Path disagreement shows responses were not held fixed, but does not by itself establish bias; the observed diagonal is descriptive.Source: §3 · p. 3
  • Target recall measures reporting of a specified violation, not unsupported findings or complete audit correctness.Matched zero-, one-, and two-defect cases, complete keys, precision, and exact-set accuracy distinguish coverage from correct auditing.Source: §3 · p. 3

4 Shared Substrate and Protocol

The studies use a shared codebase to record agent actions and artifacts, then replay execution or audit injected trajectory defects. The protocol also records model provenance and response reuse.

4 Shared Substrate and Protocol

  • A shared codebase serializes signals, target weights, risk-gate revisions, order conversion, simulated execution, and portfolio updates.Study A replays the execution stage; Study B audits injected trajectory defects.Source: §4 · p. 4
  • The studies record direct-provider and API model identifiers, but identifiers without dated revisions denote provider snapshots, not immutable model versions.Study A uses five identifiers; Study B uses deepseek-v4-pro and glm-5.Source: §4 · p. 4
  • Run manifests and content digests support consistency checks, while released scripts compare derived tables against recorded values.The passage directs readers to the Reproducibility statement for the checks' exact scope.Source: §4 · p. 4
  • Provider responses are cached with per-task checkpointing, allowing reported analyses to reuse stored responses and fixed-tape replay to run without new provider calls.The multi-label collection contains 600 responses; corrected self-consistency uses 720 responses at temperature 0.7.Source: §4 · p. 4

5 Study A: A Fixed-Tape Contrast for Execution Mechanics

Study A separates an execution-only contrast from independently sampled closed-loop comparisons, which also change response paths and portfolio states. On the fixed-response replay, stressed execution lowers total return, but ten seed clusters do not resolve model rankings.

5.1 What independent runs compare

  • 19.8% of 450 independently sampled E0/E1 pairs share a parsed 24-period decision path, so the diagonal comparison changes more than execution.The matrix has five recorded model identifiers, three synthetic settings sharing a 24-day upward phase, ten seeds, and three provider samples.Source: §5.1 · p. 4

5.2 The fixed-response control and its estimand

  • Replaying each stored response tape through both destinations fixes the pre-risk decision path and blocks feedback from execution through later portfolio prompts.The balanced destination contrast averages within-tape E1-minus-E0 differences; the tape-origin contrast bundles sampling, feedback, collection time, and provider state and is descriptive.Source: §5.2 · pp. 4–5

5.3 What the fixed-response control supports

  • −0.0170 (95% interval [−0.0230, −0.0117]): the fixed-tape destination contrast lowers total return, equal to 10.4% of the +0.164 E0 baseline.The observed diagonal is −0.0135, the sum of the mechanical destination contrast and a descriptive tape-origin term; Figure 1 shows the decomposition.Source: §5.3 · p. 5, Figure 1 · p. 6
  • The split is metric-specific: the tape-origin contrast is +0.813 on Sharpe, while the destination contrast is 21.3% of baseline Sharpe versus 10.4% of baseline return.Source: §5.3 · p. 5
  • 0.26–0.76 is the exact-order agreement frequency across seed-cluster bootstrap resamples; despite high Kendall τb point estimates, rankings remain unresolved at ten clusters.The winner changes in the jump–tail setting with E1-origin tapes; Table 1 distinguishes bootstrap agreement frequency from a posterior probability of the population order.Source: §5.3 · p. 5, Table 1 · p. 5
  • A model with zero return and all-hold decisions in 358 of 360 replay rows can appear robust through inaction, so activity should be reported beside robustness.Source: §5.3 · p. 5

6 Study B: What a Single-Defect Audit Score Misses

Study B tests whether single-defect target recall generalizes to multi-defect audits, using corrected keys and matched tasks with an explicit multi-label prompt. Its findings distinguish target coverage from complete audit correctness and show that prompt changes affect targeted recall differently across auditors.

6.1 What target recall measures

  • The legacy answer key missed a second true defect, so reporting its edit was wrongly penalized while an unreported target violation remained a miss.Re-scoring found that auditors reported exactly one of two true defects in 62–100% of legacy dual-defect tasks and both in 0–33%.Source: §6.1 · p. 7
  • The legacy design cannot identify a quantity-cue effect or establish that reporting one target label was instruction compliance.A single tampered record can violate multiple rules, and one reported target label can coexist with other findings.Source: §6.1 · p. 7, Figure 2 · p. 8

6.2 Matched controls and the primary estimand

  • The primary analysis uses 300 matched zero-, one-, and two-defect tasks across three sources, with prompts permitting multiple findings.Complete rule-based keys establish ground truth relative to the four coded rules, not defects outside them.Source: §6.2 · pp. 7, 9, §6.1 · p. 7

6.3 What the matched controls support

  • The target violation recall drop is positive in five of six matched cells (median 0.267), with three surviving Holm correction.Nine of 600 responses fail to parse, and malformed output is not credited as a correct empty answer.Source: §6.3 · p. 9, Table 4 · p. 10
  • deepseek-v4-pro includes both target labels in 70–88% of dual-defect responses, versus 0–48% for glm-5, yet has lower overall exact-set accuracy: 29/300 versus 108/300.On dual-defect tasks, exact-set accuracy is 21/100 versus 7/100, respectively; target coverage and exact correctness differ.Source: §6.3 · p. 9

6.4 Prompt sensitivity of target violation recall

  • Naming the violated rule raises targeted recall for both auditors on legacy tasks, while generic deliberation and majority self-consistency help different auditors.Precision and exact-set outcomes are not reported for these prompt-sensitivity arms; the self-consistency arm also varies temperature.Source: §6.4 · p. 10

6.5 No detectable self-favoritism at this resolution

  • Neither auditor shows detectable self-favoritism, but the design could detect only a large effect and confounds producer difficulty with self-favoritism.The minimum detectable effect is roughly 0.17 absolute recall; each auditor’s cross condition contains only one other producer.Source: §6.5 · p. 11

7 Discussion

The two studies require controls matched to their claims: fixed-response execution contrasts for mechanical effects and multi-defect specificity measures for audit quality.

7 Discussion

  • Study A holds the response tape fixed for a mechanical execution contrast, while Study B adds matched defect conditions and specificity outcomes to interpret recall.These controls address what the comparison estimates and what the scored audit success captures.Source: §7 · p. 11
  • In dual-defect tasks, deepseek-v4-pro reports both true labels in 78/100 cases but adds extra findings in 57, leaving 21/100 strictly correct sets.The corresponding counts for glm-5 are 23, 16, and 7.Source: §7 · p. 11
  • Figure 3’s pooled path agreement masks that all 89 agreeing paths come from the nearly inactive model, while active models have none.The six audit cells also show heterogeneity and one exception to the median direction.Source: §7 · p. 11
  • Report activity and ranking uncertainty beside execution contrasts, and unsupported findings and zero-defect behaviour beside recall.A rank correlation of 1.000 over ten seeds coexists with exact-order agreement in 0.26 of bootstrap resamples.Source: §7 · p. 11

8 Limitations

The studies are limited to specific synthetic settings and do not establish broad trading or auditing performance.

8 Limitations

  • Study A uses ten shared seed clusters, three synthetic settings in the same 24-day upward phase, one uncalibrated stress parameterization, and a provider mix including an almost inactive model.Adaptive responses to E1 fills and downward or sideways phases are outside the design.Source: §8 · p. 11
  • Study B tests two hosted API snapshots, synthetic substrates, and one coupled violation-plus-edit construction; other defect combinations, models, prompts, and deployment distributions are untested.Source: §8 · p. 11
  • The paper makes no claim that any studied model is better or worse at trading, and reported returns are simulator properties under stated assumptions.Source: §8 · p. 11

9 Conclusion

The conclusion distinguishes a conditional execution contrast from broader closed-loop comparisons and treats target recall as insufficient on its own to characterize audit quality. It recommends keeping claims within the controls and diagnostics each design supports.

9 Conclusion

  • −0.0170 in total return is the fixed-response E0–E1 mechanical contrast, while ten seed clusters leave the model ranking unresolved.Source: §9 · p. 11
  • Five of six audit cells show positive target-recall drops, three survive Holm correction, and one is negative.Source: Figure 3 · p. 12
  • The auditor that most often includes both targets has low precision, frequent zero-defect false positives, and only 21/100 exact dual-defect sets.Source: §9 · p. 12
  • A score supports only claims covered by its controls and scoring; report activity and ranking uncertainty with execution contrasts and specificity with recall.Source: §9 · p. 12

Reproducibility

The released artifact supports specified reanalyses and verification checks, while its provenance measures do not establish plan timing or provider authenticity.

Reproducibility

  • The artifact includes analysis code, figure sources, frozen inputs, and an offline verifier, with different reconstruction levels for Studies A and B.Study A supports statistical reanalysis of 1,800 replay outcomes, not regeneration of provider calls or execution trajectories; Study B’s primary experiment includes 300 tasks and 600 structured results.Source: Reproducibility · p. 13
  • Content digests detect disagreement but cannot prove the plan predates collection because the records are author-writable and lack a third-party timestamp.An outcome-dependent rule outside the hashed plan guided whether Study B would stand alone, merge into a broader report, or be released as a negative result.Source: Reproducibility · p. 13
  • The verifier checks manifests and required inputs, recomputes tables, and regenerates figures, but establishes neither provider authenticity nor corpus representativeness.It ships code and data rather than the manuscript, and no new provider call is needed.Source: Reproducibility · p. 13

Use of Generative AI

The author led the study, while generative AI tools assisted with coding, experimental-design feedback, analysis checks, editing, and release materials.

Use of Generative AI

  • The author planned and led the study, reviewed literature, developed methods, designed experiments, analyzed results, and wrote the paper.Source: Use of Generative AI · p. 13
  • Generative AI tools helped write code, provide experimental-design feedback, and check mathematical and statistical analyses.Source: Use of Generative AI · p. 13
  • The tools also helped edit text, prepare plotting scripts and figures, and prepare and check the data and code package for release.Source: Use of Generative AI · p. 13

Appendix: per-cell tables

The appendix reports per-cell values underlying the main-text summaries and describes two tables; the tables are generated from released CSV files by script.

Appendix: per-cell tables

  • The appendix reports per-cell values underlying the main-text summaries, and scripts/build_appendix_tables.py generates all tables from released CSV files.Source: Appendix: per-cell tables · p. 17
  • Table 5 reports every estimand of the fixed-tape 2 × 2 replay for the four outcomes used in the body.Intervals use 10,000 seed-cluster bootstrap draws, 10 clusters, and 450 matched pairs; five further outcomes are recorded in factorial_estimands.csv.Source: Table 5 · p. 17
  • Table 6 covers six auditor-by-source cells in the hash-bound multi-label plan, with target-violation recall on matched triplets.It reports discordant counts, unadjusted paired-bootstrap intervals, and exact McNemar p-values with Holm correction across six primary comparisons.Source: Table 6 · p. 18
Loading 2609.32379v1…