PaperlayerView exact revision on arXiv

Source-linked AI summary

FIDES-Live: Evaluating Behavioral Concordance in Sequential Trading Agents

Arther Tian, Alex Ding, Simon Wu, Aaron Chan

arXiv:2610.36476v1cs.CRcs.AI

TL;DR

Sequential trading agents can produce decisions, executions, account states, and reports that disagree even when portfolio values are correctly computed. FIDES-Live retrospectively audits those artifacts across paper-trading traces, finding distinct patterns in commitments, reporting, and execution while leaving long-run trading skill unestablished.

  • Problem

    Valid orders, execution outcomes, account consistency, commitment adherence, reporting accuracy, and financial returns are distinct properties that a portfolio return or aggregate score can make indistinguishable.

    Source: §1 · p. 1

  • Method

    FIDES-Live retrospectively applies deterministic, artifact-level checks to sequential trading traces, separating decision validity, stated commitments, execution and ledger reconciliation, and receipt-grounded reporting while retaining unscorable cases.

    Source: §1 · p. 1, §3.2 · p. 5, §3.3 · p. 5

  • Results

    94.36% overall cryptocurrency commitment adherence was 62.63% for constraints with nonempty proposals versus 97.56% for valid empty proposals, while all retained transitions reconciled with fills and report failures remained.

    Source: §1 · p. 2, §8 · p. 17

  • Takeaways & Limitations

    Artifact-level auditing complements financial outcomes by showing which observable claims agree with recorded actions and states, so concordance, coverage, activity, and financial outcomes should be reported separately.

    Source: §7 · p. 16, §8 · pp. 16–17

    This retrospective audit uses dependent account-rounds from two environments with different assets, dates, schedules, and execution policies, so short windows establish neither stable rankings nor long-run alpha.

    Source: §6 · p. 14

Abstract

from arXiv · show

Sequential trading agents produce decisions, orders, execution receipts, account states, and reports that need not agree, even when the final portfolio value is correctly computed. We present FIDES-Live, an operational extension of FIDES from generated strategy concordance to the audit of sequential agent traces. The protocol separates structured decision validity, optional numerical commitment adherence, execution and ledger reconciliation, and receipt- grounded reporting, with explicit coverage and unscorable outcomes. We retrospectively audit ten requested inference routes over 7,299 cryptocurrency account-rounds and a 310-round equity pilot, using real market observations and local paper execution. Cryptocurrency decision validity is 97.35% and structured reporting accuracy is 90.81% among scorable responses, but route-level decision and reporting performance can diverge sharply. Commitment adherence is 94.36% overall; it falls to 62.63% on constraints emitted with nonempty order proposals, compared with 97.56% for valid empty proposals. All retained cash and position transitions reconcile with recorded fills, while execution controls and unavailable quotes explain separate rejection populations. Short-window marked returns provide descriptive context rather than a new test of whether concordance predicts alpha. These observations motivate reporting concordance, evidence coverage, activity, and financial outcomes separately. They do not establish long-run trading skill, semantic faithfulness of explanations, or causal benefits from audit feedback.

1 Introduction

Sequential trading traces can contain valid orders that do not execute, internally consistent accounts that violate commitments, and accurate reports alongside losses. FIDES-Live audits these artifacts separately and finds that decision, commitment, execution, and reporting outcomes do not collapse into one reliability measure.

1 Introduction

  • Valid orders can fail to execute, accounts can violate allocation commitments, and accurate transaction reports can coexist with losses, distinctions obscured by portfolio return or a single aggregate score.As agents repeatedly act on market observations, evaluating the artifacts around each action becomes a separate measurement problem.Source: §1 · p. 1
  • FIDES-Live links observations, decisions, commitments, orders, receipts, account states, and reports, then separately checks action validity, stated constraints, fill arithmetic, and receipt-grounded facts.It preserves unavailable evidence and reports eligibility and coverage rather than assigning convenient scores.Source: §1 · p. 2
  • The retrospective audit covers 7,299 cryptocurrency account-rounds across ten requested routes and a 310-round equity pilot, using real market observations and local paper execution.The environments differ in risk controls, advisory signals, costs, and scheduling, and are analyzed separately.Source: §1 · p. 2
  • 94.36% overall cryptocurrency commitment adherence fell to 62.63% for constraints with nonempty proposals, versus 97.56% for valid empty proposals; all retained transitions reconciled with fills.The equity pilot had 100 orders rejected for market unavailability, while the cryptocurrency executor rejected 117 through its proposal gate.Source: §1 · p. 2
  • The study establishes measurements and observed discrepancies, not profitable trading, faithful internal reasoning, deliberate deception, or causal benefits from audit feedback.Source: §1 · p. 2

2 Related Work

Prior work establishes live, sequential financial evaluation and related state-grounded reliability methods, while FIDES-Live focuses on agreement among artifacts surrounding sequential trading actions. It complements financial outcome evaluation with an operational audit, without making scores across the earlier FIDES study and this study directly comparable.

Trading agents and live evaluation

  • TradingAgents [10], StockBench [1], Agent Market Arena [3], and LiveTradeBench [12] exemplify established live-data and sequential financial evaluation settings; FIDES-Live audits artifact agreement around those actions.AlphaForgeBench [13] instead evaluates generated factors and strategy implementations through deterministic execution.Source: §2 · pp. 2–3

State-grounded reliability and explanation faithfulness

  • State-based evaluations such as AgentBench [2] and τ-bench [11] ground reliability in observable outcomes, but consecutive trading rounds are not independent task repetitions or a direct passk measure.Ledger agreement checks observable claims, not whether a narrative reveals internal causal reasoning; discrepancies are not evidence of deliberate deception.Source: §2 · p. 3

From FIDES to sequential operational concordance

  • FIDES-Live extends FIDES from one generated strategy, code, and backtest to sequential decisions, orders, receipts, account states, and reports, separating execution events from reporting discrepancies.Accuracy after observing a receipt does not validate an earlier investment forecast, and the checks do not semantically evaluate every rationale.Source: §2 · p. 3

Measurement before aggregation

  • FIDES-Live prioritizes deterministic checks, eligible denominators, and evidence coverage, reporting operational dimensions separately rather than treating an unvalidated weighted composite as overall agent quality.The connections to PoQ [4] [5] and SFGA [7] motivate measurement choices; the corpus was not produced with those methods.Source: §2 · pp. 3–4

Relationship to the preceding FIDES study

  • The preceding FIDES study evaluated 40 generated strategies with historical backtests, whereas this study audits newly recorded forward-running traces; their scores are not directly comparable.The change is a continuation of the measurement perspective, not a replication of its three scores.Source: §2 · p. 4, Table 1 · p. 4

3 An Operational Concordance Protocol

FIDES-Live treats each sequential agent round as a set of artifacts that can disagree, then audits distinct properties from retained evidence. It separates decision validity, optional commitment adherence, receipt-grounded reporting, execution, and ledger reconciliation, with unavailable evidence kept visible.

3.1 The trace as the measurement object

  • Each round links an observation and pre-execution account to a decision, proposed orders, execution receipts, post-execution account, and subsequent report.Observations can differ across accounts, and a rejected order can leave the account unchanged.Source: §3.1 · p. 4

3.2 Evidence reconstruction and eligibility

  • Timestamp-ordered artifact reconstruction distinguishes pass, fail, and unscorable outcomes, retaining unavailable calls in accounting totals rather than awarding credit.Successful outputs divided by retained rounds are reported as retained-round yield, not as completion probabilities across every started attempt.Source: §3.2 · p. 5

3.3 Decision validity and explicit commitments

  • Decision validity checks the response schema, rationale, orders, and optional commitments; separate commitment checks evaluate declared numerical constraints against eligible post-execution states.Commitments are optional, and high compliance at low coverage is weak evidence about general adherence; platform risk limits are not voluntary commitments.Source: §3.3 · pp. 5–6

3.4 Receipt-grounded reporting

  • Receipt-grounded reporting checks structured fields against the supplied post-execution account and receipts, but does not assess narrative semantics or intent.A structured report can pass despite an unchecked narrative inconsistency.Source: §3.4 · p. 6

3.5 Execution and ledger reconciliation

  • Execution gates and recorded fills are tracked separately from decision validity; ledger reconciliation reconstructs cash and positions from filled orders and compares them with recorded states.Reconciliation establishes agreement with recorded fills and marks, not exchange execution, quote freshness, or realistic liquidity.Source: §3.5 · pp. 6–7

4 Corpus and Experimental Setting

The audit covers separate cryptocurrency and equity paper-trading environments, ten requested routes, and a provenance-tracked corpus. Their differing controls and schedules are retained as distinct settings rather than treated as a controlled market-transfer test.

4.1 Two forward-running environments

  • The cryptocurrency corpus contains 7,299 account-rounds, while the equity pilot contains 310; both use real market observations with local paper execution and differ in controls and schedules.The cryptocurrency system uses six spot assets, advisory momentum/volatility analysis, and a balanced policy; the equity system has no mandatory balanced-allocation gate or advisory analysis.Source: §4.1 · pp. 7–8

4.2 Inference routes and requested output

  • Ten requested inference routes are audited through decision and report requests, with reports receiving execution receipts and account states and no hidden repair call.Returned identifiers do not attest fixed model weights, and the retained evidence establishes neither a fixed sampling seed nor weight-level version attestation.Source: §4.2 · p. 8

4.3 Data reconstruction and validation

  • The analysis reconstructs outcomes from retained artifacts, validates stored hashes and parsed responses, and passes 36 deterministic implementation controls.The study does not treat account-rounds as independent samples for significance testing, annualize a one-week return, or infer a stable model ranking.Source: §4.3 · pp. 8–9

5 Results

The audit finds that decision validity, report accuracy, commitment adherence, execution outcomes, ledger reconciliation, and financial returns measure distinct properties. Results vary by route and activity, while the equity pilot and short-window returns provide bounded context.

5.1 A valid decision does not ensure an accurate report

  • Cryptocurrency decisions were valid in 7,052/7,244 cases (97.35%), while reports were accurate in 6,597/7,265 (90.81%); DeepSeek had 620/628 valid decisions but 90/628 accurate reports.Claude had 556/726 valid decisions and 605/732 accurate reports, while other routes reached 99.73–100% report accuracy. Of 668 report failures, 395 were JSON-parse errors and 271 parsed reports failed requested fields; 6,597/6,868 parsed reports without recorded response errors passed all checks (96.05%).Source: §5.1 · pp. 9–10, Table 3 · p. 10

5.2 Commitment coverage and activity change the interpretation

  • Commitment coverage was 25.13%, and adherence was 94.36% overall but 62.63% on nonempty proposals versus 97.56% on valid empty proposals.Only 219/7,052 valid decisions contained an order; in rejected-proposal rounds, 37/131 commitments passed despite portfolios potentially remaining unchanged after gate rejection.Source: §5.2 · p. 10, Table 4 · p. 11, Figure 3 · p. 12

5.3 Execution controls and ledger fidelity are different measurements

  • All 7,299 cryptocurrency and 310 equity retained rounds reconciled ledger transitions, while 338 contract-valid cryptocurrency orders yielded 221 fills and 117 quality_rejected rejections.Gate failures also included invalid or unavailable decision responses, which are distinct from valid empty proposals and from rejected-order counts.Source: §5.3 · pp. 11–12, Table 5 · p. 12

5.4 The equity pilot illustrates environment-error attribution

  • In the equity pilot, 297/309 decisions were valid (96.12%) and 274/309 reports accurate (88.67%); commitment adherence was 196/202 (97.03%), including 25/31 active and 171/171 empty-proposal constraints.Of 168 orders in contract-valid proposals, 63 filled, 100 received market_unavailable, and five were rejected for insufficient cash; 24/25 scorable reports on rejection rounds were accurate.Source: §5.4 · p. 13

5.5 A concrete factual discrepancy

  • A retained DeepSeek report reproduced cash, equity, and positions but misstated the filled-order count on an empty-order round; another empty-order report returned 1,584,655,368.The examples show field-level discrepancies but do not identify their mechanism or establish intentional misrepresentation.Source: §5.5 · p. 13

5.6 Accurate reporting and short-window returns answer different questions

  • Across the settled cryptocurrency session, all ten route returns were negative, ranging from −1.448% to −0.157%, although seven routes had 100% conditional reporting accuracy.The valuation path excludes earlier marks with unknown staleness and contains an approximately 97-minute gap; ten dependent one-week paths cannot establish a concordance-return relationship or benchmark passive-market skill.Source: §5.6 · pp. 13–14, Table 6 · p. 14

5.7 Recorded resource use

  • The recorded workload totaled 211,324,566 prompt and completion tokens across entries with both counts present; missing usage was not imputed as zero.These totals include attempts outside committed rounds and are not a cost-normalized model comparison or a reconstructed dollar bill.Source: §5.7 · p. 14

6 Limitations

The audit is retrospective and observational, with dependent account-rounds across different environments; its deterministic checks support narrow artifact-level claims, not broader claims about model faithfulness or market execution.

6 Limitations

  • Consecutive account-rounds share market conditions, portfolio history, and often input data, so counts are not independent replications and model differences remain descriptive.The short windows do not establish stable rankings, long-run alpha, or generalization across market regimes.Source: §6 · p. 14
  • Deterministic checks measure format, allowed values, and structured factual fields, but do not establish rationale–order agreement or reveal internal reasoning.Commitment adherence covers only emitted, evaluable constraints, and selection into commitment-making is uncontrolled.Source: §6 · pp. 14–15
  • Saved valuation evidence is incomplete: freshness is unknown for the initial 20 hours 49 minutes, and the later nonstale path has an approximately 97-minute gap.Missing ancillary execution-attempt artifacts also mean retained-round yield cannot estimate success across all started attempts; model-route labels do not attest immutable weights or serving configuration.Source: Figure 4 · p. 15, §6 · pp. 15–16
  • The environments differ in assets, dates, schedules, and execution policies, while crypto agents also receive algorithmic signals and use a deterministic proposal gate.These conditions can affect activity, exposure, returns, and apparent adherence, preventing attribution of outcomes solely to the language model.Source: §6 · p. 15
  • Fills come from a local paper executor, so reconciliation verifies arithmetic against retained fills rather than whether an exchange would execute them.The audit omits queue priority, partial fills, depth-dependent impact, and some venue-specific costs; stale prices can still reconcile.Source: §6 · p. 15

7 Discussion

FIDES-Live treats decision validity, commitment adherence, ledger agreement, and structured reporting as distinct evidential claims. Its discussion stresses that denominators and measurement coverage matter, while proposed extensions remain future experiments.

7 Discussion

  • Action-contract validity, commitment adherence, ledger reconciliation, and structured reporting accuracy each establish a different property, and none alone establishes profitability or transparent internal reasoning.Validity concerns executor interpretation; ledger reconciliation concerns arithmetic agreement with fills; reporting accuracy concerns selected factual claims matching a supplied receipt.Source: §7 · p. 16
  • Without coverage, activity, and eligibility, a combined score can reward the composition of observations rather than intended behavior.Rarely declared constraints, mostly inactive systems, and whole-proposal gates alter the opportunities and account states against which behavior is assessed.Source: §7 · p. 16
  • Malformed reports can prevent downstream applications from obtaining contracted factual output, but detecting them does not establish that a model cannot reason about finance.Reporting parse, schema, field, and infrastructure outcomes separately constrains interpretation; missing quotes and correctly reported rejections should not be treated as invented trades.Source: §7 · p. 16
  • Prospective frozen protocols, repeated independent accounts, human-annotated semantic pairs, and randomized feedback or gate ablations are proposed as future experiments, not demonstrated effects.The suggested designs address path dependence, semantic concordance, and whether audits change subsequent behavior or only activity and exposure.Source: §7 · p. 16

8 Conclusion

FIDES-Live extends FIDES from generated strategies to sequential agent traces, auditing observable decisions, commitments, execution, account states, and reports. The retained corpus supports artifact-level concordance claims, not conclusions about persistent rankings or profitable trading.

8 Conclusion

  • FIDES-Live extends the FIDES measurement perspective from generated trading strategies to sequential agent traces, distinguishing decisions, commitments, local execution, account arithmetic, and post-execution factual reports.Unavailable evidence remains visible rather than being scored as though it were observed.Source: §8 · p. 16
  • Across 7,609 retained account-rounds, ledger reconciliation coexists with report failures, commitment coverage varies, and adherence differs between active and empty proposals.The results support artifact-level auditing as a complement to financial outcome reporting, but establish neither persistent model rankings nor profitable trading.Source: §8 · p. 17
  • The source package includes generated tables, figures, normalized outcomes, source digests, and offline analysis code, but raw-evidence regeneration requires access to the retained source corpus.No real-money orders were placed.Source: §8 · p. 17

A Recorded Routes and Protocol Differences

The route labels identify requested inference routes rather than independently attested model weights. Recorded prompt stability does not imply identical complete request payloads, and recorded conditions are not pooled into a causal cross-market comparison.

A Recorded Routes and Protocol Differences

  • Model names denote request routes, not independently attested immutable model weights.Returned identifiers and call metadata do not independently attest provider routing or serving configuration.Source: Table 7 · p. 19
  • The recorded material differences are not pooled into a causal cross-market comparison.The environments differ in assets, dates, schedules, and execution policies.Source: Table 8 · p. 19
  • Decision prompts share one hash across 7,299 cryptocurrency rounds, while report prompts share one hash across 7,298; one interrupted report lacks a prompt.Two advisory-role wordings occur in 5,505 and 1,794 requests, so prompt stability does not mean every complete request payload is identical.Source: A Recorded Routes and Protocol… · p. 19

B Resource Accounting and Daily Observation Counts

The supplementary tables document cryptocurrency observation counts and call-level accounting, retaining operational gaps and bounding how usage and elapsed-time measures should be read.

B Resource Accounting and Daily Observation Counts

  • Table 9 includes cryptocurrency call-table accounting for attempts outside committed rounds.Tokens count recorded prompt and completion usage; missing usage contributes no measured amount.Source: Table 9 · p. 20
  • Usage pairs count calls with both components available, while median elapsed seconds describe the gateway/system rather than hardware-normalized speed.Source: Table 9 · p. 20
  • Table 10 reports cryptocurrency observation counts by UTC date while keeping partial first and last days and operational gaps visible.Accurate/scorable reports follow the same primary definition as the main text.Source: Table 10 · p. 20

Exact model-level denominators

Table 11 provides exact cryptocurrency commitment numerators and denominators and distinguishes coverage from evaluable-constraint passage rates.

Exact model-level denominators

  • Table 11 gives exact cryptocurrency numerators and denominators corresponding to Table 3.Source: Table 11 · p. 21
  • Declaring/valid measures commitment coverage, whereas passed/evaluable counts individual nontrivial constraints.Source: Table 11 · p. 21
  • A perfect rate based on few constraints does not establish broad coverage.Source: Table 11 · p. 21

C Audit Reconstruction and Case Traceability

The audit reconstructs retained runs under explicit inclusion and missingness rules, preserves traceability, and rechecks historical output contracts and account transitions offline.

C Audit Reconstruction and Case Traceability

  • The corpus includes populated original equity runs and indexed cryptocurrency rounds for API-driven arena paper accounts, while excluding specified demonstrations, replays, archives, and fixtures.Combining datasets means using a common audit schema, not relabeling their actual market, policy, or source.Source: C Audit Reconstruction and Case… · p. 21
  • The audit counts 840 unavailable ancillary attempt artifacts as missing provenance and does not count one orphan started call as a completed response.Unparseable or absent factual fields are report-accuracy component failures, not counts of proven numerical fabrication.Source: C Audit Reconstruction and Case… · p. 21
  • Timestamp ordering restores the recovered MiniMax observation to its account sequence while retaining both identifiers and the source digest.Source: C Audit Reconstruction and Case… · p. 21
  • Selected DeepSeek reporting cases illustrate deterministic failures already defined by the audit, not a human-labeled semantic test set.Source: C Audit Reconstruction and Case… · p. 21
  • The offline audit checks source identity, reconstructs account-rounds, rechecks action contracts and report fields, and reconciles account transitions.Normalized outcomes support aggregate reproduction; recomputing factual checks from first principles requires original account and response artifacts.Source: C Audit Reconstruction and Case… · pp. 21–22

D Historical Output Contracts

  • The historical decision contract requires JSON with a rationale and orders containing allowed ticker, side, and positive asset-unit quantity fields.Commitments are optional numerical promises about the portfolio immediately after execution, marked at execution mids.Source: D Historical Output Contracts · p. 22
  • The report contract requires factual post-execution fields based only on the receipt, including round-specific filled-order count and nonzero share holdings.It distinguishes rejected from filled orders and specifies tolerances for account values and share counts.Source: D Historical Output Contracts · p. 22

Related papers

Loading 2610.36476v1…