Source-linked AI summary
CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents
Bo Qu, Mingguang Chen
arXiv:2606.29771v3cs.AIcs.LGq-fin.CPq-fin.PM
TL;DR
Fixed-window return rankings do not establish whether an LLM trading agent’s decision process is sound. CLQT is a cost-aware, point-in-time closed-loop benchmark that audits decisions and scores capabilities. Across backtest and live cohorts, it finds a stating-versus-doing gap, while the short live horizon and differing model cohorts bound what that track establishes.
Problem
Fixed-period return leaderboards can be dominated by the market path and do not establish decision quality, while credible model rankings require more models, regimes, and repeated runs.
Source: §1 · pp. 1–2Method
CLQT runs a five-stage agent loop with auditable tool calls and DecisionRounds, using the audit trail to assess capability and distinguish agent intent from post-construction allocations.
Source: §3.1 · p. 8, §7.9 · p. 35, §9 · p. 44Results
+0.30 backtest and +0.23 live agreement–judge divergence reproduces the stating-versus-doing gap across cohorts, horizons, and held-out judges.
Source: §9 · p. 44Takeaways & Limitations
Capability and behavioral axes reveal module-value differences that returns alone cannot distinguish, and a targeted scaffold remediation eliminated both identified autonomous hold mechanisms.
Source: §9 · p. 44The four-week live track uses different model versions across cohorts and indicative-only returns, so it establishes instrument consistency on unseen data rather than model generalization.
Source: §8.5 · p. 42
Abstract
from arXiv · showhide
LLM agents are increasingly cast as autonomous portfolio managers, yet the dominant evaluation idiom, a leaderboard of returns over a fixed window, certifies neither the soundness of an agent's process nor the durability of its edge: one period's return is dominated by the market path, and apparent alpha can dissolve once look-ahead bias and trading costs are controlled. We introduce CLQT, a closed-loop benchmark that reframes LLM trading evaluation as diagnosis rather than ranking. CLQT enforces point-in-time data access through a hard TimeGate, models institutional transaction and financing costs, scores strategy consistency across rounds, and seals every gather-analyze-decide-execute-reflect cycle into a recompute-verifiable audit chain; the same model runs as a constrained investment committee or a single autonomous orchestrator, making scaffolding an experimental variable. From the audit trail CLQT computes a five-axis capability scorecard (Coherence, Acuity, Composure, Discipline, Reliability), with coherence scored partly by a held-out LLM judge to curb self-preference bias. We validate CLQT on a contamination-controlled, year-long multi-model backtest campaign with a 13-configuration ablation grid and a four-week live broker paper-trading track on post-cutoff data. The diagnosis-first read shows that the capability leader is not the Sharpe leader; that agents' allocations systematically fail to follow their own stated analysis, a stating-versus-doing gap stable across both tracks (+0.30 backtest, +0.23 live); and that module value registers on the capability and behavioral axes when returns alone cannot separate it. Net of realistic costs, agents clear defensive baselines but do not cleanly beat the index. Credible evaluation of LLM investment agents must therefore diagnose the process rather than rank a period's return, the standard CLQT operationalizes and makes auditable.
1. Introduction
Fixed-period return leaderboards do not establish whether an agent’s process is sound, because market paths can dominate results and high returns may reflect future-data access, ignored costs, or mandate drift. CLQT instead uses a closed-loop audit trail to diagnose capabilities and limitations before ranking agents.
1. Introduction
- The benchmark targets methodological gaps in point-in-time data, realistic costs, portfolio-level decisions, cross-round consistency, memory, and tool orchestration.The paper’s contributions also include a held-out LLM judge for Coherence and a campaign spanning backtests, ablations, and live broker trading.Source: §1 · p. 2
- Fixed-period return rankings can reflect the realized market path and risk profile rather than decision quality, while credible rankings require more models, regimes, and repeated runs.The paper therefore argues for diagnosis before ranking: use the decision trail to locate where the gather–analyze–decide–execute–reflect process breaks down.Source: §1 · p. 2
- CLQT computes a five-axis diagnostic capability scorecard from the audit trail and sets consistent dominance across axes and sub-periods as the standard for calling a model superior.A recompute-verifiable hash chain enables independent checks of audit-trail integrity.Source: §1 · p. 2
2. Literature Review and Positioning
CLQT positions itself as a closed-loop benchmark that combines capabilities often evaluated separately and scores the decision process, while adapting general agent infrastructure and cognitive models to portfolio management.
2.1 Benchmark Landscape
- CLQT combines point-in-time data, costs, memory, tools, live support, and process scoring in one loop; prior benchmarks may go deeper on individual features but do not provide the full stack.The shared-feature tally is a prerequisite rather than a quality score, and the full combination supports a capability profile across backtest and live.Source: §2.1 · pp. 4, 6
2.2 Key Gaps Addressed
- CLQT adds parameterized cost models and financing accrual, formalizes three-tier memory for direct ablation, and validates live execution with the same decision-level audit trail as backtest.The default cost tier’s commission and spread legs are validated against live broker fills.Source: §2.2 · p. 6
- CLQT logs and hashes tool calls, accumulates news sentiment across cycles, and scouts candidates using earnings surprise and macro-correlation alongside momentum.These extend tool orchestration, intertemporal information, and candidate screening as measurable parts of the benchmark.Source: §2.2 · p. 6
2.3 The LLM-Agent Infrastructure CLQT Builds On
- CLQT adapts general agent patterns for memory, tools, role-decomposed workflows, and profile-based scoring, then measures whether those mechanisms improve decisions.Its held-out Coherence judge addresses self-preference bias, with scores reported alongside mechanical signal–action agreement.Source: §2.3 · p. 7
2.4 Theoretical and Cognitive Foundations
- The loop draws on OODA and situation-awareness models, while declarative-versus-procedural knowledge motivates Coherence and experiential learning motivates reflection and memory.Acuity contrasts responses to informative signals with responses to a volatility distractor; Discipline and Reliability are operational constructs.Source: §2.4 · pp. 7–8
3. System Design
CLQT’s design goal is to make each mechanism in the agent system measurable from the audit fields it emits. Its substrate proceeds from the decision loop through information gathering, execution, consistency, memory, portfolio construction, and sentiment synthesis.
3.1 Architecture Overview
- The five-stage loop runs through typed, auditable MCP tools, and each DecisionRound records stage outputs and audit fields from which the paper’s metrics can be recovered.Stages comprise acquisition, signal synthesis, portfolio construction, execution, and attribution with lesson extraction.Source: §3.1 · pp. 8–9
3.2 Six-Agent Architecture
- Structured mode uses six role-restricted agents and prescribed protocols, whereas autonomous mode gives one orchestrator full tool access and discretion over workflow.Both modes use real multi-turn tool calling; the mode switch confounds Discipline and Acuity, which are compared within mode.Source: Figure 2 · p. 9, §3.2 · p. 10
3.3 Multi-Source Information Gathering and 3-Dimensional SCOUT Stage
- SCOUT ranks candidates using cross-sectional z-scores for momentum, earnings surprise, and market correlation, while data channels map backtest and live providers to equivalent inputs.The top-K candidates pass to portfolio construction; the 50/30/20 weighting is an ablation default, not a derived optimum.Source: §3.3 · pp. 10–11
3.4 Cost-Aware Execution
- Execution models configurable transaction costs, including selectable market-impact models, and accrues financing costs for leveraged or short positions.The default tier’s commission and spread legs are validated against live broker fills, while its impact leg is parameterized rather than fitted.Source: §3.4 · pp. 11–12
3.5 Strategy Consistency Measurement
- ConsistencyScore equally weights style drift, turnover consistency, signal adherence, and mandate alignment; scores below 0.7 trigger a drift warning and capped per-asset deltas.The weights and threshold are fixed defaults, not tuned against returns.Source: §3.5 · p. 12
3.6 Structured Multi-Modal Memory System
- Three-tier memory combines recent working context, decaying episodic records, and consolidated semantic patterns, with consolidation every 12 rounds.Episodic and semantic decay rates are fixed at 0.95 and 0.98 per round, respectively; memory provenance supports ablation.Source: §3.6 · pp. 12–13
3.7 Knowledge Synthesis and Decision Pipeline
- MAPC reconciles an optimizer proposal with a budget-neutral tactical overlay, projects the result onto feasible constraints, and subjects it to Risk Officer validation.Separate pre- and post-projection fields distinguish agent intent from adjustments by the optimizer and constraints; candidate rotation also observes addition, holding-period, and turnover limits.Source: §3.7 · p. 13
3.8 Intertemporal Sentiment Trajectory and News Persistence
- Persistent news records form per-symbol sentiment trajectories, classified by comparing the last three days’ mean with the first three days’ mean.The replayable trajectory enters the Acuity informative composite and can be toggled in the news-sentiment ablation.Source: §3.8 · p. 14
4. Data Integrity
CLQT protects evaluation through point-in-time access, contamination controls, and a recompute-verifiable record of every decision round.
4.1 TimeGate: Point-in-Time Enforcement
- TimeGate blocks access to data timestamped after the decision’s as-of time, applying the same rule to all modalities, memory, and news supplements.Live mode also freezes access an intraday interval before the decision timestamp.Source: §4.1 · p. 15
4.2 Contamination Prevention
- Per-model training-cutoff filtering and point-in-time universe membership address training-data contamination and survivorship bias.Source: §4.2 · p. 15
4.3 Contamination Detection
- Canary-document and future-shift probes are implemented but were not exercised; this campaign relies instead on point-in-time and cutoff controls, live post-cutoff data, and the verified hash chain.Active adversarial probing is identified as a hardening direction.Source: §4.3 · p. 15
4.4 Recompute-Verifiable Audit Trail
- Every DecisionRound is sealed into a hash chain that re-derives hashes and detects edits, though it does not prevent wholesale store replacement and recomputation.Every campaign run passes end-to-end integrity verification.Source: §4.4 · p. 15
5. Backtest and Live Evaluation Protocols
CLQT evaluates agents in contamination-aware backtests and broker paper trading, with campaign-specific rebalance settings and decision-level cross-track comparisons.
5.1 Backtest Track: Contamination-Aware Period Design
- The backtest assigns evaluation periods by training cutoff; this campaign uses a single one-year window because every model’s cutoff precedes it.The supported two-year path for older-cutoff Group B models is not exercised by this cohort.Source: §5.1 · pp. 15–16
5.2 Live Track: Broker Paper-Trading Architecture
- The live track gathers before market open, submits and reconciles broker orders at the open, and locks the portfolio between rebalances.A dedicated execution agent optimizes cost and fill quality while upstream target weights remain fixed.Source: §5.2 · p. 16
5.3 Rebalancing vs. Re-Optimization
- Calendar, threshold, and agent-discretion modes distinguish mechanical rebalancing from full re-optimization, but the campaigns use calendar mode throughout.Backtests rebalance bi-weekly and live sessions daily; the other modes are supported but untested here.Source: §5.3 · p. 16
5.4 Cross-Track Comparison Protocol
- Because the tracks differ in duration, cadence, and model generation, this campaign compares them on track-invariant decision-level axes rather than returns.When tracks share a window and configuration, the general protocol compares the same metric suite; fill-price divergence was used to validate the LOW cost tier.Source: §5.4 · p. 16
6. Evaluation Framework
CLQT combines passive benchmarks, recorded metrics, a five-axis capability scorecard, held-out judging, and behavioral analysis to diagnose agent decisions.
6.1 Ablation Dimensions
- The framework supports 12 ablation dimensions spanning module flags, configuration overrides, and structured-versus-autonomous skill mode.Table 3 presents the ablation dimension taxonomy.Source: §6.1 · p. 17, Table 3 · p. 17
6.2 Baselines and Benchmark Comparison
- The passive suite includes index ETFs, a synthetic 60/40 portfolio, the agent’s equal-weight universe, and four algorithmic baselines for net-of-cost comparisons.The eight benchmarks report cumulative and excess return, information ratio, and tracking error.Source: §6.2 · p. 17
6.3 Metric Dictionary
- The metric suite records performance, benchmark-relative results, costs, and governance measures, all recoverable from the audit trail.Sharpe and Sortino use a zero risk-free-rate convention.Source: §6.3 · p. 17, Table 4 · p. 18
6.4 The Five-Axis Diagnostic Capability Scorecard (APM-CS)
- The APM-CS scores five audit-trail-based capabilities, with returns treated as outcomes the axes explain rather than as a scorecard axis.Its axes are Coherence, Acuity, Composure, Discipline, and Reliability; it is cohort-relative, and raw axis values are the portable read-out.Source: §6.4 · pp. 18–19
- Acuity relates proposed adjustments to informative signals versus distractors, while Composure averages a volatility-response measure with strategy consistency when available.The noise composite combines volatility and 14-day RSI; Discipline is compared within mode because risk-officer approval is confounded across modes.Source: §6.4 · pp. 18–19
6.4.1 Held-out LLM-as-Judge for Coherence
- The held-out judge scores whether allocations follow agents’ stated analyses and signals, ignoring profitability; track-specific judges are calibrated but their scores are not blended.On 50 blind-rated rounds, the judge’s ordering tracks a human rater as closely as the two judges track each other.Source: §6.4.1 · p. 20
6.5 Behavioral Interpretability: Method
- Behavioral analysis renders decision behavior and stated reasoning as a human-legible decision personality, using audit records rather than model weights.It examines stated reasoning, learning, guardrail interactions, inter-agent dynamics, tool use, cognitive effort, and struggle signatures; it does not analyze hidden chain-of-thought.Source: §6.5 · pp. 20–21
7. Experiments
The experiments use a year-long backtest, ablations, and a completed live track to diagnose decision quality and behavior rather than rank models by returns alone. They assess capability and behavior across multiple dimensions, while the rising-market regime limits what returns and ablations can establish.
7.2 Setup
- Five models were tested in structured and autonomous modes over 26 bi-weekly rounds, with five seeds for structurally reliable configurations and point-in-time cutoff checks.qwen3·structured was a single showcase seed after seven seed attempts failed to complete; parser hardening recovered valid content and improved structured Reliability.Source: §7.2 · pp. 22–23
7.3 Campaign Results (RQ1, RQ2)
- Among reliable configurations, haiku·structured is the Sharpe leader at 3.21; rather than crowning qwen3 from a leaderboard, the scorecard presents a profile for each configuration.Source: §7.3 · p. 24
- Reliable structured runs beat a median ~6/8 passive baselines, but agents do not cleanly beat the cap-weighted index.The index shortfall is interpreted as a deliberate de-risking trade, not absent skill.Source: §7.1 · p. 22, §7.3 · p. 24
7.4 Radar Fingerprints (RQ2, continued)
- No configuration encloses all five capability axes; Acuity is near zero for eight of ten configurations, while haiku·structured’s 0.44 is thinly sampled.The radar profiles show uneven strengths, including deepseek·structured’s balanced profile and weaknesses in qwen3 Reliability and haiku Coherence.Source: §7.4 · p. 24, §7.3 · p. 24
7.5 RQ3: Model-Specific Reliability Failures and Their Mechanisms
- Three audit-trail reliability deficits arise through distinct mechanisms: qwen3·structured’s non-terminating tool loop, gemini·structured’s persistent parse failures, and deepseek’s mode-dependent reliability.qwen3·structured fails to complete more than 10 of 26 rounds across seven seed attempts; gemini·structured retains about 10/26 parse-fail rounds on at least one attempt, while deepseek Reliability is 0.99 structured versus 0.62 autonomous.Source: §7.5 · pp. 24–26
7.6 RQ4: Scaffold Shapes the Capability Profile
- Scaffold effects vary by model and axis: backtest composites descriptively favor structure for reliable configurations, while all six live pairs favor autonomy on mode-comparable axes.Structure broadly lifts Composure, but haiku·structured pairs the best reliable Sharpe, 3.21, with the cohort’s lowest Coherence, 0.23.Source: §7.3 · p. 24, §7.6 · pp. 26–27
7.7 RQ5–RQ6: Ablation Grid (single- and multi-module)
- No single-module or cluster knockout separates on Sharpe beyond the ±0.42 repeated-run noise band; the mis-calibrated HIGH tier instead incurs ≈527 bps/round and Sharpe −1.06.Removing the information stack leaves Sharpe within noise but collapses Acuity to ≈−0.01; removing memory eliminates the consistency-score leg. Bare-workflow is also not separable from the fully augmented agent on returns (∆Sharpe +0.26, range [−0.40, +0.66]).Source: Figure 9 · p. 28, §7.7 · pp. 28–29, Table 7 · p. 29
7.7.2 Looking Past Sharpe: Risk, Cost, and Decision Behavior
- Ablations degrade decision processes while returns stay similar: bare-workflow halves analysis and signal breadth, performs no reflection, and spends 40% less reasoning effort, yet proposes the largest trades.Every module-ablation configuration holds drawdown to −3.2…−4.7% and Calmar 5.8–8.7, while removing cost-awareness raises realized costs 10% and the execution pair 18%.Source: §7.7.2 · pp. 29–30, Figure 10 · p. 30, Table 8 · p. 30
- Module feasibility synthesizes returns, risk, costs, capability, and behavioral evidence into per-module verdicts rather than a returns ranking.The supplied passage identifies Table 9 as the location of those verdicts but does not provide the individual module outcomes.Source: §7.7.2 · p. 31, Table 9 · p. 31
7.7.3 The Regime Does Much of the Work
- In a benign rising market, returns are driven mainly by the market, muting ablation differences and limiting tests of regime-contingent modules.SPY rose 27% with an −8.9% drawdown and calm VIX; this period barely tests information modules, learning loops, or covariance-aware optimization, while returns can reward risk-taking that drawdowns would punish.Source: §7.7.3 · p. 31
7.8 RQ7: Net of Costs, Agents Clear the Defensive Baselines but Not the Index Cleanly
- Structured agents beat about 6/8 passive baselines but do not uniformly clear the cap-weighted index; autonomous configurations clear fewer baselines.Deepseek cleared 6.8/8, haiku and gemini 6.0/8, while qwen3-235b autonomous was lowest at 2.2/8. Against SPY, agents had lower volatility and roughly half the maximum drawdown (−4.1% vs −8.9%) at the same Sharpe and about 2× the Calmar; the absolute-return shortfall reflects deliberate diversification.Source: Figure 11 · p. 32, §7.8 · p. 32
7.9 Live-Track Decision-Level Validation (Completed Four-Week Campaign, 18 Valid Days)
- With GLM-5.2 over 18 valid live rounds and minimax-m3 in backtest, agreement–judge coherence gaps were +0.23 and +0.30, respectively; no configuration dominates all five axes.The live and backtest cohorts are different model versions, so this supports instrument consistency, not same-model generalization; the similar gaps are not proof of equality.Source: §7.9 · pp. 33, 35, Figure 12 · p. 35
7.10 Behavioral Interpretability: Findings
- In autonomous mode, gemini-3.5-flash held 9/18 rounds and minimax-m3 8/18 through distinct tool-budget and schema failures; structured mode had no holds.All 17 hold rounds preceded the round-14 scaffold remediation, after which both models traded every day for six valid days; this is mechanism-consistent intervention evidence, not a general efficacy estimate.Source: §7.10 · pp. 36–37, Figure 13 · p. 37
7.11 Summary
- The validation arc finds that capability diagnostics reveal distinctions obscured by Sharpe, while no configuration dominates all five axes and the cohort is not cleanly rankable.Module value registers on capability and behavioral axes, and the coherence gap on unseen live data is the summary’s strongest single exhibit.Source: §7.11 · p. 38
8. Discussion
CLQT argues for diagnosis before ranking: the campaign yields per-configuration process profiles, while no configuration dominates all five axes. It connects those axes to general agent capabilities and bounds interpretation by cohort, regime, judge, and live-track limitations.
8.1 Diagnosis Before Ranking
- No configuration dominates all five axes, and a credible cross-regime ranking would require an order of magnitude more models, regimes, and repeated runs.CLQT therefore presents a profile of where each configuration’s process breaks and under which scaffold, rather than ranking agents.Source: §8.1 · p. 38
8.2 Capability Axes as General Agent Diagnostics
- The five axes map finance measures to general capabilities, and the audit trail supports a portable behavioral lens for complex, cost-bearing, partially observable processes.The stated-reasoning/allocation gap recurs across tracks (+0.30 backtest, +0.23 live); the authors describe its ACT* interpretation as parsimonious, not causal proof.Source: §8.2 · p. 39
8.3 Scaffolding × Capability, and the Cost of Intelligence
- Structure lifts Composure and partly Discipline, but schema demands stress some models; gemini·structured spends ~280k reasoning tokens/run on retries and remains only mid-capability.Autonomous composites score higher for every live model, but hold-collapses limit some gains; a thin governance layer removed observed collapses in an uncontrolled remediation.Source: §8.3 · pp. 39–41, Figure 14 · p. 40
8.4 Benchmark Hygiene: Output Budgets, Cost Realism, Auditability
- Budget telemetry helps prevent truncation from being mistaken for incapability, while the HIGH cost tier’s ~+527 bps/round miscalibration led the authors to retain LOW.They recommend verified point-in-time enforcement, reliability telemetry, repeated-run noise bands, and held-out judges; the audit chain lacks externally anchored non-repudiation.Source: §8.4 · p. 41
8.5 Limitations
- Claims are bounded by small cohorts and configurations, one rising regime with modeled backtest costs, and four-week live evidence that tests consistency, not transfer.Axis readings depend on operationalization and the noise band is assumed transferable; held-out judging mitigates but cannot eliminate self-preference, and human anchoring used one author-rater.Source: §8.5 · pp. 41–42
8.6 Future Work
- Future work calls for broader panels and repeated runs, regime-stratified evaluation, varied mandates and reasoning budgets, active stress probes, and finer-grained exposure attribution.The authors also prioritize broader judge and human-rater panels and recalibration of the HIGH cost tier.Source: §8.6 · p. 43
8.7 Ethics and Broader Impact
- CLQT evaluates historical data and broker paper trading without real capital; the authors say its findings argue against treating any evaluated configuration as deployment-ready.They frame the instrument as a pre-deployment gate for coherence failures, reliability deficits, and scaffold-dependent collapses.Source: §8.7 · p. 43
9. Conclusion
CLQT’s audit-based diagnosis shows why return-only rankings can mislead and why ranking remains premature under the paper’s standard. It also identifies a cross-track stating-versus-doing gap and reports that capability and behavioral axes distinguish module value when returns do not.
9. Conclusion
- The nominal Sharpe winner was a 5-of-26-round reliability artifact, the capability leader sat mid-pack on returns, and no configuration dominated all five axes.By the instrument’s own standard, ranking would be premature.Source: §9 · p. 43
- CLQT makes this diagnosis auditable with a closed-loop, cost-aware, temporally gated benchmark and a hash-verifiable audit trail, plus a five-axis scorecard and held-out, two-judge Coherence protocol.Process scaffolding is treated as an experimental variable through live broker execution.Source: §9 · p. 44
- +0.30 backtest and +0.23 live agreement–judge divergence capture a stating-versus-doing gap that reproduces across cohorts, horizons, and held-out judges on unseen data.The passage characterizes this as instrument consistency across model generations, not same-model transfer.Source: §9 · p. 44
- Capability and behavioral axes reveal module-value differences that returns cannot distinguish, and targeted mid-campaign scaffold remediation eliminated both autonomous hold mechanisms.The paper concludes that credible ranking still needs more models, regimes, and repeated runs.Source: §9 · p. 44
Appendix A: Cost Tier Parameterizations
The cost-tier appendix contrasts a conservative commission assumption with a HIGH-tier parameterization that requires recalibration.
Appendix A: Cost Tier Parameterizations
- At typical $200k+ position sizes, the modeled 0.5 bps commission is below the live broker’s approximately $1-per-fill flat commission, making the LOW tier’s commission leg conservative.The passage says this comparison is based on live broker paper fills.Source: Table A1 · p. 47
- The HIGH Almgren–Chriss tier produces approximately +527 bps per round in the campaign and must be recalibrated before use.Source: Table A1 · p. 47
Appendix B: SCOUT Signal Computation and MCP Tool Inventory
The appendix specifies three SCOUT signals and their guards, then describes an auditable in-process MCP tool interface for quantitative analysis.
Appendix B: SCOUT Signal Computation and MCP Tool Inventory
- Momentum uses a 12-month price ratio with a one-month skip-lag, earnings surprise is clamped to [−5, +5], and market correlation is a 60-day rolling Pearson correlation against SPY.Earnings surprise is None/excluded when no estimate is available.Source: Appendix B · p. 47
- Market correlation is set to None and its weight redistributed when the window has fewer than 10 observations or near-zero variance.Source: Appendix B · p. 47
- Quantitative analysis uses 19 MCP-standard tools across four namespaces, invoked in-process against quant/data libraries rather than through networked server transport.Every tool call is hashed, timed, and logged; Table B1 inventories the tools.Source: Appendix B · p. 47, Table B1 · p. 47
Appendix C: Model Registry — IDs, Snapshots, and Training Cutoffs
The appendices document the model snapshots and evaluation safeguards, then map persisted audit fields to scorecard metrics. The audit trail also supports recovery of every reported metric without new instrumentation.
Appendix C: Model Registry — IDs, Snapshots, and Training Cutoffs
- The backtest registry lists model IDs, providers, training cutoffs, and cohort assignments, while the table also covers live models and held-out Coherence judges.Dated model suffixes pin snapshots; the passage describes the reasoning settings and a disclosed live-run deviation.Source: Appendix C · p. 47, Table C1 · p. 48
- The contamination protocol executes on post-cutoff data, with a cutoff gate and exchange-calendar exclusion; the scorecard tooling prevents a judge from matching any scored run.The passage notes that backtest cutoffs precede the 2025-06-16 start, with qwen3-235b within a one-month grace period.Source: Table C1 · p. 48
Appendix D: Audit-Trail Field Map for the Scorecard
- Persisted fields include behavioral lessons, memory provenance, and reference counts; every paper metric is recoverable from the per-run store without new instrumentation.Source: Appendix D · p. 48
- The audit-field map links portfolio weights and returns, execution costs, rationale and target weights, constraints, clipping, turnover, failures, token telemetry, and benchmark comparisons to metrics and axes.The map also identifies reasoning-corpus and metacognition/learning-trajectory fields as behavioral sources.Source: Appendix D · p. 48, Table D1 · p. 49
Appendix E: Behavioral Anecdotes (round-id-pinned)
The round-pinned examples contrast rich, signal-grounded decisions with process collapse and operational failures that can produce holds. They also show why allocation reasoning and portfolio outcomes need separate inspection.
Module-detach collapse
- In the 2025-11-03 round, module removal cut signal review from ~128 to ~56 and left the bare workflow with no scouting, constraints, adjustments, or reflection.The full agent instead committed a mean-variance-optimal book with explicit weights and a cost-management lesson.Source: Appendix E · p. 49
- Despite that process collapse, risk caps kept realized books safe and returns unaffected (Table 8).Source: Appendix E · p. 49
Two roads to a hold
- Autonomous holds arose through distinct recoverable failures: tool-turn exhaustion prevented allocation serialization, while schema-invalid output triggered fallback; later trade-and-hold patterns were unsynchronized.Source: Appendix E · p. 49
Coherence-judge patterns
- Low Coherence-judge scores trace to five mismatches: direct contradictions, equal weights despite signals, thesis-book name mismatch, signal-inverted weights, or no thesis.Source: Appendix E · p. 49
Competent, signal-grounded decisions
- Signal-grounded examples connect identified momentum or sentiment changes to portfolio rotations; claude-sonnet-4.6 scored Coherence 0.60 and APM-CS 83.6.Source: Appendix E · pp. 49–50
Appendix F: Capability-Axis Robustness (construct validity)
The appendix tests whether capability rankings and cross-mode readings withstand changes to score construction, axis weighting, and measurement scope. It reports stable headline conclusions alongside specific sensitivity boundaries.
Axis revisions
- The revised Discipline axis is bounded, while Composure averages its legs and falls back to a memory-free leg when memory is ablated.The prior Composure product could become uncomputable under memory ablation; the revised construction instead drops the consistency leg.Source: Appendix F · p. 50
F.1 — Rank stability
- The D3/D4 revision alone preserves configuration ranks (τ = 0.86, ρ = 0.94), while the main leader and Reliability-artifact findings persist across tested constructions.Compounding all three axis revisions yields τ = 0.58 and ρ = 0.78; the D3/D4 revision overturns the prior claim that memory removal makes Composure uncomputable.Source: Appendix F · p. 50
F.1b — Acuity noise-set revision
- The D2 noise-set revision moderately reorders Acuity (τ = 0.56, ρ = 0.78) but leaves composite ordering largely intact (τ = 0.75, ρ = 0.88).Haiku-structured Acuity rises from ≈0.2 to 0.44 when the RSI distractor is included.Source: Appendix F · p. 50
Orthogonality and weight sensitivity
- Mean |r| = 0.23 across axes (maximum 0.55 for Coherence–Discipline); weight variants retain configuration ordering (τ ≥0.71, or τ ≥0.89 for Coherence reweighting).Source: Appendix F · p. 50, Figure F1 · p. 51
F.4 — Cross-mode sensitivity
- On mode-comparable axes, backtest preferences vary by model, while all six live pairs favor autonomy by +0.02 to +0.20.The three-axis comparison uses Coherence, Composure, and Reliability; it is a robustness read, not an official composite.Source: Appendix F · pp. 51–52, Table F1 · p. 51
F.5 — Human anchoring
- A blinded author rater agrees with held-out judges at ρ = 0.60, supporting their ordering but not a calibrated score level.The authors identify an independent expert panel as future work.Source: Appendix F · p. 52
Appendix G: Reproducibility and Artifacts
The appendix describes a reproducibility pipeline in which persisted run records feed scorecards, analyses, and figures, with round-level records and hashes available for verification.
Recomputation
- Every reported number recomputes from persisted per-run audit stores through scorecard, ablation, behavioral-analysis, and figure-generation tooling.Table 9’s per-module verdicts synthesize the underlying evidence rather than being directly generated by one tool.Source: Appendix G · p. 52
Judge adjudications
- Judge scores are cached by round and judge model, so recomputation reuses the reported adjudications instead of sampling again.Source: Appendix G · p. 52
Campaign configuration
- Declarative campaign configurations pin the run schedule, seeds, model snapshots, budgets, and persisted token and dollar costs; gathered data snapshots are point-in-time-gated.The backtest used 26 bi-weekly rounds per seed, and the live track ran a four-week daily campaign.Source: Appendix G · pp. 52–53
Verification and artifacts
- Hash-chain verification re-derives each round’s hashes, allowing a third party with the stores to confirm the recorded decisions match engine output.Audit databases, scorecard and analysis tooling, and figure scripts are available from the corresponding author on reasonable request.Source: Appendix G · p. 53