PaperlayerView exact revision on arXiv

Source-linked AI summary

StableEval Arena: A Cost-Aware Agentic Benchmark for Stablecoin Price Stability Prediction

Sean Wan, Dongping Liu, Luyao Zhang

arXiv:2609.18949v1cs.LGcs.AIcs.CEecon.GNq-fin.CP

TL;DR

Stablecoin risk evaluation must distinguish ordinary peg noise from rare, high-consequence stress, while existing benchmarks do not directly assess agentic stablecoin risk prediction under cost and reliability constraints. StableEval Arena uses leakage-safe historical replay with complementary stress-enriched and natural-distribution settings. It finds that agents can produce valid structured outputs yet remain weak at rare-stress and sustained-depeg detection.

  • Problem

    Existing benchmarks do not directly test whether agentic systems can assess stablecoin risk under cost and reliability constraints, despite the need to distinguish normal peg noise from rare, high-consequence stress.

    Source: §1 · p. 2

  • Method

    StableEval Arena replays historical cases with a frozen 30-day lookback and structured next-seven-day risk assessments in a 120-case stress-enriched block and a 507-case natural-distribution block.

    Source: §1 · p. 2

  • Results

    Agents return valid structured predictions under the shared protocol, but rare-stress and sustained-depeg detection remain weak, and simple baselines remain strong in the natural-distribution evaluation.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The benchmark distinguishes protocol-following reliability from dependable detection of rare peg stress or sustained depeg.

    Source: §5 · p. 7

    The natural-distribution pool contains only 21 stress cases, so stress-recall and missed-depeg results are risk-warning signals rather than final statistical conclusions.

    Source: §7 · p. 9

Abstract

from arXiv · show

We introduce StableEval Arena, a cost-aware benchmark framework for evaluating agentic AI systems on stablecoin peg-risk prediction. StableEval Arena evaluates LLM-backed agentic systems on diagnosing peg stress and forecasting deviations from the one-dollar peg over a hidden seven-day horizon, using leakage-safe historical replay with exchange price-volume data and market-context features. We report two complementary experiment blocks: a 120-case stress-enriched validation block and a 507-case natural-distribution full-arena evaluation block. Across six LLM-backed agent configurations and baselines, StableEval Arena measures prediction quality, calibrated-label behavior, structured-output reliability, latency, token consumption, and estimated inference cost. Rather than ranking agents by accuracy alone, the framework treats trustworthiness as a joint property of forecast quality, operational reliability, and computational cost. The results show a gap between protocol-following reliability and financial-risk reliability: agents reliably produce valid structured outputs at modest measured cost, but still miss most rare severe-stress and sustained-depeg cases. To support auditing and replication, we release the benchmark dataset on Hugging Face and the source code on GitHub.

STABLEEVAL ARENA

StableEval Arena is a leakage-safe benchmark for stablecoin risk evaluation across a 120-case validation block and a 507-case full arena.

STABLEEVAL ARENA

A. Historical Data and Two Case Blocks

The historical-data workflow separates a 120-case validation experiment from a 507-case full-arena experiment and reports their outputs separately.

A. Historical Data and Two Case Blocks

Keywords

The listed keywords span agent evaluation, stablecoin stability, benchmarking, infrastructure, decentralized finance, and trustworthy AI.

Keywords

  • The keywords include Agentic AI Evaluation and AI Agent Infrastructure.Source: Keywords · p. 1
  • Stablecoin Price Stability and Cost-aware Benchmarking are listed as keywords.Source: Keywords · p. 1
  • The remaining keywords are Decentralized Finance and Trustworthy AI.Source: Keywords · p. 1

1 Introduction

Stablecoin risk assessment concerns whether a peg will remain stable or enter warning or stress, a question existing agent benchmarks do not directly address under cost and reliability constraints. StableEval Arena evaluates structured next-seven-day risk forecasts through historical replay, and its results show agents can follow the protocol without reliably detecting rare financial stress.

1 Introduction

  • Stablecoin risk assessment asks whether a peg will remain stable, enter a warning state, or experience peg stress, rather than forecasting the full future price path.Source: §1 · pp. 1–2
  • Existing benchmarks do not directly test agentic systems as stablecoin risk-assessment agents under cost and reliability constraints.Source: §1 · p. 2
  • StableEval Arena uses a frozen 30-day lookback window for structured next-seven-day risk assessments and compares agents with non-agent baselines under one protocol.It measures prediction quality, stress recall, missed-depeg risk, calibration, valid-output rate, latency, token usage, and inference cost across a 120-case stress-enriched block and a 507-case natural-distribution block.Source: §1 · p. 2
  • Agents produce valid structured outputs under the shared protocol, but rare-stress and sustained-depeg detection remain weak across the two evaluation settings.In the 507-case natural-distribution block, simple baselines remain strong and raw accuracy is shaped by stable-dominant class balance.Source: §1 · p. 2

2 Background and Related Work

Stablecoin risk monitoring must distinguish routine peg noise from stress rooted in market mechanisms, liquidity, and redemption conditions. The background motivates evaluating agentic financial systems on decision quality together with protocol compliance, reliability, and resource use.

2.1 Stablecoin Risk Monitoring

  • Stablecoin stability depends on market design, arbitrage access, reserves or collateral, redemption, liquidity, confidence, and regulation, so ordinary peg behavior may not reveal stress exposure.The cited work frames monitoring around warning or stress conditions that can affect payments, liquidation thresholds, collateral valuation, and settlement reliability.Source: §2.1 · p. 2

2.2 Agentic Financial Prediction and Evaluation

  • Agentic financial prediction synthesizes bounded evidence into structured judgments, making instruction following, valid outputs, consistency, and resource constraints part of evaluation alongside prediction quality.The section connects time-series benchmarks, multimodal cryptocurrency forecasting, and agent-evaluation work to motivate process-level assessment.Source: §2.2 · p. 2

2.3 Cost-Aware and Trustworthy Benchmarking

  • Cost-aware benchmarking checks whether systems remain reliable under operational conditions, since aggregate accuracy can coexist with missed stress events, malformed outputs, high latency, or irreproducible results.Stablecoin monitoring may query agents repeatedly across assets and time windows, making decision quality, cost, and reliability jointly relevant.Source: §2.3 · pp. 2–3
  • StableEval Arena’s six system comparators are characterized by implementation role, interface, memory, orchestration, and control; four are native/framework-level and two are adapter-based OpenClaw-family comparators.The section also motivates reporting stress recall, missed-depeg rates, false alarms, calibration, deviation error, cost, latency, and failure modes, with transparent documentation.Source: §2.3 · p. 3, Table 1 · p. 3

3 StableEval Arena

StableEval Arena is a historical-replay benchmark for structured, short-horizon stablecoin risk assessment under a shared, leakage-safe protocol. It pairs natural-distribution evaluation with stress-enriched diagnostics and records predictive, operational, and reproducibility measures.

3.1 Overview

  • StableEval Arena evaluates short-horizon stablecoin risk through two complementary blocks: a 507-case natural-distribution pool and a 120-case stress-enriched diagnostic subset.The natural pool tests natural class balance, while the enriched subset concentrates rare watch and stress outcomes for sharper stress analysis.Source: §3.1 · pp. 3–4

3.2 Task Formulation

  • Each case supplies a frozen 30-day hourly lookback and hides the next seven days, requiring a structured stable/watch/stress assessment rather than a full future price path.The output includes depeg probability, maximum peg-deviation estimate, direction, confidence, and rationale.Source: §3.2 · p. 4

3.3 Case Library and Construction

  • Cases use hourly price-volume data and market-structure features for USDT, USDC, and DAI, with BTC and ETH as context assets; all agent-facing features use only the lookback window.Hidden future labels and horizon statistics support case selection and evaluation but are excluded from agent-facing prompt packets.Source: §3.3 · p. 4
  • The frozen evaluator assigns stable, watch, or stress labels using pre-declared deviation and duration thresholds, designed to avoid ordinary peg-noise labels while retaining sensitivity to consequential deviations.The stress-enriched subset retains all watch and stress cases and adds stable controls without changing the historical-replay protocol.Source: §3.3 · p. 4

3.4 Agent Systems

  • The 507-case comparison includes four native/framework-level agents, two adapter-based OpenClaw-family extensions, and non-agent baselines such as persistence and threshold rules.Adapter-based OpenClaw and NanoClaw results are interpreted as protocol-compatibility, output-validity, and cost-logging tests because verified local execution was unavailable.Source: §3.4 · p. 4

3.5 Evaluation Protocol, Outputs, and Metrics

  • All systems receive the same case packets and strict JSON schema under a fixed backend and protocol, with Macro-F1, balanced accuracy, stress recall, missed-depeg rate, calibration, and deviation error emphasized over accuracy alone.The benchmark retains raw and calibrated outputs separately and scores probabilistic predictions with Brier score and maximum-deviation estimates with error metrics.Source: §3.5 · p. 5

3.6 Cost, Reliability, and Reproducibility

  • Each run records prediction and operational diagnostics, while preserved prompts, leakage checks, metadata, raw and calibrated predictions, cost logs, scripts, and versioned leaderboards support auditing and replication.Operational records include token usage, latency, estimated inference cost, valid-JSON status, and execution failures.Source: §3.6 · pp. 5–6

4 Results

Across the two benchmark blocks, calibrated agents improve aggregate classification in the stress-enriched setting, but simple baselines remain stronger on the natural-distribution pool. Structured-output reliability is high, while rare-stress and sustained-depeg detection remain weak.

4.1 120-Case Stress-Enriched Validation

  • Under the calibrated first-pass protocol in the stress-enriched setting, five of six agentic systems exceed historical persistence on Macro-F1, although Nanobot and OpenClaw stress recall is 0.095 versus 0.333.Source: §4.1 · p. 6

4.2 507-Case Natural-Distribution Scaling

  • In the 507-case natural pool, simple ML/logistic leads with Macro-F1 of 0.4019, while LangGraph leads agents with Macro-F1 of 0.3044; agent stress recall peaks at 0.0952.There are 441 stable cases among the 507, making raw accuracy especially misleading.Source: §4.2 · p. 6

4.3 Cross Alignment and Stress Diagnostics

  • 19 of 21 true stress cases are missed by the six agentic systems, and no agentic system lowers missed-depeg below 1.0000; historical persistence reports 0.6667.Restricted 507-case runs are generally close to or better than corresponding 120-case results, so the full-pool difference mainly reflects stable-dominant distribution.Source: §4.3 · pp. 6–7

4.5 Summary

  • In the 120-case block, calibrated agents improve aggregate Macro-F1 over simple baselines, but no agentic system exceeds simple ML/logistic on Macro-F1 in the 507-case natural pool.The two blocks test stress sensitivity and full-pool robustness, respectively.Source: §4.2 · p. 6, Table 2 · p. 7, §4.5 · p. 7

4.4 Calibration, Ablation, and Cost-Reliability Tradeoffs

  • 0.351 best calibrated first-pass Macro-F1 rises from 0.301, with best stress recall rising from 0.000 to 0.095; verifier stress recall reaches 0.143 while Macro-F1 is 0.349.For Nanobot, verifier-calibrated inference costs $0.978 versus $0.345 for calibrated first-pass inference.Source: §4.4 · p. 7
  • Anonymization produces a maximum label-change rate of 0.333, while all main 120-case agent runs achieve valid JSON rate of 1.0 and failure rate of 0.0.Completed 507-case runs also maintain reliable structured-output collection, distinct from weak rare-stress detection.Source: §4.4 · p. 7

5 Discussion and Analysis

StableEval Arena separates reliable protocol execution from reliable financial-risk detection: valid structured predictions do not translate into dependable detection of rare peg stress or sustained depeg. Its stress-enriched and natural-distribution blocks expose different aspects of that gap.

5.1 Two-Block Evaluation

  • On StableEval-120-Enriched, calibrated agents improved Macro-F1 over the strongest simple baseline, but low stress recall still left severe-risk detection unreliable.The stress-enriched block concentrates non-stable cases so stress behavior is not washed out by the stablecoin base rate.Source: §5.1 · pp. 7–8
  • On StableEval-507-Natural, simple ML/logistic remained strongest by Macro-F1 and balanced accuracy, while an always-stable baseline achieved high accuracy but recovered no stress cases.With 441 of 507 cases stable, accuracy is a weak indicator of risk sensitivity.Source: §5.1 · p. 8

5.2 Reliability vs. Financial Trustworthiness

  • The six systems completed the 507-case protocol with valid outputs, enabling comparison through one historical-replay interface despite weak financial-risk detection.The shared protocol and execution logs make malformed outputs, missing predictions, runtime failures, and unlogged costs visible in comparisons.Source: §5 · p. 7, §5.2 · p. 8
  • 19 of 21 true stress cases were missed by all six systems, and no agentic system reduced the sustained-depeg missed-depeg rate below 1.0000.Valid, low-cost JSON outputs therefore do not establish trustworthy monitoring.Source: §5.2 · p. 8

5.3 Why Rare Stress Remains Difficult

  • The 30-day hourly lookback supports reproducible, leakage-safe evaluation, but bounded price and market evidence omits reserves, redemption frictions, exchange depth, and other stress signals.Conservative stable/watch choices under ambiguity can aid aggregate performance in the enriched block, but may miss rare high-consequence events; calibration cannot recover signals absent from the packet.Source: §5.3 · p. 8

6 Industry Implications and Future Work

StableEval Arena’s weak rare-stress detection motivates extending agentic payment evaluation beyond price prediction to authorization, settlement, pricing, and stakeholder-aware risk. The proposed future work broadens evidence and tests those payment-infrastructure settings.

6 Industry Implications and Future Work

  • Weak rare-stress detection remains a barrier to using agentic systems as reliable monitoring layers for stablecoin-based payment infrastructure.Procedural reliability does not imply reliable financial-risk detection or payment safety.Source: §6 · p. 8
  • Stablecoin recurring payments make delegated authorization, consent scope, spending limits, revocation, auditability, and liability concrete engineering requirements.The cited payment approaches remain competing emerging directions rather than settled standards or broadly adopted practice.Source: §6 · p. 8
  • Payment systems need explicit rules for finality, reconciliation, escalation, and loss allocation; on-chain confirmation alone does not establish economic finality.Economic finality also depends on usable funds, available liquidity, cleared compliance checks, completed reconciliation, and assigned responsibility for failure or delay.Source: §6 · pp. 8–9
  • Venue-specific price divergence and tax handling raise accountability questions about spreads, jurisdiction, fees, tax calculation, and remittance responsibility.Future benchmarks could label business costs such as false interruptions, spread losses, tax misclassification, and compliance disputes.Source: §6 · p. 9
  • Future versions should broaden evidence and assets, use stakeholder-aware risk-sensitive costs, test multiple backends and native execution, and evaluate payment orchestration under explicit constraints.Proposed additions include liquidity, redemption, oracle, reserve, news, authorization, and settlement evidence.Source: §6 · p. 9

7 Limitations

StableEval Arena’s conclusions are bounded by rare stress observations, limited surveillance evidence, and residual risks from recognizable packet details. The current asset scope also covers only three stablecoins.

7 Limitations

  • StableEval-507-Natural contains only 21 stress cases, so stress-recall and missed-depeg results are risk-warning signals rather than final statistical conclusions.StableEval-120-Enriched concentrates stress cases for diagnostic evaluation; the asset scope is USDT, USDC, and DAI.Source: §7 · p. 9
  • The benchmark prioritizes bounded, reproducible packets over exhaustive real-world surveillance and does not include every high-cost or real-time risk signal.The current packets omit signals such as reserve attestations, redemption flows, cross-chain liquidity, oracle feeds, regulatory events, and news; the single qwen/qwen-2.5-72b-instruct backend does not test backend generality.Source: §7 · p. 9
  • Residual parametric-memory effects cannot be fully ruled out because some packets retain real stablecoin names and dates despite hidden future labels and anonymization checks.Future releases plan broader anonymized and counterfactual evaluation.Source: §7 · p. 9

8 Conclusion

StableEval Arena is a cost-aware benchmark for agentic stablecoin risk assessment, pairing stress-enriched validation with a natural-distribution evaluation. Across both settings, systems produce valid structured outputs, but rare-stress and sustained-depeg detection remain weak.

8 Conclusion

  • The benchmark pairs 120-case stress-enriched validation with a 507-case natural-distribution scaling study for rare-risk diagnosis and full-pool evaluation.Each case uses a 30-day evidence window for structured next-seven-day stable/watch/stress, depeg-probability, deviation, direction, confidence, and rationale predictions.Source: §8 · p. 9
  • Across both settings, systems produce valid structured outputs, but rare-stress and sustained-depeg detection remain weak.The central finding is a gap between execution reliability and financial-risk detection.Source: §8 · p. 9
  • StableEval Arena evaluates prediction quality, stress recall, missed-depeg risk, calibration, valid-output rate, latency, token usage, and inference cost under a unified protocol.It compares agentic systems with simple baselines.Source: §8 · p. 9
  • Future work should broaden evidence packets and asset coverage, test backend sensitivity and more robust native agent execution, and evaluate richer payment and settlement scenarios.Source: §8 · p. 9

A Shared Protocol, Data, and Execution Metadata

The appendix documents a shared replay and execution protocol across stress-enriched and natural-distribution evaluations, with audits and artifacts for tracing results. The natural-distribution block shows calibration gains in aggregate Macro-F1, while rare-stress and sustained-depeg detection remain weak.

A.2 Dataset, Labels, and Case Construction

A.3 Agentic Systems and Execution Metadata

  • Six platforms share one model backend and prompt/evaluator protocol, so comparisons measure platform-mediated execution rather than independent foundation-model ability.LLM inference is served remotely; local hardware supports orchestration, parsing, calibration, scoring, and analysis.Source: §A.3 · p. 12, §A.4 · p. 12

B StableEval-120-Enriched Validation Results

B.3 120-Case Stress and Ablation Diagnostics

  • In the 120-case ablation, calibration improves the best aggregate metrics without extra calls, while the verifier raises best stress recall but approximately doubles mean total cost.Calibration remaps the predicted class using model-estimated risk; the verifier adds a risk-auditor call using the same lookback evidence.Source: Figure 4 · p. 13, Table 12 · p. 15, §B.3 · p. 15

C.3 507-Case Stress and Calibration Diagnostics

  • On the 507-case stable-heavy set, calibration improves every agent’s Macro-F1, but non-agent baselines remain strong and rare-stress detection remains weak.The full-pool results are a scaling check, not a replacement for stress-enriched validation.Source: Figure 8 · p. 19, §C.3 · p. 20, §D.1 · p. 21

D.1 120-Versus-507 Scaling Context

  • Across blocks, the 507-case results reflect distribution-sensitive scaling, while strict sustained-depeg detection remains weak.The 507-case pool has lower stress prevalence and many more stable controls than the 120-case block.Source: §D.1 · p. 21

D.2 Anonymization and Leakage-Safety Audit

  • The matched 12-case anonymization audit keeps numerical evidence fixed while changing coin names and dates; it is an audit signal, not proof of no leakage.The comparison uses real identifiers versus anonymized aliases and relative time labels.Source: §D.2 · p. 21

E Reproducibility Artifacts and Release Checklist

  • The release preserves frozen cases, prompt packets, raw predictions, calibration rules, evaluator outputs, cost logs, and cross-variant summaries for tracing results.Metric-consistency and alignment files serve as audit artifacts rather than additional leaderboard tables.Source: §D.3 · p. 21, E Reproducibility Artifacts and… · p. 22

Related papers

Loading 2609.18949v1…