Source-linked AI summary
SURE-EVAL: A Systematic and Unified Agentic Framework for Reproducible Evaluation
Jing Peng, Junhao Du, Yixuan Wang, Bowen Wang, Hanqi Li, Chaolei Liu, Weihan Chen, Haohui Xie, Ruichen Sun, Chenghao Wang, Wen Wen, Guanyu Chen, Xiaoyu Gu, Haoyu Li, Yiwei Guo, Bohan Li, Tao Liu, Yucheng Wang, Yu Xi, Yihua Zhou, Qiang Zhou, Feng Lu, Shuai Wang, Kai Yu
TL;DR
Speech-model evaluation lacks a unified workflow connecting heterogeneous model onboarding with controlled inference and versioned scoring, even though execution and scoring choices can both affect results. SURE-EVAL combines agentic onboarding and protocol commitment with deterministic scoring; across its evaluations, onboarding completion improves and reported-to-unified score differences vary by model and language.
Problem
Speech benchmarks standardize selected datasets or scoring procedures, but heterogeneous model onboarding, controlled inference, and versioned scoring are rarely connected in one executable workflow.
Source: §1 · p. 2Method
SURE-EVAL uses a Tool Agent to create verified callable tools and a Main Agent to commit inference and scoring protocols executed by versioned deterministic programs.
Source: §1 · p. 3, §3.3 · p. 8, Inference protocol · p. 9Results
Across 18 releases, the Codex-only baseline completes 12 one-shot onboardings while the Tool Agent Workflow completes all 18; TTS reported-to-unified shifts vary by model and language.
Source: §5 · p. 15, §4.5 · p. 15Takeaways & Limitations
Explicit, versioned, rerunnable conditions make reported-versus-unified score gaps auditable rather than attributable vaguely to reproduction error.
Source: §4.5 · p. 15Table 3 measures bounded onboarding completion, not downstream model quality or repeated-run statistical reliability.
Source: §4.2 · p. 12
Abstract
from arXiv · showhide
Audio and speech models are released rapidly, but reported scores often conflate model capability with deployment and evaluation choices. The same checkpoint can produce different predictions under different runtimes, hardware, decoding settings, or fallback policies. Even fixed predictions can receive different scores under different normalization and metric implementations. Existing speech benchmarks standardize selected datasets or scoring procedures, but rarely connect heterogeneous model onboarding, controlled inference, and versioned scoring in one executable workflow. We introduce SURE-EVAL, a Systematic and Unified Agentic framework for Reproducible Evaluation of audio and speech systems. A Tool Agent Workflow converts model releases into isolated, verified callable tools. A Main Agent Workflow commits task-specific inference and scoring protocols, then delegates all score-bearing operations to versioned deterministic programs. Each result retains its runtime, protocol, pipeline nodes, predictions, and audit artifacts. Across 18 public releases covering automatic speech recognition, text-to-speech, voice conversion, speaker diarization, speaker-attributed recognition, and multi-task audio understanding, a Codex-only baseline completes 12 models in one shot, while the same agent with the SURE-EVAL Tool Agent Workflow completes all 18. We also conduct unified evaluations over seven ASR test conditions and two TTS subsets. A protocol analysis of three TTS systems finds absolute differences of 0.02-0.52 points between paper-reported and unified results, with the direction varying by model and language. These results show that reproducible evaluation requires controlling both model execution and output scoring.
1 Introduction
Speech-model scores vary because both execution conditions and scoring pipelines shape results, while existing benchmark surfaces leave heterogeneous onboarding and inference protocols under-specified. SURE-EVAL links verified model onboarding to committed inference and versioned scoring, then evaluates the design across diverse releases and controlled ASR/TTS settings.
1 Introduction
- Execution variance means a checkpoint alone does not specify runtime, hardware, precision, decoding, fallback, or other conditions that shape its behavior.Results may not transfer to another implementation when these conditions are incomplete or implicit.Source: §1 · p. 2
- Scoring variance lets fixed predictions receive different scores through choices such as normalization, tokenization, ASR frontends, and metric implementations.These choices can make apparent rankings reflect evaluation-pipeline decisions rather than model capability.Source: §1 · p. 2
- SURE-EVAL binds verified callable models, inference protocols, datasets, output contracts, versioned scoring, and run evidence into executable evaluation claims.Agents interpret incomplete evidence, while deterministic programs perform score-bearing operations.Source: §1 · p. 3
- The framework combines evidence-guided onboarding, committed evaluation planning, and versioned metric pipelines, and is evaluated across 18 releases and controlled ASR/TTS settings.The evaluation includes a same-agent comparison with and without the Tool Agent Workflow, alongside unified evaluations and a reported-versus-unified protocol analysis.Source: §1 · pp. 3–4
2 Related Work
Existing speech benchmarks standardize task suites or selected data and scoring surfaces, but generally assume models are already callable. SURE-EVAL complements them by onboarding heterogeneous releases and evaluating them under explicit, versioned execution and scoring contracts.
2.1 Reproducible Speech Benchmarking
- SUPERB standardizes downstream evaluation over frozen speech representations, while the Open ASR Leaderboard provides common normalization, WER, and efficiency reporting for ASR systems.SUPERB’s scope is representation learning; the Open ASR Leaderboard remains an ASR-specific benchmark with a predefined evaluation surface.Source: §2.1 · p. 4
- SURE organizes speech-understanding data and scoring, whereas SURE-EVAL extends the scope to heterogeneous releases, generation, conversion, runtime, decoding, and evaluator versions.Adding a release is an agentic onboarding step followed by deterministic evaluation, rather than manual integration into a benchmark entry.Source: §2.1 · p. 4
2.2 Agentic Deployment and Evaluation Automation
- SURE-EVAL bridges deployment and evaluation automation by turning heterogeneous speech releases into verified callable systems, then applying explicit, versioned inference-and-scoring contracts.Repo2Run targets repository executability; One-Eval orchestrates LLM benchmarking after an evaluation interface is available.Source: §2.2 · p. 4
3 SURE-EVAL Framework
SURE-EVAL connects heterogeneous model onboarding to committed inference and scoring protocols, with score-bearing work performed by deterministic programs. Its contracts and versioned routes keep the conditions and artifacts behind each result inspectable.
3 SURE-EVAL Framework
- The framework routes requests through the Main Agent, conditionally invokes the Tool Agent when callable readiness is unresolved, and assigns score-bearing work to deterministic programs.Agents interpret requests and evidence, while decisions that change score semantics must be committed as structured protocol inputs.Source: §3 · pp. 4–5, §3.1 · p. 6
3.1 Executable Evaluation Claims
- Each result binds a verified callable model, execution and scoring contracts, a score, and a run-level audit record.The contracts capture inference conditions and the dataset, output handling, canonicalization, and metric procedure; changes to relevant fields create a distinct evaluation claim.Source: §3.1 · pp. 5–6
3.2 Tool Agent: Evidence-Guided Model Onboarding
- The Tool Agent uses selected repository and execution evidence to build an isolated callable tool, then validates import, loading, inference, and output-contract gates.Failures are handled during onboarding; passing all gates yields a verified executable interface for evaluation.Source: §3.2 · pp. 6–8
3.3 Main Agent: Committing Comparable Evaluation
- The Main Agent commits the tool, datasets, inference profile, scoring route, entry point, paths, and expected artifacts before score-bearing execution.A bounded smoke test checks the committed end-to-end surface; failures block execution rather than authorize a bypass or partial score.Source: §3.3 · pp. 8–9, Inference protocol · p. 9
3.4 Versioned Speech Evaluation Registry
- The registry represents evaluation routes as directed sequences of versioned nodes and stores resolved node versions with predictions and aggregate metrics.Explicit routes expose task-specific choices; later updates to a normalizer or evaluator cannot silently redefine an archived result.Source: §3.4 · pp. 10–11
4 Framework Evaluation and Analysis
The evaluation tests SURE-EVAL’s onboarding workflow and committed evaluation protocols across 18 releases, unified ASR and TTS conditions, and a comparison of paper-reported with unified TTS scores.
4.1 Experimental Setup
- The experiments hold the coding agent and evidence fixed for onboarding, and commit common hardware, inference, dataset, output, and scoring settings for model evaluation.Main Agent inference uses an RTX 4090, batch size one, and no quantization; ASR spans seven test conditions and TTS uses zh/en Seed-TTS-Eval subsets.Source: §4.1 · pp. 11–12
4.2 Tool Agent Onboarding Coverage
- 18/18 releases pass one-shot onboarding with the Tool Agent, versus 12/18 with the same Codex agent alone.The six baseline misses span recognition, synthesis, diarization, and speaker-attributed recognition, with varied integration ambiguities.Source: §4.2 · p. 12
4.3 Unified ASR Evaluation
- Qwen3-ASR provides the most consistent cross-lingual coverage in the evaluated rows, while SenseVoice is competitive on Mandarin despite weaker English results.English-only systems’ Mandarin conditions are marked unsupported rather than scored with non-meaningful CER values.Source: §4.3 · p. 13
4.4 Unified TTS Evaluation
- TTS rankings vary by metric: Qwen3-TTS’s default mode leads on content error and UTMOS, while its alternative mode improves speaker-similarity estimates and English WV-MOS.The evaluated systems use shared registered audio frontends and fixed model versions.Source: §4.4 · pp. 13–14, Table 5 · p. 14
4.5 Protocol Analysis: Reported Results versus the Unified Contract
- On zh, unified TTS scores shift by -0.02 to +0.10 points; on en, all three are 0.23–0.52 points lower than reported.The direction of the reported-versus-unified gap varies by model and language; the comparison changes multiple execution and scoring factors.Source: §4.5 · pp. 14–15, Table 6 · p. 15
5 Conclusion and Outlook
SURE-EVAL binds a verified callable model, inference protocol, dataset, output contract, versioned scoring route, and run evidence into an executable evaluation claim. Across 18 releases, the Tool Agent completes all one-shot onboardings compared with 12 for the Codex-only baseline; future work will broaden the registry and support community reproduction and protocol-aware comparison.
5 Conclusion and Outlook
- SURE-EVAL binds model readiness, inference, data, outputs, versioned scoring, and run evidence into an executable evaluation claim.The framework combines agentic onboarding with controlled evaluation.Source: §5 · p. 15
- 18 releases yield 18/18 one-shot onboardings with the Tool Agent Workflow, versus 12 with the Codex-only baseline.Unified ASR/TTS evaluation and the reported-versus-unified analysis also show that execution and scoring choices can materially change result interpretation.Source: §5 · p. 15
- Future work will broaden the model, task, dataset, and metric-node registry and support community reproduction services and protocol-aware model comparison.The stated goal is more transparent, comparable, and reproducible speech research.Source: §5 · p. 15