Source-linked AI summary
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
Chengyu Shen, Yanheng Hou, Minghui Pan, Runming He, Zhen Hao Wong, Meiyi Qiang, Zhou Liu, Hao Liang, Peichao Lai, Zeang Sheng, Wentao Zhang
TL;DR
Practical LLM evaluation often requires manual benchmark discovery, heterogeneous configuration, and metric interpretation. One-Eval turns natural-language requests into traceable, customizable workflows through intent-based planning, benchmark resolution, and task-aware reporting. In experiments on 100 requests, 99 achieved executable plans, while 85% completed automated schema-related configuration and 84% completed full planning.
Problem
Existing evaluation workflows burden users with benchmark discovery, heterogeneous configuration, and result interpretation, while outputs often remain isolated scalar metrics.
Method
One-Eval converts natural-language evaluation requests into executable workflows using intent structuring, benchmark resolution and normalization, task-aware reporting, and human-in-the-loop refinement.
Results
99 out of 100 requests achieved Plan Executable Rate, with Auto-Complete Rate at 85% and Full Plan Rate at 84%.
Takeaways & Limitations
One-Eval supports traceable end-to-end evaluation with minimal manual effort and actionable evidence for auditing and debugging.
Abstract
from arXiv · showhide
Reliable evaluation is essential for developing and deploying large language models, yet in practice it often requires substantial manual effort: practitioners must identify appropriate benchmarks, reproduce heterogeneous evaluation codebases, configure dataset schema mappings, and interpret aggregated metrics. To address these challenges, we present One-Eval, an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. One-Eval integrates (i) NL2Bench for intent structuring and personalized benchmark planning, (ii) BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization to ensure executability, and (iii) Metrics \& Reporting for task-aware metric selection and decision-oriented reporting beyond scalar scores. The system further incorporates human-in-the-loop checkpoints for review, editing, and rollback, while preserving sample evidence trails for debugging and auditability. Experiments show that One-Eval can execute end-to-end evaluations from diverse natural-language requests with minimal user effort, supporting more efficient and reproducible evaluation in industrial settings. Our framework is publicly available at https://github.com/OpenDCAI/One-Eval.
1 Introduction
Model evaluation increasingly informs development and deployment decisions, but existing workflows leave users to discover benchmarks, configure heterogeneous systems, and interpret results manually. One-Eval addresses this gap by transforming natural-language requests into customizable, executable workflows with decision-oriented reporting and human review.
- Evaluation results increasingly serve as decision-making signals for model comparison, deployment readiness, and risk assessment.
- Existing practices require users to reproduce benchmark repositories or configure static frameworks, creating substantial burdens in benchmark discovery, setup, and result interpretation.
- Agentic evaluation reframes the task as transforming abstract evaluation intents into reliable and actionable pipelines, beyond merely executing evaluations or aggregating scores.
- Existing tools rarely support intent interpretation, personalized benchmark selection, configuration validation, or decision-tailored analysis, leaving outputs as insufficient isolated scalar metrics.
- One-Eval integrates intent decomposition, benchmark retrieval and configuration, task-oriented reporting, and human checkpoints for reviewing and refining agent decisions.
2 Related Work
Prior evaluation research provides benchmarks and standardized toolkits, while agent systems automate complex multistep tasks. One-Eval builds on these lines by treating evaluation as an end-to-end, intent-driven decision process with personalized reporting.
- Model-evaluation research spans domain-specific benchmarks such as GSM8K, MATH, and MMLU, alongside standardized toolkits including lm-eval-harness and OpenCompass.
- Agent-based systems reduce manual effort by decomposing high-level goals into sequential decisions for complex, multistep workflows.
- Existing automation generally targets isolated evaluation components rather than an end-to-end agent-driven decision process, leaving fragmented practical support.
- Most evaluation reports use single or aggregated metrics, while multidimensional approaches often retain fixed dimensions and static formats that are weakly aligned with user goals.
3 System Design
One-Eval models evaluation as a modular, human-refinable pipeline from natural-language intent to executable benchmark workflows and task-oriented reports. NL2Bench selects and normalizes benchmarks, while resolution and configuration adapt heterogeneous data for execution.
- 3.1 Framework Overview: One-Eval transforms high-level natural-language evaluation requests into executable and verifiable workflows driven by user intent.
- 3.1 Framework Overview: The three-stage pipeline interprets intent and constructs a plan, resolves benchmarks and settings into an executable workflow, then produces task-oriented results and reports with human validation.
- 3.1 Framework Overview: Metric Recommendation and Reporting selects task-based metrics and generates structured reports rather than isolated scalar scores, connecting workflow construction with result interpretation.
- 3.2 NL2Bench: NL2Bench structures requests into intents, recommends aligned benchmarks, and supports iterative user refinement before producing a benchmark plan for downstream execution.
- 3.2 NL2Bench: Candidate retrieval combines a 77-benchmark local gallery with embedding or TF-IDF ranking and live HuggingFace search for insufficient quality matches or long-tail benchmarks.
- 3.3 Benchmark Resolution and Configuration: BenchResolve acquires datasets, resolves identifiers, and normalizes heterogeneous benchmarks into validated configurations containing sources, splits, mappings, and task metadata.
4 Experiments
The experiments assess One-Eval’s industrial usability through a case study, end-to-end success-rate evaluation, and feature comparison. Across 100 requests, the system produced executable plans at high rates without manual correction while preserving an auditable workflow trail.
- 4 Experiments: The experiments evaluate actionable outputs, executable plans without human edits, and capabilities beyond existing frameworks through three complementary studies.
- 4.1 Case Study: The case study shows how a single natural-language request becomes an executable, auditable workflow with preserved evidence for diagnosing failures.
- 4.1 Case Study: The case study translates a request for broad knowledge and light reasoning into a benchmark plan spanning MMLU, TruthfulQA, CommonsenseQA, GSM8K, and MATH-500.
- 4.1 Case Study: BenchResolve records runnable configurations and splits, caches resolved datasets, and normalizes heterogeneous schemas through key mappings in the demonstrated workflow.
- 4.2 End-to-End Success Rate: Table 1 reports cumulative success checkpoints and efficiency, while Table 2 compares workflow-critical features including customization, automation, and intent-conditioned recommendations.
5 Conclusion
One-Eval executes natural-language evaluation requests as traceable end-to-end workflows through NL2Bench, BenchResolve, and task-aware Metrics & Reporting. Experiments show reliable execution with minimal manual effort and actionable evidence for auditing and debugging.
- One-Eval executes natural-language evaluation requests as traceable end-to-end workflows through NL2Bench, BenchResolve, and task-aware Metrics & Reporting.
- Experiments demonstrate reliable execution with minimal manual effort and actionable evidence for auditing and debugging.
A Alignment with Representative Evaluation Frameworks
One-Eval’s framework comparison focuses on workflow capabilities relevant to industrial evaluation, including customization, automation, intent-conditioned recommendations, and reproducible reporting. Its native workflow records concrete artifacts and distinguishes first-class capabilities from behavior requiring manual configuration or external scripting.
- Scope of the comparison: The comparison evaluates custom assets, setup automation, intent-conditioned recommendations, and workflow-level reporting rather than overall quality, speed, or benchmark coverage.
- Criterion 1: Custom benchmarks and metrics: Custom support requires maintainable interfaces for registering benchmarks or datasets and attaching custom evaluation logic or metrics without modifying core runner internals.
- Criterion 2: End-to-end automation: End-to-end automation starts from a high-level requirement and produces a runnable plan with minimal manual specification of benchmarks, splits, schema mappings, and settings.One-Eval resolves identifiers, downloads assets, normalizes schemas, and records intermediate artifacts for traceability.
- Criterion 3: Benchmark recommendation: Benchmark recommendation is intent-conditioned, combining structured goals and constraints with candidate retrieval and an inspectable benchmark plan with justifications.
- Criterion 4: Metric recommendation: Metric recommendation proposes task- and objective-conditioned metric suites with reporting templates, including breakdowns, slices, and diagnostic signals.
- Interpretation of blanks/✗: Blank or ✗ entries indicate capabilities are not exposed natively as intent-conditioned first-class modules, although external scripting or engineering effort may reproduce similar behavior.
- Reproducibility and reporting: One-Eval persists plans, resolved benchmark identifiers, configuration choices, and report outputs so checked features can be mapped to concrete system artifacts.The comparison applies criteria consistently to user-facing workflow primitives such as interfaces, registries, and planning modules.
B Selected Custom Metrics
One-Eval’s curated custom metrics target diagnostic insight rather than only generic aggregate scores. They support more actionable, decision-oriented reporting across tasks by addressing specific failure modes.
- One-Eval’s custom metrics target symbolic equivalence, schema and format compliance, and judge-based error attribution.These metrics are designed to expose specific failure modes for diagnostic insight.
C Additional Discussion
One-Eval is designed for deployment-oriented evaluation in which stakeholders need structured, traceable evidence for decisions and iterative refinement. Its modular workflow and human checkpoints accommodate changing domains, benchmarks, reporting needs, and configuration choices.
- Decision-oriented evaluation outputs: Industrial evaluation supports model selection, release gating, regression monitoring, and targeted iteration, requiring signals beyond aggregate scores to identify where and why performance changes.
- From scores to actionable signals: One-Eval connects aggregate results with slices, categories, and representative examples to support regression identification, data collection, prompt or template adjustments, and cross-role communication.
- Traceability and auditability for iterative evaluation: Plans, configurations, resolved benchmark identifiers, schema mappings, caches, and per-sample traces are preserved as first-class state for auditing and reproduction.Reproducible evidence reduces ambiguity and accelerates review when results inform stakeholder explanations or release decisions.
- Human-in-the-loop as controlled refinement: Human-in-the-loop checkpoints let users confirm benchmark coverage, adjust constraints, and approve reporting preferences during controlled refinement.These interactions focus on validating high-impact decisions.
- Supporting evolving and heterogeneous requirements: Separating intent understanding, benchmark resolution and configuration, and evaluation and reporting enables new sources, domain-specific templates, and metric suites without changing the stable workflow.
D Consistency with Benchmark Cards and Standard Protocols
One-Eval grounds benchmark configuration and execution in benchmark metadata and records protocol decisions as reproducible artifacts. It also normalizes heterogeneous schemas and checks executability before runs.
- Protocol grounding via benchmark metadata: Benchmark metadata disambiguates identifiers, selects canonical configurations, and surfaces split availability and constraints affecting executability.Dataset cards provide task definitions, subsets, splits, label semantics, and field descriptions used during resolution.
- Split and subset selection policy: One-Eval prioritizes canonical test splits, falls back to supported alternatives, and records split-selection rationale in benchmark configuration state.For multiple subsets or configurations, metadata cues guide selection of a representative subset.
- Versioned identifiers and reproducible artifacts: Resolved identifiers, configuration parameters, cache locations, and supported version information link reported results to exact evaluated data.Recorded repository revisions, commits, or snapshot metadata strengthen reproducibility when available.
- Schema alignment with benchmark definitions: Explicit key mappings normalize heterogeneous feature schemas into a unified input-output interface while remaining inspectable and refinable.Mappings are stored with benchmark configurations to preserve transparency and consistency with benchmark definitions.
- Executability and validity checks: Validity checks verify required fields and evaluable targets, while per-run artifacts and sample traces support auditability and reproduction.Recorded plans, resolved benchmark sets, configuration decisions, and cache references let practitioners verify protocol choices.
E Prompts
One-Eval uses specialized prompts to structure evaluation requests, recommend benchmarks and metrics, resolve dataset configurations, and map heterogeneous datasets to evaluation tasks. These prompts enforce machine-readable outputs and support executable workflow construction.
- Query understanding: QueryUnderstandAgent converts natural-language requests into structured fields covering task status, modality, domains, benchmarks, models, and special requirements.Its output is restricted to a parsable JSON object without explanation.
- Benchmark search: BenchSearchAgent recommends publicly available, widely used benchmark names aligned with the user’s latest requirements.The prompt requests benchmark descriptions and permits full repository identifiers when known.
- Metric recommendation: MetricRecommendAgent matches metrics to benchmark metadata and sample data according to the fundamental nature of each task.Recommendations use registry-aligned metric names, priorities, arguments, and descriptions in JSON format.
- Benchmark configuration: BenchConfigRecommend selects dataset splits and configurations using an explicit priority: test, test-like, validation, development, or training with a warning.It returns a JSON object containing the recommended configuration, split, and rationale.
- Task mapping: Evaluation task mapping classifies datasets by task type and assigns required input, target, choice, label, or context fields.Supported categories include text scoring, single-reference generation, multiple-reference generation, and multiple-choice evaluation.