Source-linked AI summary
Benchmark Everything Everywhere All at Once
Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai, Bokang Yang, Wencheng Han, Xiao-Hui Li, Xiangyu Yue
TL;DR
Existing benchmark construction is labor-intensive and benchmarks can saturate quickly, limiting sustainable and discriminative evaluation. Benchmark Agent automates customizable benchmark construction through planning and controlled execution, producing reliable, discriminative, and cost-efficient benchmarks while supporting rapid iteration.
Problem
Human-driven, benchmark-specific construction requires repeated manual effort, while rapid performance saturation limits differentiation among state-of-the-art methods.
Method
Benchmark Agent uses a Planner to formulate benchmark specifications and an Executor to instantiate them into evaluation-ready items with quality control.
Results
Generated benchmarks are reported as reliable, discriminative, and cost-efficient across human, consistency, and efficiency evaluations.
Takeaways & Limitations
The framework supports customized, fine-grained benchmarks and faster updates, reducing human effort and turnaround time for evolving evaluation needs.
Takeaways & Limitations
Generated benchmarks are mainly validated on representative tasks, tool coverage bounds the framework, and lightweight human review remains useful.
Abstract
from arXiv · showhide
Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.
1 Introduction
Benchmark Agent is introduced as a fully autonomous framework for customizable benchmark construction, addressing the labor, reuse, iteration, and saturation limitations of existing benchmarks. Its automated, reusable pipeline is evaluated for quality, discriminative power, efficiency, and cost effectiveness.
- Motivation: Existing benchmark pipelines are largely human-driven, requiring substantial effort and often rebuilding each benchmark from scratch.This creates repeated manual work and slow iteration cycles.
- Motivation: Existing benchmarks can reach performance saturation shortly after release, with Qwen-series accuracy scores exceeding 80%.This leaves modest headroom for differentiating state-of-the-art methods.
- Framework: Benchmark Agent automatically designs, adapts, and manages benchmarks according to user-specified evaluation needs.The framework uses a dual-component design inspired by a brain-cerebellum hierarchical architecture for this long-horizon task.
- Framework: Benchmark Agent supports user-oriented customization by adjusting task formats, domains, and evaluation criteria to specific requirements.It focuses on customized and personalized evaluation rather than only broad, uniform general-purpose assessment.
- Evaluation: Benchmark Agent standardizes and automates construction stages, enabling reusable pipelines with lower human costs and continual benchmark refreshability.Experiments assess generated-benchmark quality through human evaluation, LLM-as-a-judge assessment, and consistency checks, alongside efficiency and cost-effectiveness analyses.
2 Related Works
Related work spans agent-based evaluation, agent-based data synthesis, and broader agent systems for complex interactive tasks. Benchmark Agent combines these directions into an autonomous pipeline that plans, transforms, and verifies benchmark items.
- Agent-based Evaluation: LLM agents have been used to evaluate or compare model outputs, with MLLM-as-a-Judge extending judging across diverse modalities.Recent work also explores agent-based evaluation frameworks that automate parts of the evaluation process.
- Agent-based Data Synthesis: Agent-based data synthesis uses iterative reasoning, interaction, or tool use to generate large-scale training data while reducing human annotation.Multi-round refinement has been applied to code generation and multimodal content creation.
- Agent Systems: Agentic systems now support complex interactive domains by perceiving multimodal observations, maintaining long-horizon plans, and executing interdependent actions.Applications include web browsers, mobile devices, and scientific discovery.
- Benchmark Agent: Benchmark Agent decomposes user requirements, grounds subtasks in real datasets, allocates resources under global constraints, then transforms and verifies samples against quotas.Its pipeline comprises Benchmark Planner and Benchmark Executor.
3 Benchmark Agent
Benchmark Agent converts abstract evaluation requirements into standardized, executable benchmark specifications through coordinated planning, grounding, and allocation. It then realizes validated specifications into evaluation-ready items while continuously enforcing semantic validity and structural compliance.
- Framework: Benchmark Agent uses a Benchmark Planner and controlled execution framework to transform abstract requirements into high-quality, evaluation-ready benchmarks.The Planner handles specification design, while execution instantiates feasible specifications into concrete items.
- Benchmark Planning: The Design Agent converts informal requirements into independent, testable subtasks and revises them until their formulations are well-posed and meaningful.It uses proposal and revision tools before passing stabilized subtasks to grounding.
- Benchmark Planning: The Grounding Agent searches for compatible datasets, validates executable transformation plans, and rejects specifications lacking at least one valid dataset–transformation grounding per subtask.Rejected specifications return to the Design Agent, creating a design–grounding feedback loop.
- Benchmark Planning: The Allocation Agent checks grounded instantiations against global quota and resource constraints, diagnosing bottlenecks or incompatible quotas when feasibility is violated.A feasible instantiation records a dataset, validated transformation plan, and allocated sample quota.
- Benchmark Realization: For each grounded instantiation, the system generates the allocated number of items through adaptive orchestration and execution, feeding intermediate outputs back into subsequent planning.Planning remains constrained by the dataset-level transformation plan to prevent uncontrolled divergence across samples.
- Quality and Quota Control: Only samples passing continuous semantic-validity and structural-compliance checks enter the benchmark; invalid samples are discarded or technically fixed.Verification tests whether each sample reflects the intended evaluation objective and conforms to the required output format.
4 Experiments
Benchmark Agent generates customized benchmarks with high human acceptance and usable item-level quality across text, audio, image, and audio-text-image scenarios. Experiments show that its agentic workflow preserves user intent, supports model discrimination, reduces construction cost, and reveals uneven multimodal model performance.
- Experimental Setup: Benchmark Agent covers 15 benchmarks across four scenarios: 4 text-only, 3 audio-text, 7 image-text, and 1 audio-text-image.The benchmarks are constructed from user demands, with details and visualizations provided in the appendices.
- User-oriented Customization: Human acceptance rates reach around 96–98%, while UIA scores range from 68.54 to 81.48 across five representative customized benchmarks.Human experts assess correctness, clarity, answerability, and relevance; UIA measures alignment with the user’s evaluation goal.
- Item-level Quality: FSQ and QAC remain consistently high, whereas variation in CQC and lower TSD/SSC scores expose challenges in grounding, target-signal dependence, and difficulty control.The results indicate that generated samples are generally well-structured and semantically coherent but less consistent in evidence grounding and difficulty design.
- Model Discriminativeness: Qwen3.5 scores scale from 71.06–87.23 on Multi-Perspective, 40.96–56.38 on Art-Reasoning, and 45.26–54.49 on Math-Reasoning as model size increases from 2B to 27B.These consistent scaling trends suggest that the generated benchmarks provide useful signals for model comparison.
- Cost and Components: Human annotation requires higher labor and time costs than Benchmark Agent, which can further accelerate updates through increased deployment bandwidth.Component ablations also show distinct roles for the Design Agent, TC + Scoring, sample-level planning, and verification.
- Ablation Studies: Direct LLM generation scores much lower overall, especially on UIA, TSD, and SSC, while different backbone models remain usable with scores of 73.17–79.88, 65.64–70.75, and 70.67–73.82 on three benchmarks.The comparison indicates that agentic workflow is needed to preserve user intent and control evaluation signals, while performance is relatively stable across tested backbones.
- Multimodal Findings: Qwen models perform strongly in multimodal geography and math but struggle on fine-grained art and animal tasks, indicating unbalanced domain performance.The observed weaknesses may reflect architectural or training-data limitations.
5 Discussion and Conclusion · Appendix Contents · A Experiment Details
Benchmark Agent is presented as an autonomous framework for customizable benchmark construction that addresses the scalability and sustainability limits of human-driven evaluation. Its distinguishing strengths are fine-grained customization and rapid, efficient iteration, while experiments indicate reliable, discriminative, and cost-efficient benchmarks.
- 5 Discussion and Conclusion: Benchmark Agent is the first autonomous agent-based framework presented for customizable benchmark construction.It is designed to address the scalability and sustainability limitations of human-driven evaluation.
- 5 Discussion and Conclusion: Experiments show that Benchmark Agent generates reliable, discriminative, and cost-efficient benchmarks.
- 5 Discussion and Conclusion: Unlike conventional generic and coarse-grained benchmarks, Benchmark Agent targets customized and fine-grained evaluation demands.It adapts to different application scenarios, user goals, and domain-specific requirements.
- 5 Discussion and Conclusion: Benchmark Agent can construct benchmarks that better reflect real-world usage and nuanced evaluation criteria.
- 5 Discussion and Conclusion: Benchmark Agent supports fast and efficient benchmark iteration despite traditional construction being time-consuming and costly.The framework enables quick benchmark updates, refinements, and extensions as models and tasks evolve.
- 5 Discussion and Conclusion: The agent-based framework significantly reduces human effort and benchmark turnaround time.
A.1 Benchmarks Generated from Benchmark Agent … A.3.2 Programmatic Transformers
Benchmark Agent turns high-level evaluation intents into explicit subtasks and evaluates generated benchmarks through complementary benchmark- and item-level criteria. Its implementation combines deterministic synthesis and restructuring tools to create, modify, organize, and standardize samples without uncontrolled semantic changes.
- A.1 Benchmarks Generated from Benchmark Agent: High-level user queries are decomposed into explicit subtasks that capture their core cognitive challenges rather than fitting fixed benchmark formats.Examples include integrating dispersed evidence and reconciling differing information.
- A.2 LLM-as-Judge protocol details: UIA, FSQ, QAC, CQC, TSD, and SSC provide one benchmark-level and five item-level metrics, each scored on a 1–5 scale.The metrics assess user-intention alignment, structural validity, question–answer coherence, contextual grounding, target-signal dependence, and skill-specific challenge.
- A.2 LLM-as-Judge protocol details: The scoring principles assign errors to the most relevant dimension, distinguishing schema, question–answer, context, shortcut, and reasoning-depth failures.Valid and grounded items can still receive lower TSD or SSC scores when they are easy or shortcut-solvable.
- A.3 Implementation Details: Pure tools are non-LLM deterministic operators that apply parameterized data transformations without semantic interpretation or free-form generation.They support benchmark realization as a low-level implementation layer.
- A.3.1 Synthesis Tools: Synthesis tools create or modify sample content through traceable parameters for perceptual inputs, controlled variations, and auxiliary information.Examples include text-to-speech, image resizing, audio mixing, noise injection, and web search.
- A.3.2 Programmatic Transformers: Programmatic transformers deterministically restructure formats, metadata, fields, and sample organization while preserving intended meaning and avoiding new semantic information.File conversion, metadata editing, content decomposition, and structured field patching standardize data for downstream processing.
B Experiment Results
The experiments report representative results demonstrating that generated benchmarks provide useful signals for comparing models. Supplementary results extend coverage to the remaining text-only and audio-text benchmarks.
- Representative results show that generated benchmarks can provide useful signals for model comparison.
- The main paper reports these representative experiment results.
- Supplementary materials provide full results for the remaining text-only and audio-text benchmarks.
B.1 Text-only benchmark experiments
Text-only benchmark experiments reveal clear performance gaps across model families and scales, with larger models generally achieving higher accuracy. Strong scaling trends across Qwen2.5, Qwen3, and Qwen3.5 indicate that the generated benchmarks are valid and discriminative.
- Scaling and discrimination: Text-only results show clear performance gaps across model families and model scales, with larger models generally achieving higher accuracy.The benchmarks cover Multi-Perspective (T), Long-Dialog Reasoning (T), Multilingual (T), and Math-Reasoning (T).
- Scaling and discrimination: Strong scaling trends across Qwen2.5, Qwen3, and Qwen3.5 suggest the generated benchmarks support comparisons among models with different capability levels.The results indicate both evaluation validity and sufficient discriminative power.
B.2 Audio-based and Omni-based benchmark experiments
This section evaluates audio-text and image-audio-text benchmarks generated by Benchmark Agent across noise robustness, audio reasoning, multilingual, and omni-understanding scenarios. The results show a substantial performance gap between Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, indicating strong benchmark discrimination.
- The experiments cover audio-text and image-audio-text benchmarks for Noise Robustness (A), Audio-based Reasoning (A), Multilingual (A), and Omni-Understanding (O).
- A substantial performance gap separates Qwen2.5-Omni-3B and Qwen2.5-Omni-7B on the audio-text benchmarks.The passage identifies this gap as evidence of the discriminative power of benchmarks generated from Benchmark Agent.
C Limitations · D Benchmark Visualization
The framework has limitations in domain coverage, tool availability, and the remaining value of lightweight human review. Future work will broaden domains, expand and improve tool use, and add interactive expert feedback mechanisms.
- C Limitations: Generated benchmarks are mainly validated on representative tasks, leaving more domain-specific scenarios for further exploration.The authors plan to expand the benchmark pool to cover broader domains and more diverse user requirements.
- C Limitations: Benchmark Agent is partly bounded by the coverage of available tools.Its modular agentic design allows additional tools to be integrated into the tool library.
- C Limitations: Future work will expand the available tool set.This is intended to improve the system’s coverage beyond its current tool library.
- C Limitations: The agent is intended to flexibly select, combine, and invoke different tools.This capability is part of the planned extension of the modular agentic system.
- C Limitations: Although Benchmark Agent substantially reduces human effort, lightweight human review remains useful.Human review can help refine generated benchmarks while preserving reduced annotation costs.
- C Limitations: Future work will incorporate interactive feedback mechanisms for efficient expert refinement.The goal is to enable experts to refine generated benchmarks with minimal additional cost.
D.1 Text-only Cases · D.2 Audio-based Cases
The generated text-only and audio-based cases are clear, unambiguous tests that expose failures in multilingual understanding, mathematical reasoning, temporal reasoning, speaker attribution, and discourse reference resolution. These examples demonstrate the discriminative value of Benchmark Agent across written and spoken inputs.
- D.1 Text-only Cases: Text-only cases are clear and logically decidable, so observed model errors reflect genuine reasoning failures rather than ambiguity.The examples are intentionally self-contained with unambiguous ground-truth labels.
- D.1 Text-only Cases: In multilingual NLI, the model predicts contradiction for unrelated premise and hypothesis situations that should receive a neutral label.This indicates confusion between semantic difference and logical incompatibility in cross-lingual reasoning.
- D.1 Text-only Cases: In multi-dialog reasoning, the model predicts before although two statements anchored to the same call moment have an identity temporal relation.The error suggests reliance on dialogue turn order instead of explicit cues such as “at the same time.”
- D.2 Audio-based Cases: Audio-based cases evaluate speaker attribution, cross-lingual understanding, and discourse-level reference resolution from spoken interactions with unambiguous answers.Transcripts are shown for clarity, but the original inputs are audio.
- D.2 Audio-based Cases: In mixed-language audio, the model attributes the clarification that Ana’s case is not a typical lawsuit to Ana instead of the lawyer.The lawyer describes the case as more like mediation or negotiation, despite clear semantic content.
- D.2 Audio-based Cases: In spoken discourse reasoning, the model links “That” to an earlier conversational moment instead of Speaker 1’s local explanation that the long-term relationship is exciting.This reveals difficulty resolving discourse-level anaphora while maintaining local spoken context.
- D.2 Audio-based Cases: A model incorrectly resolves “That” to the earlier period when the speakers were constantly together rather than to Speaker 1’s realization that their lasting relationship is exciting.The correct answer identifies Speaker 1’s realization as the antecedent.
D.3 Image-based Cases
The image-based cases test high-level visual-label inference under domain-specific conventions, with errors reflecting failures in visual reasoning and evidence integration rather than annotation ambiguity. The examples show category-boundary confusion in art and cue-prioritization failure in medical diagnosis.
- Overall finding: Both image-based examples contain multiple converging visual cues and little genuine ambiguity, so the observed errors indicate failures in visual reasoning and evidence integration.The benchmark annotations are described as well-supported rather than noisy or underspecified.
- Art: The art case distinguishes an early modern revival of classical landscape painting from late traditional practice using the saturated blue–green palette and revivalist presentation.The model instead assigns the artwork to a late traditional literati period by overemphasizing classical motifs and calligraphy.
- Medical: The medical case requires integrating FLAIR MRI findings with neurological symptoms to localize the embolic source to the distal basilar artery.Progressive drowsiness is a high-weight cue for distal basilar syndromes, but the model predicts the proximal basilar artery by overemphasizing generic brainstem involvement.