PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models

José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins

arXiv:2504.01001v2cs.CLcs.AI

TL;DR

Human-annotated test sets and robust task-specific metrics are costly to create and become less sufficient as language models expand across tasks and modalities. ZSB uses language models to generate synthetic test data and evaluate it in flexible benchmarks. Across the tested tasks, its rankings correlate strongly with human rankings and often outperform widely used standard benchmarks.

  • Problem

    As models expand to new tasks and modalities, robust task-specific metrics are difficult to develop and human-annotated test sets are costly and saturate more quickly; prior automated approaches rely on existing data or focus on limited tasks.

    Source: §1 · pp. 1–2

  • Method

    ZSB uses language models to generate synthetic test instances and evaluate them through prompts for data generation and judgment.

    Source: §1 · p. 2

  • Results

    Across five text tasks and one multimodal task, ZSB rankings correlate strongly with human rankings and often outperform widely adopted standard benchmarks.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    ZSB enables rapid benchmark development for specialized tasks where collecting human-annotated data is costly or impractical.

    Source: §7 · p. 10

    Synthetic data may encode generator-model biases, and automated benchmark results should complement rather than replace human evaluation and real-world testing.

    Source: Ethics Statement · p. 11

Abstract

from arXiv · show

As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets -- which are expensive to create -- saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.

1 Introduction

Automatic evaluation depends on robust task-specific metrics and human-annotated test sets, which are difficult to develop and become costly or saturated as models expand to new tasks and modalities. ZSB combines synthetic test-data generation and language-model evaluation to build flexible benchmarks, which the authors test across text and vision-language tasks.

1 Introduction

  • Prior automated approaches either rely on pre-existing data or focus on limited settings, while human-dependent crowdsourcing constrains scalability across tasks.The cited examples focus on English chat capabilities or summarization.Source: §1 · p. 2
  • ZSB combines language-model generation of synthetic test data with language-model evaluation, using two prompts to create flexible, scalable benchmarks.The prompts specify data generation and evaluation; the judgment prompt can support a Likert scale.Source: §1 · p. 2
  • Because ZSB is model-agnostic, its data-generation and evaluation components can be updated with newer models as capabilities improve.The framework is intended for specialized or emerging tasks where real-world data collection is costly or impractical.Source: §1 · p. 2
  • The evaluation covers general capabilities in English, Chinese, French, and Korean, translation, and English vision-language capabilities.The benchmarks span five text tasks and one multimodal task.Source: §1 · pp. 2–3
  • Across the evaluated systems, ZSB rankings correlate strongly with human rankings and often outperform widely adopted standard benchmarks.The authors also report ablations on open models, judge size, and dataset variety.Source: §1 · p. 3

2 Zero-shot Benchmarking (ZSB)

ZSB uses language models to generate benchmark instances and evaluate them. Its prompts support varied task instances and flexible evaluation, with instance-level safety metadata also provided.

2 Zero-shot Benchmarking (ZSB)

  • ZSB automates benchmark creation from test-data generation through evaluation using language models.The framework is presented as applicable to any task.Source: §2 · p. 3
  • A meta prompt specifies the task and placeholder attributes for generated instances, which can yield extensive combinations of test cases.For English chat, the listed attributes produce 9,832,320 possible combinations.Source: §2 · p. 3
  • A judgment prompt evaluates instances with a six-point Likert scale, and direct assessment can use existing scoring frameworks.The authors choose direct assessment over pairwise comparison, noting similar ranking quality; evaluation uses one model in this work.Source: §2 · p. 3
  • Each benchmark instance includes a description and a six-point safety score generated with Claude-Sonnet-3.5.The authors report that fewer than 0.3% of instances score three or below.Source: §2 · p. 3

3 Experimental Setup

The experiments evaluate benchmarks for multilingual general capabilities, translation, and English vision-language tasks, comparing system rankings with gold rankings and established baselines. They describe the evaluated systems, task settings, ranking-correlation measure, and baseline construction.

3 Experimental Setup

  • Each task uses 500 test instances generated with Claude-Sonnet-3.5, and the evaluation covers up to 22 systems scored by a six-point Likert judgment.Systems are ranked by their average score across instances.Source: §3 · pp. 3–4

3.1 Meta-evaluation

  • Pairwise-Accuracy measures the fraction of system pairs ranked in agreement with gold rankings and orders benchmarks like Kendall τ.Spearman’s ρ is also reported in an appendix, with the same conclusions.Source: §3.1 · p. 4

3.2 Tasks

  • General-capabilities evaluation generates open-ended queries in English, Chinese, French, and Korean and compares rankings with Chatbot Arena.The task targets queries involving core knowledge, instruction-following, and conversational capabilities.Source: §3.2 · p. 4
  • Translation benchmarks cover eleven language pairs and compare up to seven systems against WMT24 General MT Shared Task standings.Professional translators create the WMT test sets and evaluate submitted translations.Source: §3.2 · p. 4
  • The English vision-language task evaluates 12 VLMs against gold rankings from the vision section of Chatbot Arena.The task extends general capabilities to include visual understanding.Source: §3.2 · pp. 4–5

3.3 Baselines

  • For general capabilities, standard baselines use static multiple-choice test sets ranked by average scores, Borda Count, or an oracle Top-1 selection.Top-1 selects the benchmark with the highest correlation to gold rankings.Source: §3.3 · p. 5
  • Translation baselines include the WMT24 organizer metric AUTORANK and three established MT metrics: XCOMET-XXL, METRICX-24, and COMETKIWI-XXL.The metrics include reference-based and reference-free approaches.Source: §3.3 · p. 5
  • M-ArenaHard is treated as an upper bound because its queries come directly from the source of the gold rankings.The authors apply their judging framework to isolate the impact of test-data provenance.Source: §3.3 · p. 5

4 Main Results

ZSB ranks systems competitively with established benchmarks across multilingual general-capability, translation, and vision-language evaluations. Its strongest reported comparisons include multilingual results across all four languages, translation results close to leading metrics, and a multimodal advantage over standard benchmarks.

4.1 LLM General Capabilities

  • ZSB outperforms standard-benchmark test-set averages across all four languages, beats the Borda-count baseline on French and Korean, and reaches 0.845 PA in French, above the reported upper bound.The English test set cost roughly $5, compared with $500 for ArenaHard and upward of $100,000 for GPQA.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Translation

  • At 0.79 PA, ZSB is two points below AUTORANK and COMETKIWI-XXL and close to reference-based XCOMET and METRICX on translation rankings.The comparison covers up to seven systems across eleven language pairs, totalling 143 system comparisons.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 VLM English General Capabilities

  • ZSB outperforms multimodal standard benchmarks by 4.5 PA points regardless of aggregation method, despite using 500 instances versus the leaderboard's eight datasets.All but one test set, MMMU, are outperformed by ZSB.Source: §4.3 · p. 6, Table 1 · p. 6

5 Ablations

The ablations examine benchmark quality across open-model choices, judging approaches, reference and prompt variation, and dataset size and variety. They find that open models can produce strong rankings, while judge size and dataset variety are important factors.

Open models

  • Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct generate benchmarks with strong human-ranking correlations, and Qwen outperforms averaged standard benchmarks across the board.The passage also reports strong correlations outside English and says model choice matters: Qwen is especially strong in Korean, while Llama lags behind.Source: §5 · p. 7

Open-model judges

  • High judge performance is achievable from 14B parameters onwards, especially for Chinese and Korean, while open-model pairings can outperform Claude-only pipelines outside English.With Claude-generated data, Llama-3.3-70B-Instruct judges improve ranking correlations on Chinese and French, outperforming M-ArenaHard on French.Source: §5 · p. 7

Model-family matching

  • Qwen judges often rank Qwen-generated data better than Claude-generated data, suggesting gains from tailoring data generation and judging to open models.This trend is reported for Qwen judges from 7B to 72B; the authors describe the explanation as possible, not established.Source: §5 · pp. 7–8

Judgment approach

  • Direct assessment and pairwise comparison have comparable performance in English; direct assessment slightly outperforms pairwise comparison across systems or against a Claude-Sonnet-3.5 baseline.The table compares judgment approaches for English general capabilities.Source: §5 · p. 8, Table 4 · p. 8

Prompt and reference robustness

  • Generated instances consistently follow meta-prompt specifications, but only a small fraction of references are perfect despite most receiving a score of 5.Benchmark PA is also reported across the default meta-prompt and two rewrites.Source: §5 · p. 8, Table 5 · p. 9

Dataset size

  • Beyond 100 instances, dataset-size gains stagnate, reaching only 2.5 extra PA points at 500 instances.The authors note that small datasets are more accessible and easier to create and curate.Source: §5 · p. 8

Dataset variety

  • Adding more attribute variety in the meta-prompt can improve performance by more than 6 PA points, while adding a reference yields only marginal gains.The figure analyzes factors affecting English general-capabilities PA, including dataset size, attribute count, and reference use.Source: §5 · pp. 8–9, Figure 2 · p. 9

6 Related Work

Related work includes human-annotated static test sets, synthetic test sets, crowdsourced evaluation, and automated benchmark creation. ZSB builds on these approaches by covering a wider range of languages and tasks, including multimodal ones.

Fully-automated evaluation suites

  • Prior automated benchmark work draws on crowdsourced or pre-existing data, or focuses on limited tasks and languages; ZSB explores more languages, tasks, and modalities.The cited examples include methods centered on chatbot capabilities, dialogue response generation, summarization, video, or English benchmarks.Source: §6 · pp. 9–10

Automatic evaluation with static test sets

  • Human-annotated static sets and task-specific metrics remain common, while LLMs are increasingly used as judges for automatic evaluation.The cited examples include intrinsic metrics and learned metrics for machine translation and text generation.Source: §6 · p. 10

Synthetic test sets

  • Synthetic test sets offer scalability and can be enhanced as language models improve, but some benchmark efforts focus on multiple-choice tasks.MuSR is cited as a synthetic multiple-choice benchmark used in the Open LLM Leaderboard.Source: §6 · p. 10

7 Conclusion

The paper concludes that ZSB automates benchmark data creation and evaluation, producing rankings that correlate well with human judgments across text and multimodal tasks. Its ablations identify open models, direct assessment, and dataset variety as useful, with judge size also important.

Conclusion

  • Across five text tasks and one multimodal task, ZSB rankings correlate well with human judgments and outperform widely used standard benchmarks.The ablations find that open models can create high-quality benchmarks, judge size is crucial, direct assessment is viable, and dataset variety matters more than size.Source: §7 · p. 10

Implications and future work

  • ZSB can support rapid benchmark development for costly or impractical data-collection settings, and the authors identify real-world utility as an open question.They release benchmarks and code, and propose future work on other modalities and tasks, synthetic image generation, and increasing dataset variety.Source: §7 · p. 10

Reproducibility Statement

Reproducibility may be constrained by the use of closed models that could become unavailable.

Reproducibility Statement

  • Some experiments rely on closed models, which may become unavailable and pose a potential reproducibility challenge.The authors release benchmarks and code to reproduce experiments and create new benchmarks.Source: Reproducibility Statement · p. 11

Ethics Statement

The ethics statement identifies risks from synthetic-data bias and misuse of automated evaluation. It calls for review of generated instances and human evaluation alongside real-world testing.

Ethics Statement

  • Synthetic data may encode the biases of its generating models, and safety scores do not fully address this risk.Users should carefully review generated instances before deployment.Source: Ethics Statement · p. 11
  • Automated evaluations could be misused to claim superiority, so ZSB benchmarks should complement—not replace—human evaluation and real-world testing.The authors release code and data to enable scrutiny and improvement.Source: Ethics Statement · p. 11

A Prompts and ZSB Examples

The appendix documents the prompts and examples used in ZSB, covering benchmark-data generation and response judging across text, translation, and vision-language tasks, along with meta-prompt attribute distributions, safety metadata prompts, and evaluation baselines.

A.1 ZSB Meta-prompts for Data Generation

  • General-capability data prompts use requested attributes to elicit an engaging, open-ended challenge and a reference answer.The attributes include language, topic, subtopic, difficulty, style, writer, writing proficiency, and prompt length.Source: §A.1 · p. 18
  • Translation prompts specify source and target languages, topic, subtopic, length, and style, then generate source text and a natural-sounding reference translation.Source: §A.1 · p. 19
  • VLM data prompts pair an image with a question requiring visual analysis and reasoning, avoiding image descriptions, simple detection, and model-specific bias.Suggested capabilities include spatial reasoning, object relationships, contextual understanding, and interpretation of abstract or cultural content.Source: §A.1 · p. 20

A.2 Meta-prompt Attributes and Distributions

A.3 ZSB Judgement Prompts

  • General-response and translation judges rate outputs on a 1–6 scale, using task-specific criteria such as accuracy, completeness, clarity, and fidelity to source.General-response criteria also include relevance, evidence, and objectivity; translation criteria include naturalness, terminology consistency, and technical precision.Source: §A.3 · pp. 34, 36
  • VLM judges score responses on a 1–6 scale for visual detail recognition, reasoning, prompt adherence, image-based accuracy, and depth of understanding.Source: §A.3 · p. 38

A.4 ZSB Generated Data Examples

  • Generated-data examples include an English prompt about electric vehicles and automotive supply chains and a French prompt about digital education paradoxes.Source: §A.4 · p. 39, Figure 12
  • Examples span French general capabilities, English–Czech translation, and English vision-language tasks, alongside historical-source analysis and a craft-beer-label image prompt.Source: Figure 15 · p. 40, §A.4 · p. 40, Figure 13

A.5 ZSB Safety Metadata

  • Text-only safety metadata prompts ask an expert auditor to rate a prompt from 1–6 against risks including harmful outputs, privacy extraction, manipulation, guardrail circumvention, appropriateness, and ethics.Source: §A.5 · p. 41, Figure 16 · p. 41, Figure 17 · p. 42
  • The six-level safety rubric ranges from highly safe prompts with minimal risk to prompts indicating high likelihood of harm and serious ethical problems.Source: §A.5 · p. 41
  • For multimodal safety, the auditor rates the prompt–image combination and considers risks such as personal-data extraction, harmful interactions, and context-dependent interpretations.Source: §A.5 · p. 43, Figure 18 · p. 43, Figure 19 · p. 44
  • English general-capability examples include a prompt rated 5 (Generally Safe) and one rated 2 (Risky), with feedback identifying responsible privacy discussion versus a minor’s unsafe late-night outing request.Source: §A.5 · p. 45, Figure 20, Figure 21

B.1 Evaluated Systems

B.2 Baselines for LLM and VLM General Capabilities

  • General-capability baselines use Open LLM and VLM Leaderboard benchmarks for English and popular language-specific benchmarks for Chinese, French, and Korean; individual benchmark results appear in Table 23.Source: §B.2 · p. 48, Table 23 · p. 49

B.3 Baselines for Translation

  • Translation baselines use WMT24 test data for all language pairs; AUTORANK scores come from Kocmi et al. (2024), while XCOMET, METRICX, and COMETKIWI scores come from Freitag et al. (2024).Source: §B.3 · p. 49

C.1 Main Results measured in Spearman Correlation

The appendix tables cover Spearman correlations, data-generator and judge ablations, output-format failure rates, Llama 4 Maverick results, and translation results by language pair.

C.1 Main Results measured in Spearman Correlation

  • Table 24 reports Spearman correlations for LLM and VLM general capabilities across languages, with conclusions stated to match pairwise accuracy.The passage does not provide the correlation values.Source: Table 24 · p. 50

C.2 Data Generator and Judge Ablations for More Languages

  • Tables 25–27 cover pairwise-accuracy ablations of data generators and judges for Chinese and French general-capability benchmarks.Tables 25 and 27 are labeled Chinese; Table 26 is labeled French.Source: Table 25 · p. 50, Table 26 · p. 50, Table 27 · p. 51

C.3 Data Generator and Judge Ablations Output Format Failure Rates

  • Table 28 reports failed-judge-generation percentages by data-generator model for the English general-capabilities benchmark.The passage does not provide the percentages.Source: Table 28 · p. 51

C.4 General Capabilities Results with Llama 4 Maverick

  • Table 29 reports pairwise accuracy when Llama 4 Maverick serves as both data generator and judge on LLM general-capabilities benchmarks.The passage does not provide the accuracy values.Source: Table 29 · p. 51

C.5 By-LP results for MT

  • Table 30 presents translation-evaluation results by language pair.The passage does not provide the results or language pairs.Source: Table 30 · p. 52

D Topic Distributions of ArenaHard and ZSB General Capabilities Data

The topic distributions contrast ArenaHard’s coding emphasis with ZSB’s focus on open-ended writing, while noting that ZSB can be adapted to include coding questions.

D Topic Distributions of ArenaHard and ZSB General Capabilities Data

  • ArenaHard is mostly coding, whereas ZSB’s general-capabilities data focuses on open-ended writing and contains no coding; this difference may explain their differing pairwise accuracies with Chatbot Arena rankings.Unlike static ArenaHard, ZSB can be adapted to generate more coding questions and better match Chatbot Arena’s distribution.Source: Table 31 · p. 52
Loading 2504.01001v2…