PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-as-a-Judge: Evaluate Agents with Agents

Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber

arXiv:2410.10934v2cs.AI

TL;DR

Existing evaluations inadequately capture the intermediate processes of agentic systems or require substantial manual effort. The paper introduces Agent-as-a-Judge and the DevAI benchmark for realistic AI development tasks, finding closer agreement with human consensus than LLM-as-a-Judge and similar performance to expert human evaluators. The authors position this framework as a basis for richer feedback and scalable agentic-system improvement.

  • Problem

    Existing agentic-system evaluations often focus on final outcomes rather than intermediate stages, while realistic code-generation benchmarks and thorough human evaluation remain limited or costly.

  • Method

    The paper introduces Agent-as-a-Judge, applies it to code generation, and evaluates three agentic systems on DevAI, a benchmark of 55 realistic AI development tasks with hierarchical requirements.

  • Results

    Agent-as-a-Judge aligned with human consensus in 90% of tested cases versus 70% for LLM-as-a-Judge and performed similarly to an ensemble of expert human evaluators.

  • Takeaways & Limitations

    Agent-as-a-Judge provides rich intermediate feedback while reducing reliance on human evaluators, opening the possibility of scaling agentic-system evaluation and improvement.

  • Takeaways & Limitations

    The proof of concept does not fully exploit advanced agentic optimization methods for Agent-as-a-Judge, which the authors leave for future work.

Abstract

from arXiv · show

Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes -- ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems -- by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement.

1 Introduction

Agentic systems require evaluations that capture intermediate task-solving behavior without the expense of extensive human judgment. The paper introduces Agent-as-a-Judge and DevAI, then shows stronger agreement with human evaluators than LLM-as-a-Judge.

  • Motivation: Current agentic-system evaluations either overlook intermediate task-solving stages or require prohibitively expensive human assessment.Agentic systems often act step-by-step, so final-outcome-only evaluation cannot pinpoint what affects performance.
  • Agent-as-a-Judge: Agent-as-a-Judge uses agentic systems to evaluate agentic systems, extending LLM-as-a-Judge with rich intermediate feedback while retaining cost-effectiveness.The framework evaluates the full thought and action trajectory rather than only final outcomes.
  • DevAI and evaluation: DevAI contains 55 comprehensive AI development tasks designed to address limitations of benchmarks focused on algorithmic, simple, or automated-repair problems.The benchmark targets realistic development challenges rather than isolated programming skills.
  • DevAI and evaluation: The paper benchmarks three open-source code-generation agentic frameworks on DevAI using human judges, LLM-as-a-Judge, and Agent-as-a-Judge.This provides a comparative evaluation across three judging approaches.
  • Results: 90% alignment with human consensus was achieved by Agent-as-a-Judge, compared with 70% for LLM-as-a-Judge across tested cases.Agent-as-a-Judge also aligned more closely with the human ensemble than individual human evaluators did.
  • Results: Agent-as-a-Judge saved 97.72% of evaluation time and 97.64% of evaluation cost compared with involving three human experts.These savings address the substantial time and expertise required for human evaluation.

2 DevAI: A Dataset for Automated AI Development

DevAI is a benchmark of 55 realistic AI development tasks designed to evaluate agents across comprehensive, dependency-structured workflows rather than isolated final outcomes. The section also reports preliminary comparisons of three open-source code agents using cost, time, token, and file-output statistics.

  • Motivation: The benchmark addresses a gap in existing code-generation evaluations, which cover limited portions of development and insufficiently represent intermediate stages or long-horizon work.The section contrasts realistic development needs with benchmarks focused on algorithmic problems, simple programming tasks, or automated repair.
  • Dataset design: DevAI contains 55 AI development tasks with 365 hierarchical requirements and 125 softer preferences.Each task combines a user query with requirements connected by dependencies and optional preferences.
  • Dataset design: The benchmark targets realistic, relatively small-scale AI development problems spanning supervised learning, reinforcement learning, computer vision, natural language processing, and generative models.Tasks are intended to resemble work assigned to research engineers while remaining computationally inexpensive to evaluate.
  • Dataset design: Requirements form a directed acyclic graph of milestones, enabling denser feedback than binary success metrics and making simple memorization inadequate.Dependencies can connect stages such as data loading, modeling, and result visualization; hierarchical requirements further structure evaluation.
  • Preliminary benchmark: The preliminary experiment runs MetaGPT, GPT-Pilot, and OpenHands with gpt-4o-2024-05-13, a 1,800-second task limit, and recorded development artifacts and trajectories.The recorded outputs support later human, LLM-based, and agent-based evaluations.
  • Preliminary benchmark: MetaGPT has the lowest average cost at 1.19 USD, while OpenHands has the highest at 6.38 USD; GPT-Pilot saves 273.33 code lines versus MetaGPT’s 11.15.OpenHands averages 362.41s per task, GPT-Pilot 1622.38s, and GPT-Pilot saves 3.84 files versus MetaGPT’s 0.42.

3 Human-as-a-Judge: Manual Evaluation on DevAI

The study manually evaluates AI developers on DevAI, using expert review and consensus to assess requirement satisfaction and human-evaluation reliability. The results show both substantial evaluator disagreement and improved agreement after consensus and majority voting.

  • Human Evaluation Setup: Three expert evaluators reviewed whether AI developer outputs satisfied each DevAI requirement, then debated their assessments to produce final consensus results.The initial evaluation took an estimated 58 human hours, followed by 28.5 additional hours for consensus.
  • Performance Analysis: GPT-Pilot and OpenHands satisfied about 29% of requirements, but only one task achieved complete requirement satisfaction.The figure rises to about 44% when prerequisites are ignored.
  • Disagreement Analysis: 10% to 30% disagreement occurred between evaluator pairs, indicating that individual human judgments were not fully reliable.The paper attributes errors partly to missed information in complex, multi-step development tasks and differing interpretations of ambiguity.
  • Error Analysis: Majority voting reduced the overall error rate to 6.01%, correcting most individual evaluator errors relative to consensus.One evaluator made errors on 23.77% of GPT-Pilot evaluations before aggregation.
  • Conclusion: The authors recommend debate after judgments or larger expert panels, favoring debate because engaging more experts is costly and often impractical.The paper notes that small-group majority voting can still differ from consensus by around 5%.

4 Agent-as-a-Judge: Evaluating Agents with Agents

Agent-as-a-Judge extends LLM-based evaluation with agentic components that inspect task-solving trajectories and provide intermediate feedback. In the proof of concept, it more closely matches human consensus than LLM-as-a-Judge while substantially reducing evaluation cost and time.

  • 4 Agent-as-a-Judge: Evaluating Agents with Agents: Agent-as-a-Judge evaluates code-generating agents with agentic features that provide intermediate feedback throughout the task-solving process.The framework retains the cost-effectiveness of LLM-as-a-Judge while operating as an agentic system.
  • 4.1 Proof-of-Concept: Eight modular components were initially designed, including graph, locate, read, retrieve, ask, memory, and planning modules.The provided component description covers seven named modules while identifying the initial design as eight components.
  • 4.1 Proof-of-Concept: The highest-performing ablation combined graph, locate, read, retrieve, and ask, while planning and memory introduced issues in the described design.The selected combination achieved the highest performance; historical memory could propagate earlier judgment errors into later decisions.
  • 4.2 Judging Agent-as-a-Judge and LLM-as-a-Judge: Agent-as-a-Judge outperformed LLM-as-a-Judge on Judge Shift, reaching 0.27% for Requirement (I) versus 31.24% for LLM-as-a-Judge on OpenHands.Lower Judge Shift indicates closer alignment with the Human-as-a-Judge consensus.
  • 4.2 Judging Agent-as-a-Judge and LLM-as-a-Judge: 90% alignment with human consensus was achieved by Agent-as-a-Judge, compared with 70% for LLM-as-a-Judge across all tested cases.For OpenHands, Agent-as-a-Judge reached 92.07% and 90.44% in gray-box and black-box settings, versus 70.76% and 60.38%.
  • 4.2 Judging Agent-as-a-Judge and LLM-as-a-Judge: PR-curve analysis found that Agent-as-a-Judge outperformed any single human evaluator on OpenHands and aligned most closely with majority voting.The paper uses PR Curves because requirement satisfaction is class-imbalanced and shift or alignment metrics can mislead.
  • Ablation Studies: Adding graph to ask increased alignment from 65.03% to 75.95%, while retrieve provided task-dependent benefits rather than a consistent gain.Retrieve was not significantly beneficial in one case but added valuable trajectory information for MetaGPT and GPT-Pilot.
  • Evaluation Cost: Agent-as-a-Judge required 30.58 USD and 118.43 minutes, or 2.29% of the cost and 2.36% of the time of human evaluation.The human baseline took 86.5 hours and cost approximately 1297.50 USD.

5 Related Work

Related work spans agentic systems, AI software-development agents, benchmarks for AI tasks, and AI-based judging. DevAI is positioned as a realistic complete-cycle development benchmark, while Agent-as-a-Judge applies agentic evaluation beyond traditional setups.

  • Agentic systems: The paper frames agentic-system evaluation as an active research area whose current methods and benchmarks have struggled to keep pace with deployment advances.This motivation connects the related work to the paper’s evaluation framework and benchmark contributions.
  • AI Developers: Recent work applies agentic systems to software development, debugging, data science, and academic-paper writing.The related systems include agents that imitate software companies and perform specialized development tasks.
  • Benchmarks for AI developments: Existing AI-development benchmarks cover tasks such as machine learning, data science, and engineering, whereas DevAI targets realistic queries for complete development cycles.DevAI also includes hierarchical requirements and a more comprehensive evaluation setup.
  • AI Judges: Prior AI-judge work extends LLM-as-a-Judge with multiple LLMs or applies single evaluators to multimodal tasks, but does not match Agent-as-a-Judge’s setup.The cited comparison distinguishes those systems by their use of trivial agentic components or traditional evaluation settings.

6 Discussion and Conclusion

The paper proposes Agent-as-a-Judge as a framework for evaluating agentic systems with intermediate feedback and presents future directions for iterative self-improvement. It concludes that the framework and DevAI benchmark support more scalable evaluation of agentic code generation.

  • Outlook 1: Intermediate Feedback for Agentic Self-Improvement: The framework may enable process-supervised reward models that identify and fix issues during complex, multistage tasks.The paper describes intermediate feedback as useful for effective and efficient optimization.
  • Outlook 2: Flywheel Effect Driven by Agent-as-a-Judge: The authors hypothesize that iterative feedback between the judge and evaluated agents could create a flywheel effect of mutual improvement.They also suggest this interaction could complement LLM reasoning data and help embed agentic capabilities into foundation models.
  • Discussion and Conclusion: Agent-as-a-Judge uses agentic systems to evaluate agentic systems and extends LLM-as-a-Judge with intermediate feedback.The framework is applied to evaluating code-generating systems.
  • Conclusion: The conclusion presents Agent-as-a-Judge and DevAI as a basis for scaling evaluation of agentic systems.The paper reports that Agent-as-a-Judge outperforms existing methods on the task and performs similarly to an ensemble of expert human evaluators.

B.1 Summary of Experiments

The experiments evaluate developer agents, human judgments, and AI-based judges across multiple levels. The design emphasizes step-by-step performance on structured, dependency-aware AI development tasks rather than only final outcomes.

  • Summary of Experiments: The experimental program contains basic agent statistics, human evaluations, human-evaluation error analysis, AI-judge baselines, and Agent-as-a-Judge ablations.These experiments are organized across three evaluation levels.
  • Summary of Experiments: The benchmark compares human and automated judges when evaluating developer agents.The summarized experiment table distinguishes developer agents, human judges, LLM-as-a-Judge, and Agent-as-a-Judge.
  • Benchmark Design: DevAI prioritizes step-by-step task-solving ability because its relatively small, well-studied tasks can be overfit under conventional performance metrics.This design targets practical applicability while avoiding task-performance evaluation as the primary development measure.
  • Benchmark Design: Each task uses explicit, binary requirements as milestones, alongside preferences for subjective or ambiguous characteristics.Requirements cover AI-development elements including data processing, methods, metrics, visualization, and human-computer interaction.
  • Benchmark Design: DevAI models automated AI development as a dependency-structured process spanning preprocessing, feature engineering, model selection, tuning, reporting, and interactive applications.The seven-step process is inspired by established AI-development methodologies.
  • Dataset Refinement: The dataset underwent two rounds of review by different participants to check logical consistency, language correctness, ambiguity, and domain applicability.The review process identified a moderate number of dataset errors.

E.5 Analyse the dataset

DevAI represents AI development tasks through categorized requirements, dependencies, preferences, and execution-related tags. Its sample format encodes concrete criteria for data, modeling, metrics, visualization, and user interaction.

  • Dataset Analysis: Requirements are categorized by dataset, machine-learning method, visualization, metrics, human-computer interaction, and data processing.The authors note that categories can overlap or miss nuances.
  • Dataset Analysis: Preference classifications range from common-sense inferences to explicitly stated characteristics and may be subjective.The paper cautions that these classifications may not fully capture practical importance.
  • Dataset Analysis: DevAI tasks can be tagged for dataset or environment, preprocessing, machine-learning method, model saving, metrics, human-computer interaction, visualization, and other requirements.An additional tag indicates whether Kaggle credentials are needed.
  • Sample Task Structure: The sample speech-emotion task requires loading RAVDESS, preprocessing audio through noise removal and normalization, and extracting MFCC features.These requirements are assigned to data and preprocessing categories.
  • Sample Task Structure: A sample task specifies prerequisites and criteria such as implementing a CNN-LSTM model, saving recognition accuracy, generating a confusion matrix, and creating a Flask upload API.The criteria are linked to categories including machine-learning method, performance metrics, visualization, and human-computer interaction.
  • Sample Task Structure: Task metadata records whether Kaggle access, training, and web navigation are needed, while preferences describe robustness and API usability.The sample includes separate preference criteria for audio preprocessing and user-facing instructions.

F User experiences of code-generation agentic systems

The evaluated code-generation agents exhibit distinct user-experience profiles. OpenHands emphasizes interaction, MetaGPT emphasizes decomposition, and GPT-Pilot emphasizes detailed delegation at the cost of fluidity.

  • User Experience: OpenHands offers the most refined user experience through an interactive frontend for direct engagement and task execution.The paper associates this interface with a smoother workflow and operational efficiency.
  • User Experience: MetaGPT excels at task decomposition through Directed Acyclic Graphs that support structured and scalable modularization.It may be less aggressive in file management and preservation during practical deployments.
  • User Experience: GPT-Pilot delegates work across more than 20 specialized agents but provides a more fragmented user experience because of reduced interactivity.The paper identifies user engagement and operational fluidity as improvement areas.

G More DevAI dataset samples

DevAI includes realistic AI development tasks with detailed project queries and required artifacts. Examples cover PyTorch style transfer and Rossmann Store Sales forecasting.

  • Task 13 asks for a PyTorch perceptual-loss project for image style transfer.
  • The sample tasks specify implementation files, datasets, saved models, and result figures as development outputs.
  • Figures 10 and 11 present DevAI examples corresponding to Tasks 13 and 19.
  • Task 19 asks for sequence-to-sequence LSTM forecasting using the Rossmann Store Sales dataset.

H Human Evaluation Procedure

Human evaluation used three AI experts, multiple evaluation rounds, task-specific criteria, and recorded agent trajectories. The procedure combined independent scoring with discussion-based consensus.

  • Three AI experts evaluated outputs from agentic code-generation systems.
  • First round: In the first round, evaluators spent 16.5, 19.5, and 22.0 hours using minimal instructions and scorecards.
  • Second round: The second round involved discussing disagreements to correct errors and reduce evaluator bias before reaching consensus.
  • Evaluators judged execution success and requirement satisfaction using task descriptions, requirements, preferences, workspaces, and trajectories.
  • DevAI imposes a 30-minute solution limit, includes execution time, and does not guarantee GPU availability.
  • Trajectories use structured records containing steps, user messages, agent thoughts and actions, environment responses, and usage information.

J.2 A Sample of Trajectory

The trajectory sample records an agent’s development process step by step, including actions, environment feedback, token usage, cost, and elapsed time. The examples show iterative implementation and error correction.

  • OpenHands trajectories record every step so the development process can be replayed.
  • The sample shows the agent responding to a FileExistsError while implementing the data loader and later correcting an indentation issue.
  • A trajectory step records user messages, agent thoughts and actions, and environment responses.
  • Usage records include input and output tokens, model name, inference time, cost, and total step execution time.
  • Accumulated usage tracks total trajectory cost and time through the current step.

K.1 Component Abalations

Component ablations show that ask, graph, read, and locate improve Agent-as-a-Judge performance, while search and planning provide weaker or negative results in the tested setting. Retrieval effectiveness depends on trajectory and workspace complexity.

  • Component ablations: Adding ask, graph, read, and locate produced significant performance gains for Agent-as-a-Judge.
  • Search: Introducing search caused a noticeable performance decline, while alternative search engines did not improve alignment.
  • Search: Search may become more valuable as workspaces grow more complex, whereas direct retrieval and evaluation suffice in simpler workspaces.
  • Planning and memory: Planning did not noticeably improve performance, and memory also showed no improvement in the experiments.
  • Search: Sentence-BERT reached 87.70% alignment, below the 90.44% configuration without search.
  • Retrieve: Retrieval is difficult in trajectories; truncating beginnings improved large-trajectory retrieval, while truncating middle sections worked best for individual steps.
  • Prompt design: The judge prompt evaluates whether each requirement is satisfied or unsatisfied using evidence from code, data, and outputs.
Loading 2410.10934v2…