Source-linked AI summary
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi Gu, Hui Su, Xunliang Cai, Xiangnan He
TL;DR
Reliable verification of agent behavior in complex, novel environments remains difficult because rule-based and text-grounded judges have limited generalization. AJ-Bench evaluates Agent-as-a-Judge through interactive evidence gathering across three domains, finding consistent gains over LLM-as-a-Judge while exposing unresolved performance and scope limitations.
Problem
Verifying agent behaviors in novel environments remains challenging because existing rule-based and LLM-based approaches are limited beyond narrow or surface-text settings.
Method
AJ-Bench evaluates environment-interacting judges across search, data systems, and GUI tasks, assessing information acquisition, state verification, and process verification.
Results
Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, with an average improvement of 0.13 in F1 and an average F1 of 0.72.
Takeaways & Limitations
AJ-Bench provides a benchmark for systematic Agent-as-a-Judge evaluation and highlights the promise of agent-based judging paradigms.
Takeaways & Limitations
Most AJ-Bench tasks are modified from existing benchmarks, and search evaluation reliability may be affected by unstable network connectivity.
Abstract
from arXiv · showhide
As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.
1 Introduction
AJ-Bench addresses the challenge of verifying agent behavior in novel environments by evaluating judges that acquire evidence through tools and interaction. It shows that Agent-as-a-Judge improves over text-grounded judging while remaining imperfect on complex, multi-step behaviors.
- AJ-Bench targets the challenge of verifying agent behaviors in novel environments at scale.
- Rule-based verifiers are effective for narrow tasks but do not generalize well, while LLM-as-a-Judge relies primarily on surface-level textual signals.
- Agent-as-a-Judge actively interacts with environments to reproduce trajectories, verify intermediate states, and assess tool usage.
- AJ-Bench comprises 155 tasks and 516 binary-labeled trajectories across search, data systems, and graphical user interfaces, measuring information acquisition, state verification, and process verification with F1.
- Agent-as-a-Judge consistently outperforms LLM-as-a-Judge by an average 0.13 F1, but its average F1 of 0.72 remains unsaturated.
- Experiments across three domains show imperfect performance on complex, multi-step behavior evaluation, leaving substantial room for improvement.
2 Related Work
Prior judge benchmarks emphasize agreement, factuality, or reasoning, while agent benchmarks evaluate task solving rather than behavior verification. Agent-as-a-Judge shifts evaluation toward active evidence acquisition and environment-aware auditing.
- Earlier LLM-judge benchmarks primarily measured alignment with human judgments, while newer datasets target instruction following, reasoning, factual accuracy, and reasoning quality.
- Agent-as-a-Judge incorporates agentic capabilities into verification, including external tool use for reasoning tasks.
- Existing agentic judging efforts have largely remained within reasoning benchmarks, leaving broader environment-aware evaluation less addressed.
- Task-solving benchmarks assess agents’ problem-solving and execution, whereas judge-agent benchmarks verify behaviors by acquiring evidence, inspecting states, and auditing execution traces.
3 Benchmark Construction
AJ-Bench is constructed from interactive, reproducible tasks and trajectories across search, data systems, and GUI environments, with labels validated through automated procedures and human inspection. Its replayable environments let judges acquire contextual evidence beyond static trajectories.
- AJ-Bench task design jointly considers interactivity, complexity, reproducibility, and LLM–environment interaction for reliable evaluation.
- Search Domain: The search domain combines Mind2Web2 deep-search tasks requiring multi-hop reasoning with WideSearch tasks emphasizing broad information coverage.
- DS Domain: The data-systems domain uses Filesystem and Postgres tasks whose outcomes can be verified directly from environment state.
- GUI Domain: The GUI domain adapts PowerPoint, Word, and Excel tasks requiring precise execution positions and planned action sequences, while excluding unstable GUI states.
- Trajectory Construction: Trajectories combine existing benchmark data with LLM-regenerated samples, followed by human verification, quality filtering, and normalization across diverse model sources.
- Labeling: Labels are binary, with search labels assigned at item level and data-systems and GUI labels assigned at trajectory level; GUI labels receive manual inspection because scripts can miss execution details and edge cases.
- Dataset Statistics: The final benchmark contains 155 tasks, 516 trajectories, and 60 tools across its domains.
- Evaluation Environment: Replay reconstructs independent live environments from action sequences, enabling judges to interact after the final recorded state and acquire contextual information beyond static trajectories.
4 Experiments
Experiments show that tool-enabled Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, while performance depends on reasoning effort, interaction budget, and input modality. Despite these gains, stronger reasoning alone does not guarantee reliable tool use or evaluation.
- Main Experiments: Agent-as-a-Judge consistently outperforms LLM-as-a-Judge, with tool usage improving performance by approximately 13 percentage points across three domains.The comparison holds for both thinking and chat models built on the same base model.
- Main Experiments: Agent-as-a-Judge built on weaker models can achieve performance comparable to LLM-as-a-Judge based on state-of-the-art models.The implemented Agent-as-a-Judge baseline also consistently outperforms existing state-of-the-art LLM-as-a-Judge models.
- Thinking Ablation: Increasing reasoning effort does not necessarily improve Agent-as-a-Judge performance across domains.For gpt-5-mini, medium generally exceeds low effort, whereas high does not consistently exceed medium; deepseek-v3.2 thinking performs worse than no-thinking.
- Interaction Turns Ablation: Increasing the interaction budget improves F1 across tasks, with the largest gains at smaller budgets.Word and PPT tasks benefit more from extended interactions, indicating greater reliance on iterative information gathering.
- Multimodal Ablation: Mixed accessibility-tree and screenshot inputs consistently outperform either single modality in the GUI ablation.The effect varies substantially across subdomains, and additional modalities can introduce noise and redundancy in some scenarios.
5 Conclusions
The paper introduces AJ-Bench to address the lack of systematic evaluation for Agent-as-a-Judge. Its benchmark spans three domains and provides a platform for studying agent-based judging.
- Contributions: AJ-Bench is introduced as the first benchmark specifically designed for systematic Agent-as-a-Judge evaluation.It targets a gap in existing benchmarks.
- Benchmark Scope: AJ-Bench covers Search, DS, and GUI with 155 tasks and 516 trajectories.The benchmark is intended to support comprehensive evaluation and future research.
- Future Use: The authors position AJ-Bench as a foundational evaluation platform and resource for future Agent-as-a-Judge research.The conclusion frames the benchmark as supporting continued community investigation.
Limitations
AJ-Bench has limitations in task diversity and environmental stability. Many tasks are adapted from existing benchmarks, and web-environment instability can affect search-domain reliability.
- Task Diversity and Scalability: Most AJ-Bench tasks are adapted from existing benchmarks rather than created entirely from scratch.The authors plan to construct more independent tasks and scale data generation for broader coverage and possible training use.
- Environment Stability: Network instability in external web environments may affect evaluation reliability in the Search domain.This limitation arises because Agent-as-a-Judge relies on interactions with those environments.
A.1 Mind2Web2 Task Design
Mind2Web2 tasks are organized by answer characteristics and filtered to support reliable task design. Ground-truth tasks have fixed answers, whereas no-ground-truth tasks permit multiple valid solutions.
- Task Categorization: Tasks are categorized as ground_truth, no_ground_truth, or time_sensitive before further filtering.Time-sensitive tasks are filtered or rewritten into the other two categories.
- Task Categorization: Ground_truth tasks have fixed, well-defined answers, while no_ground_truth tasks allow multiple valid solutions or subsets of answers.The latter includes tasks requiring only a specified subset, such as three or five items.
A.2 Human Annotation
Human annotators labeled Mind2Web2 responses using task-specific evaluation rubrics. The process combined trained full-time and student annotators with representative examples and criterion-level judgments.
- A dedicated annotation team labeled Mind2Web2 data for the Search domain.The team included full-time and student annotators compensated at rates comparable to local market roles.
- Annotators first formulated an evaluation rubric for each response before assigning criterion-level labels.
- Representative Mind2Web2 examples were provided as references before annotation began.
A.3.1 Search Domain
The Search-domain pipeline summarizes lengthy returned pages before giving their content to the agent. Summarization uses the same model as the agent, with model-specific reasoning settings.
- Lengthy Search-domain pages are summarized before being provided as agent context.The resulting summary supplies the extracted page content used by the agent.
- Summarization uses the same model as the agent with model-specific reasoning configurations.gpt-5-mini uses low reasoning effort, while deepseek-v3.2 disables explicit reasoning during summarization.
A.3.2 GUI Domain
The GUI evaluation pipeline integrates OSWorld with MCPMark through an MCP server and client. Parallel evaluation is supported by AWS hosts and independent OSWorld instances for trajectory replay.
- The GUI pipeline uses OSWorld components within MCPMark.An OSWorld MCP server and MCP client provide the integration.
- AWS infrastructure enables highly parallelized GUI evaluation.An AWS host manages task allocation across evaluation instances.
- Each trajectory is replayed and evaluated on an independent AWS instance using the OSWorld project’s AWS AMI.
A.4 Agent Framework Ablation
The ablation and robustness analyses examine Agent-as-a-Judge across frameworks, models, subdomains, and statistical uncertainty. Results support broad improvement over LLM-as-a-Judge, while revealing framework variation, sampling-related uncertainty, and recurring failure modes.
- Framework and model analysis: Agent-as-a-Judge is evaluated beyond MCPMark to assess whether findings generalize across agent frameworks.A ReAct reimplementation uses explicit reasoning and actions, unlike MCPMark’s more autonomous and implicit reasoning process.
- Framework and model analysis: Additional judge models are compared with and without Agent-as-a-Judge on representative Search, GUI, and DS subdomains.The models span closed-source and open-source families, with evaluation restricted to Wide, FileSystem, and PPT because agentic judging is more costly.
- Statistical reliability: 95% confidence intervals from three independent runs assess the reliability of subdomain-level performance estimates.The intervals are estimated using the t-distribution.
- Statistical reliability: Upward-shifted confidence intervals across most subdomains support Agent-as-a-Judge improvements, while smaller PPT and WORD subsets show wider variance.For DEEP, GPT-5-mini-low intervals are strictly non-overlapping: [66.18, 70.66] versus [74.89, 76.49].
- Failure analysis: Four failure modes are analyzed: omitted or failed tool calls, incorrect tools, misinterpreted outputs, and wrong reasoning despite correct evidence.The analysis combines proportional distributions across Search, DS, and GUI with qualitative examples.
- Failure analysis: Failure examples include reading 51 papers without creating answer.html and accepting an incomplete duplicate-file interpretation as successful.The duplicate-file case considered strict and lenient interpretations before judging the task.
- Failure analysis: Other examples show incorrect reasoning despite accurate evidence and successful GUI verification supported by coordinates, slide status, and accessibility-tree evidence.The GUI example verifies the image’s right-side position on Slide 2 through live and trajectory evidence.
- GUI limitations: Tool-formatting deviations can break the GUI execution pipeline, making the setup sensitive to protocol changes.