PaperlayerView exact revision on arXiv

Source-linked AI summary

Counsel: A Meta-Evaluation Dataset for Agentic Tasks

Sashank Pisupati, Henry Broomfield, Eujeong Choi, Antonia Calvi, Charlie Wang, Roman Engeler, Max Bartolo, Patrick Lewis

arXiv:2606.21627v1cs.AIcs.LG

TL;DR

Human annotation of increasingly complex agent trajectories is costly, while the soundness of scalable LLM-as-a-judge critiques remains insufficiently measured. Counsel addresses this gap with human meta-evaluations of process-level critiques across customer-support and coding agents, finding that more capable judges and greater reasoning effort improve agreement, reaching 88% on error location and 65% on reasoning.

  • Problem

    Human annotation of increasingly complex agent trajectories is costly, while the soundness of scalable LLM-as-a-judge critiques remains insufficiently measured.

  • Method

    Counsel pairs open-weight LLMJ process critiques of trajectories from customer-support and coding benchmarks with human labels of error-location and explanation correctness.

  • Results

    88% location agreement and 65% reasoning agreement were achieved by the strongest judge, with more capable models and greater reasoning effort producing higher human agreement.

  • Takeaways & Limitations

    Counsel provides a dataset and practical foundation for studying and improving LLMJ critiques that identify failure points and articulate diagnostic explanations.

  • Takeaways & Limitations

    Counsel measures critique precision and reasoning when judges flag errors, not recall when they fail to raise issues, and covers only two agentic environments and model families.

Abstract

from arXiv · show

As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.

1 Introduction

As agentic tasks grow longer and more complex, human trajectory annotation becomes increasingly burdensome, motivating widespread use of LLM-as-a-judge critiques whose soundness often remains unmeasured. Counsel addresses this gap with human meta-evaluations of process-level critiques across customer-support and coding benchmarks, supporting evaluation and improvement of judges.

  • Motivation: Growing task complexity increases the human cognitive burden of annotating agent trajectories for performance evaluation and training-data curation.The 50% task-completion time horizon was estimated to double every 7 months.
  • Motivation: LLM-as-a-judge systems generate process- and outcome-level trajectory critiques and are widely used to evaluate agents.Their applications span data curation, training, and deployment.
  • Motivation: Despite widespread adoption as diagnostic artifacts, the soundness of LLMJ critiques often goes unmeasured.Critiques provide richer descriptions of agent behavior than numeric scores, while most meta-evaluations focus on final judgments.
  • Counsel: Counsel is a public dataset of human meta-evaluations for process-level LLMJ critiques on τ-bench customer-support and DA-Code coding tasks.Open-weight LLMJs critique open-weight agents by flagging trajectory-step errors and explaining their causes; humans assess error-location and explanation correctness.
  • Counsel: Counsel supports evaluating critique quality, training meta-judges to select better critiques, and directly improving judges through supervision or reward signals.These uses extend evaluation beyond agreement on final scores to error localization and reasoning quality.

2 Related work

Counsel contributes a public, permissively licensed dataset of human meta-evaluations for LLM-as-a-judge critiques across two agentic domains. It extends prior benchmarking, which has largely focused on comparing final judgments with human judgments, by supporting LLMJ benchmarking and meta-judge training.

  • Dataset contribution: Counsel provides high-quality human meta-evaluations of LLMJ critiques and judgments across two agentic domains.The dataset is public and permissively licensed.
  • Dataset contribution: The dataset supports benchmarking LLMJ quality on agentic tasks and training meta-judges to improve LLMJs for agents.Its uses span both evaluation and evaluator improvement.
  • Prior work: Prior LLMJ benchmarking has largely meta-evaluated final judgments by comparing scores or preferences with human judgments.Examples include JudgeBench, RewardBench, AgentRewardBench, Agent-as-a-Judge, and MAST.

3 Methods … 3.4 Judgment Generation

Counsel constructs agent trajectories, span-level LLM-as-a-Judge critiques, and human-readable judgments across τ-bench and DA-Code using open-weight models. Its methodology varies agent and judge configurations while evaluating each step independently from available context.

  • 3.1 Terminology: Counsel defines trajectories as ordered agent–environment messages, spans as individual model or tool invocations, judgments as categorical LLMJ predictions, and critiques as textual explanations.Judge outputs contain the judgment and critique in that order.
  • 3.2 Agent environments: The study uses τ-bench for customer service and DA-Code for code generation, selecting 115 retail tasks and 50 Data Insights or Data Manipulation tasks.τ-bench uses multi-turn tool-agent-user interactions, while DA-Code evaluates realistic data-science workflows in a sandbox.
  • 3.3 Trajectory generation: Agent trajectories are generated by rolling out benchmark tasks with open-weight agent, judge, and user models selected to produce realistic, heterogeneous failure modes.The design varies model families, scales, and reasoning configurations while avoiding both superficial underthinking and excessive deliberation or latency.
  • 3.3.1 Agent models: The agent pool includes GPT-OSS-20B with medium reasoning and Qwen3-235B-A22B-Instruct-2507 without reasoning, providing varied latency and deliberation profiles.GPT-OSS-20B is a 20B-parameter MoE with 3.6B active parameters; Qwen3 has 235B parameters and 22B active parameters.
  • 3.3.2 Generating agent trajectories: Minimal repository changes enable model querying through Together AI; τ-bench uses its standard ToolCallingAgent with Qwen3-235B-A22B-Instruct-2507 as the capable simulated user.This configuration aims to attribute observed failures and critiques to the agent rather than user-side errors or underspecification.
  • 3.4.1 Judge models: Judge models vary by family and reasoning effort to diversify judgments, cover error localizations and explanations, and reduce dependence on any single model’s failure modes.The study includes GPT-OSS-120B at low and high reasoning settings and Qwen3-235B-A22B-Instruct-2507 without reasoning.
  • 3.4.1 Judge models: Same-family and cross-family judge–agent pairings enable analysis of self-preference bias, including GPT↔GPT, Qwen↔Qwen, and GPT↔Qwen configurations.Self-preference bias refers to models being more lenient toward their own generations.
  • 3.4.2 Generating judge evaluations: Each model-call span is independently judged using only the trajectory context, available tools, and current output, without privileged future information or previous judge outputs.The judge is prompted to identify task-relevant errors and provide concise, human-digestible critiques in addition to any internal reasoning.

3.5 Human annotation

Human annotators evaluated flagged LLMJ errors in complete τ-bench and DA-Code trajectories using privileged context and a unified three-way scheme covering location and critique soundness. Three experienced annotators achieved substantial reliability, with Krippendorff’s α ≈0.78.

  • Annotation protocol: Human meta-judgments assessed both whether the error location was correct and whether the LLMJ critique was accurate.Annotators used full-trajectory information, including past and future judge span outputs unavailable to the judges.
  • Annotation protocol: Annotators reviewed complete trajectories containing at least one judge-flagged error, reducing burden by focusing evaluation on potential issues.They evaluated outputs from three LLMJs across τ-bench and DA-Code.
  • Labeling scheme: The three-way scheme labels each flagged span as Spot On, Poor Reasoning but Correct location, or Should Not Have Flagged.The middle category captures correct localization paired with an incorrect or inadequate explanation.
  • Quality and reliability: α ≈0.78 indicates substantial inter-annotator agreement and high reliability in the meta-judgment task.Agreement was estimated from triple-annotated sample batches using Krippendorff’s alpha.

4 Dataset analysis

Counsel’s analyses characterize trajectory structure, judge behavior, and human-assessed critique quality across τ-bench and DA-Code. They identify domain-dependent precision, improved critique quality with greater reasoning effort, and no detected length or self-preference bias.

  • Trajectory characteristics: DA-Code trajectories typically use fewer agent steps than τ-bench but generate more output tokens per step because of greater task complexity.The comparison concerns output tokens excluding reasoning tokens.
  • Judge behavior: Length bias was not a concern: span counts did not differ significantly (p = 0.929), while Qwen3’s longer output-token generation had a small effect size (Cohen’s d = 0.329).The token-length difference was statistically significant (p = 0.0001) but judged unlikely to substantially affect analyses.
  • Judge behavior: On τ-bench, judges assessed Qwen3 as the more performant agent, while GPT-OSS-120B was the stricter judge.The assessment was based on fewer errors attributed to Qwen3 by the judges.
  • Judge behavior: No significant self-preference bias was detected for any judge in one-sided normal tests of proportions.Self-preference was assessed by comparing same-family and different-family evaluations.
  • Critique quality: Greater reasoning effort improved critique quality, with GPT-OSS-120B:high producing fewer “poor reasoning” or “should not have flagged” evaluations than its low-reasoning counterpart across both benchmarks.On DA-Code, GPT-OSS-120B:high also had 74% greater recall than Qwen3 (33/19).

5 Discussion

Counsel introduces a meta-evaluation dataset for assessing the precision and diagnostic quality of LLM judges’ flagged critiques across coding and customer-service agents. It supports scalable judge improvement while leaving recall, broader generalization, and systematic downstream applications for future work.

  • Contribution: Counsel pairs real agent trajectories and step-level LLMJ critiques with human labels to assess whether critiques locate failures correctly and explain them usefully.Labels include “spot on,” “poor reasoning,” and “should not have flagged.”
  • Contribution: The dataset provides a practical, scalable foundation for improving evaluation of agentic systems, which are difficult to observe directly at scale.It is intended to support models that produce critiques useful for debugging and guardrails.
  • Limitations: Counsel measures judges’ precision and flagged-critique quality, but not recall of errors that judges fail to identify.Its focus is critique quality when an error has already been flagged.
  • Limitations: Testing only two agentic environments and model families limits how well the dataset may generalize to dissimilar agentic domains.The paper explicitly identifies domain coverage as a generalization constraint.
  • Future work: Future work should study downstream uses of meta-judgments as learning signals and develop automated meta-judges that score, rank, or filter critiques.Potential applications include selecting higher-quality evaluations, filtering spurious flags, and calibrating strictness; a preliminary few-shot experiment is presented in Section D.

Impact statement

Counsel aims to improve diagnostic critiques of agent behavior, supporting safer deployment, debugging, training, and reduced reliance on costly human oversight. The work emphasizes responsible use by complementing human judgment, acknowledging risks of over-trust or misuse, and enabling broad research through open-weight models and permissive licensing.

  • Benefits: Counsel can support safer deployment, better debugging, and more effective training for agents performing complex real-world tasks.Higher-quality diagnostic critiques may also reduce reliance on costly human oversight while improving transparency and accountability.
  • Limitations: Automated judges risk being over-trusted or misapplied outside their validated domains.The paper therefore emphasizes that Counsel should complement, not replace, human judgment.
  • Availability: Counsel uses open-weight models and permissive licensing to encourage broad, responsible research into scalable and aligned evaluation methods.

A Additional DA-Code filtering

To address underspecified DA-Code tasks, the dataset discarded tasks that frontier-model agents completed correctly on fewer than 50% of samples. This favors an easier subset to reduce misleading false negatives and increase judge-error opportunities for evaluating judges.

  • Filtering criterion: Tasks were discarded when fewer than 50% of samples were completed correctly by both o3 and Claude-3.5-Sonnet agents.Each model was run through all Data Insights and Data Manipulation tasks.
  • Rationale: Although this biases the dataset toward easier tasks, filtering reduces false negatives that could mislead human meta-annotators.The benchmark’s internal evaluation criteria could otherwise mark semantically correct agent runs as incorrect.
  • Rationale: The easier subset also increases agent attempts that may produce false-positive error flags, supporting the dataset’s focus on judge evaluation quality.This tradeoff is accepted because the dataset measures how well judges evaluate agent trajectories.

B Judge prompt

The judge prompt frames evaluation as QA analysis of whether an AI agent has an error in its current step, guided by examples of error types. An example prompt is provided, with abbreviated sections marked by angular brackets.

  • Judge task: The judge is instructed to evaluate whether an AI agent suffers from an error in its current step.The prompt casts the judge as a QA analysis expert.
  • Judge task: The prompt guides analysis by listing types of errors that may help identify agent failures.These error types are presented as guidance for the judge’s analysis.
  • Prompt example: Figure 5 presents an example judge prompt, with omitted sections delineated by angular brackets.The figure notes that sections are omitted for brevity.

C Dataset analysis

The section analyzes agent trajectory statistics across τ-bench retail and DA-Code, including step counts and per-step output tokens. It also examines self-preference bias by comparing same-family and different-family judge critiques.

  • Trajectory statistics: Figure 6 compares agent-step distributions across τ-bench retail and DA-Code for GPT-OSS-20B and Qwen3 agents.The figure includes τ-bench retail results for both GPT-OSS-20B and Qwen3, and DA-Code results for Qwen3.
  • Trajectory statistics: Figure 6 also analyzes distributions of output tokens per agent step, excluding reasoning tokens, on τ-bench retail for GPT-OSS-20B and Qwen3.The figure separately reports the number of output tokens per agent step.
  • Self-preference bias: Figure 7 measures self-preference bias by comparing the proportion of agent spans critiqued by judges from the same versus different model families.One-sided normal-test p-values are displayed above the corresponding bars for whether same-family judgments are less prevalent than different-family judgments.

D Evaluation in-the-loop

Counsel’s meta-annotations were tested as in-context examples for agents and an in-loop guardrail judge on τ-bench retail. Across configurations, effects were generally small, with only spot-on feedback to the judge producing a statistically significant improvement, while generalization remained unestablished.

  • Method: Two settings supplied Counsel examples either to the agent’s system message or to an in-loop judge that could request one retry.The judge appended its critique to context before giving the agent a second chance; performance was measured by average task reward.
  • Method: Four configurations compared no examples, 20 spot-on judgments, 20 poor-reasoning or incorrect judgments, and a 10–10 mixed split.Examples were provided as pairs of the current agent output and critique.
  • Results: 10 full τ-bench iterations supported statistical claims, but most deviations from the “No-feedback” baseline were insignificant.The experiment used τ-bench retail average reward as its performance measure.
  • Results: Only spot-on feedback supplied to the judge produced a statistically significant improvement.Figure 8 evaluates average reward across 115 τ-bench retail tasks, with 10 benchmark runs for each configuration.
  • Discussion: The experiment provides an early signal that Counsel can improve task completion, but limited prompt engineering and same-environment annotations prevent demonstrating generalization.The authors nevertheless suggest developers could use the method to align agents within their environment of interest.
Loading 2606.21627v1…