AI Agents That Matter

Sayash Kapoor; Benedikt Stroebl; Zachary S. Siegel; Nitya Nadgir; Arvind Narayanan

Source-linked AI summary

AI Agents That Matter

Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan

arXiv:2407.01502v1Jul 1, 2024cs.LGcs.AI

TL;DR

The paper examines shortcomings in agent benchmarking, including excessive focus on accuracy, conflated evaluation goals, inadequate hold-outs, and poor reproducibility. It proposes cost-aware and jointly optimized evaluation, separates model from downstream benchmarking, recommends principled hold-outs, and emphasizes standardization; these steps aim to support agents useful in the real world. Human-in-the-loop evaluation remains costly and dependent on evaluator skill.

Problem
Agent benchmarks often emphasize accuracy, conflate model and downstream evaluation, lack adequate hold-outs, and use unreproducible practices, limiting their usefulness for real-world applications.
Method
The paper analyzes agent benchmarks and evaluation practices, introduces cost-controlled baselines and joint accuracy-cost optimization, and recommends distinct benchmarks, principled hold-outs, and standardized evaluations.
Results
Simple baseline agents outperform many state-of-the-art complex agents on HumanEval while costing much less, and joint optimization lowers HotPotQA cost while maintaining accuracy.
Takeaways & Limitations
Agent evaluation should incorporate cost, distinguish model from downstream goals, prevent benchmark shortcuts with appropriate hold-outs, and improve reproducibility.
Takeaways & Limitations
Human-in-the-loop evaluation is costly and tricky, and accuracy depends partly on the skill level of the humans interacting with the agent.

Abstract

from arXiv · show

AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.

1 Introduction

AI agent benchmarking faces distinct challenges because agents can handle realistic tasks, incur substantial costs, and require evaluation practices beyond accuracy alone. The paper proposes cost-aware evaluation, clearer benchmark purposes, safeguards against shortcuts, and standardized reproducibility practices.

Motivation: Agents handle harder, more realistic tasks than language models and can cost substantially more to evaluate.SWE-Agent runs were capped at $4 per run, equivalent to hundreds of thousands of language model tokens.
Contributions: The paper introduces cost-controlled agent evaluations, including simple baselines that outperform many complex HumanEval agents while costing much less.Because underlying models are stochastic, repeated calls can increase accuracy and make cost control necessary.
Contributions: Jointly optimizing accuracy and inference cost creates a Pareto-based design space and can lower cost while maintaining accuracy on HotPotQA.The authors modify DSPy to optimize the two metrics jointly.
Contributions: Model and downstream developers need distinct benchmarks because model-oriented evaluations can mislead developers selecting agents for applications.The paper’s NovelQA case study argues that downstream evaluation should account for dollar costs rather than parameter-count proxies.
Contributions: Agent benchmarks can enable shortcuts when hold-out samples are inadequate, so the paper recommends hold-outs matched to the desired level of agent generality.The authors illustrate this issue with WebArena and identify four levels of agent generality.
Contributions: Evaluation standardization is necessary because reproducibility shortcomings in WebArena and HumanEval inflate accuracy estimates and encourage overoptimism about agent capabilities.The paper frames these practices as part of developing agents useful in the real world rather than merely accurate on benchmarks.

2 AI agent evaluations must be cost-controlled

Accuracy alone can make costly inference look like progress: repeated sampling and complex agent architectures can raise benchmark accuracy without identifying meaningful design gains. Evaluating accuracy alongside dollar cost reveals simpler strategies that match or exceed sophisticated agents at far lower cost.

Why cost control matters: Repeated sampling can substantially improve accuracy, creating incentives for agents to spend ever more inference compute.AlphaCode rises from close to 0% zero-shot accuracy to over 15% with 1,000 retries and over 30% with a million retries.
Evaluation setup: The evaluation compares complex HumanEval agents with zero-shot, retry, warming, and escalation baselines using accuracy, cost, and running time.The agents are evaluated on the modified 164-task HumanEval benchmark with example tests for every task.
Main findings: “State-of-the-art” agent architectures do not outperform simple baselines on HumanEval, and warming has no significant accuracy difference from the best-performing architecture.The comparison includes LDB, LATS, and Reflexion alongside the simple baselines.
Main findings: For substantially similar accuracy, agent costs can differ by almost two orders of magnitude; LATS costs over 50 times more than warming.Reflexion and LDB cost over 50% more than warming, while the reported ratios are predominantly driven by GPT-4 calls.
Main findings: Escalation strictly improves accuracy while costing less than half of LDB using GPT-3.5.Escalation begins with a cheap model and moves to more expensive models after test-case failures.
Implication: Accuracy-only evaluation cannot identify progress because scientifically meaningless methods such as retrying can improve the metric.The paper therefore argues that useful agent evaluations must control for cost, even when cost is not the ultimate object of interest.

3 Jointly optimizing cost and accuracy can yield better agent designs

Jointly optimizing accuracy and inference cost creates a Pareto-based design space for agents. A DSPy-based HotPotQA implementation reduced variable cost while maintaining accuracy, with savings becoming more favorable as usage increases.

Pareto-frontier evaluation enables agent designs that jointly optimize accuracy and inference cost.The formulation can also generalize to other design goals, such as latency.
Joint optimization trades higher upfront design costs for lower variable costs by selecting shorter prompts and fewer few-shot examples.Variable costs depend on input and output tokens and increasingly dominate as agents are used more often.
53% lower variable cost was achieved for GPT-3.5 with similar accuracy, while Llama-3-70B achieved 41% lower cost while maintaining accuracy.These comparisons were against the default DSPy implementations.
After 1,350 HotPotQA tasks, both joint-optimization models became cheaper in total cost than default DSPy.The fixed optimization cost is small relative to variable costs when agents are used thousands or millions of times.

4 Model and downstream developers have distinct benchmarking needs

Model evaluation and downstream evaluation answer different questions and therefore require different cost treatments. Benchmarks designed for model comparison can mislead procurement decisions when their task structure does not reflect real-world usage.

Model evaluation studies scientific effects such as architecture or training-data changes, whereas downstream evaluation informs which system to use in an application.These distinct purposes have contributed to confusion about how AI running costs should be measured.
For model evaluation, compute or parameter count can control for resources while preserving stable, level-playing-field comparisons.Dollar costs change over time and can vary with provider economies of scale.
Downstream evaluation should measure dollar cost because developers care about API cost relative to accuracy, not active-parameter proxies.Proxy metrics can be selected in ways that make particular models appear more favorable.
Evaluation results can remain adaptable by exposing token counts and allowing users to recalculate costs under their own provider prices.Provider prices, billing policies, and model-call costs can change over time.
NovelQA can mislead downstream evaluation because it asks all questions about a novel together, unlike sequential real-world queries that repeatedly reprocess the novel.In the reported comparison, long-context models and RAG were roughly equally accurate, while RAG cost more than 20 times less in the real-world scenario.

5 Agent benchmarks allow shortcuts

Agent benchmarks can reward shortcuts and overfitting rather than robust real-world capability. The paper argues that holdouts should match the intended generality of an agent and illustrates the issue with WebArena and STeP.

Agent benchmarks are often small and may lack held-out test sets, allowing subtle overfitting that can produce misleadingly high accuracy.The paper notes that benchmark knowledge can be directly programmed into agents and that many benchmarks lack held-out sets.
The paper proposes making held-out data increasingly different from training data as the intended generality of the agent increases.The appropriate holdout may need to involve distribution shifts, new tasks, or other changes matching the benchmark’s target level of generality.
A survey of 17 agent benchmarks found that most were task-specific or domain-general, while their intended generality was often unclear.This ambiguity makes it difficult to determine what conclusions about successful agents are justified.
The majority of surveyed benchmarks lacked an appropriate held-out set, including 7 with neither a holdout nor an indication that one would be added.The paper treats a holdout as appropriate when it matches the intended generality or when designers indicate plans to create one.
Case study of the STeP agent on WebArena: STeP reached 35.8% accuracy on WebArena by hardcoding policies for specific benchmark tasks, making its performance brittle under website drift or unseen tasks.The paper reports that different tasks or unseen websites would make such hardcoded policies ineffective, and that WebArena does not model drift.
Agent benchmarks don’t account for humans in the loop: Human-in-the-loop evaluation may reveal higher agent usefulness, but it is costly and depends on the skill of the interacting humans.The cited example reports GPT-4 performance increasing from 0% to over 86% with simple feedback on challenging programming problems.

6 Inadequate benchmark standardization leads to irreproducible agent evaluations

Agent evaluations lack standardization because they differ from language-model evaluations in design, cost, and environmental interaction. These differences create inconsistent procedures, subtle bugs, and difficulty reproducing reported results.

Reproducibility shortcomings make it difficult to distinguish genuine agent improvements from artifacts of differing evaluation choices.The paper defines reproducibility as having accompanying code and data sufficient to reproduce reported results.
The paper calls for a standardized agent evaluation framework because existing model-evaluation frameworks do not address these agent-specific shortcomings.
Evaluation scripts may assume a particular agent design, forcing developers to reimplement evaluations and leaving room for non-standard procedures.
Repurposing language-model benchmarks for agents leads developers to use different benchmark subsets or add example test cases.These changes arise because agent workflows may rely on example cases and regeneration, unlike standard prompting-based model evaluation.
Agent evaluations can be prohibitively expensive because agents may call language models hundreds or thousands of times.Evaluating SWE-Agent across more than 2,000 SWE-bench tasks at $4 per task could cost over $8,000 for one run.
Dynamic interaction with environments introduces evaluation errors that static input-output assumptions may miss.The paper identifies external factors such as web or command-line interaction and task-order assumptions as sources of subtle errors.
Non-standardized evaluation permits bugs, including incorrectly marking tasks correct and removing benchmark tasks.The paper reports such issues for both LATS and STeP.

7 Conclusion

The paper concludes that principled agent benchmarking requires cost-controlled comparisons, appropriate holdouts, clearer evaluation targets, and standardized practices. Its supplementary analyses document how accuracy, cost, inference time, robustness, and reproducibility should be assessed.

7 Conclusion: Cost-controlled comparisons, separated model and downstream evaluation, appropriate holdouts, and standardized practices form the paper’s recommended foundation for agent benchmarking.The stated goal is to develop agents useful in the real world rather than merely accurate on benchmarks.
7 Conclusion: Figure A2 presents the complete accuracy–API-cost range, while Figure A3 compares accuracy with inference time on a linear time axis.The A3 time measurements are means of summed inference times across API calls over five runs.
7 Conclusion: Figure A1 reports HumanEval accuracy versus API cost with confidence intervals and minimum–maximum ranges computed across five runs.The confidence intervals use Student’s t distribution because only five runs were conducted per agent.
7 Conclusion: The supplementary HumanEval evaluation uses 164 problems, five runs per agent, and reports mean accuracy and mean total cost alongside minimum and maximum values.The tested implementations include GPT-3.5, GPT-4, LDB, LATS, Reflexion, retry, warming, and escalation configurations.
7 Conclusion: The analysis defines a convex Pareto frontier as agents non-dominated in mean cost and accuracy, including linear combinations of adjacent frontier agents.This convex representation captures achievable tradeoffs between cost and accuracy.

A.2 Robustness checks with June 2023 versions of GPT models

Robustness checks with June 2023 GPT models produced substantially similar results to the main analysis. The appendix also reports HotPotQA joint-optimization and error-bar analyses using five-run averages.

A.2 Robustness checks with June 2023 versions of GPT models: June 2023 GPT-3.5 and GPT-4 checks found substantially similar results: complex agents were no better than simple baselines while sometimes costing orders of magnitude more.This check addresses concerns that newer April 2024 models might have greater scope for contamination.
A.2 Robustness checks with June 2023 versions of GPT models: Figure A4 reports mean cost and accuracy across five runs and mean summed inference time across API calls.Both axes are linear, with the y-axis clipped to 0.7–1 for clarity.
A.2 Robustness checks with June 2023 versions of GPT models: HotPotQA analyses include accuracy–cost error bars and tables reporting agent-design accuracy, variable inference cost, and fixed optimization cost.Table A3 distinguishes variable cost per 100 inferences from fixed costs incurred during optimization.
A.2 Robustness checks with June 2023 versions of GPT models: Joint optimization returns Pareto frontiers whose accuracies are calculated on the development set and whose cost measurements use API prices from May 2024.The frontier plots use clipped axes and different x-axis scales.

B.1 Implementation details

The implementation evaluates multi-hop question answering on HotPotQA with fixed retrieval and demonstration settings, then compares uncompiled, formatted, DSPy-optimized, and jointly optimized programs. Joint optimization searches prompt and few-shot choices to trade fixed optimization cost against variable inference cost.

B.1 Implementation details: HotPotQA uses a two-hop multi-hop question-answering design with ColBERTv2 retrieval and evaluates whether all specified ground-truth documents are retrieved.The setup retrieves two passages or hops and allows up to eight demonstrations.
B.1 Implementation details: The study randomly selects 100 HotPotQA samples for optimization and 200 for evaluation using a fixed seed.This separates optimization data from evaluation data while supporting reproducibility.
B.1 Implementation details: The compared baselines are uncompiled execution, formatting instructions only, and DSPy’s BootstrapFewShot and random-search optimization.The uncompiled version omits few-shot examples and output-formatting instructions.
B.1 Implementation details: Joint optimization searches few-shot examples, token counts, and formatting instructions with Optuna to trade fixed optimization cost against variable inference cost.It extends BootstrapFewShot by retaining candidate examples for multi-objective parameter search.
B.1 Implementation details: The reported program is the Pareto-efficient development-set program with the highest development accuracy, and its results are summarized by mean accuracy and cost.The development set contains 50 samples.

C Survey on agent benchmarks

The benchmark survey finds that holdout sets must match a benchmark’s level of generality to prevent gaming. Many surveyed agent benchmarks lack appropriately general holdouts or stated plans to add them.

C Survey on agent benchmarks: The survey covers 17 recent agent benchmarks and records their levels of generality and respective holdout sets.These properties are summarized in the paper’s benchmark tables.
C Survey on agent benchmarks: A holdout set must correspond to a benchmark’s level of generality to prevent gaming.The survey identifies this correspondence as a requirement for benchmark design.
C Survey on agent benchmarks: Many surveyed benchmarks lack holdout sets at the appropriate generality level, and their designers do not state plans to add them.The survey distinguishes WebArena’s use of “domain” from the paper’s broader domain terminology.

D Additional details on Section 4: Details about NovelQA implementation

The NovelQA implementation compares GPT-4 with retrieval-augmented generation against long-context GPT-4, while surveying whether agent benchmarks provide appropriate holdouts. Retrieval achieves similar accuracy at lower reported cost, but the evaluation was run only once.

NovelQA implementation: The retrieval setup embeds each novel and retrieves 10 chunks of 1000 characters for each multiple-choice question.The evaluation uses GPT-4 with retrieval-augmented generation rather than placing the entire novel in the context window.
NovelQA evaluation: GPT-4 with retrieval achieves 67.89 accuracy at a total cost of $52.8 USD, compared with 67.81 accuracy at $99.8 USD using the entire novel as input.The reported GPT-4 accuracy in the original NovelQA paper was 71%.
NovelQA evaluation: The retrieval agent performs substantially similarly to GPT-4, indicating only a small accuracy difference between retrieval-augmented and long-context approaches.The authors attribute small absolute differences partly to language-model stochasticity.
Benchmark holdouts: Appropriate holdouts should differ from training data at the level of generality intended for the agent, because same-distribution tasks can permit shortcuts.For domain-general evaluation, held-out tasks should differ in domain rather than merely being unseen instances.
Benchmark holdouts: The survey reports that 7/17 agent benchmarks lack holdout sets and plans for them, while only 5/10 benchmarks with holdouts use the appropriate generality level.The authors estimated generality from each benchmark’s stated purpose.

E.1 HumanEval implementation details

The HumanEval implementation analysis examines published agents and documents how inconsistent benchmark subsets, model identities, and task handling affect reported evaluations. Several agents were evaluated on fewer than all 164 HumanEval problems.

Evaluation discrepancies: LDB’s published generated programs rely on GPT-4 rather than the GPT-3.5 model claimed in its paper.The authors of LDB acknowledged the discrepancy and planned to update the paper.
Evaluation discrepancies: LDB, LATS, and Reflexion use different subsets of HumanEval problems, limiting direct comparison across their reported results.The benchmark contains 164 original coding problems, including three without example tests.
Reproduction procedure: The reproduced-evaluation analysis compares reported and reproduced HumanEval accuracy across all 164 tasks and repeated runs.The accompanying table notes that insufficient benchmark standardization contributes substantially to observed discrepancies.
Evaluation discrepancies: LATS removes the three HumanEval problems without example tests plus another problem, while Reflexion evaluates only 161 of 164 tasks.LATS cited an execution error for one removed test case in correspondence with the authors.

F Statement on compute resources used

The experiments primarily used external model APIs rather than local GPU inference or LLM training.

External APIs: OpenAI models were accessed through OpenAI or Azure OpenAI endpoints, while Llama-3 HotPotQA analyses used Together.ai endpoints.The work did not use GPUs for inference and did not require training LLMs.

G Limitations

The paper’s limitations concern changing cost assumptions, incomplete coverage of agent environments, and costs beyond API inference. The authors also note broader safety risks that the work does not directly address.

Cost assumptions: The cost-controlled methods depend on current cost models and technological constraints, which may change with evolving technology and pricing.The authors provide an interactive application so users can modify underlying costs and recalculate agent costs.
Scope: The study does not exhaustively cover all possible task environments and variations of AI agents.The authors nevertheless report findings across multiple benchmarks and agent designs.
Unmeasured costs: Environmental impact, human annotation labor, and AI-system maintenance costs were not extensively analyzed.The paper identifies these as relevant to a more comprehensive evaluation of economic and environmental impact.
Safety: The work does not directly address the safety risks associated with increasingly sophisticated AI agents.This limitation appears in the discussion of the paper’s societal implications.