Source-linked AI summary
"LLM Agent Performance" Is Not a Single Evaluation Target
Pengyu Zhu, Li Sun, Philip S. Yu, Sen Su
TL;DR
Agent benchmark scores reflect both models and surrounding execution conditions, while the label “LLM agent performance” covers several distinct evaluation targets. The paper distinguishes model comparisons, complete-system comparisons, and robustness through declared candidate boundaries and condition policies, and derives reporting and leaderboard implications from that distinction.
Problem
Agent benchmarks are used for model, complete-system, and robustness evaluations, but a score alone does not specify which question it answers.
Method
The paper classifies evaluations by candidate boundary and condition policy, separating reference-stack model comparisons, complete-agent comparisons, and robustness tests across declared conditions.
Results
The paper concludes that “LLM agent performance” is not a single evaluation target and that incompatible results should not be collapsed into one ranking.
Takeaways & Limitations
Leaderboards and reports should declare candidate boundaries and condition policies, separate comparison classes, and report robustness only over tested conditions.
Takeaways & Limitations
The position does not resolve task-construct validity, evaluator bias, benchmark contamination, or statistical adequacy, and finite perturbation sets cannot establish robustness to all deployment settings.
Abstract
from arXiv · showhide
LLM agent benchmark scores are shaped not only by the model but also by the agent harness, environment, evaluator, and inference budget. Unified execution controls these non-model factors by evaluating candidate models under the same configuration, making observed differences more attributable to the models themselves. However, model comparison is only one use of agent benchmarks. Other evaluations compare complete agent systems or test whether a fixed model or system remains stable across predeclared changes in its operating conditions. These results can all be reported under the common label of "LLM agent performance." Our position is that "LLM agent performance" does not denote a single evaluation target. Model comparisons under a reference stack and comparisons of complete agent systems answer different questions, while robustness asks whether either conclusion persists across predeclared conditions. The claim supported by a score therefore depends on the declared candidate boundary and condition policy. We derive implications for leaderboards, result reporting, and benchmark versioning, showing how distinguishing these classes preserves fair comparison while accommodating system innovation and robustness analysis.
1 Introduction
LLM agent benchmarks serve distinct purposes—comparing models, evaluating complete systems, and testing robustness—yet their scores are often grouped as “LLM agent performance.” The supported conclusion depends on the declared candidate boundary and condition policy.
- Benchmarks compare backbone models, complete agent systems, or performance stability under changing interfaces, environments, and resource constraints.
- A benchmark score alone does not specify which evaluation question it answers.
- Under shared execution, score differences support conditional model comparisons; separately optimized agents instead yield complete-system comparisons.
- Robustness asks whether a conclusion persists when candidates face predeclared changes outside their boundary.
- Agent outcomes reflect interactions among the model, harness, inference interface, environment, evaluator, and resource budget, so controlled execution improves model-level attribution.
- “LLM agent performance” is an umbrella term whose comparative meaning is fixed by candidate boundary and condition policy, not a context-independent quantity.
2 Why Model-Level Attribution Requires Controlled Execution
Model-level attribution requires controlling or matching non-model execution factors because small differences in inference, scaffolding, memory, tools, parsing, and environments can alter agent trajectories. The required controls differ from complete-system and robustness evaluations.
- Direct model comparisons should fix, match, or measure inference, prompting, memory, tool mediation, evaluator behavior, and environment state.
- Inference Configuration: Inference settings and provider protocols can filter requests, constrain tool schemas, or record interface rejections as task failures rather than model failures.
- Inference Configuration: Identical weights and tokenized inputs can produce different outputs across inference engines, with early divergences propagating through multi-step trajectories.
- Prompting and Planning Strategies: Prompts and planning scaffolds alter elicited behavior, so they should be fixed or semantically matched for model comparisons but may remain candidate-owned in system evaluations.
- Memory Mechanisms: Memory serialization and length-management policies change the context presented to models; unequal policies measure model–memory configurations rather than LLMs alone.
- Tools and External Environments: Tool schemas, parsing, recovery, and live environments affect executable actions and evidence, requiring controlled or replayable conditions for model-level attribution.
3 Control Is Necessary, but Does Not Determine the Evaluation Target
Controlled execution makes model comparisons interpretable by fixing non-model conditions, but it does not determine whether the candidate is a model or a complete agent system. The evaluation question and candidate boundary must therefore be specified before controls are chosen.
- Controlled execution fixes non-candidate factors so score differences can be attributed to the intended source of variation.For model comparisons, this includes the harness, environment, evaluator, inference interface, and resource budget.
- Model-level comparisons use a fixed reference stack, whereas complete-system comparisons allow candidate-owned execution choices to vary.A full-system result cannot be attributed to the backbone model alone.
- A shared runtime can support model-level, agent-system, or robustness evaluations, which answer different questions.The same API or sandbox does not make their scores interchangeable.
- The evaluation question must be specified before controls are chosen, including the candidate, component ownership, condition policy, and compatible comparison results.These declarations determine what claims a score can support.
4 Candidate Boundaries and Condition Policies Behind "LLM Agent Performance"
“LLM agent performance” depends on the declared candidate boundary and condition policy rather than denoting one context-independent quantity. Model-level, agent-system, and robustness evaluations therefore assign different roles to the same components and support different claims.
- The candidate boundary identifies the compared entity and its candidate-owned components, while the condition policy specifies what is fixed, matched, or varied.An uncontrolled material difference is neither candidate-owned nor part of a declared condition sweep.
- Model-level evaluation: A model-level evaluation compares declared LLM releases within the same reference agent and fixed execution conditions.Its claim is conditional on that reference configuration and does not characterize separately optimized systems.
- Agent-system evaluation: An agent-system evaluation compares complete systems, including their models, prompts, planning, memory, routing, parsing, and other candidate-owned choices, under common external rules.Tools and external services must be assigned explicitly to the candidate or evaluation environment.
- Robustness evaluation: A robustness evaluation holds a declared model or system fixed while testing it across a predeclared sweep of conditions outside its boundary.Fixed corpora, live-web access, snapshots, alternative harnesses, and external budgets can be declared conditions; a single fixed corpus is not itself a robustness study.
- Target-dependent component roles: A component’s role changes with the candidate: prompts and routing are fixed setup for model comparisons but candidate components for complete-system comparisons.Changing a prompt tests robustness only when the candidate is the LLM and the prompt lies outside its boundary.
5 Implications for Comparing and Reporting Agent Scores
Fair comparison requires matching evaluation targets, candidate-boundary rules, and compatible condition policies rather than relying on a shared benchmark name. Leaderboards and reports should preserve separate target classes, document provenance, and version benchmarks with bridge evidence.
- Direct comparability requires the same evaluation target, candidate-boundary rule, and compatible condition policy.Model comparisons match non-model conditions; system comparisons use common external rules; robustness comparisons share or explicitly relate condition sets.
- Within each condition, evaluations should align tasks, scoring, aggregation, uncertainty treatment, and other non-candidate factors.Stochastic evaluations should report uncertainty from repeated runs or another justified sampling procedure.
- Benchmark revisions should be bridged with common anchor candidates under both configurations before results are linked across versions.Bridge evaluations can reveal score shifts, rank correlations, rank reversals, and candidate-specific effects; unstable results define distinct comparison regimes.
- Leaderboards should maintain separate tracks for model capability and agent-system performance, with robustness reported as a condition-aware layer within each track.Different targets may be shown together for diagnosis but should not be merged into one scalar ranking.
- Reports should state the target, candidate boundary, fixed or varied components, versioned evaluation setup, resource and scoring policies, uncertainty, and comparability limitations.The target-first principle requires declaring the candidate boundary and condition policy before aggregating scores.
6 Alternative Views
The paper addresses objections by distinguishing evaluation targets, preserving agent-system innovation, and treating controlled and live evaluations as complementary. It also argues that shared protocols alone do not make results comparable across candidate boundaries and condition policies.
- Static benchmarks can test prerequisites such as instruction following, planning, and tool-selection knowledge, but not how a candidate’s actions alter later observations.
- A common harness need not govern every evaluation: model-level comparisons fix or match a reference stack, while system-level evaluations allow candidate-owned internal mechanisms.
- Existing frameworks already instantiate distinct comparison designs, including fixed-stack model comparisons and analyses spanning models, scaffolds, and benchmarks.
- Production agents are valid end-to-end candidates, but their scores support system-level claims rather than isolated attribution to the backbone model.
- MCP improves interoperability but does not specify the candidate boundary, implementation policies, or evaluation design needed for direct score comparability.
- Controlled and live evaluations provide complementary evidence because environments should match the evaluation target rather than obey universal determinism.
7 Conclusion
The conclusion argues that “LLM agent performance” covers multiple evaluation targets rather than one. It recommends reporting candidate boundaries, condition policies, incompatible comparisons, and tested robustness conditions explicitly.
- “LLM agent performance” may mean reference-stack LLM comparisons, common-rule agent-system comparisons, or robustness testing across declared conditions.
- Leaderboards should declare the candidate boundary and condition policy, separate incompatible comparisons, and report robustness only over tested conditions.
- The same benchmark and metric can support different conclusions when the declared candidate boundary or condition policy changes.
8 Limitations
The paper limits its claim to comparative attribution through candidate boundaries and condition policies, leaving several validity questions and broader domains unresolved.
- The position does not resolve task-construct validity, evaluator bias, benchmark contamination, or statistical adequacy.
- A reference harness is not neutral and may interact differently with different model families.
- No finite perturbation set establishes robustness to all deployment settings, and extending the argument to embodied or safety-focused evaluations requires further study.
A Prompt Artifacts Across Candidate Boundaries
Prompt-side text has distinct evaluation roles: benchmark instructions define task requirements, while demonstrations and control logic form a procedural scaffold whose status depends on the candidate boundary. The paper therefore treats prompt artifacts as classification evidence, not proof of causal scaffold effects.
- Prompt roles: Benchmark-specific instructions define the task and admissible output contract, whereas demonstrations, planning rules, and control logic form the procedural scaffold.
- Source illustrations: BFCL supplies function-calling templates, while smolagents supplies configurable agent templates; these source-level artifacts are not alternative prompts for a shared task suite.
- Source illustrations: The BFCL excerpt illustrates an output contract, while the smolagents excerpt illustrates a procedural control rule.
- Interpretive limit: The excerpts support classification rather than causal scaffold-effect claims; causal testing would require matching tasks, models, interfaces, tools, environments, evaluators, and budgets while varying the scaffold.
- Reporting: Because the artifacts are versioned and dynamically assembled, reports should record repository commit, template path, invocation mode, and enabled options.
B Memory Policy Across Candidate Boundaries
Memory policy must be assigned according to the declared candidate boundary: fixed or matched for model comparisons, but candidate-owned for complete agent-system comparisons. Robustness analysis keeps the candidate fixed while varying external memory conditions systematically.
- Memory-policy dimensions: Memory policy includes serialization, retention and retrieval, persistence, and feedback or recovery signaling.These dimensions describe how interaction history and failures affect the model-visible context and behavior.
- Memory-policy dimensions: Logging-only metadata matters for evaluation only when the harness exposes it to, or acts on it for, the candidate.Timing records, token counts, internal object types, and diagnostic annotations are otherwise behavior-irrelevant.
- Model comparison: For model comparisons, memory policy belongs to the reference stack and should be fixed or semantically matched across candidate models.Associated context, storage, and retry budgets should also remain constant, supporting a comparison under the declared policy rather than a memory-independent claim.
- Agent-system comparison: For complete agent-system comparisons, memory implementation and update policy are candidate-owned, so differences are system-level advantages rather than backbone-model effects.Systems may differ in how they store, compress, retrieve, or present prior interactions.
- Robustness analysis: Under robustness analysis, the candidate remains fixed while external memory policies or constraints are varied across a declared condition set.Replacing a complete system’s internal memory module defines a different system, whereas context limits, storage capacity, episode length, and observation noise may be varied as external constraints.
- Reporting and claim scope: Reports should disclose memory-policy source and version, model-visible assembly, retention and retrieval, reset behavior, recovery signaling, and resource limits.These disclosures clarify whether memory is fixed, candidate-owned, or systematically varied, while causal memory-policy claims require controlled intervention on the remaining stack.