Source-linked AI summary
PACE: A Proxy for Agentic Capability Evaluation
Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
TL;DR
Agentic benchmarks are costly and infrastructure-intensive, motivating whether inexpensive non-agentic instances can predict their results. PACE selects compact source subsets and learns predictors from calibration models, achieving strong cross-benchmark accuracy at far lower cost while depending on representative calibration models.
Problem
The predictive connection between inexpensive non-agentic benchmark performance and costly agentic benchmark performance is not yet well understood.
Method
PACE selects compact subsets of non-agentic instances and combines complementary selection signals with noise-aware regression to predict agentic benchmark scores.
Results
Across four agentic targets and 14 models, PACE achieves 3.80% MAE, 0.81 Spearman correlation, and around 85% pairwise accuracy under LOOCV at roughly 100× lower cost than matched random target sampling.
Takeaways & Limitations
PACE can support cheaper model ranking, denser training monitoring, and screening before expensive full-harness evaluations.
Takeaways & Limitations
PACE depends on calibration models representing future models, and proxy error may increase for distribution shifts or substantially different architectures and training paradigms.
Abstract
from arXiv · showhide
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.
1 Introduction
PACE addresses the high cost and complexity of agentic evaluation by predicting agentic benchmark performance from a compact subset of inexpensive non-agentic instances. Across broad experiments, it achieves strong predictive accuracy at a small fraction of full evaluation cost while also exposing capability requirements.
- Motivation: Agentic benchmarks require long-horizon interaction, tools or environments, error recovery, complex infrastructure, and substantial time and API costs.A single model evaluation can cost thousands of dollars and take hours or days.
- Approach: PACE selects a compact, budget-constrained subset of non-agentic benchmark instances whose aggregate scores predict target agentic performance.Its candidate pool spans capabilities such as instruction following, tool calling, and multimodal understanding.
- Approach: PACE differs from benchmark compression by selecting inexpensive source instances from a separate pool without modifying target-benchmark scoring.The method predicts performance on an agentic benchmark rather than reducing the cost of evaluating that same benchmark.
- Evaluation: PACE is evaluated across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks spanning 11 LLM capabilities.The agentic targets include GAIA, SWE-Bench Multimodal, SWE-Bench Verified, and SWT-Bench.
- Results: Under LOOCV, PACE predicts agentic scores with MAE under 4%, Spearman correlation above 0.80, and pairwise ranking accuracy around 85% at much less than 1% of full evaluation cost.A 100-instance proxy achieves strong prediction, with roughly 100× lower dollar cost than random target sampling at equal quality.
- Interpretability: Analyzing selected instances identifies the capabilities most strongly associated with each target agentic task, providing interpretable evidence about capability structure.The analysis distinguishes skills related to successes and failures across agentic benchmarks.
2 Background
Agentic and non-agentic evaluations differ in interaction demands and infrastructure, but overlap in the underlying capabilities they measure. PACE formulates proxy construction as selecting source instances and learning predictors from calibration models, including both performance and pairwise preference objectives.
- Evaluation Context: Agentic evaluations require longer horizons, tools or environments, and error recovery, often involving sandboxes, browsers, repositories, and custom runtimes.Their results can also be sensitive to environmental noise, harness design, and external dependencies.
- Evaluation Context: The shared capability basis includes 11 categories, including instruction following, planning, code generation, tool calling, reasoning, multimodal understanding, and verification.These overlapping capabilities make prediction across the two evaluation protocols plausible.
- Related Work: Prior work predicts complex behavior from simpler signals through matrix factorization, sample-level unification, and aggregated static benchmark scores.These approaches provide related evidence for connecting simpler evaluations with broader outcomes.
- Problem Formulation: PACE represents target-benchmark performance as each model’s mean score across target instances and source performance as per-instance scores across non-agentic benchmarks.The formulation organizes model outcomes into target and source score matrices.
- Problem Formulation: For held-out evaluation, calibration models fit selection and prediction while unseen models measure generalization.This separates the models used to construct the proxy from those used to evaluate it.
- Problem Formulation: With a source-instance budget C, PACE selects a set of indices and predicts each model’s target score from scores on the selected instances.A second objective learns pairwise preference predictions indicating which of two models is stronger on the target benchmark.
3 PACE: A Proxy for Agentic Capability Evaluation
PACE predicts target agentic performance from a compact subset of non-agentic instances. It combines noise-aware regression with complementary local and global selection strategies, using held-out validation to tune the final ensemble.
- Framework: PACE uses selected source-instance scores to predict target mean performance and pairwise model preferences on agentic benchmarks.The framework separates prediction into Goal A and Goal B.
- Regression: Target-instance bootstrapping reduces sensitivity to sampling noise when fitting the regression objectives.Bootstrap replicates of target means are used so learned weights reflect the target sampling distribution rather than one instance sample.
- Instance Selection: PACE decouples instance selection from regression because joint regularized fitting is under-determined and overfits when training models are far fewer than candidate instances.Independent filter signals are computed before regression.
- Instance Selection: Local selection ranks instances by target relevance, while Global selection multiplies target relevance by SVD leverage to capture globally informative instances.Global leverage measures contribution to the source pool’s latent structure; Local recomputes an SVD on its selected subset.
- Instance Selection: PACE combines the complementary Local and Global subsets into a C-instance proxy, extending overlapping selections to ensure exactly C unique instances are evaluated.The final prediction is an ensemble whose weight and budget split are optimized through held-out validation.
4 Experiments and Results
PACE is evaluated across four diverse agentic benchmarks, 14 models, and 19 non-agentic source benchmarks under strict LOOCV. With 100 proxy instances, it predicts absolute scores and pairwise preferences accurately while substantially reducing evaluation cost.
- Experimental Setup: PACE evaluates GAIA, SWE-Bench Verified, SWE-Bench Multimodal, and SWT-Bench across 14 models and 19 source benchmarks.The agentic targets cover browser-based question answering, repository-level coding, multimodal software engineering, and test construction.
- Evaluation Protocol: Under strict LOOCV, each model is held out while the remaining models drive source-instance selection and regression.The protocol directly tests generalization to unseen models.
- Main Results: 3.80% average MAE and 0.807 Spearman correlation show that 100 source instances predict absolute agentic scores accurately.GAIA and SWT-Bench have slightly larger absolute errors but retain Spearman correlation of at least 0.79.
- Main Results: 84.4% average LOOCV pair-accuracy substantially exceeds the 50% random baseline for pairwise preference prediction.PACE reuses the same selected instances with a pairwise logistic model.
- Cost–Quality Tradeoff: PACE matches random target-sampling quality at roughly 1/100 the cost, with quality improving as budget increases before saturation.The practical sweet spot is C = 100, and PACE dominates random target-sampling below saturation on MAE, Spearman correlation, and ranking accuracy.
5 Analysis and Discussion
The selected proxy instances reveal both shared and target-specific capability requirements across agentic benchmarks. Bootstrap pooling improves predictive metrics on every target, supporting its default use.
- Capability Allocation: Instruction Following and Reasoning reach 100 selected instances for all four targets, while other capability allocations vary substantially.The allocation patterns indicate shared requirements alongside benchmark-specific capability signatures.
- GAIA: GAIA emphasizes Instruction Following plus Verification and Test, matching its browser-based, multiclause question-answering demands.Its allocation is dominated by IFEval and PlanBench instances and includes verification-focused selection.
- SWE-Bench Verified: SWE-Bench Verified selects 65 Multimodal Understanding instances, 34 Information Retrieval instances, and 24 Verification and Test instances.Visual and retrieval items are discriminative despite the target task being text-only patch generation; Code Generation contributes less because it offers limited discrimination.
- SWE-Bench Multimodal: SWE-Bench Multimodal emphasizes Long-Context Aggregation (38), Verification and Test (35), Planning (27), Multimodal Understanding (25), and Code Generation (22).The distribution reflects integration across long issues, repository code, and screenshots.
- SWT-Bench: SWT-Bench peaks at 56 instances each for Verification and Test and Planning, followed by Multimodal Understanding (44) and Information Retrieval (34).These selections match the need to plan test sequences and reason about assertions exposing bug-triggering paths.
- Bootstrap Ablation: Bootstrap improves average MAE by 0.77% and average Spearman by +0.15, with no target regressing on either metric.The ablation compares no-bootstrap and with-bootstrap LOOCV results while holding selection and regression fixed.
6 Conclusion
PACE predicts agentic-benchmark performance from compact, automatically selected non-agentic instance sets using complementary geometric and target-relevance signals. Across four targets and 14 models, it combines strong predictive performance with much lower cost and interpretable capability profiles.
- Contribution: PACE combines noise-aware bootstrap regression with SVD leverage and rank correlation, ensembled through a learned per-target weight.The resulting subsets are compact and tailored to each agentic benchmark.
- Results: 3.80% MAE, 0.81 Spearman correlation, and around 85% pairwise accuracy are achieved across four targets and 14 models.These results use leave-one-out cross-validation and are obtained at roughly 100× lower cost than matched-quality random target sampling.
- Implications: PACE supports cheaper model ranking, denser checkpoint monitoring, and screening before expensive full-harness evaluations.The paper describes these as practical uses during model development and selection.
- Limitation: PACE’s use cases depend on calibration models representing future models, so proxy error may increase outside that calibration distribution.The paper recommends periodically refreshing the calibration set as model distributions shift.
B Benchmark and Capabilities
The paper organizes a broad source benchmark pool around 11 LLM capabilities. These categories describe the atomic skills used to construct and interpret PACE proxies.
- Benchmark Inventory: Table 1 lists standard static benchmarks spanning the source evaluation pool used by PACE.The listed benchmarks include ACPBench, AIME, BEIR, BFCL, DebugBench, GPQA, HumanEval, IFEval, and others.
- Capability Taxonomy: The capability taxonomy contains 11 categories, including instruction following, long-context aggregation, error recovery, planning, code generation, retrieval, code search, tool calling, and reasoning.The categories organize the abilities represented by the source benchmarks.
C Per-source-benchmark allocation
PACE allocates its 100-instance proxy budget across source benchmarks differently for each agentic target, while retaining a shared set of predictors. These allocations reflect target-specific demands such as visual reasoning, web interaction, planning, code navigation, and instruction following.
- Shared predictors across targets: PACE consistently selects PlanBench, VisualPuzzles, and VisualWebBench across targets, providing shared planning, multimodal-reasoning, and web-interaction signals.PlanBench is selected heavily except for SWE-bench Verified, while VisualPuzzles and VisualWebBench are selected for every target.
- Target-specific benchmark allocation: SWE-bench Verified concentrates on VisualWebBench (34) and VisualPuzzles (29), despite evaluating text-only patch generation.The allocation emphasizes visual and web-interaction reasoning as discriminative signals for frontier models.
- Target-specific benchmark allocation: SWE-bench Multimodal draws on PlanBench (22), RepoBench (20), and InFoBench (18) for navigation across code, screenshots, and issues.The selected benchmarks match the target’s multimodal, long-context surface form.
- Target-specific benchmark allocation: GAIA is dominated by PlanBench (37) and IFEval (26), matching browser-based question answering with strict instruction following and multi-step planning.These selections align with GAIA’s combination of instruction-following and planning demands.
- Target-specific benchmark allocation: SWT-bench concentrates on PlanBench (56) and VisualWebBench (34), consistent with multi-step test generation and specification-based validation.The allocation reflects planning and output-validation requirements.
D Budget Sweep
PACE remains informative with small source-instance budgets, while larger budgets improve absolute prediction and pairwise ranking differently. A budget of 100 offers a practical balance between predictive quality and evaluation cost.
- Table 4 setup: Table 4 averages Goal A and Goal B LOOCV performance across the four agentic targets as the source-instance budget C increases from 25 to 500.The table reports absolute-score prediction and pairwise-ranking performance as functions of the source-instance budget.
- Budget-performance trade-offs: 3.30% LOOCV MAE and 0.868 Spearman occur at C = 400, whereas pairwise accuracy reaches 89.27% at C = 500.Absolute prediction begins to worsen at C = 500, suggesting overfitting, while ranking continues to benefit from additional instances.
- Budget-performance trade-offs: At C = 25, PACE achieves 4.02% MAE, 0.832 Spearman, and 83.98% pair accuracy across the four agentic targets.This is within approximately 0.7% MAE of the C = 400 optimum and approximately 5% pair accuracy of the C = 500 optimum.
- Budget-performance trade-offs: C = 100 is the headline budget because it remains competitive with larger budgets while keeping per-model evaluation cost low.The paper uses this budget elsewhere as a practical operating point.
E Lasso and Ridge Baseline
PACE generalizes better under LOOCV than Lasso and Ridge, despite weaker in-sample fitting. The comparison indicates that robust source selection matters more than maximizing in-sample fit for model-efficient prediction.
- Methods compared: The baselines select features through penalized regression, whereas PACE uses a decoupled selection procedure before prediction.Lasso uses nonzero columns, Ridge uses top-C coefficient magnitudes, and the selected inputs feed a separate pair-logistic for Goal B.
- Generalization: PACE’s in-sample-to-LOOCV degradation is smaller than the baselines’, with MAE increasing from 2.88% to 3.80%.Lasso rises from 0.51% to 5.69% MAE, while Ridge rises from 0.82% to 5.16%.
- Generalization: Lasso and Ridge fit nearly perfectly in-sample but degrade sharply under LOOCV, indicating overfitting on the small model set.Their held-out performance is substantially weaker than their in-sample metrics.
- LOOCV comparison: 3.80% LOOCV MAE, 0.81 Spearman, and 84.37% pairwise accuracy make PACE the strongest method versus Lasso and Ridge.Lasso and Ridge achieve 5.69% and 5.16% LOOCV MAE, with lower Spearman and pairwise accuracy.
- Robustness to α: PACE outperforms both baselines at every tested regularization strength α.Table 6 reports average LOOCV MAE and Spearman across the four agentic targets for five α values.
F Limitations
PACE’s evidence is bounded by proxy gaming risk, a small calibration set, the evaluated agentic-benchmark coverage, and the capabilities represented in its static source pool. The paper therefore treats generalization and proxy reliability as scope-limited.
- Proxy gaming: Known proxy instances can be optimized directly, creating proxy-gaming risk analogous to benchmark contamination.Suggested mitigations are refreshing the set, keeping it private until evaluation, or sampling a fresh subset.
- Small calibration set: The 14-model calibration set is small relative to the C = 100 feature dimensionality, so individual learned regression weights may be unreliable.The paper interprets the weights as aggregate signals and expects selection quality to improve with more evaluated models.
- Coverage of agentic benchmarks: PACE’s generalization beyond the four agentic benchmarks sharing the OpenHands framework remains unstudied.Different scaffolds, tool sets, or evaluation protocols may define an untested scope boundary.
- Static source pool: PACE cannot recover capabilities absent from its 19-benchmark source pool, regardless of which instances it selects.Proxy quality is constrained by whether the source pool measures the capabilities required by the target agentic benchmark.
- Claim scope: The introduction’s cost and prediction claims are supported by the reported LOOCV results and cost-quality comparison.The supplied review text assesses the claims as accurately reflecting the paper’s contributions and scope.
2. Limitations
The paper identifies limitations involving calibration assumptions, uncertainty reporting, and reproducibility provisions. It also describes planned access to code, data, and experimental protocols.
- Predictions rely on calibration models being representative of future models, and proxy error may grow under distribution shift.
- The reported experimental protocol specifies LOOCV over 14 models, C = 100, bootstrap B = 300, a fixed seed, 19 source benchmarks, and 4 agentic targets.
- The paper plans to release anonymized code, score matrices, and exact commands for reproducing Table 2 and Figure 1.
- Headline LOOCV results are point estimates without error bars, although bootstrap pooling and an ablation address target-instance label noise.