Source-linked AI summary
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jiachi Chen, Mingwei Liu, Zibin Zheng
TL;DR
Full SWE-agent benchmarks are costly, while result-only efficient evaluation discards process signals from how agents solve tasks. PTA-IRT uses historical trajectories as privileged information for calibration selection and ability estimation, and consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks.
Problem
Efficient SWE evaluation methods largely discard trajectory information by relying on pass/fail outcomes, despite costly multi-step benchmark tasks.
Method
PTA-IRT uses historical execution trajectories as privileged information for trajectory-aware item selection and full-benchmark performance estimation.
Results
PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks under low calibration budgets.
Takeaways & Limitations
Privileged trajectory information makes budgeted SWE-agent evaluation more informative within the paper’s evaluation protocol.
Takeaways & Limitations
Historical trajectories are available for training interactions but are not available or required when estimating a new agent’s performance at inference time.
Abstract
from arXiv · showhide
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.
1 Introduction
Realistic SWE-agent benchmarks are costly because tasks require long, multi-step interaction, while prior efficient evaluation methods largely discard trajectory information. PTA-IRT addresses this gap by using historical trajectories as privileged information and consistently improves budgeted score and ranking recovery across four benchmarks.
- Motivation: Full-benchmark SWE-agent evaluation is expensive because repository-level tasks require code exploration, tool use, editing, and test execution.SWE-bench contains more than 2,000 programming tasks, and a full SWE-agent run can exceed $8,000 under a $4-per-task limit.
- Motivation: Prior IRT methods estimate benchmark performance from small subsets but typically represent each task only through final pass/fail outcomes.This discards process-level signals about explored context, actions, edits, and verification paths.
- Approach: PTA-IRT incorporates historical agent trajectories as privileged information for trajectory-aware item selection and performance estimation.It summarizes trajectories, learns trajectory-aware 4PL item characteristics, selects a difficulty-stratified calibration subset, and transfers measurement knowledge through LUPI.
- Results: PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery under low calibration budgets across four SWE benchmarks.The evaluation compares PTA-IRT with prior IRT baselines under a budgeted subset-to-full benchmark protocol.
- Evaluation setting: The paper reports experiments on four SWE benchmarks and frames efficient benchmarking as recovering full-benchmark performance from a limited calibration subset.Historical trajectories are used offline, while a new agent is executed only on the calibration subset.
2 Background
SWE benchmarks increasingly require costly long-horizon agent interactions, motivating smaller evaluation subsets and IRT-based performance estimation. The background introduces IRT and LUPI, while positioning trajectories as privileged process information unavailable when evaluating a new agent.
- Benchmark context: Modern SWE-agent benchmarks require long-horizon reasoning, tool use, editing, and testing, making them more expensive than conventional single-turn evaluations.These requirements arise in repository-level and agentic benchmark settings.
- Efficient evaluation: IRT reduces evaluation cost by modeling agent–instance responses to estimate item properties and latent agent abilities from fewer test instances.Its item-level properties include difficulty and discriminability.
- Prior work: Prior efficient SWE evaluation includes neural IRT extensions and agent-driven item selection, while SWE benchmarks also use rule-based or human-filtered subsets.Examples include Deep-IRT, PSN-IRT, AutoJudger, SWE-bench Lite, and SWE-bench Verified.
- Item Response Theory: The IRT Item Characteristic Curve models the conditional probability that an agent correctly solves an instance from the agent’s latent ability and item difficulty.The one-parameter logistic formulation uses θ_i for agent ability and b_j for instance difficulty.
- Learning Using Privileged Information: LUPI uses privileged information during training to improve a predictor that relies only on standard inputs at inference time.In this setting, historical agent trajectories are privileged information and are not required when estimating a new agent’s full-benchmark performance.
3 PTA-IRT
PTA-IRT uses historical agent trajectories as privileged process evidence to select an informative, difficulty-stratified calibration subset and estimate full-benchmark ability from limited outcomes.
- Trajectory representations: PTA-IRT compresses heterogeneous execution logs into structured summaries covering task goals, explored context, executed edits, and solution paths.The summaries describe observable actions without inferring correctness or unobserved reasoning.
- Calibration selection: The calibration subset allocates items across pass-rate difficulty bins and selects high-information tasks within each bin to cover easy-to-hard regimes.This balances informativeness with representativeness for recovering full-benchmark scores from a small budget.
- Trajectory-aware scoring: A trajectory-aware 4PL IRT model injects summary-derived residuals into item discriminability and difficulty while keeping guessing and feasibility bounds shared.The shared item parameters are (aj, bj, cj, dj); trajectory information adjusts only discriminability and difficulty.
- Calibration selection: PTA-IRT scores tasks with trajectory-aware Fisher information weighted by effective sample size, reducing priority for items with low-quality privileged evidence.Incomplete or malformed summaries are down-weighted, and the ESS factor lowers scores when privileged evidence is sparse or unreliable.
- Ability estimation: A teacher–student LUPI estimator uses privileged summaries offline, while the deployable student estimates ability from calibration outcomes and extrapolates across all tasks.At test time, network weights are frozen and a scalar ability θ* is fit on the calibration subset before predicting benchmark-wide pass probabilities.
4 Experimental Setup
The experiments evaluate PTA-IRT across four SWE-bench versions against classical, neural, and agent-driven IRT baselines using score-error and ranking-recovery metrics with four-fold cross-validation.
- Benchmarks: The evaluation covers SWE-bench Lite, Verified, Full, and Pro, spanning curated subsets and longer-horizon enterprise-style tasks.Lite and Verified are curated subsets of Full, while Pro extends the benchmark across broader repositories and languages.
- Baselines: Baselines span classical IRT methods, neural IRT methods, and AutoJudger’s agent-driven calibration-item selection.Classical baselines include MLE, MCMC, VI, and VIBO; advanced methods include Deep-IRT and PSN-IRT.
- Metrics: The study measures Mean Absolute Error, Kendall’s τ, and Spearman’s ρ to assess score accuracy and ranking recovery.MAE compares predicted with ground-truth full-benchmark performance, while Kendall’s τ and Spearman’s ρ evaluate ranking agreement and monotonic correlation.
- Evaluation protocol: Four-fold cross-validation trains on 75% of evaluated models and tests on the remaining 25%.Process summaries are generated with DeepSeek-V4-Flash1 and embedded with all-MiniLM-L6-v2.
5 Main Results
Across four SWE benchmarks, PTA-IRT improves budgeted score and ranking recovery, with gains supported by trajectory-aware selection and process-aware representations. Ablations and trajectory analyses show that summary content, stratified selection, and privileged supervision each contribute to performance.
- RQ1: Effectiveness: At 10% calibration, PTA-IRT achieves the best MAE, Kendall’s τ, and Spearman’s ρ across all four SWE benchmarks.Average performance is MAE 0.041±0.015, τ = 0.888, and ρ = 0.973.
- RQ1: Effectiveness: 5% calibration already reaches τ = 0.768 on SWE-bench Lite, while 20% raises τ from 0.768 to 0.886 and tends to reduce MAE.Figure 2 evaluates calibration budgets from 5–25% using MAE, Kendall’s τ, and Spearman’s ρ.
- RQ2: Ablation Analysis: Removing trajectory-aware scoring, LUPI, Top-K selection, or clustering worsens average score and ranking recovery relative to the full configuration.The only local exception is a marginal MAE improvement on Verified without LUPI; it does not overturn the average result.
- RQ3: Trajectory Summaries: Summary dropout generally weakens recovery, and full dropout without ESS approaches the trajectory-scorer ablation on Lite, indicating that gains depend on summary content.At extreme dropout, both ESS settings degrade as usable process signal disappears.
- RQ4: Summary Characteristics: Trajectory summaries divide labor across channels: Task Goal aligns with issue text, Edits Executed with submitted patches, and Context Explored and Path Overview remain process-specific.Whole-summary question similarity is 0.714 with Δ = +0.473 versus a random-question baseline; Task Goal reaches 0.776 and Edits Executed reaches 0.670 against submitted answers.
- RQ4: Summary Characteristics: Submitted answers concentrate outcome geometry, whereas summaries preserve diverse solving paths that binary labels and patches alone do not collapse.Answer distances are PP 0.175 versus FF 0.290 with Δout = 0.033; full-summary distances are PP 0.143 versus FF 0.163 with Δout = 0.005.
6 Conclusion
PTA-IRT addresses the cost of full SWE-agent evaluation by using historical execution trajectories as privileged information in efficient IRT-based evaluation. Across four SWE benchmarks and low calibration budgets, it consistently outperforms prior IRT baselines on score and ranking recovery.
- Conclusion: PTA-IRT builds structured trajectory summaries, selects a difficulty-stratified calibration subset, and estimates full-benchmark score and ranking from limited observations via LUPI.Historical execution trajectories provide privileged information for trajectory-aware measurement and ability estimation.
- Conclusion: Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks.The evaluation protocol uses limited calibration observations to estimate full-benchmark performance.