PaperlayerView exact revision on arXiv

Source-linked AI summary

HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Wanyi Chen, Wanqin Shen, Yuanchen Shi, Zhensu Sun, Fang Kong

arXiv:2610.24611v2cs.AI

TL;DR

Evaluating software agents across full benchmarks is expensive, and small subsets can preserve rankings while yielding inaccurate scores. HSPR combines historical response factors, uncertain group success rates, and posterior-guided task selection to recover full-benchmark scores. At a 10% budget, it has the lowest MAE on three of four SWE-bench banks and the lowest MaxErr on all four among compared methods.

  • Problem

    Full-benchmark software-agent evaluation is expensive, while a small task subset can preserve rankings yet produce inaccurate scores.

    Source: §1 · p. 1

  • Method

    HSPR models source-fitted response factors and uncertain success rates within source-signature groups, updates posteriors from target outcomes, and uses posterior-guided acquisition and a calibrated posterior median.

    Source: §1 · p. 2

  • Results

    At a 10% task budget, HSPR achieves the lowest MAE point estimates on three of four SWE-bench banks and the lowest MaxErr on all four among compared methods.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results highlight the role of historical response structure in task-efficient full-score recovery, with the contribution of group uncertainty varying across task banks.

    Source: §5 · p. 8

    Generalization under substantial shifts in model families or agent scaffolds remains unevaluated, and posterior uncertainty is conditional on the assumed response model, including within-group exchangeability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets. Code and project resources are publicly available.

1 Introduction

The introduction frames full-benchmark score recovery as a costly inference problem: small subsets may preserve rankings while misestimating scores. HSPR uses historical response structure and adaptive queries, and reports stronger benchmark results than the compared methods at a 10% task budget.

1 Introduction

  • Full software-agent evaluation can require repository exploration, code editing, tool use, and execution across hundreds or thousands of tasks.The study evaluates four SWE-bench task banks and includes hierarchy controls, estimation and acquisition ablations, paired analysis, mechanism diagnostics, and budget sweeps.Source: §1 · pp. 1–2
  • At a 10% budget on SWE-bench Pro, Mid-Range selection preserves model ordering (τb = .967) but its raw subset mean has MAE of 22.798 pp.Using the same outcomes, a source-fitted Ridge estimator reduces MAE to 3.434 pp.Source: §1 · p. 1
  • Historical tasks with identical source-response signatures can have different outcomes for a new target agent, despite sharing a response tendency.The introduction asks how to recover a target’s full-benchmark score from few executions while accounting for uncertainty in transfer from source response patterns.Source: §1 · p. 1
  • HSPR combines source-fitted response factors with uncertain success rates within source-signature groups, updating both target latent-state and group-rate posteriors after each observed outcome.Posterior-guided acquisition selects subsequent tasks, and observed successes plus predictive counts for unqueried tasks define a full-score posterior whose median is calibrated using source-only validation.Source: §1 · p. 2
  • At a 10% task budget, HSPR has the lowest MAE point estimates on Lite, Full, and Pro and the lowest MaxErr on all four SWE-bench banks among compared methods.In controls, removing group uncertainty or replacing adaptive queries increases macro MAE; equal-bank macro MAE decreases across 5–20% budgets.Source: §1 · p. 2

2 Related Work

Prior work addresses efficient benchmark prediction, psychometric evaluation, subset objectives, and broader agent-evaluation settings. HSPR draws on historical responses but focuses on estimating the numerical full-benchmark score through a finite-score posterior and score-oriented acquisition.

Efficient benchmark prediction

  • Efficient-testing methods combine informative queries with correction or learned prediction, while compact subsets and response-matrix methods optimize evaluation from historical outcomes.HSPR instead uses a model-conditional finite-score posterior with global factors and uncertain group rates.Source: §2 · p. 2

Psychometric evaluation

  • Psychometric approaches include item-response calibration, adaptive testing, agent-response models, and historical-trajectory methods; HSPR adds residual response dimensions and conditional target-specific group rates.Unlike ATLAS’s latent-ability objective, HSPR’s score-variance criterion targets full-benchmark score while accounting for dependence among unobserved outcomes.Source: §2 · p. 2

Subsets and objectives

  • Prior subset work studies ranking preservation, subset quality, and rank-correlation prediction, whereas score recovery requires an accurate numerical scale.Source: §2 · p. 2

Agent evaluation context

  • Agent-evaluation benchmarks cover tool use, multi-turn interaction, and end-to-end engineering; some methods use trajectories or process-aware observations, while HSPR uses source binary responses and observed target outcomes.Source: §2 · p. 2

3 Hierarchical Score Posterior Recovery

HSPR proceeds from historical response modeling through adaptive target queries to full-score recovery. It combines source-derived task structure with observed outcomes and predictive counts for unqueried tasks.

3.1 Finite-Score Estimation

  • HSPR estimates the full-benchmark score from exact observed outcomes and a joint predictive distribution over unobserved outcomes that preserves their dependence.The evaluator observes at most B distinct target-task outcomes; acquisition uses the source response matrix and observed history.Source: §3.1 · p. 3

3.2 Historical Factors and Groups

  • A regularized logistic item model and residual factors represent historical responses, while tasks with identical source responses are grouped and their fitted parameters averaged.The target latent state combines scalar ability with residual coordinates; the fitted item parameters remain fixed during target inference.Source: §3.2 · pp. 3–4

3.3 Target-Specific Group Uncertainty

  • Within each source-signature group, target outcomes share an uncertain success rate, so observing an outcome updates group-rate and latent-state posteriors and informs unqueried predictions.Integrating over group rates couples outcomes within a group, while integrating over the global latent state links predictions across groups.Source: §3.3 · p. 4

3.4 Posterior-Guided Acquisition

  • The score-variance acquisition rule selects a group using expected score-variance reduction; this greedy criterion is not established as optimal for final absolute error.Its variance reduction includes both resolving the queried outcome and updating predictions for other unqueried group members.Source: §3.4 · p. 4

3.5 Predictive Score Decision

  • Under absolute score loss, the predictive score median is the posterior-optimal decision, with a source-selected calibration applied to the estimate.Predictive draws include observed successes exactly and sample unqueried group success counts; at full budget, the estimate equals the score.Source: §3.5 · p. 5

3.6 Source-Only Configuration

  • Staged leave-one-source-agent-out validation selects model and acquisition settings before target evaluation, and importance sampling approximates the continuous posterior for score prediction.The configuration is frozen after refitting item parameters, residual factors, and groups on all source agents.Source: §3.6 · p. 5

4 Experiments

Across four archived SWE-bench task banks, the experiments compare matched-budget procedures and test hierarchy, adaptive querying, and budget sensitivity. At 10%, HSPR has the lowest MAE on three banks and the lowest MaxErr on all four, while the controls examine contributions from group uncertainty and query adaptivity.

4.1 Experimental Setup

  • Four archived SWE-bench banks use held-out agent configurations, fold-disjoint source and target identifiers, and primary 10% budgets of 30, 50, 229, and 73 tasks.Table 1 compares complete procedures at matched target-task budgets, allowing query policies and estimators to differ.Source: §4.1 · pp. 5–6

4.2 Main Results

  • At 10%, HSPR has the lowest MAE on Lite, Full, and Pro, is within 0.055 pp of ATLAS-port on Verified, and has the lowest MaxErr on all four banks.Equal-bank macro MAE is 1.664 pp for HSPR versus 2.478 pp for PTA-IRT, the strongest baseline on that aggregate.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • With reconstruction fixed, replacing adaptive queries raises macro MAE from 1.664 to 3.085 pp with Uniform queries or 2.934 pp with Mid-Range queries.Removing group uncertainty raises macro MAE from 1.664 to 1.808 pp; the paired bootstrap interval for the difference is [−0.012, 0.302]. The median and mean yield 1.664 and 1.674 pp on shared paths.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Mechanism Analysis

  • Full averages 8.32 tasks per source-signature group, versus 426 groups among Verified’s 500 tasks; source-zero tasks still have target success rates of 1.84% on Full and 2.07% on Pro.In a controlled Full comparison, source-zero tasks receive 13.2% of queries with hierarchy versus 0.19% without it.Source: §4.4 · pp. 7–8

4.5 Budget Sensitivity

  • Across 5–20% budgets, HSPR’s equal-bank macro MAE falls from 2.562 to 1.262 pp, using fixed 10% source-selected settings and nested query paths.Baselines retain budget-specific settings; Figure 3 reports MAE and MaxErr in percentage points across four task banks.Source: §4.5 · p. 8, Figure 3 · p. 8

4.6 Discussion

  • The score-acquisition contribution from group-rate uncertainty scales with the number of unqueried group members and the posterior variance of their common rate.As queries reveal outcomes, remaining group sizes and rate posteriors change; larger source-signature groups permit more within-group sharing than Verified’s groups.Source: §4.6 · p. 8

5 Conclusion

HSPR combines source-fitted response factors, uncertain group success rates, and posterior-guided acquisition to recover full-benchmark scores from limited observations. At a 10% budget, it has the lowest MAE on three of four SWE-bench task banks and the lowest MaxErr on all four among the compared methods.

5 Conclusion

  • HSPR combines source-fitted response factors, uncertain group success rates, and posterior-guided acquisition to estimate full-benchmark scores from limited observations.Its final estimate uses a posterior median with source-selected calibration.Source: §5 · p. 8

Limitations

The evaluation covers four SWE-bench-family task banks, but generalization to substantial shifts in model families or agent scaffolds remains unevaluated. HSPR also depends on historical source-bank coverage, and its posterior uncertainty is conditional on the response model and within-group exchangeability.

Limitations

  • Generalization under substantial shifts in model families or agent scaffolds remains to be evaluated.HSPR’s response factors and source-signature groups depend on historical source-bank coverage.Source: Limitations · p. 9
  • Posterior uncertainty is conditional on the assumed response model, including within-group exchangeability.Source: Limitations · p. 9
  • Evaluation budgets count distinct target-task executions, but task-count reductions alone do not establish token, wall-clock, or monetary savings.Historical data collection, source fitting, and configuration selection add costs.Source: Limitations · p. 9

Ethical Considerations

The study uses existing software-agent response matrices and does not involve human participants or private user data. It cautions that recovered scores may mislead comparisons, especially when historical source data poorly represent a target configuration.

Ethical Considerations

  • Treating recovered scores as fully observed results may mislead model comparisons, particularly when historical source data poorly represent a target configuration.Source: Ethical Considerations · p. 9
  • The study uses existing software-agent response matrices and does not involve human participants or private user data.Source: Ethical Considerations · p. 9
  • Reports should state the evaluation budget and distinguish recovered scores from full-benchmark measurements.Source: Ethical Considerations · p. 9

A Implementation and Source-Only Configuration

HSPR fits source response structure and selects its deployed configuration through source-only validation. The implementation freezes the selected model and settings before target evaluation.

A Implementation and Source-Only Configuration

  • Source abilities are standardized logits of Jeffreys-smoothed full scores, and regularized logistic item fitting estimates task intercepts and positive slopes.The numerical fitting settings are fixed for source fitting.Source: A Implementation and Source-Only… · p. 10
  • A source-response residual SVD supplies task directions whose loadings represent departures from the scalar ability pattern.The source-fitted item parameters are fixed during target inference.Source: A Implementation and Source-Only… · pp. 10–11
  • Source-signature groups average source-fitted intercepts, slopes, and residual directions, while the configuration grid varies residual rank and precision.Source: A Implementation and Source-Only… · p. 11
  • Source-only validation selects the model rank, precision, group-uncertainty strength, acquisition rule, and calibration through staged configuration grids.Each held source is excluded before fitting; the selected configuration is refit on all source agents and frozen before target replay.Source: A Implementation and Source-Only… · p. 11
  • An auxiliary logistic-normal group MAP proposes latent states; conjugate updates reweight existing nodes and update observed counts between proposal refreshes.The proposal is refreshed after each eight observed outcomes.Source: A Implementation and Source-Only… · p. 11
  • Predictive medians use 131,072 complete finite-score draws, while query seeds 0–4 generate the five target paths.Separate random generators keep predictive sampling independent of within-group query randomization.Source: A Implementation and Source-Only… · p. 11

B Derivation of the Acquisition Criterion

The derivation uses the law of total variance to characterize one-step uncertainty reduction and distinguish direct task information from global transfer across groups.

B Derivation of the Acquisition Criterion

  • The score-variance calculation retains both Bernoulli outcome variance and shared group-rate uncertainty when forming predictive finite-score variance.Source: B Derivation of the Acquisition… · p. 11
  • Marginalizing the conditional score covariance adds a term for global transfer between groups; omitting the direct item term instead favors information about other unqueried tasks.Source: B Derivation of the Acquisition… · p. 11
  • The law of total variance establishes the identity for score T = S and coordinatewise for latent states, exactly under the assumed distribution, including a finite weighted-node approximation.Source: B Derivation of the Acquisition… · pp. 11–12

C Evaluation Protocol and Reproducibility

The evaluation freezes model, query-policy, and calibration choices before target replay, then scores predictions only after recording them. Paired statistical comparisons use configuration-level errors conditional on the archived task bank.

C Evaluation Protocol and Reproducibility

  • Source validation removes the held-out row before fitting, and all model, query-policy, and calibration choices are frozen before target replay.The selected configuration is then refit on all source rows.Source: C Evaluation Protocol and… · p. 12
  • Each query path uses B distinct executions, excludes exhausted groups, and samples uniformly within the selected group before revealing the outcome.Predictive draws use a numerical generator separate from query selection.Source: C Evaluation Protocol and… · p. 12
  • Full target outcomes are accessed only after each score estimate and queried-task identities are recorded, separating inference from scoring information.Historical bank collection and source fitting are additional costs outside the task budget.Source: C Evaluation Protocol and… · p. 12
  • FAQ-port and ATLAS-port are adapted procedures, and matched task budgets do not equalize methods’ historical information or source-fitting costs.RSL uses Uniform queries; Mid-Range variants share a source-selected task set rather than HSPR’s adaptive paths.Source: C Evaluation Protocol and… · p. 12

D Complete Paired Statistical Comparisons

  • Paired comparisons use one seed-averaged error per configuration; bootstrap intervals condition on the archived task bank, while Wilcoxon tests use a global 16-contrast Holm correction.Baseline seed counts differ, so the paired unit is the configuration.Source: D Complete Paired Statistical… · p. 12

E Additional Controlled Analyses and Numerical Validation

The controlled analyses examine hierarchy and matched-path reconstruction, numerical precision, historical response structure, inference overhead, and score recovery across model families. The hierarchy control raises macro MAE, while family-level errors vary across banks.

E Additional Controlled Analyses and Numerical Validation

  • Removing hierarchy raises equal-bank macro MAE from 1.664 to 1.808 pp, but the 95% interval for the 0.143 pp difference includes zero.The controls retain source-selected acquisition settings and calibration, while each generates its own posterior and distinct-task path.Source: E Additional Controlled Analyses and… · p. 12
  • Numerical checks compare 8,192 and 32,768 integration nodes at a 0.05 pp tolerance, and compare Verified draws at 65,536 and 131,072.The median’s central 95% numerical interval is checked against one finite-task increment, 1/N.Source: E Additional Controlled Analyses and… · p. 13
  • The matched-path comparison finds the largest group-uncertainty improvement on Full, while Pro slightly favors the no-group mean.The comparison uses posterior means on identical query paths; the mean-versus-median comparison also uses identical paths with calibration selected separately by decision family.Source: E Additional Controlled Analyses and… · p. 13, Table 6 · p. 13
  • The 10% budget sweep preserves primary predictions and checks source checkpoints at the same numerical tolerances; acquisition controls retain HSPR settings while comparing Uniform and Mid-Range query policies.Uniform reuses saved baseline permutations, whereas Mid-Range orders tasks by source-rate distance from .5.Source: E Additional Controlled Analyses and… · pp. 13–14

F Historical Response Structure and Transfer Diagnostics

G Computational Overhead

  • The inference batch took 510.1 seconds under concurrent evaluation and 3.31 CPU hours total, excluding source fitting, configuration selection, and agent task execution.All replayed paths matched archived results, with a largest absolute prediction difference of 2.22 × 10−16 on the unit score scale.Source: G Computational Overhead · p. 14

H Score Recovery Across Model Families

  • On Lite, GPT/o-series mean error is 1.71 pp versus Claude’s 2.15 pp; on Verified, it is 1.70 pp versus 1.52 pp.Full errors are below 1.71 pp for every configuration, and its family means are 0.39, 0.80, and 0.74 pp for GPT/o-series, Claude, and unspecified groups, respectively.Source: H Score Recovery Across Model Families · p. 14, Figure 4 · p. 15

Related papers

Loading 2610.24611v2…