Source-linked AI summary
On Randomness in Agentic Evals
Bjarni Haukur Bjarnason, André Silva, Martin Monperrus
TL;DR
Agentic evaluations often rely on a single pass@1 run, but the reliability of such scores is uncertain despite their influence on deployment, research, and leaderboard decisions. This paper analyzes 60,000 SWE-Bench-Verified trajectories across models and scaffolds, finding substantial single-run variance and recommending repeated runs and complementary metrics for more reliable evaluation.
Problem
Single-run agentic evaluation scores may be unreliable even though they inform model deployment, research direction, and leaderboard-based progress assessments.
Method
The paper conducts ten independent runs of six model–scaffold configurations on SWE-Bench-Verified, analyzing outcomes, performance distributions, and trajectory divergence at different sampling temperatures.
Results
Single-run pass@1 estimates vary by 2.2 to 6.0 percentage points, with standard deviations exceeding 1.5 percentage points even at temperature 0; trajectories diverge early and can yield different solution strategies.
Takeaways & Limitations
Reliable agentic evaluation should use multiple independent runs, statistical power analysis, and metrics such as pass@k and pass^k to characterize performance under favorable and unfavorable randomness.
Takeaways & Limitations
Multiple-run evaluation increases cost, particularly for GPU-poor organizations and academic settings.
Abstract
from arXiv · showhide
Agentic systems are evaluated on benchmarks where agents interact with environments to solve tasks. Most papers report a pass@1 score computed from a single run per task, assuming this gives a reliable performance estimate. We test this assumption by collecting 60,000 agentic trajectories on SWE-Bench-Verified, spanning three models and two scaffolds. We find substantial variance: single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected, with standard deviations exceeding 1.5 percentage points even at temperature 0. This variance has critical implications: reported improvements of 2--3 percentage points may reflect evaluation noise rather than genuine algorithmic progress. Through token-level analysis, we show that trajectories diverge early, often within the first few percent of tokens, and that these small differences cascade into different solution strategies. To enable reliable evaluation of agentic systems, we recommend three concrete practices: (1) estimate pass@1 from multiple independent runs per task, especially when measuring small improvements, (2) use statistical power analysis to determine the number of runs needed to detect expected effect sizes, and (3) consider metrics like pass@k (optimistic bound) and pass^k (pessimistic bound) with k>1 to better characterize the full performance envelope. While these practices increase evaluation cost, they are essential for distinguishing genuine scientific progress from statistical noise.
1 INTRODUCTION
Agentic evaluations commonly report pass@1 from one run per task, but this study finds that such scores can vary substantially even under temperature-0 settings. The authors trace this variance to early trajectory divergence and recommend multi-run, statistically powered evaluation with broader metrics.
- Evaluation practice: Single-run pass@1 is standard in agentic benchmarks, despite estimating success from only one binary outcome per task.The practice is used across research papers, model releases, and community leaderboards.
- Evaluation variance: 2.2 to 6.0 percentage points: single-run pass@1 estimates vary depending on which run is observed.Standard deviations exceed 1.5 percentage points even at temperature 0, and variance persists across all tested configurations.
- Study design: 60,000 trajectories across three models and two scaffolds quantify randomness in SWE-Bench-Verified agentic evaluations.The study collects over 25.58B tokens and 1.88M tool calls while analyzing outcomes, performance distributions, and divergence points.
- Mechanism: Early trajectory differences can cascade through autoregressive conditioning into fundamentally different solution strategies.The token-level analysis finds divergence often within the first few percent of tokens.
- Interpretation: 2–3 percentage-point improvements may reflect evaluation noise rather than genuine algorithmic progress.The authors illustrate that a one-run change from 31% to 33% could arise from sampling a favorable run.
- Recommendations: Reliable evaluation requires multiple independent runs, statistical power analysis, and pass@k and passˆk metrics with k > 1.These practices are intended to characterize the full performance envelope and distinguish progress from statistical noise.
2 CHARACTERIZING RANDOMNESS IN AGENTIC EVALS
The study measures randomness in agentic evaluations across models, scaffolds, temperatures, and repeated SWE-Bench-Verified runs. It finds substantial run-to-run variation, early trajectory divergence, and performance bounds that expose dependence on stochastic exploration.
- Experimental setup: The experiments run six model-scaffold configurations ten times each on SWE-Bench-Verified, including temperature 0 and nonzero sampling settings.The study analyzes 60,000 trajectories, over 25.58B tokens, and 1.88M tool calls.
- Evaluation variance: 2.2 to 6.0 percentage points: single-run resolution rates vary across the twelve evaluated configurations, so one run can materially change the reported score.Table 1 aggregates ten independent evaluations per configuration and reports means, standard deviations, minima, and maxima.
- Evaluation variance: Temperature 0 does not eliminate variance: DeepSWE-preview on nano-agent records 20.4 ± 1.0% with an 18.2%–21.4% range.Qwen3-32B on R2E-Gym similarly records 22.3 ± 1.8% with a 19.8%–25.2% range.
- Performance bounds: 18.5 percentage points: DeepSWE-preview on R2E-Gym rises from 34.4% pass@1 to 52.9% pass@5, while passˆ5 is 15.5%.The 18.9-point pass@1-to-passˆ5 gap indicates that favorable random choices affect part of the observed capability.
- Performance bounds: 24.9 percentage points: the largest pass@1-to-pass@5 improvement occurs for Devstral-2 on R2E-Gym at temperature 0.For Devstral-2 on nano-agent, narrower bounds still coexist with substantial benefit from stochastic exploration.
- Trajectory divergence: First divergence typically occurs within the first 1% of a trajectory, although temperature 0 delays rather than suppresses it.For DeepSWE-preview on nano-agent, median divergence moves from token 5 at temperature 1.0 to token 56 at temperature 0.0; small differences can then cascade into different strategies and outcomes.
3 IMPLICATIONS AND MITIGATION STRATEGIES
Single-run scores can obscure whether observed differences reflect genuine capability gaps or evaluation noise, especially for long-horizon agentic tasks. The paper therefore recommends repeated evaluation, power analysis, and performance-envelope metrics, while noting the added cost.
- Implications: Single-run leaderboards may reflect evaluation noise rather than true capability ordering.This makes it difficult to determine whether observed differences are genuine capability gaps or different samples from overlapping performance distributions.
- Implications: Early divergence in long-horizon trajectories can cascade through subsequent actions and amplify evaluation variance.Pass@1 originated in shorter, independent code generations, whereas agentic tasks involve multi-step interactions.
- Mitigation strategies: Approximately 9 runs detect a 2% improvement at p < 0.05 with 80% power, while 1% improvements require 36 runs.The required run count depends on improvement magnitude, desired power, and observed variance.
- Mitigation strategies: At median variance, detecting a 1% improvement requires 36 runs, compared with 8 runs at the lowest observed variance of σ = 0.7%.Large improvements such as 10% can require far fewer runs and may sometimes be detectable with single runs.
- Mitigation strategies: Reporting pass@1, pass@k, and passˆk characterizes expected performance, optimistic retry performance, and pessimistic consistency.The pass@k–passˆk gap indicates how much stochasticity may be beneficial or detrimental.
- Trade-offs: Multiple runs improve reliability but increase evaluation cost, especially for GPU-poor organizations and academic settings.The paper presents this cost as a practical concern accompanying more reliable evaluation.
4 RELATED WORK
Prior work identifies substantial nondeterminism and reproducibility challenges in language-model and agentic evaluations, while related research also leverages behavioral variance for diverse solution search.
- Infrastructure effects, repeated queries, and prompt sensitivity can produce differing outputs even under nominally deterministic or greedy sampling.Reported sources include floating-point operations, rounding, hardware, batch size, and prompt variation.
- Intraclass correlation has been proposed to quantify agentic evaluation stability, which varies with task complexity and model capability.
- Behavioral diversity and self-consistency exploit multiple solution paths to improve performance on some tasks.
- This paper complements variance-exploitation work by analyzing when and why multi-step agent trajectories diverge and how variance should be interpreted in evaluations.
- Prior reproducibility research reports difficulty reproducing baselines and advocates standardized assessment with multiple runs and confidence measures.
5 CONCLUSION
The paper concludes that randomness materially limits the reliability of agentic evaluations, with variance persisting at temperature 0 and trajectories diverging early before cascading into different strategies.
- 60,000 trajectories across six agentic systems reveal single-run pass@1 ranges of 2.2–6.0 percentage points, persisting at temperature 0.
- Trajectory divergence occurs at a median within the first 1% of tokens and cascades through autoregressive conditioning.
- Optimistic and pessimistic performance bounds differ by up to 24.9 percentage points.
- Future work should examine dynamic context strategies and longer-horizon tasks, which were excluded or not covered in this study.
A STATISTICAL POWER ANALYSIS FOR DETERMINING THE NUMBER OF RUNS
The power analysis models run-level pass@1 differences with a two-sample hypothesis test and shows that detecting smaller improvements requires rapidly more runs, especially under greater variance or higher power.
- The analysis determines runs needed to detect a pass@1 difference ∆ = |µ1 − µ2| at chosen significance and power levels.It treats two experimental conditions with true values µ1 and µ2, sample means x̄1 and x̄2, and standard deviations σ1 and σ2.
- The framework uses a two-sample t-test with equal sample sizes and known equal standard deviation to calculate runs per agent under test.
- The test rejects H0 when the absolute statistic exceeds the two-tailed critical value, while power requires the expected statistic to exceed that value by a Zβ margin.
- At p < 0.05 and 80% power, required runs increase exponentially as targeted improvements shrink across variance scenarios from σ = 0.7% to σ = 1.8%.
- At median variance σ = 1.5%, detecting a 2% improvement requires 9 runs at 80% power and 15 runs at 95% power.
B NORMALITY OF P A S S@1 SCORES
The paper evaluates whether pass@1 scores across ten runs are compatible with a normal approximation and finds results broadly consistent with using that approximation for power analysis.
- Eleven of 12 configuration–scaffold groups pass the Shapiro-Wilk test at p ≥ 0.05.
- The sole exception is DeepSWE-preview on nano-agent, with W = 0.80 and p = 0.016.
- Because n = 10 gives limited power to detect non-normality, passing the test does not confirm exact normality.
- The results are nevertheless consistent with the normal approximation being adequate for the power analysis.
C INFERENCE HYPER-PARAMETERS
The experiments specify model hosting, inference settings, and scaffold tool-call limits. Temperature 0 runs use greedy decoding across all models.
- Qwen3-32B and DeepSWE-preview were hosted with vLLM on NVIDIA A100 80GB GPUs, using approximately 3,500 GPU hours.
- Devstral-2 was accessed through Mistral’s API.
- Both scaffolds used default configurations, with nano-agent capped at 500 tool calls and r2e-gym at 100.
- Temperature 0 experiments used greedy decoding for all models.
D PASS@K PLOTS
The appendix provides additional pass@k and pass^k plots across the evaluated model–scaffold pairs. Figure 1 covers DeepSWE-preview, while the additional plots cover the remaining pairs.
- Figure 1 shows pass@k and pass^k curves for DeepSWE-preview with both nano-agent and r2e-gym.
- Additional pass@k plots are provided for all other model–scaffold pairs evaluated in the study.
- Figures 6 and 7 present additional pass@k and pass^k curves for all model–scaffold pairs in two parts.