Source-linked AI summary
Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
Jiajun Jiang, Sharon Zheng, Natan Vidra, Spurthi Setty
TL;DR
AI coding benchmarks can mismeasure reliability by confusing unit-test count with independent attempts and by treating functional correctness as deployment safety. This paper corrects the estimator and proposes security-adjusted reliability, finding that synthetic scores drop from 0.96–0.98 to 0.00–0.12.
Problem
Current agent benchmarks misuse pass@k by treating unit-test count as independent attempts, while functional correctness alone does not establish deployment safety.
Method
The paper applies pass@k with independent rollouts, evaluates it in a synthetic multi-rollout benchmark, and proposes a security-adjusted complementary metric.
Results
Corrected reliability estimates radically altered synthetic absolute scores, reducing them from 0.96–0.98 to 0.00–0.12.
Takeaways & Limitations
Multiple independent rollouts are necessary for trustworthy reliability estimates, and security-adjusted reliability remains a complementary lens requiring stronger evaluation.
Takeaways & Limitations
Findings are primarily synthetic rather than production-based, while the security-adjusted experiment covered only three agents, limiting statistical power.
Abstract
from arXiv · showhide
AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number of unit tests in a single submission rather than the number of independent rollout attempts, conflating test-suite size with attempt independence. We diagnose this operationalization error, prove it by counterexample, and propose reliability@k, the same estimator applied correctly, with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair. In a synthetic multi-rollout benchmark, the misapplied metric inflates reported scores by 0.85-0.97 in absolute terms (0.96-0.98 reported vs. 0.00-0.12 corrected), and a cheap single-rollout proxy fails to substitute for repeated runs (Spearman $ρ= 0.417$). Motivated by evidence that functional correctness does not imply security safety, we additionally propose security-adjusted reliability@k, which counts only rollouts that are both functionally correct and free of high-severity insecure patterns. In an initial live-API test with three agents, the adjustment did not change any ranking under our current scanner and threshold, so we present it as a proposed complementary lens whose decisive evaluation requires better-powered future runs. Finally, a preliminary 5-task SWE-bench Verified pilot observes the same core concern in a real repository setting: macro-averaged hidden-test pass rate was 0.80 while strict task resolution was 0.20.
1 Introduction
The introduction identifies two independent failures in AI coding-agent evaluation: misapplying pass@k by treating unit tests as independent samples and overlooking security risks in functionally correct code. It presents CodeBench and proposes corrected reliability and security-adjusted metrics, supported by synthetic, live-API, and preliminary SWE-bench evidence.
- Evaluation failures: Current pass@k practice substitutes unit-test counts for independent full-solution rollouts, violating the estimator’s i.i.d. assumption.The paper formalizes this operationalization error and provides a counterexample involving agents with identical true reliability.
- Evaluation failures: 0.85–0.97 absolute inflation results from the pass@k operationalization error, potentially inverting agent rankings.The estimator is mathematically correct, but its current implementation uses correlated sub-results rather than independent samples.
- Evaluation failures: Functional test success does not establish deployment safety: Veracode reports 2.74× more vulnerabilities in AI-generated code and approximately 45 percent of fully passing solutions contain high-severity flaws.Examples include SQL injection, OS command injection, and unsafe deserialization.
- Proposed metrics: reliability@k correctly sets n to independent rollouts and c to fully successful rollouts, reducing scores from 0.96–0.98 to 0.00–0.12 in a synthetic benchmark.The proposal applies the Chen et al. estimator per task-agent pair using full execution success as the rollout criterion.
- Validation: The security-adjusted metric preserved every ranking in an initial three-agent live-API test, with Kendall τ = 1.000 under the current scanner and lenient threshold.A 5-task SWE-bench Verified pilot likewise found strict task resolution of 0.20 versus macro-averaged hidden-test pass rate of 0.80.
2 Background
Background establishes that stochastic decoding makes repeated nonzero-temperature completions vary, while pass@k assumes independent samples and sufficient sample size. Agentic coding differs from cheap completion sampling because each attempt is costly, stateful, and environment-dependent, motivating reliability-focused evaluation.
- Decoding stochasticity: At temperature 0, greedy decoding deterministically selects the highest-probability token at each step.Higher temperatures flatten the distribution and increase stochasticity in generated completions.
- pass@k assumptions: pass@k estimates the probability that at least one of k randomly chosen samples is correct using n generated samples and c correct ones.The estimator was introduced to reduce the high variance of naively generating k samples and dividing the number that pass.
- pass@k assumptions: The estimator requires n independent draws from the model and a sample size reasonably large relative to k.These assumptions hold when n counts independent completions for the same prompt, but fail when n is reinterpreted otherwise.
- Agentic evaluation: Agentic coding attempts are expensive, stateful processes involving tool calls, API tokens, time, and resettable environmental side effects.Unlike stateless completion sampling, agents are generally run once in production, creating the operational motivation for reliability@k.
3 Experimental Setup
The experiments use a reproducible synthetic CodeBench benchmark spanning ten tasks, three agent profiles, and eight independent rollouts per task-agent pair. Sequential experiments test the broken pass@k metric, i.i.d. violations, score inflation, and proxy correlation, with a five-task SWE-bench Verified pilot as preliminary external validation.
- Benchmark construction: CodeBench contains 10 curated coding problems across easy, medium, and hard tiers, covering algorithms, data structures, and lightweight repository scenarios.The tiers contain 3 easy, 3 medium, and 4 hard tasks.
- Benchmark construction: The benchmark evaluates three synthetic profiles—agent-x, claude-code, and codex—rather than production systems.Independent rollouts are generated synthetically for each task-agent pair.
- Benchmark construction: The synthetic benchmark produces 240 ExecutionResult entries from 10 tasks, 3 agents, and 8 rollouts per task-agent pair under seed=42.Rollouts use latent skill sampled from U[0.30, 1.0] and truncated-Gaussian performance variation.
- Experimental sequence: Experiments 0–3 sequentially examine broken pass@k, i.i.d. violations, correct-versus-broken score inflation, and single-rollout proxy correlation.The correlation analysis uses Spearman’s ρ across all 30 task-agent pairs.
- Real-world pilot: A preliminary SWE-bench Verified pilot evaluates the first 5 deterministic test-split instances with one Claude Code CLI attempt per task and the official Docker harness.It is framed as external validation rather than a full benchmark.
4 Results
Results confirm that test-case-based pass@k mismeasures agent reliability by treating unit tests as independent attempts, substantially inflating scores. Corrected multi-rollout evaluation and preliminary real-repository evidence show that trustworthy assessment requires strict task resolution and repeated independent runs, while the security adjustment remains inconclusive.
- Measurement error: 0.0238 points separated Agent-B from Agent-A despite identical 0.40 test pass rates, proving test-suite size alone changes the broken metric.The implementation violates the i.i.d. assumption by treating tests within one submission as independent samples.
- Security-adjusted reliability@k: 1.000, 0.988, and 0.793 were unchanged by security adjustment for codex, agent-x, and claude-code, respectively, preserving codex > agent-x > claude-code.Kendall’s τ = 1.000 (p = 0.333), so H4 was not confirmed; the result was limited by only 3 agents and a lenient ≥0.80 threshold.
- Reliability@k: ρ = 0.417 fell below the pre-specified 0.70 surrogate threshold, showing that single-rollout proxies cannot accurately estimate multi-rollout reliability@5.The proxy was only moderately associated with rollout-level reliability across 30 synthetic task-agent pairs.
- SWE-bench Verified pilot: 0.8049 macro-averaged hidden-test pass rate contrasted with reliability@1 = 0.20, since exactly 1/5 SWE-bench Verified tasks was strictly resolved.All five patches applied cleanly, so failures were hidden-test outcomes rather than infrastructure errors; the pilot used one attempt per task.
5 Limitations
The findings are limited by reliance on synthetic, controlled data rather than production codebases. The experiments demonstrate when reliability metrics diverge under specified assumptions, not real-world performance of any particular commercial agent.
- Data and evaluation scope: The experiments use synthetic, controlled data rather than production codebases.The pass@k and reliability@k distributions were constructed to stress-test estimator behavior under specific assumptions.
- Interpretation: The results are a methodological demonstration of metric divergence, not a production benchmark of any specific commercial agent.
6 Ethical and Societal Considerations
The paper cautions that security-adjusted reliability results on a specific synthetic test suite must not be mistaken for general security certification. It anticipates no direct harms and aims to make security regressions in AI-assisted coding more visible, encouraging rigorous evaluation.
- Limitations: Security-adjusted reliability on CodeBench should not be misread as certifying that an agent generally produces secure code.CodeBench measures relative performance on a specific, synthetic test suite and does not replace standard security auditing, static analysis, or penetration testing.
- Societal impact: The authors foresee no direct risks of harm from this research.
- Societal impact: The intended societal benefit is to make security regressions in AI-assisted coding more visible and measurable.This is intended to encourage more rigorous evaluation practices across the field.
8 Conclusion
The paper introduces CodeBench, a synthetic framework for evaluating AI coding-agent reliability and security. Four experiments examine estimator bias, reliability metrics, proxy divergence, and security-adjusted rankings.
- CodeBench is introduced as a synthetic benchmarking framework for evaluating the reliability and security of AI coding agents.
- Four experiments examine pass@k estimation bias, reliability@k relative to unit-test count, single-rollout proxy divergence, and security-adjusted rankings across three coding agents.
- The results show that ranking inversions from proxy metrics are possible in principle.