PaperlayerView exact revision on arXiv

Source-linked AI summary

ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation

Hui Sun, Yun-Ji Zhang, Zheng Xie, Ren-Biao Liu, Yali Du, Xin-Ye Li, Ming Li

arXiv:2604.03922v2cs.LG

TL;DR

Selecting code with generated tests is difficult because test correctness is unknown and entangled with code correctness. ACES uses leave-one-out AUC to weight tests by ranking consistency, with ACES-C offering closed-form correction and ACES-O optimizing the objective iteratively. Across multiple code-generation benchmarks, ACES achieves state-of-the-art Pass@k, while the two variants address different quality-assumption regimes.

  • Problem

    Generated tests may be incorrect, while assessing test correctness requires knowing which generated codes are correct, creating a circular dependency.

  • Method

    ACES evaluates each test by leave-one-out AUC consistency with rankings induced by the remaining tests, then derives weights through closed-form correction or differentiable optimization.

  • Results

    ACES achieves state-of-the-art Pass@k on multiple code-generation benchmarks, with ACES-O leading standalone and ACES-C excelling as a plug-in execution-only scorer.

  • Takeaways & Limitations

    Test votes can be used to rank candidates by discriminative power without externally determining test correctness, using only the binary pass matrix.

  • Takeaways & Limitations

    The oracle-optimal test weights require unknown discriminative powers in practice, and ACES-C’s guarantees rely on an average-quality assumption that ACES-O does not require.

Abstract

from arXiv · show

Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a \emph{circular dependency}. Our key insight is that we need not determine test correctness at all: \emph{test votes should rank, not merely count}. What matters is not how many codes pass a test, but whether the test can \emph{distinguish} correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose \textbf{ACES}~(\textbf{A}UC \textbf{C}onsist\textbf{E}ncy \textbf{S}coring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@$k$ on multiple code generation benchmarks.

1 INTRODUCTION

ACES reframes test-based code selection as ranking: tests should be weighted by their ability to distinguish correct from incorrect candidates rather than counted equally. Leave-one-out AUC breaks the code–test quality circularity, supporting two lightweight weighting algorithms that improve Pass@k across benchmarks.

  • Motivation: Test-based code selection is circular because code and generated-test correctness are both unknown.Existing approaches either count tests uniformly, use heuristics, or require substantially more computation.
  • Core insight: A test’s value is its ability to distinguish correct from incorrect code, not merely the number of candidates it accepts.Easy tests dilute ranking signal, while tests favoring incorrect candidates can corrupt it.
  • Core insight: Leave-one-out evaluation ranks candidates using the remaining tests and checks whether each held-out test agrees with that ranking without knowing code correctness.The procedure uses only the internal structure of the binary pass matrix.
  • Theory: LOO-AUC formalizes this agreement, and its expected value is proportional to each test’s latent discriminative power.The proportionality coefficient depends on pass-rate variance and the ranking quality of the remaining tests.
  • Algorithms: ACES-C uses closed-form weights under an average-quality assumption, while ACES-O iteratively optimizes a differentiable LOO-AUC objective without that assumption.Both methods operate only on the binary pass matrix with negligible overhead.
  • Empirical results: ACES advances the state of the art in Pass@k across multiple benchmarks, with ACES-O strongest standalone and ACES-C effective as an execution-only plug-in scorer.ACES-C is especially effective when combined with other pre-filtering mechanisms.

2 THEORETICAL FOUNDATIONS

The paper formulates code selection as weighted voting over a binary pass matrix, where test value is determined by discriminative power rather than correctness alone. It derives a Pass@k-relevant signal-to-noise framework and motivates leave-one-out AUC as a correctness-free proxy for that latent power.

  • Problem setup: Code candidates are ranked by weighted sums of their binary test outcomes, with majority voting as the uniform-weight baseline.The pass matrix records every candidate–test execution, and ties are broken uniformly at random.
  • Problem setup: Each test’s pairwise vote is the pass difference between a correct and incorrect code, so ranking becomes a weighted voting problem.The aggregate score difference determines which code is placed higher.
  • Discriminative power: Discriminative power δj = αj −βj measures whether correct codes are more likely to pass a test than incorrect codes.Tests are informative when δj > 0, uninformative when δj = 0, and misleading when δj < 0.
  • Discriminative power: For ranking, incorrect tests can still provide useful signal when they distinguish correct from incorrect code, so correctness-based filtering may discard constructive tests.Constant test columns provide no ranking signal and are removed in preprocessing.
  • Pass@k bound: Pass@k is bounded through a signal-to-noise ratio that is maximized by oracle weights based on unknown discriminative powers.Non-uniform weighting can improve over majority voting, but the optimal weights cannot be computed directly because δj is unknown.
  • Leave-one-out evaluation: Leave-one-out AUC estimates a test’s informativeness by checking whether its pass/fail pattern agrees with rankings induced by the remaining tests.The procedure uses only the pass matrix: values above 1/2 indicate agreement, while values below 1/2 indicate disagreement.

3 ACES: AUC CONSISTENCY SCORING

ACES uses the LOO-AUC identity to assign non-uniform test weights from the binary pass matrix. ACES-C provides a corrected closed-form estimator under an average-quality assumption, while ACES-O optimizes the weighting objective when that assumption may fail.

  • Theoretical motivation: The LOO-AUC identity links observable test-ranking consistency to latent discriminative power through a test-dependent coefficient.The coefficient depends on pass-rate variance and the quality of the leave-one-out ranking.
  • Theoretical motivation: Under Assumption 4, LOO-AUC scores above 1/2 identify informative tests in expectation, while scores below 1/2 identify misleading tests.The assumption requires above-random ranking quality on average rather than for every test and permits misleading tests.
  • ACES-C: closed-form weighting: ACES-C corrects LOO-AUC excess by pass-rate variance and clips non-positive values to produce non-negative closed-form test weights.The correction removes the coefficient’s pass-rate distortion and assigns zero weight to tests scoring at or below 1/2.
  • ACES-C: closed-form weighting: E[qj] > 0 if and only if δj > 0, so ACES-C recovers the informative-versus-misleading sign and satisfies E[qj] ∝δj.This result holds under Assumption 4 and targets oracle weights proportional to max(0, δj).
  • ACES-C: closed-form weighting: ACES-C achieves near-oracle signal-to-noise ratio, with R ≥ρ^2R* and ρ →1 as the number of tests grows.The guarantee is stated as a corollary under the paper’s assumptions.
  • ACES-O: optimized weighting: ACES-O jointly optimizes test weights and their induced rankings through a differentiable LOO-AUC objective when the average-quality assumption may fail.It softly down-weights tests below 1/2 and can later recover them as the leave-one-out ranking improves.

4 EXPERIMENTS

The experiments evaluate ACES on three code-generation benchmarks, showing strong execution-only reranking and complementary gains when combined with static analysis. Analyses examine assumption satisfaction, robustness to misleading tests, and test-quality detection.

  • Setup: ACES is evaluated on HumanEval, HumanEval+, and MBPP using approximately 200 candidates and 500 generated tests per problem, with Pass@k reported for k ∈{1, 2, 5}.The study compares post-hoc reranking methods using code×test execution with methods using additional information.
  • Main results: ACES achieves the best Pass@k among execution-based methods on all three benchmarks and the best overall results when combined with complementary static-analysis methods.The combined result uses only the binary pass matrix for ACES and additional static-analysis signals from the complementary method.
  • Main results: 84.15% Pass@1 on HumanEval and 74.39% on HumanEval+ make ACES-O the leading execution-only method on those benchmarks, while MBPP reaches 72.37% and trails DS3.On HumanEval and HumanEval+, ACES-O also exceeds DS3 despite using only the pass matrix; on MBPP, DS3 reaches 75.88%.
  • Main results: ACES-C + DS3 improves over DS3 alone on all three benchmarks, while ACES-O + DS3 has its largest gain on MBPP at +0.70 Pass@1.ACES-C + DS3 is stronger than ACES-O + DS3 on HumanEval and HumanEval+, including 85.37% versus 83.54% Pass@1 on HumanEval.
  • Variant comparison: ACES-O outperforms ACES-C across all reported benchmark–k settings, whereas ACES-C improves over Majority Voting on all three benchmarks.For example, ACES-C reaches 82.93% versus 80.49% Pass@1 on HumanEval.
  • Analysis: In the Middle region, ACES-O passes 52/89 tasks versus 46/89 for ACES-C and 35/89 for Majority Voting, while Easy tasks already have near-perfect pass rates.When the average-quality assumption fails, ACES-O substantially outperforms ACES-C at 16/52 versus 9/52 tasks.
  • Analysis: For the most misleading tests, removal reduces Pass@1 by 0.030 for ACES-O versus 0.056 for Majority Voting; informative tests contribute 0.030 versus 0.012 at δj ≈0.3.The asymmetric effects are consistent with ACES down-weighting misleading tests and up-weighting informative ones.
  • Analysis: ACES-C weight signs identify at least 94.8% of informative tests, with errors concentrated near δj ≈0 where discriminative power is weak.MBPP has a higher false-positive fraction than HumanEval, 17.3% versus 9.5%, largely among borderline tests.

5 RELATED WORK

Related work frames code selection as weighted bipartite ranking under noisy evaluation. ACES differs by identifying reliable test signals from the binary pass matrix without external supervision or additional training.

  • Code selection with generated tests: CodeT scores consensus sets by passed-test counts, whereas MBR-exec and SRank require pairwise output comparison beyond the binary pass matrix.Other approaches co-evolve code and tests through reinforcement learning or evolutionary search, requiring more computation.
  • Ranking and noisy evaluation: ACES treats each test as a noisy annotator and uses LOO-AUC to identify reliable comparators without external supervision.The framework fits multi-label AUC optimization and differs from RankBoost by requiring an average rather than per-ranker quality condition.
  • Ranking and noisy evaluation: Unlike verifier and LLM-as-judge paradigms, ACES requires no additional training.Its weighting mechanism operates on execution-derived test outcomes rather than a separately trained or prompted evaluator.

6 CONCLUSION

ACES replaces vote counting with leave-one-out AUC consistency to assess test informativeness without external supervision. Its two variants provide complementary weighting strategies, with strong execution-only and combined benchmark performance.

  • ACES uses leave-one-out AUC to assess test consistency and break the circular dependency between code and test quality.The framework operates without external supervision.
  • ACES-C provides closed-form weights approximating the oracle in expectation, while ACES-O optimizes a differentiable LOO-AUC objective without the average-quality assumption.
  • Both variants use only the binary pass matrix with negligible overhead and achieve state-of-the-art Pass@k among execution-only methods.
  • Combining ACES with complementary static-analysis signals yields the best overall results across all benchmarks.
  • Future work includes modeling correlations among generated tests and applying internal-consistency evaluation to other noisy-evaluator settings.

A.1 PROOF PRELIMINARIES

The proofs model binary test outcomes through class-conditional pass rates and analyze weighted code rankings using pairwise comparisons. They establish the LOO-AUC identity and derive conditions under which ACES-C weights approximate oracle weighting.

  • The proof framework distinguishes correct and incorrect code sets using class counts, correctness fraction, and marginal test pass rates.It defines α_j and β_j as class-conditional pass rates and p_j as the marginal pass rate.
  • Under the model, codes are independently sampled, and test outcomes are conditionally independent across tests given correctness labels.
  • Pass@k is reduced to pairwise ranking errors and bounded with Hoeffding’s inequality over weighted per-test votes.
  • LOO-AUC compares leave-one-out scores for codes passing and failing a held-out test, with ties counted as 1/2.
  • Under the average-quality assumption, population ACES-C weights are proportional to positive discriminative power and approach oracle performance as the number of tests grows.
  • ACES-O uses a smooth logistic surrogate with softmax weights, but its objective is non-convex and gradient ascent guarantees only convergence to a stationary point.

B.3 IMPLEMENTATION DETAILS

The implementation combines pass-matrix-based ACES scoring with optional DS3 static and behavioral signals. ACES-O uses shortlist pre-filtering and iterative optimization, while the combined formulation exposes controls for blending signals.

  • ACES-O pre-filters candidates by majority vote, reducing per-step cost from O(nm^2) to O(Km^2), whereas ACES-C processes all candidates directly.
  • DS3 filters malformed code and invalid tests before combining static code quality with pairwise behavioral consensus.
  • The combined formulation blends ACES candidate scores, static quality, and DS3-style consensus through coefficients α and β.β controls the blend between ACES scores and static quality, while α controls independent versus consensus-weighted aggregation.
  • Setting α = β = 0 recovers the original DS3 score P · q.
  • The supplementary experiments cover additional benchmarks, Pass@k curves, assumption checks, sensitivity, convergence, runtime, vote statistics, and prompts.

C.1 ILLUSTRATIVE EXAMPLE

Controlled Easy and Hard instances illustrate why ACES-C and ACES-O are complementary. ACES-C is sufficient when informative tests dominate, while ACES-O recovers from strong misleading tests through iterative refinement.

  • C.1 ILLUSTRATIVE EXAMPLE: The Easy instance contains eight informative and two misleading tests, while the Hard instance contains six informative and four strongly misleading tests.Both instances use an 8 × 10 pass matrix with three correct and five incorrect codes.
  • Easy case: In the Easy case, ACES-C concentrates weight on discriminative tests and breaks the majority-vote tie affecting the lowest correct code.
  • Hard case: In the Hard case, only 2 of 6 informative tests initially exceed LOO-AUC 1/2, limiting ACES-C’s correction under misleading uniform rankings.
  • Easy case: Easy-case AUC is 0.90 for MV and 1.00 for both ACES-C and ACES-O.
  • Hard case: Hard-case AUC is 0.60 for MV, 0.77 for ACES-C, and 1.00 for ACES-O.
  • Hard case: ACES-O’s iterative co-evolution recovers all six informative tests and reaches AUC 1.00 in the Hard case.

C.2 ADDITIONAL GENERATION MODELS AND BENCHMARKS

Across additional models and harder benchmarks, ACES improves candidate ranking over Majority Voting, with complementary strengths between ACES-C and ACES-O. The gains are largest at Pass@1 and on non-trivial tasks, while ACES-C is especially robust on LeetCodeDataset.

  • At least one ACES variant improves Pass@1 over Majority Voting on every model–benchmark pair.
  • On HumanEval, the best ACES variant gains +1.2 to +2.4 points over MV across the three models.
  • On HumanEval+, the best ACES variant gains +0.6 to +2.4 points over MV across the three models.
  • ACES-O leads ACES-C on HumanEval and HumanEval+ for both Qwen models, whereas ACES-C leads for DeepSeek-Coder-V2-16B and across LeetCodeDataset.
  • ACES-C improves over Majority Voting on LeetCodeDataset by +3.5 to +7.9 Pass@1 points across the three models.
  • On non-trivial LeetCodeDataset tasks, ACES-C improves over MV by +9.4, +11.0, and +23.4 points at Pass@1 for the three models, respectively.
  • With DS3 pre-filtering, ACES-C leads at small k on HumanEval and HumanEval+, while the variants converge at larger k and perform comparably on MBPP.

C.4 ASSUMPTION SATISFACTION ANALYSIS

The assumption analysis finds that Assumption 4 holds for most non-trivial tasks, while ACES variants specialize across difficulty regimes. ACES-C’s weighting provides larger gains than filtering alone, and ACES-O is more effective when average test quality is low.

  • Assumption 4 is satisfied for the majority of non-trivial tasks on all benchmarks, while MBPP contains a higher proportion of hard tasks.
  • ACES provides the most value in the Middle difficulty region, where informative and misleading tests coexist.
  • When average test quality is low, ACES-O more effectively identifies and up-weights the few reliable tests than ACES-C.
  • When average test quality is high, ACES-C’s closed-form solution matches or approaches ACES-O without iterative optimization.
  • As k increases, the performance gap between methods narrows consistently across benchmarks because top-ranking quality becomes less sensitive.
  • ACES-C Filter improves over Majority Voting by +0.61 on HumanEval, +0.00 on HumanEval+, and +0.70 on MBPP.
  • Non-uniform weighting adds +1.83 on HumanEval, +1.83 on HumanEval+, and +1.87 on MBPP beyond ACES-C Filter.

C.6 SENSITIVITY TO NUMBER OF TESTS

ACES scales better than Majority Voting as more tests become available. Majority Voting plateaus around 50–100 tests, whereas both ACES variants continue improving and can outperform the full-test baseline with only 100 tests.

  • MV plateaus around m′ = 50–100, while both ACES variants continue improving as the test pool grows.
  • At m′ = 100, ACES-C already outperforms MV using the full m′ = 500 test budget on all benchmarks.
  • More tests improve LOO-AUC estimates, enabling ACES to differentiate informative from misleading tests more finely.
  • The gap between ACES-O and ACES-C generally widens with m′, as optimization exploits richer test signals.
  • A modest test budget of approximately 100 tests suffices for LOO-AUC weighting to capture most of the available signal.

C.7 SENSITIVITY TO NUMBER OF CANDIDATE CODES

ACES-C is robust with limited candidate data, whereas ACES-O benefits most from larger candidate pools and focused pre-filtering. ACES-O converges quickly and remains stable across broad hyperparameter ranges, while all reranking methods stay below one second per task.

  • All methods remain stable as candidate count changes, with standard deviations ≤1.6% throughout.
  • MV is approximately flat across n′ and decreases on MBPP from 70.8% at n′ = 20 to 68.6% at n′ = 200.
  • ACES-C maintains performance close to its full-data result even at n′ = 20, while ACES-O improves steadily and leads at n′ = 200.
  • ACES-C is robust to limited tests and candidates, whereas ACES-O performs best when both m′ and n′ are sufficiently large.
  • ACES-C improves as the pre-filtering cutoff K increases, reaching its best performance at K = 200 on all benchmarks.
  • ACES-O performs best at small to moderate K values of 8–32 and declines gradually for larger candidate pools.
  • ACES-O’s surrogate objective nearly converges by iteration 80–100, after which Pass@1 fluctuates by at most 0.6 percentage points through iteration 300.
  • ACES-C takes 9 ms per task versus 4 ms for MV, while ACES-O takes approximately 0.85 s; all methods finish within one second.
Loading 2604.03922v2…