PaperlayerView exact revision on arXiv

Source-linked AI summary

Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning

Dong Huang, Mark Harman, Jie M. Zhang, Zhijiang Guo, Mingzhe Du, See Kiong Ng

arXiv:2608.24473v1cs.SEcs.CL

TL;DR

Software testing faces diminishing returns, while prompt-based generation does not directly optimize whether a suite achieves an effectiveness–size trade-off. Ockhamareto combines Pareto-gated suite optimization with token-level segment credit, shifting the mutation–test-count frontier and scaling to larger models.

  • Problem

    Software testing faces diminishing returns, and prompt-based generation optimizes test quality only indirectly rather than the suite's overall effectiveness–size trade-off.

  • Method

    Ockhamareto combines a Pareto-gated conciseness bonus with token-level segment credit in a single-shot GRPO framework.

  • Results

    Ockhamareto-4B strictly Pareto-dominates RL baselines on a held-out benchmark, saturates mutation within three tests, and adds +30–35 pp mutation at 9B and 27B scales.

  • Takeaways & Limitations

    The combined framework shifts the entire (mutation, −#tests) frontier and front-loads bug-catching power into the first test.

  • Takeaways & Limitations

    Prompt-based generation does not train the model directly against whether the suite as a whole provides an effectiveness–size trade-off.

Abstract

from arXiv · show

We introduce \textbf{Ockhamareto}, a single-shot GRPO framework for unit-test generation and selection, based on the principles of \emph{Ockham's Razor} and \emph{Pareto Optimality}. Ockhamareto has two principal components: (i)~a \emph{Pareto-gated Bonus} that rewards only rollouts non-dominated in~(mutation, $-$\#tests) space, and (ii)~\emph{Token-level Segment Credit}, which attributes each test's marginal mutation kills back to the tokens of its unit-test block. On the \emph{UnLeakedTestBench~(ULT)}, Ockhamareto \emph{strictly Pareto-dominates} the strongest RL baseline~(\emph{MIST-RL}). Furthermore, it dominates on {\em each and all} optimization objectives, catching more bugs ($49.9\%$ vs $31.3\%$ mutation score at $N{=}5$), using \emph{fewer} tests ($2.60$ vs $4.67$ on average), thereby achieving $3.4\times$ the per-test trade-off improvement. The advantage is found in all four benchmarks~(\emph{HumanEval+}, \emph{MBPP+}, \emph{CodeContests}, \emph{TestGenEval-Lite}): Ockhamareto leads both mutation and coverage metrics on every one, always with the smallest suite. Ockhamareto also outperforms the state-of-the-art at all model scales, adding $+30$--$35$~pp mutation at 4B, 9B, and 27B model sizes. We also show that the knee point of the optimal trade-off between efficiency and effectiveness on the Pareto front is not correlated with obvious more easily computed proxy metrics, such as function size. This finding motivates the Pareto front computation; it is needed to identify this crucial engineering trade-off for each function under test.

1 Introduction

Unit-test generation must jointly optimize fault-detection effectiveness and suite conciseness, but existing single-shot and multi-turn approaches lack either whole-suite trade-off control or fine-grained credit assignment. Ockhamareto addresses both with a Pareto-gated bonus and token-level segment credit, improving mutation detection while producing smaller suites.

  • Problem: Single-shot generation avoids sequential inference but requires the policy to choose which tests belong in the complete suite.Standard GRPO assigns one scalar advantage to the entire trajectory, obscuring individual test contributions.
  • Problem: A single scalar trajectory reward provides neither an explicit whole-suite conciseness criterion nor fine-grained attribution among tests.The coupled optimization requires distinguishing efficient suites from redundant ones and identifying tests responsible for fault detection.
  • Approach: Ockhamareto introduces a group-relative Pareto gate that awards a conciseness bonus only to non-dominated rollouts in effectiveness–size space.Unlike a fixed test-count penalty, the gate preserves suites that trade off mutation effectiveness and suite size without committing to one universal weight.
  • Approach: Token-level segment credit maps each test’s marginal mutation kills to its own token span, reinforcing high-yield tests and penalizing redundant or failing tests.This supplies single-shot generation with fine-grained supervision that multi-turn methods obtain through sequential interactions.
  • Results: 49.9% mutation score with 2.60 tests at N=5 outperforms MIST-RL’s 31.3% mutation score with 4.67 tests on ULT.The improvement is 18.6 percentage points while using 44% fewer tests; mutation largely saturates within the first three tests.
  • Results: Across HumanEval+, MBPP+, CodeContests, and TestGenEval-Lite, Ockhamareto leads evaluated methods on mutation and coverage while using the smallest suites.At 4B, 9B, and 27B scales, it improves mutation over corresponding untuned models by approximately 30–35 percentage points.

2 Background

Software testing balances fault-detection effectiveness against the resources required to construct, execute, review, and maintain tests. The paper uses Pareto fronts and mutation-based execution feedback to reason about this trade-off and about credit assignment in single-shot generation.

  • Testing trade-offs: The central testing question is when additional fault-detection benefit no longer justifies the cost of more testing.The paper frames this as a diminishing-returns problem rather than a universal stopping criterion.
  • Testing trade-offs: In this work, the number of tests serves as a simple observable proxy for suite cost, although it does not capture every component of testing effort.Other effort dimensions include execution time, monetary cost, maintenance burden, and developer attention.
  • Pareto analysis: A suite is Pareto-optimal when no alternative is at least as effective with no more tests and strictly better on one dimension.The resulting front retains suites where improving effectiveness requires sacrificing suite size, or vice versa.
  • Pareto analysis: Unlike prior test-selection settings that assume an existing collection of tests, this work lets the learning objective influence which tests are generated and how many are generated.This moves conciseness from post-hoc selection into the generation process.
  • Mutation testing: Mutation score measures the fraction of mutants killed by a suite and provides an execution-derived signal for both evaluation and learning.Mutation testing is more computationally costly than structural coverage, but is used here for suite-level quality and test-level segment credit.
  • Credit assignment: Single-shot generation preserves one-call inference but loses the structural per-test credit assignment that multi-turn generation obtains from sequential execution.Ockhamareto treats each test as a segment and uses its marginal fault-detection contribution for credit.

3 Method

Ockhamareto uses single-shot GRPO to generate complete pytest suites, combining suite-level quality and Pareto-gated conciseness with token-level credit for each test’s marginal mutation value. Centered per-test offsets preserve the trajectory-level advantage while rewarding high-yield tests and penalizing redundant ones.

  • Single-shot generation: Each task produces a complete pytest suite in one shot, with GRPO scoring sampled suites in a sandbox.The single-shot design front-loads discriminating tests and keeps inference cost constant regardless of suite size.
  • Suite-level reward: The suite-level quality reward combines reference-test correctness and mutation score, while assigning no weight to coverage.The reward uses Wc=0.2 and Wm=0.81, and failed suites receive zero reward.
  • Pareto-gated conciseness: The Pareto-gated bonus rewards only rollouts that are non-dominated in mutation and negative test-count space.An optional rank term favors fewer tests within the Pareto frontier, while suites trading quality for size receive nothing.
  • Token-level segment credit: Token-level segment credit assigns each test’s marginal mutation kills to the tokens inside that test’s source span.The sandbox reports first-kill counts per test, which are converted into per-segment rewards and mapped through character offsets.
  • Token-level segment credit: Centered segment offsets leave the trajectory-level GRPO advantage unchanged while giving high-yield tests positive and low-yield tests negative credit.In the worked example, a test with 40 first kills receives +0.13 per token, while redundant tests receive negative offsets.
  • Implementation: The implementation locates test blocks by source spans, validates re-tokenization, and falls back to scalar GRPO credit on mismatch.The resulting update can apply centered offsets to independently scorable segments beyond pytest when span localization and per-segment scores are available.

4 Experimental Setup

The evaluation trains Qwen3.5 models with single-shot GRPO on leakage-controlled test-generation data and evaluates them across five held-out benchmarks. Mutation scoring uses fixed per-task fault pools or real incorrect submissions, with common prompts and per-test attribution.

  • Data and decontamination: Training uses approximately 10,500 examples from PLT, with strict decontamination and no overlap with any evaluation benchmark.Functions appearing verbatim in common LLM training corpora are excluded from training.
  • Benchmarks: Evaluation spans ULT, HumanEval+, MBPP+, CodeContests, and TestGenEval-Lite across four distinct code distributions.ULT contains 2,126 leakage-free Python tasks, while TestGenEval-Lite requires module-level context and cross-function reasoning.
  • Fault sources: Cosmic-ray mutant pools provide fault sources for four benchmarks, whereas CodeContests uses incorrect human submissions instead of synthetic mutants.The latter measures rejection of real buggy implementations and complements syntactic mutation scoring.
  • Scoring protocol: All methods use the same single-shot prompt and per-test scoring harness, with TestGenEval-Lite restricted to tasks where every method produces a passing test.This shared usable subset is used to ensure fair comparison under varied execution environments.
  • Models and training: Experiments use Qwen3.5 at 4B, 9B, and 27B parameters with LoRA fine-tuning and greedy decoding during evaluation.LoRA reduces trainable parameters by approximately 99% relative to full fine-tuning.
  • Reinforcement learning setup: Each training step samples K=8 suites for each of 32 tasks, scores them in parallel sandboxes, computes GRPO and segment advantages, and applies a KL-constrained update.The reference policy is frozen at initialization and the KL coefficient is 0.1.
  • Baselines: Baselines include the untuned model, suite-level +GRPO, and MIST-RL, a multi-turn mutation-aware method that penalizes redundancy.MIST-RL is reimplemented on the same Qwen3.5-4B base and training data for comparison.
  • Metrics and questions: The study measures mutation score, statement and branch coverage, suite size, and per-test efficiency, with RQ2 testing the two core components.The evaluation also uses first-N aggregation to expose redundancy hidden by suite-level metrics.

5 Results

Ockhamareto improves fault detection and conciseness together: it Pareto-dominates RL baselines on ULT and retains the same qualitative advantage across all four out-of-distribution benchmarks. Its gains are concentrated in early tests, and its suites remain smallest across the evaluated settings.

  • ULT results: 49.9% mutation with 2.60 tests lets Ockhamareto strictly Pareto-dominate both RL baselines on ULT.It improves mutation by 18.6 percentage points and uses 44% fewer tests than MIST-RL.
  • ULT results: 19.2% per-test efficiency is 3.4× the base model and 2.9× MIST-RL.The comparison is reported at N=5 on ULT.
  • Cross-benchmark generalization: Across HumanEval+, MBPP+, CodeContests, and TestGenEval-Lite, Ockhamareto leads mutation and coverage while using the smallest suites.The same ranking holds across all five held-out benchmarks when ULT is included.
  • Suite compression: On TestGenEval-Lite, Ockhamareto has a median suite size of 3 tests versus 13 for MIST-RL while still leading every metric under N=5.The benchmark uses a shared usable subset requiring at least one passing test from every method.
  • Cross-benchmark generalization: 44.6% fault detection with 3.04 tests on CodeContests exceeds MIST-RL’s 32.4% with 4.90 tests.This is a 12.2 percentage-point improvement using 38% fewer tests on real buggy submissions.
  • Budget behavior: Ockhamareto saturates by N=3, capturing 99% of its N=5 mutation in three tests.Its first test reaches 33.3% mutation, exceeding MIST-RL’s fifth test at 31.3%.
  • Suite compression: Ockhamareto emits 3.02 tests per task, compared with 12–16 for the base and RL baselines.Its median is 3 and p90 is 3, while baseline first-5 metrics discard approximately 60% of generated tests.

5.2 RQ2: What is the contribution of each component?

Ablations show that Pareto gating and token-level segment credit are complementary: the former controls whole-suite conciseness, while the latter improves the value assigned to individual tests. The same reward design remains effective from 4B through 27B models.

  • Component ablation: Removing the conciseness bonus lowers mutation from 49.9% to 40.0% and per-test efficiency from 19.2% to 12.5%.Average suite size rises from 2.60 to 3.21 tests because Pareto pressure is removed.
  • Component ablation: Removing segment credit lowers mutation from 49.9% to 41.2% and increases suite size from 2.60 to 3.07 tests.Per-test efficiency falls to 13.4%, and the policy recovers some mutation by generating more tests.
  • Component ablation: Both mechanisms are necessary and complementary: Pareto gating controls suite conciseness, while segment credit improves individual-test fault-detection value.Removing either component substantially reduces mutation efficiency.
  • Model scale: +30–35 pp mutation lifts persist at 4B, 9B, and 27B with the same fixed Ockhamareto configuration.The reported lifts are +35.4 at 4B, +34.5 at 9B, and +30.5 at 27B.
  • Model scale: Scaling the untuned base from 4B to 27B raises mutation from 14.5% to 30.5%, while scaling inside Ockhamareto adds 11.1 points.The paper reports reward design as the larger gain in these experiments.
  • Model scale: Ockhamareto-4B reaches 49.9% mutation versus base-27B’s 30.5%, using approximately seven times fewer parameters and 28% fewer tests.This comparison illustrates that the framework’s benefit persists across model scales.

5.4 RQ4: How sensitive is Ockhamareto to its hyper-parameters?

RQ4 finds that Ockhamareto’s default weights provide the strongest mutation performance, while moderate changes preserve benefits and stronger pressure favors smaller suites at some mutation cost.

  • Segment-credit weight: 41.2% mutation at n=3.07 results when segment credit is removed, compared with the default’s stronger performance and smaller suites.Lower segment-credit weights also reduce mutation, while over-weighting segment credit produces smaller suites but lower mutation.
  • Conciseness-bonus weight: 40.0% mutation at n=3.21 occurs when the conciseness bonus is disabled, producing both lower mutation and larger suites.The result indicates that generating more tests does not recover the quality lost without explicit whole-suite pressure.
  • Conciseness-bonus weight: 49.9%→47.2% mutation accompanies suite compression from n=2.60→2.31 when WP=WN=0.30 replaces the default.The stronger conciseness setting raises per-test efficiency to 20.4%, but the default WP=WN=0.15 remains the raw mutation peak.
  • RQ4 answer: The default weights maximize mutation among tested settings, whereas stronger segment credit or conciseness pressure shifts the balance toward smaller suites and lower mutation.Moderate changes preserve the overall benefit.

5.5 RQ5: How Many Tests Should You Maintain? The Empirical Pareto Front

RQ5 shows that appropriate suite size is function-specific: empirical Pareto fronts expose a small set of effectiveness–size choices, with a median knee at three tests but knees ranging from one to fourteen.

  • Per-function knees: 3 tests is the median and modal knee, but observed knees range from 1 to 14 across functions.The typical function repays about three tests, while individual functions vary by an order of magnitude.
  • Engineering use: Empirical front extraction exposes a handful of defensible operating points for function-specific suite sizing, and Ockhamareto supplies most pooled-front solutions.The front’s median size is two non-dominated points rather than a continuous range.
  • Empirical front: 97% of sampled suites are dominated, making front extraction important for avoiding almost surely suboptimal suite choices.The study samples approximately 165 valid suites per task and finds a median dominated-to-front ratio of 47:1.
  • Policy comparison: Ockhamareto tracks the pooled front within 3–5 pp and supplies 60.8% of pooled-front points, while MIST-RL trails by 20 pp.Under generous first-k prefix competition, Ockhamareto still contributes 57% of pooled-front points.
  • Predictability: Knee position is not reliably predicted by lines of code or cyclomatic complexity, with Kendall’s τ_b between -0.02 and 0.08 and all p>0.3.The only significant association is internal to the front: larger fronts knee later, with τ_b=0.39 and p<0.001.

6 Discussion

The discussion attributes Ockhamareto’s behavior to complementary reward components: Pareto pressure shapes suite selection, while segment credit concentrates learning on valuable tests. Its practical advantage is largest at small budgets.

  • Pareto-gate effect: Removing the conciseness bonus changes mutation from 49.9% to 40.0% and average suite size from 2.60 to 3.21 tests.The policy does not recover the lost quality simply by generating more tests.
  • Segment-credit effect: Removing segment credit lowers mutation to 41.2% and increases suite size to 3.07 tests, while the first test alone reaches 33.3% mutation.The evidence supports fine-grained credit concentrating fault-detection value in earlier, higher-yield tests.
  • Complementary roles: Pareto pressure shapes which suites are worth generating, while segment credit shapes which tests within those suites carry the learning signal.These components therefore have complementary roles in the training signal.
  • Deployment implications: At maintainable test budgets, Ockhamareto delivers more fault detection than alternatives, with the largest gap at small budgets dominating CI and review cost.Most suite value survives aggressive truncation because suites saturate by three tests.

7 Threats to Validity

The paper identifies threats involving reward-signal alignment, execution determinism, benchmark scope, model-family coverage, and the validity of mutation score as a proxy for real faults.

  • Signal alignment: Segment credit succeeds for 67.8% of rollouts; approximately 32% fall back to scalar GRPO because tokenizer offset mappings fail.This partial application could dilute the segment-credit signal, although ablations indicate a substantial contribution remains.
  • External validity: Results may not generalize beyond Python unit testing because syntax, testing frameworks, mutation operators, and cross-module challenges differ elsewhere.The evaluation covers Python-only benchmarks and does not directly establish transfer to other languages or integration and system testing.
  • Model coverage: A single Qwen3.5 model family underlies the three tested scales, limiting evidence across architectures and pre-training distributions.The cited scope boundary concerns observed benefits of Pareto-gated conciseness and segment credit across model families.
  • Construct validity: Mutation score is reproducible and validated but imperfect because equivalent or trivial mutants can distort its relationship to real production bugs.It does not directly measure a suite’s ability to catch real bugs in production.
  • Credit attribution: First-kill attribution gives full credit to the first test killing a mutant even when later tests would also detect it, biasing the learning signal.This is a limitation of the per-test credit target rather than of mutation scoring itself.

8 Related Work

Prior test-generation methods often optimize coverage or compilation and may produce redundant suites, while Ockhamareto explicitly optimizes mutation quality and test-count cost in a single-shot framework.

  • Classical and LLM-based test generation: Earlier automated test-generation approaches include search-based, random, prompt-based, usage-example mining, and hybrid methods.These approaches include EvoSuite and Pynguin, path-coverage prompting, and search hybrids.
  • Classical and LLM-based test generation: Coverage-focused tools can produce large, redundant, or smell-laden suites, with minimization handled post hoc when used.Ockhamareto instead makes conciseness a first-class training objective.
  • Fine-grained credit assignment: Ockhamareto's segment credit uses exact execution-derived marginal mutation kills mapped to test tokens, leveraging independently scorable test functions.The mechanism is presented as a domain-specific analogue of process rewards.
  • Ockhamareto's positioning: Ockhamareto treats mutation score as the quality target and test count as an explicit cost, optimizing their trade-off rather than coverage alone.This reframes unit-test generation around effectiveness and suite size.
  • Reinforcement learning for test generation: TestDecision and MIST-RL obtain per-test credit through multi-turn generation, requiring sequential calls and optimizing each addition's value.Ockhamareto combines single-shot generation with whole-suite Pareto scoring and intra-trajectory token-level credit.

9 Conclusion

The conclusion reports that Ockhamareto combines Pareto-gated conciseness with token-level segment credit to improve the effectiveness–size frontier, including at larger model scales.

  • Conclusion: Ockhamareto combines a Pareto-gated conciseness bonus with token-level segment credit in a single-shot GRPO framework.The conclusion frames these components as shifting the mutation–test-count frontier.
  • Conclusion: Ockhamareto-4B strictly Pareto-dominates both RL baselines on a held-out benchmark, catching more bugs with fewer tests.Its mutation score saturates within three tests and front-loads bug-catching power into the first test.
  • Conclusion: +30–35 pp mutation is added at every reported 9B and 27B scale, while Ockhamareto-4B outperforms base-27B.The conclusion states that the framework scales cleanly across these model sizes.
  • Conclusion: The conclusion characterizes principled reward shaping as a stronger lever than raw model scale for this task, while the two compose.This claim is stated within the paper's unit-test-generation setting.

10 Generative AI Disclosure

The authors disclose using Claude and GPT-5.6 Sol for code development, proofreading, text clarity, and literature identification, with human-author review and responsibility.

  • Generative AI Disclosure: Claude assisted with code development, proofreading, improving author-written text, and identifying potentially relevant literature.The disclosure states that the assisted content was reviewed by at least one human author.
  • Generative AI Disclosure: GPT-5.6 Sol assisted with proofreading, improving author-written text, and identifying potentially relevant literature.The disclosure includes these uses alongside Claude's assistance.
  • Generative AI Disclosure: The authors disclose these uses for transparency and to future-proof the scholarly record against possible policy and standards changes.They note that ACM policy did not require disclosure for editing and refining author-written text at the time of writing.
Loading 2608.24473v1…