PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning

Aozhe Wang, Yuchen Yan, Nan Zhou, Zhengxi Lu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2603.15611v1cs.CL

TL;DR

Code generation reinforcement learning needs test rewards that are broad, valid, and adaptive, but static suites are limited and self-play trades targeted testing against self-collusion. Code-A1 separates Code and Test LLMs with opposing objectives, adds validity-aware rewards and Mistake Book replay, and matches or exceeds human-test-trained code generation while improving bug-revealing test generation.

  • Problem

    Static test suites have limited coverage and cannot adapt to evolving code models, while self-play either misses implementation-specific bugs or enables self-collusion.

  • Method

    Code-A1 jointly optimizes separate Code and Test LLMs with opposing objectives, using validity-aware rewards and Mistake Book experience replay.

  • Results

    Code-A1 matches or exceeds reinforcement learning with human-annotated golden tests on code generation and produces high-quality, bug-revealing tests.

  • Takeaways & Limitations

    Adversarial co-evolution provides a scalable path beyond annotation-dependent training while enabling targeted white-box testing without self-collusion.

  • Takeaways & Limitations

    Code-A1 requires ground-truth solutions to validate generated tests during training, limiting use where reference implementations are unavailable.

Abstract

from arXiv · show

Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates. Yet high-quality test suites are scarce, existing datasets offer limited coverage, and static rewards fail to adapt as models improve. Recent self-play methods unify code and test generation in a single model, but face a inherent dilemma: white-box access leads to self-collusion where the model produces trivial tests for easy rewards, yet black-box restriction yields generic tests that miss implementation-specific bugs. We introduce Code-A1, an adversarial co-evolution framework that jointly optimizes a Code LLM and a Test LLM with opposing objectives. The Code LLM is rewarded for passing more tests, while the Test LLM is rewarded for exposing more defects. This architectural separation eliminates self-collusion risks and safely enables white-box test generation, where the Test LLM can inspect candidate code to craft targeted adversarial tests. We further introduce a Mistake Book mechanism for experience replay and a composite reward balancing test validity with adversarial difficulty. Experiments on Qwen2.5-Coder models demonstrate that Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests, while significantly improving test generation capability.

1 Introduction

Code-A1 addresses scarce, static, and misaligned test rewards by adversarially co-evolving separate Code and Test LLMs. Experiments show strong code-generation performance and improved bug-revealing test generation.

  • Motivation: Static golden tests are scarce, narrow, and unable to adapt reliably as code models improve.Benchmarks typically provide only three to five tests per question, allowing coincidental solutions to pass and making reward signals overly lenient or stringent.
  • Motivation: Self-play faces a trade-off: black-box tests miss implementation-specific bugs, whereas white-box access enables self-collusion and trivial tests.The unified model can exploit weak testing because passing-code rewards offset penalties for poor tests.
  • Approach: Code-A1 separates code and test generation into opposing policies that co-evolve through dynamically adapting rewards.The Code LLM is rewarded for passing tests, while the Test LLM is rewarded for exposing defects in candidate code.
  • Approach: The framework combines validity-aware test rewards with a Mistake Book that replays historical failures and stabilizes training.The composite reward balances executable tests against adversarial difficulty, while replay preserves resolved bugs and provides stable reward baselines.

2 Related Works

Related work uses reinforcement learning and automated test synthesis to improve code generation through executable feedback. Existing approaches address sparse rewards, test quality, scalability, and inference-time self-verification.

  • Reinforcement Learning for Code: Code generation suits reinforcement learning because compiler feedback and unit-test outcomes provide executable reward signals.This distinguishes code generation from open-ended text generation, where direct functional verification is unavailable.
  • Reinforcement Learning for Code: RLTF supplies granular compilation, runtime, and logic-error feedback to mitigate sparse binary rewards in code tasks.The framework uses multiple error levels to guide online exploration more precisely.
  • Automated Test Generation: LLMs increasingly synthesize unit tests to reduce the cost and scalability constraints of manual test creation.Common pipelines generate tests from question descriptions and filter hallucinated tests using proxy solutions.
  • Automated Test Generation: Generated tests support both reinforcement-learning rewards and inference-time self-verification such as Best-of-N selection.Candidates can be ranked by how many self-generated tests they pass, supporting iterative debugging workflows.

3 Methods

Code-A1 trains separate Code and Test LLMs through adversarial rollouts, validated white-box tests, replayed historical failures, and opposing rewards. Its design stabilizes co-evolution by preserving challenging failures while balancing test validity against adversarial difficulty.

  • Adversarial Co-Evolution: Code-A1 jointly optimizes separate Code and Test LLMs with opposing objectives during adversarial rollouts.The Code LLM generates candidate solutions, while the Test LLM generates tests conditioned on each candidate solution.
  • Adversarial Co-Evolution: The Test LLM uses white-box access to candidate code to craft targeted tests that probe implementation weaknesses.For each candidate solution, the Test LLM generates multiple test suites conditioned on both the question and candidate code.
  • Rollout and Validation: Generated tests are validated against the ground-truth solution, retaining correct or repairable tests and discarding other invalid tests before code evaluation.Validation checks execution, uniqueness, and predicted-answer correctness; incorrect predictions may be replaced with ground-truth return values.
  • Experience Replay: The Mistake Book stores historically failed tests, removes tests that become solved, and supplies a frontier of failures for stable, curriculum-aware training.Historical tests reduce reward variance, expose progress through pass-rate gaps, and help prevent previously fixed bugs from returning.
  • Reward Design: Code rewards combine historical and newly generated test pass rates, while Test rewards combine validity with adversarial difficulty.The Test reward penalizes tests easier than historical failures, and α controls the validity–adversariality trade-off; extreme settings encourage trivial or invalid tests.
  • Policy Optimization: GRPO with token-level loss aggregation is applied to both models, and high-reward-variance test groups are selected to balance compute and learning value.Asymmetric sampling generates M solutions and M × N test suites, then selects the top-ℓtest groups with the highest reward variance, with ℓ × N = M.

4 Experiments

Code-A1 is evaluated across model scales and baselines for code and test generation, with ablations probing reward balance, answer prediction, and the Mistake Book.

  • Experimental Setup: The rollout procedure samples candidate code, retrieves historical failures, validates generated tests, executes them, and updates policies through GRPO.The Mistake Book is updated by adding newly failed tests and removing tests newly passed by candidate solutions.
  • Experimental Setup: Experiments use Qwen2.5-Coder models at 1.5B, 3B, and 7B scales, with code and test generation evaluated against multiple baselines.The setup trains on 9,688 KodCode-V1 questions and evaluates code on three benchmarks and tests on UnLeakedTestBench.
  • Main Results: 56.95% average accuracy on the 1.5B model exceeds Golden Tests at 56.23% and Self-Play at 55.88%.Code-A1 consistently achieves the highest average scores across the three model scales.
  • Main Results: 15.29 Mul for the 3B Code-A1 Test LLM surpasses the 7B Base model’s 14.72, while Code-A1 improves test generation over Base and SFT models.The reported Mul scores are 7.14 versus 4.35 at 1.5B, 15.29 versus 8.53 at 3B, and 19.74 versus 14.60 at 7B.
  • Ablation Studies: α = 0.5 achieves 56.95 Avg for code generation and 7.14 Mul for test generation, balancing validity with adversarial difficulty.Lower α destabilizes training by emphasizing difficulty without correctness incentives.
  • Ablation Studies: Removing predicted answers degrades test generation, with pass@5 dropping to 11.18%, while removing core components is reported to degrade performance.The predicted answer helps avoid impossible tests and preserves complete unit-test synthesis.

5 Analysis

Additional analyses test whether Code-A1-generated tests can replace human annotations, whether rewards co-evolve stably, and whether adversarial tests improve inference-time selection.

  • Generated Tests as Golden Tests: 56.75% average accuracy from Code-A1-generated golden tests exceeds human annotations at 56.23% and SFT-generated tests at 56.22%.On HumanEval+ and MBPP+, Code-A1-guided training reaches 71.67% and 64.42%, versus 71.15% and 63.30% for human annotations.
  • Training Dynamics: α = 0.5 closely tracks the Golden Tests baseline for Code LLM training and achieves the highest sustained Test LLM reward.α = 0 causes erratic learning, while α = 1.0 produces inflated rewards from trivial tests.
  • Test-Time Scaling: 67.81% is the highest overall performance under parallel test-time scaling, achieved by the full Code-A1 plus Code-A1 combination.The Code-A1 Test LLM reaches 65.82% when verifying the Base Code LLM, compared with 65.13% for Base self-verification.
  • Test Difficulty & Diversity: On threeSum, the evolving Test LLM discovers highly repetitive-element edge cases that expose a duplicate-triplet bug missed by golden tests and earlier generated tests.The example edge case includes [-2, 1, 1, 1, 1].

6 Conclusion

The conclusion presents Code-A1 as adversarial co-evolution of separate Code and Test LLMs, stabilized by experience replay and supported by results against annotation-based and scaling baselines.

  • Conclusion: Code-A1 decouples code and test generation into models with opposing objectives to avoid self-collusion and enable white-box testing.The Code LLM seeks to pass tests, while the Test LLM seeks to expose defects.
  • Conclusion: The Mistake Book stabilizes training by replaying historical failures, preventing catastrophic forgetting and providing stable reward baselines.The conclusion describes this mechanism as part of Code-A1’s experience replay design.
  • Conclusion: Experiments show Code-A1 matches or exceeds RL with human-annotated golden tests while producing bug-revealing tests more effectively than supervised fine-tuning or parameter scaling.The paper frames adversarial co-evolution as a scalable path beyond annotation-dependent training.

A Experimental Details.

The experimental implementation uses sandboxed batch execution to validate generated tests and evaluate code, while the Mistake Book stores and updates historical failures for replay.

  • Prompt Design: The Test LLM prompt requests eight assertion-based tests targeting logic gaps, complex inputs, and nonredundant coverage.Tests are conditioned on the question and buggy code, with assertions formatted as function calls and expected answers.
  • Sandbox Execution: Training execution validates generated assertions against ground-truth functions before evaluating candidate code against the retained tests.The sandbox executes batched requests, filters invalid cases, and combines validated tests with historical failures for reward computation.
  • Sandbox Execution: The validation templates use batched loops, duplicate detection, exception handling, and explicit markers for generated-test and attack-test outcomes.These mechanisms record valid assertions, invalid tests, ground-truth failures, and attack passes or failures through sandbox output.
  • Mistake Book: The Mistake Book maps each question to failed tests and their frequencies, retrieving historical failures and updating them as code fails or passes.A failed test is added or incremented; a passed historical test is decremented and removed when its frequency reaches zero.
  • Mistake Book: The Mistake Book is serialized to JSON after every training step so adversarial experience can be reloaded after interruption.The mechanism is implemented as a persistent global variable and supports training resumption.

A.4 Training Details

Training uses sandboxed rollouts and TopVar selection to focus Test LLM updates on informative reward variation, while Self-Play isolates test generation from candidate code.

  • Training Infrastructure: Rollouts sequentially sample the Code LLM and Test LLM, concatenate their outputs, and execute them in a sandbox with a strict 10-second timeout.The training stack uses verl, vLLM, and FSDP for reinforcement-learning experiments.
  • Asymmetric Sampling and TopVar: TopVar filters test-suite groups by reward variance and restricts Test LLM gradients to the highest-variance groups.The variance is computed over rewards for suites conditioned on each candidate solution, prioritizing informative samples.
  • Self-Play Comparison: Self-Play prevents reward hacking by using one model for both roles while restricting test generation to the question description without candidate-code access.This differs from Code-A1’s targeted adversarial generation, which conditions tests on candidate solutions.
  • Self-Play Comparison: The Self-Play rollout generates M = 8 candidate solutions and N = 8 test-suite sets per programming question before cross-running code against tests.Subsequent execution and reward steps follow the Code-A1 design.
  • SFT Comparison: The SFT comparison uses the same training source as RL but simplifies the prompt and supplies failed solution candidates with corresponding ground-truth labels.This setup is intended to compare Test LLM training under identical data distributions.
  • Test-Time Scaling: At inference, the scaling setup samples M = 16 solutions and N = 16 test suites, selecting the solution that passes the most unique tests.The test suites are generated solely from the question description, and the baseline symbol ‘/’ denotes direct avg@32 results.

C Performance Comparison with CURE

Code-A1 is compared with ReasonFlux-Coder while acknowledging base-model differences; it reports strong parameter efficiency but a test-generation gap against a Long-CoT model.

  • Comparison Setup: Direct comparison with ReasonFlux-Coder is confounded by different base models: Code-A1 uses Qwen2.5-Coder-Instruct, while ReasonFlux-Coder uses Qwen2.5-Instruct or Long-CoT Qwen3.The paper nevertheless presents cross-scale analysis of adversarial co-evolution and self-play.
  • Parameter Efficiency: 66.15% vs 64.33% Avg and 15.29 vs 10.95 Mul: Code-A1-3B outperforms ReasonFlux-Coder-7B on code and test generation.The comparison is reported as evidence of strong performance with fewer parameters.
  • Parameter Efficiency: 70.72% vs 68.05% Avg and 19.74 vs 17.14 Mul: Code-A1-7B surpasses ReasonFlux-Coder-14B on both reported tasks.The paper links this pattern to more efficient capability acquisition under decoupled adversarial co-evolution.
  • Base Model Architecture: 56.95% vs 58.29% Avg: Code-A1-1.5B approaches ReasonFlux-Coder-4B on code generation despite three times fewer parameters.Test generation shows the opposite pattern, with 7.14 vs 21.30 Mul.
  • Base Model Architecture: 7.14 vs 21.30 Mul: Code-A1-1.5B remains substantially behind ReasonFlux-Coder-4B on test generation.The paper attributes this gap to the Long-CoT base model’s inherent reasoning advantages for test synthesis and proposes combining Long-CoT with Code-A1.

D Training Pipeline of Code-A1

The Code-A1 pipeline generates candidate code and white-box tests, validates tests against ground truth, evaluates code on combined tests, and updates rewards and historical failures.

  • Code Generation: The Code LLM generates M candidate solutions for each training question.These candidates feed the subsequent white-box test-generation stage.
  • Test Generation: The Test LLM inspects each candidate solution and generates N assertion-based test suites conditioned on the question and code.White-box conditioning enables tests targeted at candidate weaknesses.
  • Mistake Book: Historical failures are retrieved from the Mistake Book using the highest failure frequencies to form a stable evaluation baseline.The retrieved tests are combined with newly generated tests during evaluation.
  • Test Validation: Generated tests are executed against ground-truth code, with incorrect predictions corrected using oracle outputs and other invalid tests discarded.Validity requires executable, unique calls with correct predicted answers.
  • Code Evaluation: Candidate solutions are evaluated on newly generated tests plus historical failures, and pass rates determine adversarial scores.The combined test set supplies the evaluation signal for both models.
  • Reward and Memory Update: The Test LLM reward is RT = α · Valid + (1 −α) · Adv, with TopVar selecting the highest-variance test-suite group for its update.The Code LLM receives rewards based on pass rates, while the Mistake Book records new failures and removes mastered tests.

E Evaluation Metrics Definition

Mul combines test validity with adversarial difficulty to measure the proportion of generated tests that are both executable and effective at exposing faults. Its multiplicative design penalizes trivial tests and highly complex tests with low validity.

  • Mul metric: Mul combines pass@k and mut@k to evaluate the yield of high-quality tests.Mutation score is conditional on tests first being valid, so the metrics capture complementary properties.
  • Validity–difficulty trade-off: High pass@k can reflect trivial tests with weak fault-detection capability and low mut@k.Validity alone does not ensure that generated tests expose implementation defects.
  • Validity–difficulty trade-off: High mut@k can coexist with low pass@k when only a small fraction of highly complex tests are valid.This may occur when evaluation selects the best tests from a sparse set of valid candidates.
  • Mul metric: Multiplying the metrics normalizes mutation score against the total sampling budget.The resulting score reflects the proportion of generated tests that simultaneously maintain validity and fault-detection capability.

F Limitations and Future Work

The paper identifies three boundaries for Code-A1: dependence on ground-truth solutions, assertion-format tests, and evaluation limited to Python function-level generation. It proposes broader validation and test-format support while leaving transfer to other domains open.

  • Dependence on Ground Truth Code: Code-A1 requires ground-truth solutions to validate generated tests during training.This limits applicability where reference implementations are unavailable; consensus or execution-based validation is proposed as future work.
  • Test Format Constraints: The framework currently requires assertion-format tests for reliable extraction and execution.This excludes stateful interactions, I/O-based testing, and property-based testing; diverse formats are proposed for future expansion.
  • Generalization to Other Domains: Code-A1 is evaluated exclusively on Python function-level code generation.Transfer to other programming languages, longer code contexts, or non-code reasoning tasks remains an open question.
Loading 2603.15611v1…