PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test

Zhangyi Hu, Chenhui Liu, Tian Huang, Jindong Li, Yang Yang, Jiemin Wu, Zining Zhong, Menglin Yang, Yutao Yue

arXiv:2605.23491v2cs.LGcs.AIcs.CL

TL;DR

Ground-truth unit tests are costly to curate, while self-generated tests can be unreliable for code selection without trusted code. CoSPlay jointly evolves code candidates and unit tests at test time, raising average BoN from 22.1% to 33.2% on Qwen2.5-7B-Instruct across four challenging benchmarks.

  • Problem

    Ground-truth unit tests are costly to curate, while existing ground-truth-free methods remain less competitive or require costly model-weight updates.

  • Method

    CoSPlay jointly evolves code candidates and self-generated unit tests through exploration, execution-matrix pass counts, iterative refinement, and output-consensus clustering.

  • Results

    Across four challenging coding benchmarks, average BoN rises from 22.1% to 33.2% for Qwen2.5-7B-Instruct, while UT accuracy increases from 14.6% to 78.3%.

  • Takeaways & Limitations

    CoSPlay provides a scalable, ground-truth-free inference strategy that generalizes across backbones and improves over existing ground-truth-free test-time scaling methods under comparable token budgets.

  • Takeaways & Limitations

    CoSPlay relies on executable environments and consumes more tokens because of iterative code–unit-test evolution, limiting direct extension to non-executable reasoning tasks.

Abstract

from arXiv · show

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) and Test-Time Scaling (TTS) have advanced LLM code generation through executable verification. Yet Ground-Truth Unit Tests (GT UTs) remain a bottleneck: SOTA RLVR methods require them for costly training, while existing TTS methods lose competitiveness without them. This motivates GT-free TTS, where existing methods directly use self-generated UTs to refine and select code candidates. Yet such UTs are often noisy or spuriously coupled with wrong code, and UT quality in turn cannot be validated without reliable code. The key challenge is therefore to jointly improve both. To this end, we present CoSPlay, a GT-free, training-free framework that jointly improves codes and UTs through cooperative self-play. It first explores diverse solution ideas and identifies their potential failure modes to produce discriminative UT ideas. It then uses bidirectional pass-count signals from the Code-UT execution matrix to iteratively prune or fix weak codes and refresh or replace unreliable UTs, letting the two pools co-evolve. Finally, when multiple codes remain tied at the highest pass count, it picks the final code from the largest output-consensus cluster, since correct codes agree on the same inputs while wrong codes diverge. Experiments on four challenging benchmarks show that CoSPlay on Qwen2.5-7B-Instruct improves average BoN from 22.1% to 33.2% and UT accuracy from 14.6% to 78.3%, matching or surpassing the RLVR model CURE-7B. When applied to CURE-7B, it further improves BoN by 5.7%. CoSPlay also generalizes across diverse backbones and outperforms GT-free TTS baselines under comparable token budgets, with continued gains as the budget scales up. These results suggest a scalable inference strategy for competitive code generation without any GT data.

1 Introduction

CoSPlay addresses the dependence of code-generation methods on trusted ground-truth unit tests by enabling self-generated codes and tests to co-evolve at test time. Its pipeline grounds tests in solution-specific failure modes, uses execution-matrix signals to refine both pools, and demonstrates effectiveness, generalizability, and scalability across coding benchmarks.

  • Motivation: RLVR methods use large collections of ground-truth unit tests for training, while test-time scaling needs trusted checks to refine, filter, and select code.Unit tests provide executable input-output checks that make functional correctness verifiable.
  • Framework: CoSPlay is a GT-free, training-free framework that creates a co-evolving loop between code candidates and self-generated unit tests at test time.The framework is presented as a scalable inference strategy for competitive code generation without GT data.
  • Pipeline: CoSPlay explores plausible solution strategies and derives targeted unit-test ideas from potential failure modes to distinguish plausible code candidates.This bootstraps an initial test pool grounded in solution-specific assumptions.
  • Empirical findings: Experiments across four challenging coding benchmarks demonstrate CoSPlay’s effectiveness, generalizability, and scalability, matching or surpassing strong RLVR models trained with massive GT unit-test collections.The framework also improves best-of-N performance across instruct and RLVR-tuned models.
  • Pipeline: Its Code-UT execution matrix supplies bidirectional pass-count signals for code cleaning, spurious coupling breaking, code refinement, and unit-test replacement.The matrix measures how many tests each code passes and how many codes each test accepts.

2 Methodology

CoSPlay constructs diverse solution plans and failure-oriented unit-test ideas, then co-evolves code and unit-test pools using bidirectional pass-count signals. Its self-play alternates code pruning, unit-test replacement or refreshing, code refinement, and final pass-count-based selection.

  • Pool Construction: CoSPlay expands diverse algorithmic hints into one- or two-hint solution plans, balancing solution diversity against combinatorial explosion.Each selected plan generates one code candidate, while failure-mode analysis produces attack ideas for discriminative unit tests.
  • Pool Construction: Unit tests combine random valid inputs with attack-derived inputs, retaining cases whose expected outputs are supported by repeated-output self-consistency.Attack inputs target plausible failure modes but are harder to solve reliably, so random inputs supplement them with broader sanity checks.
  • Pass-count Signal: Higher code and unit-test pass counts provide positive posterior signals when correct codes pass generated tests and correct tests are passed by generated codes.The method uses these bidirectional signals under stated quality thresholds to diagnose both pools.
  • Self-play Pipeline: Each self-play iteration prunes all-failing codes, refreshes suspicious low-support tests, refines codes with high-support non-trivial tests, and replaces zero-discrimination tests.The execution matrix is recomputed after every step, and the loop terminates when all entries become 1.
  • Final Selection: Final code selection scores candidates by pass count and applies execution-consensus clustering when multiple candidates share the top score.This resolves ties that discrete pass counts alone cannot break.

3 Experiments

Across four challenging coding benchmarks, CoSPlay improves base-model code and unit-test quality without ground-truth data or additional training. It outperforms scaled test-time-scaling baselines broadly, while ablations show that exploration and each self-play component contribute to final performance.

  • Experimental setup: Evaluation uses LiveBench, LiveCodeBench (v2), CodeContests, and CodeForces, containing 128, 511, 239, and 467 problems, respectively.Experiments other than the main full-benchmark results use a 200-problem benchmark to reduce computational cost.
  • Main results: CoSPlay w/ Cluster raises Qwen2.5-7B-Instruct average Signal, UT accuracy, code accuracy, and BoN from 22.1%, 14.6%, 19.0%, and 22.1% to 31.9%, 78.3%, 23.3%, and 33.2%.It slightly surpasses CURE-7B in average BoN, 33.2% vs. 32.9%, and achieves higher UT accuracy, 78.3% vs. 54.9%.
  • Comparison with baselines: The Cluster-only variant matches CodeT comparably, but both are outperformed by the full CoSPlay framework.
  • Comparison with TTS baselines: CoSPlay achieves higher Pass@1 than scaled TTS methods across both 7B and 14B models on most benchmarks.CoSPlay w/ Cluster further improves selection of BoN-tied codes through execution consensus, with one exception on Qwen2.5-14B-Instruct on LiveBench.
  • Ablation study: The full CoSPlay framework achieves the best final-round BoN accuracy, while removing exploration, Step 2, Step 3, non-trivial-best, or self-consistency degrades performance.The w/o exp-atk variant performs worst, highlighting the importance of idea-level exploration for constructing the initial Code-UT pool.

4 Discussion

The discussion shows that CoSPlay generalizes across backbones, improves verification signals through exploration and pass-count-based refinement, and uses execution consensus to select stronger code candidates. Its benefits grow with inference-time budgets, although effectiveness depends on pass-count statistics aligning with correctness.

  • Generalization across model families: CoSPlay with Cluster raises average BoN from 37.8% to 45.3% over Best-of-16, a +7.5% absolute gain across model families.The evaluation covers instruct, RLVR-tuned, and frontier-scale models, with gains attributed to jointly improving code and UT accuracy.
  • Verification signal quality: Exploration attack raises UTAcc from 12.5% to 37.2% and Signal from 26.5% to 33.7% before self-play or self-consistency.It improves initial UT accuracy and discriminativeness by reducing invalid, overly difficult, or spuriously code-tied tests.
  • Verification signal quality: Higher code and UT pass counts correlate positively with functional correctness, while correct candidates concentrate in high pass-count regions.These relationships support using execution-matrix pass counts to estimate code quality and UT reliability without GT tests.
  • Execution-consensus selection: Correct codes concentrate in larger execution-consensus clusters, and cluster scoring provides a more robust reliability estimate when invalid executions make raw cluster size unreliable.The scoring rule preserves the largest-cluster intuition while adapting selection to runtime-error settings for both CoSPlay-7B and CoSPlay-14B.
  • Inference-time scaling: At k ≥16, Qwen2.5-7B-Instruct + CoSPlay becomes comparable to CURE-7B, and at k = 64 CoSPlay improves Qwen2.5-7B-Instruct by 13.2%.CURE-7B + CoSPlay also continues benefiting from larger candidate pools, while CoSPlay outperforms competing TTS methods under comparable or larger token costs.

5 Conclusion and Future Directions … A.1.2 Support-Count Posterior

CoSPlay jointly evolves code candidates and unit tests through execution-driven cooperative self-play, while its pass-count analysis explains when support counts reliably indicate correctness. The framework performs best among evaluated GT-free TTS methods under comparable budgets, but remains limited by executable-environment dependence and token cost.

  • 5 Conclusion and Future Directions: CoSPlay jointly optimizes code candidates and unit tests through execution-driven co-evolution, using failure modes, pass counts, and output consensus.It explores diverse solution strategies, refines both pools through cooperative self-play, and resolves BoN ties using the largest output-consensus cluster.
  • 5 Conclusion and Future Directions: CoSPlay requires executable feedback and more tokens, but achieves the best evaluated performance under comparable test-time compute budgets.Future directions include improving token efficiency and extending the framework to broader reasoning tasks.
  • S Average evolution of the UT and code pass-count distributions across iterative self-play rounds 45: The supplementary analyses track how UT and code pass-count distributions, BoN, and code accuracy evolve across iterative self-play rounds.These analyses provide detailed round-by-round views of the framework’s iterative behavior.
  • A.1 Analysis of Pass-Count Signals: The i.i.d. standard-BoN analysis models independently sampled codes and UTs to explain pass counts as statistical signals for judging both object types.It introduces execution-matrix thresholds, posterior direction, and convergence rates for binomial support counts.
  • A.1.1 When Pass Counts Indicate Correctness: Pass-count discrimination requires ΔU > 0 when ρC > ρ⋆ and ΔC > 0 when ρT > ρ⋆, with the code-side threshold relatively higher.Correct codes always pass correct tests and fail incorrect tests; incorrect-code pass noise is governed by ε1 and ε2, while runtime errors count as failures.
  • A.1.2 Support-Count Posterior: Under the binomial support-count model, larger support increases posterior correctness when q1 > q0, has no effect when q1 = q0, and decreases it when q1 < q0.The posterior odds depend exponentially on the support count through the likelihood-ratio factor.
  • A.1.2 Support-Count Posterior: As evaluator count m grows, support ratios on the q1-favored side of the unique threshold η⋆ drive posterior correctness exponentially toward 1, while the opposite side drives it toward 0.At η = η⋆, posterior odds remain only constant-order, so no exponential decision is obtained.

A.2 Analysis of Execution-Consensus Clustering · B Detailed Algorithm Design · C Related Work

The analysis establishes when execution-consensus clustering identifies the correct output signature, while the algorithm operationalizes CoSPlay through iterative code–unit-test refinement and final clustering.

  • A.2 Analysis of Execution-Consensus Clustering: Under α := P(C+) > 0 and incorrect-output probability bounded by β < 1, the correct signature σ+ becomes the unique maximum-probability signature for sufficiently large R.The result assumes every incorrect code, output value, and input index has conditional probability at most β.
  • A.2 Analysis of Execution-Consensus Clustering: For independently sampled candidates and finite signature space, the largest empirical cluster converges almost surely to the correct output-consensus cluster.Eventually, σ+ has a strictly larger empirical cluster than every incorrect signature.
  • B Detailed Algorithm Design: CoSPlay takes a problem, code and UT pool sizes, clustering input size R, and iteration limit Tmax, and returns the best generated code candidate c∗.The framework begins by exploring code ideas and generating failure-oriented UT ideas.
  • B Detailed Algorithm Design: The framework initializes Nt UTs using half random valid tests and half self-consistency-generated tests, then refreshes the execution matrix after each self-play step.REFRESH computes Mij = I(Exec(ci, xj) = yj) and UT pass rates.
  • B Detailed Algorithm Design: Each self-play iteration cleans failing codes, regenerates replacements, breaks spurious code–UT coupling by replacing the worst UT, and repairs targeted codes using execution feedback.The algorithm regenerates UTs from failure-oriented ideas using the worst UT and its passed codes.
  • B Detailed Algorithm Design: The algorithm replaces zero-discrimination UTs, executes surviving codes on random inputs with runtime errors mapped to ERR, and partitions Chigh by compatibility.Final selection therefore uses runtime-error-aware execution behavior to form candidate clusters.

C.1 Fine-tuning Methods for Code Generation … C.4 Search in Natural Language Space

Prior work spans ground-truth-dependent fine-tuning, GT-challenged test-time scaling, and self-play methods that often retain external-supervision or training requirements. Natural-language search instead structures reasoning as exploration over intermediate thoughts, with code-generation methods beginning to use high-level plans.

  • C.1 Fine-tuning Methods for Code Generation: Most code-generation training methods rely heavily on external ground-truth or synthetic data for supervised fine-tuning and reward optimization.µCode and AceCoder use supervised fine-tuning, while Focused DPO and CTRL use RLVR with reward design.
  • C.1 Fine-tuning Methods for Code Generation: Some methods fine-tune models specifically for unit-test generation by leveraging ground-truth code.O1-Coder and UTGen are cited as examples.
  • C.2 TTS for LLM Code Generation: Test-time scaling commonly starts with repeated sampling such as Best-of-N, but selecting the best candidate remains difficult without ground-truth data.This motivates selection mechanisms beyond repeated sampling alone.
  • C.2 TTS for LLM Code Generation: Consensus-based methods use self-generated unit tests for voting, while CodeT faces unreliable tests and MPSC incurs substantial extensive-sampling overhead.The passage identifies these as limitations of existing consensus-based selection.
  • C.3 Self-Play in LLM: LLM self-play methods in coding generally remain dependent on external supervision and heavy training.CURE relies on ground-truth unit tests, whereas UTRL requires ground-truth code to improve model performance.
  • C.3 Self-Play in LLM: Absolute-Zero is cited as achieving a fully self-supervised setup without the external-supervision dependence described for most coding self-play approaches.The supplied passage ends after introducing this setup, so no further property is attributed here.
  • C.4 Search in Natural Language Space: Chain-of-Thought strategies have been incorporated into structured searches over natural-language reasoning steps.Tree of Thoughts and Reasoning via Planning represent reasoning as search trees over intermediate thoughts.
  • C.4 Search in Natural Language Space: In code generation, PlanSearch generates high-level natural-language plans before implementation.It is presented as a domain-specific example of natural-language-space search.

D Detailed baseline setting

The baselines follow each method’s original prompts and LLM parameters, while TTS comparisons match total code and unit-test generation budgets. All TTS unit tests are model-generated without external supervision, with method-specific settings and scaling configurations.

  • RL Baselines: RL baselines use the original prompts and LLM parameters specified in each method’s code or paper.Training data sources and base models are listed in Table 2.
  • TTS Baselines: TTS baselines control total code and unit-test generation budgets comparably, generating all unit tests without external supervision.The comparison uses method-specific settings described below.
  • TTS Baselines: SFS uses num_seed = 10, max_iters = 40, total budget of 50, scaled to 80, generated UT = 6, Role seed, and temperature 0.2.Role seed is the best-performing variant reported in the original paper.
  • TTS Baselines: CodeTree uses max budget = 50, generated UT = 6, max depth = 5, search width = 10, with a scaled setting of max budget = 1000 and generated UT = 16.The scaled setting uses max depth = 20 and search width = 100; temperature is 0.0.
  • TTS Baselines: MPSC uses solution = 60, specification = 50, test case = 100, while scaled settings use solution = 300, specification = 250, and test case = 600.The MPSC-label mode is used, with sampling temperature 0.8.

E Analysis of the cost and performance balance for TTS methods

CoSPlay delivers the strongest pass@1 accuracy on both model scales while using computation to improve verification quality rather than merely increasing sample counts. Ablations show that lightweight variants retain strong accuracy at substantially lower token costs, under comparisons with generously scaled baselines.

  • Performance and cost: 37.2% pass@1 on 7B and 45.0% on 14B make CoSPlay the best method on both model scales.These improve the strongest competing TTS baseline from 32.0% to 37.2% on 7B and from 40.5% to 45.0% on 14B.
  • Performance and cost: Removing self-consistency cuts 7B token cost from 682K to 244K while retaining 34.2% pass@1, above all competing TTS baselines.Further removing UT-iteration reduces cost to 90K tokens and 198 calls, while the passage indicates accuracy remains strong.
  • Performance and cost: In the GT-free setting, prior TTS methods rely on noisy self-generated UTs, so scaling does not necessarily improve performance.For example, S* drops from 20.5% at Round 3 to 19.2% at Round 6 on 7B.
  • Performance and cost: CoSPlay spends additional computation on targeted tests, code pruning, UT refreshing, buggy-code refinement, and replacement of zero-discrimination UTs.The method strengthens the verification signal itself rather than merely increasing the number of sampled programs.
  • Performance and cost: Baselines are evaluated with generous sampling or search budgets, including convergence for methods with built-in stopping criteria.The comparisons therefore target strong scaled configurations rather than under-explored baseline settings.

F Detailed data about generalization of our method

CoSPlay generalizes across instruct and RLVR-tuned backbones from 7B-scale models to DeepSeek-V3.2-685B, with average BoN improving from 37.8% to 45.3%. Its gains are largest when initial Code-UT signals are useful but unsaturated, while self-play can hurt near-optimal solutions or fail with overly noisy signals.

  • Cross-backbone generalization: 45.3% average BoN improves from 37.8% with CoSPlay w/ Cluster across evaluated backbones.This corresponds to a +7.5% absolute gain across models spanning 7B-scale backbones to DeepSeek-V3.2-685B.
  • Cross-backbone generalization: 10.0 percentage points: Seed-Coder-8B-Instruct BoN rises from 32.3% to 42.3%.DeepSeek-Coder-V2-Lite-Instruct-16B similarly improves from 30.8% to 41.3%.
  • Strong-model generalization: 68.2% average BoN improves from 65.7% for DeepSeek-V3.2-685B with CoSPlay w/ Cluster.On CodeForces, BoN increases from 39.3% to 50.0%, showing gains even for a strong model on difficult tasks.
  • Boundary cases: 64.7% BoN declines from 68.0% for Gemini-2.0-Flash on LiveBench after CoSPlay, while CoSPlay w/ Cluster recovers only to 66.0%.DeepSeek-V3.2-685B also drops from 85.1% to 78.4% on LiveBench after CoSPlay.
  • Boundary cases: CoSPlay is most effective when initial Code-UT signals are useful but not saturated; near-optimal baselines can be perturbed, while weak-model signals may be too noisy for reliable co-evolution.Execution feedback can guide codes and unit tests to improve each other in the favorable regime.

G Dataset Overview · H Detailed Ablation Results

The paper evaluates code generation on four competitive-programming benchmarks, including contamination-controlled and temporally annotated datasets. It also reports CoSPlay’s generalization across base and RL-tuned models using unit-test, code, signal, and BoN metrics.

  • G Dataset Overview: LiveBench is a frequently updated, contamination-limited benchmark spanning math, coding, reasoning, language, instruction following, and data analysis.The experiments use its coding subset.
  • G Dataset Overview: LiveCodeBench v2 contains 511 problems released between May 2023 and May 2024 from LeetCode, AtCoder, and CodeForces contests.Release-time tags support temporally controlled evaluation and help reduce train-test leakage.
  • G Dataset Overview: The CodeContests evaluation uses 239 problems sampled from the CURE-provided subset of a dataset built from Codeforces, Description2Code, and APPS.CodeContests was introduced with AlphaCode.
  • G Dataset Overview: The CodeForces evaluation uses 467 randomly sampled problems below difficulty level 2 from the CURE-provided open-r1/codeforces dataset.The full dataset contains more than 10k unique contest problems and emphasizes verifiability for multiple valid outputs.
  • G Dataset Overview: Table 6 evaluates CoSPlay’s generalization across different base models and RL-tuned models.It compares standard BoN baselines with CoSPlay using average unit-test accuracy, code accuracy, signal quality, and final selected-code BoN accuracy.
  • H Detailed Ablation Results: The supplied passages identify H Detailed Ablation Results as a merged section, but provide no separate ablation findings beyond the Table 6 evaluation description.The available table passage defines the reported metrics and explains that bold indicates the best performance.

H.1 Ablation Variant Descriptions · H.2 Analysis · I Random UT Initialization Enhances UT Diversity

The ablations show that solution-aware exploration and iterative self-play are important for improving code and unit-test quality, while random valid-test initialization preserves broader unit-test diversity across self-play.

  • H.1 Ablation Variant Descriptions: CoSPlay evaluates component contributions through variants removing solution-aware exploration, self-play, random unit-test initialization, and other components.These variants isolate the effects of the framework’s exploration, refinement, and initialization strategies.
  • H.2 Analysis: Removing solution-aware exploration lowers BoN performance even when raw code accuracy increases, indicating that exploration primarily improves unit-test discrimination.The ablation suggests that better test quality, rather than naive code accuracy, drives the performance gain.
  • H.2 Analysis: Removing self-play supports the conclusion that CoSPlay depends on jointly improving unit-test quality and code quality.The self-play ablations show that iterative refinement is central to the method’s cooperative behavior.
  • H.2 Analysis: Removing random initialization, self-consistency, or unit-test regeneration all hurts final BoN, demonstrating the importance of multiple unit-test refinement mechanisms.These findings are reported in the ablation study summarized by Table 7.
  • I Random UT Initialization Enhances UT Diversity: Randomly generated valid unit tests produce a higher UT rank across all self-play rounds than initialization without random tests.Both variants decline after the first update, but the full method recovers better in later rounds.
  • I Random UT Initialization Enhances UT Diversity: Random unit tests preserve broader UT diversity, complement attack-oriented tests, and prevent collapse into narrowly similar failure-focused cases.This explains why random-valid-input initialization remains useful throughout self-play.

J Detailed data of UT pass count distribution about Idea-Level Exploration vs. direct sample

At UT initialization, directly sampled unit tests often fail every code candidate, whereas Exploration-Attack produces UTs with fewer zero-pass outliers and a broader moderate-pass distribution. Execution-consensus clustering becomes more reliable as the number of random valid inputs increases.

  • UT initialization distributions: The w/o exp-atk variant concentrates zero-pass UTs across all four datasets, indicating many directly sampled UTs are likely incorrect or noisy.These UTs are not passed by any code candidate.
  • Execution-consensus clustering: Increasing clustering inputs from 0 to 64 provides a more reliable execution signature for grouping functionally equivalent candidates.The effect is evaluated on CodeContests, CodeForces, LiveBench, and LiveCodeBench.
  • UT initialization distributions: Exploration-Attack shifts UT pass counts toward roughly 2-10 and produces a more uniform, discriminative distribution.The full method reduces the zero-pass spike, suggesting its UTs are more accurate and less trivially failing or passing all candidates.

K Scalability Analysis of Random Input Size for Clustering

CoSPlay’s execution-consensus clustering remains robust as the number of random valid inputs increases from 0 to 64. More inputs generally improve or stabilize performance, especially on LiveBench and LiveCodeBench, despite non-monotonic fluctuations.

  • Experimental setup: With Qwen2.5-7B-Instruct and BoN (N = 16), the study varies clustering inputs k from 0 to 64 across four datasets.This experiment evaluates how random valid input count affects execution-consensus cluster selection.
  • Performance trend: Increasing k generally improves or stabilizes performance across all four datasets, with especially visible gains on LiveBench and LiveCodeBench.More random inputs produce more reliable execution signatures for grouping equivalent candidates and separating spurious agreements.
  • Scalability: Although improvement is not strictly monotonic, larger k does not cause performance collapse and yields more stable selection while exploiting additional test-time execution budget.Random inputs differ in discriminative strength, explaining fluctuations while preserving a favorable overall scaling trend.

L Variance analysis … New generated UT for CoSPlay-14B

Across variance, distribution, metric-evolution, and case-study analyses, CoSPlay’s self-play signals become more reliable and effective over rounds, while cluster consensus helps distinguish correct code and drive concrete fixes. The case studies also show that generated UTs can expose code–requirement mismatches and identify code–UT coupling.

  • L Variance analysis: Small standard deviations across most datasets and metrics indicate that CoSPlay’s observed improvements are stable across three independent runs.Table 8 reports mean accuracy and standard deviation over three runs.
  • M Detailed Evolution of Cluster-Size Density for Correct and Incorrect Codes: Correct codes tend to occupy larger execution-signature clusters than wrong codes, although overlap increases on harder datasets such as CodeForces.The separation is strongest on CodeContests and LiveCodeBench, supporting cluster size as a GT-free correctness proxy with limitations.
  • N Detailed Evolution of UT Pass-Count Density: Across datasets, UT pass-count density shifts toward higher values as self-play removes weak tests and replaces zero-discrimination UTs, improving pool reliability without trivializing it.UT pass count measures how many code candidates pass a generated UT, and it is positively correlated with true UT correctness.
  • O Detailed Evolution of Code Pass-Count Density: Code pass-count density likewise moves toward higher values across datasets and model scales, reflecting iterative removal or refinement of weak candidates and increasingly useful UT supervision.Code pass count measures how many generated UTs a code candidate passes, and it is positively correlated with true code correctness.
  • P Detailed Evolution of average true accuracy vs. cluster size: Larger clusters consistently correspond to higher average true code accuracy across datasets, model scales, and self-play rounds, supporting output consensus as a GT-free correctness proxy.The relationship is reported for CodeContests, CodeForces, LiveBench, and LiveCodeBench.
  • Q Detailed Breakdown of the Relationship between UT Pass Count and True Accuracy; R Detailed Breakdown of the Relationship between Code Pass Count and True Accuracy: UT and code pass counts are positively correlated with true accuracy across all benchmarks, with the 14B model achieving higher accuracy rates than the 7B model at the same pass counts.The detailed plots show that these correlations are not artifacts of aggregation and hold consistently across tasks.
  • S Average evolution of the UT and code pass-count distributions across iterative self-play rounds; T Detailed metrics evolution during self-play stage; T.1 Detailed Signal accuracy evolution during self-play rounds; T.2 Detailed BoN evolution during self-play rounds; T.3 Detailed Code accuracy evolution during self-play rounds; T.4 Detailed UT accuracy evolution during self-play rounds: Signal Accuracy, BoN Accuracy, Code Accuracy, and UT Accuracy generally rise during self-play, with cluster-based selection improving Signal Accuracy and BoN Accuracy over their non-cluster variants.Code Accuracy gains are largest between Round 0 and Round 1, while UT Accuracy shows a dramatic Round 0-to-Round 1 surge before largely plateauing; CodeForces is consistently challenging.
  • U Case study; U.1 Case Study of successful code fixing; U.1.1 CoSPlay-7B; U.1.2 CoSPlay-14B; U.2 Case study of successful locating Code–UT Coupling; New generated UT for CoSPlay-14B: Case studies show generated UTs exposing semantic mismatches that enable code fixing, while a new UT corrects a wrong oracle that had rewarded buggy code in a Code–UT coupling case.Examples include reciprocal sums versus total weights, one-row versus two-row capacity, and an impossible permutation condition where the correct answer is `No`.

U.2.3 Analysis for case study of UT-code coupling … Prompt for direct UT generation

The case study shows that self-generated unit tests can spuriously validate flawed code, while the prompt suite generates solutions, attacks, tests, outputs, repairs, and direct candidates to improve code–UT discrimination. CoSPlay targets suspicious low-but-nonzero-pass UTs through regeneration and uses structured prompting to support iterative refinement.

  • U.2.3 Analysis for case study of UT-code coupling: In the 7B case, a wrong code passes a low-pass UT before regeneration but fails the corresponding regenerated UT, breaking the spurious agreement.The initial agreement creates a false positive that can inflate the wrong solution’s pass count.
  • U.2.3 Analysis for case study of UT-code coupling: In the 14B case, a suspicious second-low UT accepts the wrong or BoN-selected code before resampling, whereas the wrong code fails the new UT afterward.Unlike the 7B example, this is not a unique-pass column; it is a low-pass suspicious UT whose agreement misleadingly supports the wrong candidate.
  • U.2.3 Analysis for case study of UT-code coupling: Step 2 replaces suspicious low-but-nonzero-pass-rate UTs while leaving the rest of the pool unchanged, removing false positives and restoring discrimination.Execution matrices mark codes as rows and UTs as columns, with pass/failure cells; regenerated UTs are distinguished from suspicious originals.
  • U.2.3 Analysis for case study of UT-code coupling: The examples expose coupling from both flawed inputs and incorrect or implementation-aligned oracles, including rectangle-overlap counting and swap-based array dominance.The rectangle candidate reduces a two-dimensional overlap problem to one-dimensional events, while the swap example’s oracle says Yes although the true answer is No.
  • Prompt for solution hints generation; Prompt for specific solution idea generation: The method prompts models to produce numbered solution observations and then derive additional observations without returning code.These prompts support solution-hint generation and specific solution-idea generation before code synthesis.
  • Prompt for code generation from solution idea; Prompt for UT attack idea generation: Code-generation prompts convert problem observations into Python scripts using input() and print(), while attack prompts request numbered ideas targeting design flaws, risks, and edge cases.The attack-idea prompt frames the model as a software-engineering tester and asks for ideas at the same level as the supplied observation.
  • Prompt for UT input generation from UT attack idea: UT-input prompts enforce constraint analysis, attack mapping, construction, verification, decomposition checks, and a strict explanation-plus-raw-input format.The instructions explicitly require checking element counts, formatting, explicit and implicit constraints, and validity before final output.
  • Prompt for UT output generation from specific input; Prompt for generating non-coupling UT; Prompt for code fixing; Prompt for random UT input generation; Prompt for direct code generation; Prompt for direct UT generation: Other prompts generate or regenerate outputs, fix failed code from runtime or logical-error reports, create random valid cases, and directly produce code or test examples.Regeneration instructs recomputation rather than copying previous attempts; random generation requires exactly new valid cases, and direct UT generation requires test input, output, and explanation.
Loading 2605.23491v2…