PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test

Zhangyi Hu, Chenhui Liu, Tian Huang, Jindong Li, Yang Yang, Jiemin Wu, Zining Zhong, Menglin Yang, Yutao Yue

arXiv:2605.23491v2cs.LGcs.AIcs.CL

TL;DR

Ground-truth unit tests are costly to curate at scale, yet leading RLVR methods rely on them for training. CoSPlay jointly evolves code candidates and self-generated tests at test time; across four challenging benchmarks, Qwen2.5-7B-Instruct’s average BoN rises from 22.1% to 33.2%.

  • Problem

    Ground-truth unit tests support executable code verification, but their expert curation is too costly to scale and leading RLVR methods rely on large collections of them for training.

    Source: §1 · p. 2

  • Method

    CoSPlay uses solution strategies and their potential failure modes to generate discriminative tests, then uses execution-matrix pass counts to co-evolve code and test pools, with output-consensus clustering for tied candidates.

    Source: §1 · p. 2, §5 · p. 11

  • Results

    Across four challenging coding benchmarks, CoSPlay improves base models; on Qwen2.5-7B-Instruct, average BoN rises from 22.1% to 33.2%.

    Source: §1 · p. 2, §3.2 · p. 6

  • Takeaways & Limitations

    CoSPlay offers a GT-data-free inference strategy that outperforms evaluated GT-free TTS methods under comparable output-token budgets and remains competitive with strong RLVR models.

    Source: §1 · p. 2

    CoSPlay requires an executable environment and consumes more tokens through iterative Code-UT evolution, making direct extension to non-executable reasoning tasks more challenging.

    Source: §5 · p. 11

Abstract

from arXiv · show

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) and Test-Time Scaling (TTS) have advanced LLM code generation through executable verification. Yet Ground-Truth Unit Tests (GT UTs) remain a bottleneck: SOTA RLVR methods require them for costly training, while existing TTS methods lose competitiveness without them. This motivates GT-free TTS, where existing methods directly use self-generated UTs to refine and select code candidates. Yet such UTs are often noisy or spuriously coupled with wrong code, and UT quality in turn cannot be validated without reliable code. The key challenge is therefore to jointly improve both. To this end, we present CoSPlay, a GT-free, training-free framework that jointly improves codes and UTs through cooperative self-play. It first explores diverse solution ideas and identifies their potential failure modes to produce discriminative UT ideas. It then uses bidirectional pass-count signals from the Code-UT execution matrix to iteratively prune or fix weak codes and refresh or replace unreliable UTs, letting the two pools co-evolve. Finally, when multiple codes remain tied at the highest pass count, it picks the final code from the largest output-consensus cluster, since correct codes agree on the same inputs while wrong codes diverge. Experiments on four challenging benchmarks show that CoSPlay on Qwen2.5-7B-Instruct improves average BoN from 22.1% to 33.2% and UT accuracy from 14.6% to 78.3%, matching or surpassing the RLVR model CURE-7B. When applied to CURE-7B, it further improves BoN by 5.7%. CoSPlay also generalizes across diverse backbones and outperforms GT-free TTS baselines under comparable token budgets, with continued gains as the budget scales up. These results suggest a scalable inference strategy for competitive code generation without any GT data.

1 Introduction

CoSPlay addresses fragile GT-free code generation by making code candidates and self-generated unit tests co-evolve at test time. Its three-stage framework uses targeted test ideas, bidirectional pass-count signals, and output consensus to refine and select code without ground-truth data.

1 Introduction

  • GT test curation is costly to scale, while self-generated tests can have wrong outputs, miss distinctions among candidates, or spuriously agree with incorrect code.SOTA RLVR methods use large collections of ground-truth tests; GT-free RLVR methods such as Absolute-Zero [47] remain less competitive and require costly weight updates [48].Source: §1 · p. 2
  • CoSPlay is presented as a GT-free, training-free inference framework that lets code candidates and self-generated tests co-evolve without ground-truth data.The introduction reports experiments across four challenging benchmarks, claims matching or surpassing strong RLVR models, outperforming GT-free TTS methods under comparable output-token budgets, and gains as budgets scale.Source: §1 · p. 2
  • The paper proposes pass counts as GT-free quality signals: test pass counts estimate test reliability, code pass counts estimate candidate quality, and output-consensus clusters estimate selection confidence.The introduction says theory and before/after self-play studies show these signals correlate with true quality and support co-evolution.Source: §1 · p. 2
  • CoSPlay’s three-stage pipeline generates failure-oriented test ideas, uses bidirectional pass counts to co-evolve code and tests, then selects among tied codes by output consensus.The consensus step compares outputs on random valid inputs, based on the expectation that correct codes agree while incorrect codes tend to diverge.Source: §1 · p. 2, Figure 3 · p. 3

2 Methodology

CoSPlay builds code and unit-test pools from diverse solution and failure ideas, then co-evolves them using execution pass counts before selecting among top-scoring codes by output consensus.

2.1 Exploration-Attack-Guided Code and UT Idea Generation

  • CoSPlay explores diverse algorithmic hints and one- or two-hint solution plans, then derives targeted unit-test ideas from each plan’s potential failure modes.The bounded hint subsets balance solution diversity against combinatorial growth; failure-oriented ideas target hidden edge cases and implementation pitfalls.Source: §2.1 · p. 3

2.2 Code and UT Initialization

  • The initial code pool samples candidates from shuffled solution plans, while the test pool combines random valid inputs with attack-derived inputs whose expected outputs pass a self-consistency filter.Multiple sampled outputs must agree before an input-answer pair is retained, raising the initial test pool’s correctness prior; random inputs supplement harder-to-infer attack probes with broader sanity checks.Source: §2.2 · pp. 3–4

2.3 Execution-Matrix-Driven Iterative Self-Play

  • Under the stated quality thresholds, higher code and test pass counts indicate higher posterior correctness, a relationship also observed in Round-0 results.The framework uses these pass-count signals to diagnose the execution matrix and guide self-play.Source: §2.3 · p. 4, Figure 4 · p. 4
  • The loop refines codes failing the highest-support nontrivial test and replaces tests passed by all codes or none, re-executing every code-test pair after each step.It skips steps without targets and ends when every matrix entry is 1 or the iteration budget Tmax is reached.Source: §2.3 · pp. 4–5
  • Self-play removes all-failing codes but refreshes low-support, nonzero-pass tests to break potentially spurious coupling with the minority code behavior that passes them.The selected test is a practical proxy for likely spurious coupling, not necessarily an incorrect test; zero-support tests may still expose shared bugs.Source: §2.3 · p. 4

2.4 Output-Consensus-Based Cluster Selection

  • Among top-pass-count codes, CoSPlay selects by output-consensus clusters on random valid inputs; with execution errors, it ranks clusters and codes by valid pairwise-agreement reliability scores.In the error-free case, maximizing the cluster score is equivalent to choosing the largest cluster; errors count as missing evidence rather than output conflicts.Source: §2.4 · p. 5

3 Experiments

Across four coding benchmarks, CoSPlay improves self-generated unit-test and code quality without ground-truth data or additional training, while ablations and comparisons examine its selection and self-play components.

3.1 Settings

  • Evaluation uses LiveBench, LiveCodeBench (v2), CodeContests, and CodeForces, with the main full-benchmark results distinguished from other experiments’ 200-problem setting.The 200-problem setting samples 50 problems per dataset for each of three random seeds and averages results. Baselines include Qwen2.5 instruct models, other TTS methods, and 7B- and 14B-scale instruct and RLVR-trained models.Source: §3.1 · pp. 5–6

3.2 Results

  • 33.2% average BoN and 78.3% UT accuracy for CoSPlay w/ Cluster on Qwen2.5-7B-Instruct surpass CURE-7B’s 32.9% BoN and 54.9% UT accuracy.CoSPlay raises the base model’s average Signal, UT accuracy, code accuracy, and BoN from 22.1%, 14.6%, 19.0%, and 22.1% to 31.9%, 78.3%, 23.3%, and 33.2%; on CURE-7B and CURE-14B, it raises BoN from 32.9% to 38.6% and from 41.8% to 46.2%, respectively. The Cluster-only variant is comparable to CodeT, while full CoSPlay outperforms both.Source: Table 1 · p. 6, §3.2 · pp. 6–7

3.1 Settings

  • CoSPlay experiments use code and UT pools of 16 each and a maximum of five iterations, with UT outputs retained when at least three of four sampled expected outputs agree.Half of UT inputs are random valid inputs and half come from A; experiments use two NVIDIA H100 GPUs with temperature 0.8, top-p 0.95, and top-k 40.Source: §3.1 · p. 6

3.2 Results

  • CoSPlay achieves higher Pass@1 than scaled TTS methods on most benchmarks at both model scales, with Qwen2.5-14B-Instruct on LiveBench as an exception.On that exception, scaling BoN to N = 256 surpasses CoSPlay-14B; the passage notes simple sampling alone is insufficient on harder benchmarks. CoSPlay w/ Cluster further expands the radar-plot envelope.Source: §3.2 · p. 7, Figure 5 · p. 7
  • The full framework has the best final-round BoN accuracy, while removing idea-level exploration or individual self-play components degrades performance.Its Signal increases consistently across rounds; ablated variants have weaker or unstable trajectories, and w/o exp-atk declines over time.Source: §3.2 · p. 7, Figure 5 · p. 7

4 Discussion

The discussion reports CoSPlay’s gains across model families and candidate budgets, and examines when execution-matrix signals and consensus clusters are informative and how they affect UT quality, diversity, and efficiency.

4.1 Generalizability of CoSPlay

  • CoSPlay w/ Cluster raises average BoN from 37.8% to 45.3% (+7.5% absolute) across instruct and RLVR models, and improves frontier-model BoN too.For DeepSeek-V3.2-685B, average BoN rises from 65.7% to 68.2%; for Gemini-2.0-Flash, it rises from 53.8% to 55.3%. Self-play is most useful when pass counts align with correctness while the pool still has improvable low-support items; very weak backbones can have noisy tests, while saturated pools may fluctuate after refreshes.Source: §4.1 · p. 7, Figure 6 · p. 8

4.2 Exploration-Attack Guided Idea Generation Improves Initial UT Quality

  • Exploration alone raises UTAcc from 12.5% to 37.2% and Signal from 26.5% to 33.7%, while increasing average UT pass count by δ = 2.39.It also reduces extreme low-support UTs by 29.0%, consistent with a shift toward UTs supported by multiple candidate codes.Source: Figure 6 · p. 8, §4.2 · p. 8

4.3 Pass Count Reflects True Accuracy

  • Correct codes and UTs are more likely to have higher pass counts, making execution-matrix pass counts a GT-free proxy for candidate quality.Correct candidates concentrate more heavily in high pass-count regions than wrong candidates.Source: Figure 7 · p. 8, §4.3 · p. 8

4.4 Cluster-Based Selection Reflects Functional Consistency

  • Larger execution-consensus clusters correlate with higher true code accuracy and aggregate cluster scores; valid pairwise agreements make the score more robust when executions are invalid.The t-SNE visualizations show correct codes tending to form large, compact clusters, while incorrect codes are often scattered or isolated.Source: §4.4 · pp. 8–9, Figure 6 · p. 8, Figure 9 · p. 9

4.5 Self-play progressively shifts pass-count distributions to the right

  • Across 7B/14B models, self-play reduces low-pass-density regions by 29.9%/27.3% for UTs and 25.2%/23.3% for codes.High-pass-density regions increase as the distributions shift toward higher-support regions across rounds.Source: §4.5 · p. 9, Figure 8 · p. 9

4.7 Scalability of CoSPlay with Increasing Candidate-Pool Size

  • At k = 64, CoSPlay improves Qwen2.5-7B-Instruct by 13.2%, and Qwen2.5-7B-Instruct + CoSPlay becomes comparable to CURE-7B when k ≥16.Both CoSPlay settings show no clear saturation at k = 64; CURE-7B + CoSPlay also continues to benefit from larger budgets.Source: Figure 10 · p. 10, §4.7 · p. 10

4.8 UT Accuracy-Diversity Trade-off of CoSPlay

  • CoSPlay moves models toward a better UT accuracy-diversity trade-off, with CURE-14B illustrating how high UT accuracy can coincide with relatively low UT rank [48].Idea-level exploration promotes diversity, while pass-count filtering preserves accuracy; Qwen-based models and CURE-7B mainly gain UT accuracy with slightly decreased yet competitive rank.Source: Figure 10 · p. 10, §4.8 · p. 10

4.9 Efficiency Analysis of CoSPlay

  • At comparable or larger token costs, scaled TTS baselines achieve lower Pass@1 than full CoSPlay, whose performance improves faster as inference budget increases.Ablations that remove costly components such as self-consistency and UT iteration reduce token consumption while still outperforming most baselines, and sometimes larger-budget variants.Source: Figure 11 · p. 10, §4.9 · p. 11

5 Conclusion and Future Directions … A.1.2 Support-Count Posterior

CoSPlay jointly refines code candidates and unit tests through execution-driven self-play, using pass-count signals and output consensus to select solutions. The appendix analyzes when support counts indicate correctness and how their posteriors behave as evaluator counts grow.

5 Conclusion and Future Directions

  • CoSPlay co-evolves code and unit-test pools through execution feedback, then uses output-consensus clustering to choose among codes tied at the highest pass count.It explores diverse solution strategies and failure modes to generate discriminative tests; within the most reliable consensus cluster, it selects the highest-reliability code.Source: §5 · p. 11
  • CoSPlay requires executable feedback and uses more tokens, but leads evaluated methods under comparable test-time budgets; improving efficiency and extending to broader reasoning tasks remain future directions.Source: §5 · p. 11

A.1 Analysis of Pass-Count Signals

  • The appendix models independently sampled codes and tests to explain when execution-matrix pass counts provide statistical signals of code and test quality.It introduces conditions for mutual reliability judgments and derives the posterior direction and convergence rate associated with binomial support counts.Source: §A.1 · p. 17

A.1.1 When Pass Counts Indicate Correctness

  • Pass counts discriminate correct from incorrect tests when the code pool has sufficient correctness, while distinguishing codes requires the unit-test pool to exceed a noise-dependent correctness threshold.The UT-side advantage is easier to obtain, especially when incorrect code is no more likely to pass incorrect tests than correct tests; runtime errors count as failures.Source: §A.1.1 · p. 17

A.1.2 Support-Count Posterior

  • With binomial support counts, larger counts increase posterior correctness when correct objects are more likely to gain support, have no effect when support rates match, and decrease correctness otherwise.The posterior odds ratio changes exponentially with the support count.Source: §A.1.2 · p. 18
  • As evaluator counts grow, support ratios on the side favoring the more-supported hypothesis drive posterior correctness exponentially toward 1, while ratios on the other side drive it toward 0.A unique threshold lies between the two support probabilities when they differ; at that threshold, or when the probabilities match, no exponential decision occurs.Source: §A.1.2 · pp. 18–19

A.2 Analysis of Execution-Consensus Clustering · B Detailed Algorithm Design · C Related Work

The analysis shows when output-consensus clustering identifies the correct signature, while the algorithm details specify how CoSPlay explores code and unit-test ideas, iteratively updates both pools, and selects a code cluster.

A.2 Analysis of Execution-Consensus Clustering

  • Under the stated anti-concentration condition, the correct output signature is the unique population mode for sufficiently large R, and the largest empirical cluster converges to it.The proposition assumes a positive probability of sampling a correct code, a finite signature space, and independently sampled candidates.Source: §A.2 · pp. 19–20

B Detailed Algorithm Design

  • The algorithm explores code and failure-oriented unit-test ideas, then initializes unit tests with half random valid tests and half self-consistency-generated tests.The supplied algorithm specifies problem input, code and test pool sizes, clustering input size, and a maximum number of self-play iterations.Source: B Detailed Algorithm Design · p. 21
  • During self-play, CoSPlay refreshes the execution matrix and unit-test pass rates, removes failing codes and replenishes them with newly generated candidates.The algorithm calls REFRESH after each self-play step.Source: B Detailed Algorithm Design · p. 21
  • CoSPlay regenerates the weakest unit test from its passed codes to break spurious coupling, and repairs target codes using the best test and execution feedback.Source: B Detailed Algorithm Design · p. 21
  • The algorithm replaces zero-discrimination unit tests with newly generated tests via self-consistency before completing iterative self-play.Source: B Detailed Algorithm Design · p. 21
  • After BoN filtering, CoSPlay partitions high-scoring codes by runtime-error-aware compatibility on R inputs for execution-consensus cluster selection.Runtime errors are mapped to ERR when computing outputs.Source: B Detailed Algorithm Design · p. 21

C.1 Fine-tuning Methods for Code Generation … D Detailed baseline setting

The related work distinguishes training-time methods that depend on external supervision from test-time search and self-play approaches, while the baseline section describes data, model, and inference settings used for comparison.

C.1 Fine-tuning Methods for Code Generation

  • Most code-generation fine-tuning methods use ground-truth or synthetic data, and even self-play approaches retain ground-truth dependencies or require costly RLVR training.µCode and AceCoder use supervised fine-tuning; Focused DPO and CTRL use RLVR; UTRL and CURE require ground-truth code and unit tests, respectively, while Absolute-Zero is computationally prohibitive.Source: §C.1 · p. 22

C.2 TTS for LLM Code Generation

  • Test-time code-generation methods include self-generated-test voting, structured search, and execution-guided refinement, but generated tests can be unreliable.CodeT and MPSC use self-generated unit tests for voting; SFS, CodeTree, and ThinkCoder structure search, while ORPS, S*, LDB, and Reflexion use runtime feedback.Source: §C.2 · p. 22

C.3 Self-Play in LLM

  • CoSPlay applies self-play at inference time, using execution-matrix pass counts to refine and filter code and unit-test candidates without parameter updates.Unlike CURE and UTRL, which retain ground-truth dependencies, CoSPlay does not require training-time optimization.Source: §C.3 · p. 22

C.4 Search in Natural Language Space

  • Natural-language planning and search can diversify code solutions, but existing methods are described as static and do not use dynamic execution feedback or iterative self-play.Tree of Thoughts and Reasoning via Planning search intermediate thoughts; PlanSearch generates high-level plans before implementation.Source: §C.4 · p. 22

D Detailed baseline setting

  • The RL baseline settings follow each method’s original prompts and LLM parameters, with data usage and base models listed in Table 2.The supplied table caption identifies its scope as training data source and base model for baselines.Source: D Detailed baseline setting · p. 22, Table 2 · p. 23
  • For TTS comparisons, total code and unit-test generation budgets are controlled to be comparable, and all unit tests are model-generated without external supervision.The listed methods include SFS, CodeTree, MPSC, PowerSampling, ThinkCoder, S*, and Best-of-N.Source: D Detailed baseline setting · pp. 22–23
  • The detailed TTS settings also specify S* pipeline sampling and Best-of-N code and unit-test sample counts.S* uses generated unit tests and code samples per round; Best-of-N uses N values of 16, 64, 128, and 256 for both code and unit tests.Source: D Detailed baseline setting · p. 23

E Analysis of the cost and performance balance for TTS methods

CoSPlay achieves higher pass@1 accuracy than competing TTS methods on both Qwen2.5 model scales, including against scaled baselines at comparable or larger token budgets. Its ablations retain strong accuracy at lower costs, while comparisons also show that scaling noisy self-generated feedback can hurt performance.

E Analysis of the cost and performance balance for TTS methods

  • CoSPlay achieves 37.2% pass@1 on 7B versus the strongest competing TTS baseline’s 32.0%, and 45.0% on 14B versus 40.5%.Table 5 reports per-dataset results and average performance across four datasets.Source: E Analysis of the cost and performance… · p. 23, Table 5 · p. 25
  • At 682K tokens, CoSPlay reaches 37.2% pass@1, versus 22.5% for BoN (N=256; 745K tokens) and 27.8% for ThinkCoder (Round 20; 1.16M tokens).MPSC (Scale) uses 662K tokens for 23.5% pass@1, and CodeTree (Scale) uses 638K tokens for 14.3%, supporting the comparison against scaled baselines.Source: E Analysis of the cost and performance… · p. 23, Table 3 · p. 24
  • Without self-consistency, 7B CoSPlay uses 244K tokens and reaches 34.2% pass@1; removing UT-iteration too reduces cost to 90K tokens and accuracy to 32.5%.The latter variant uses 198 calls and slightly exceeds SFS (Round 80) at 32.0%; on 14B, a lightweight variant uses 111K tokens and 202.9 calls for 41.5% pass@1, versus full CoSPlay’s 839K tokens and 763.7 calls.Source: E Analysis of the cost and performance… · p. 23, Table 3 · p. 24, Table 4 · p. 24
  • Scaling self-generated feedback can lower accuracy: S* falls from 20.5% at Round 3 to 19.2% at Round 6, and scaling MPSC from 60 to 300 code candidates drops pass@1 from 25.3% to 23.5%.CodeTree and CodeTree (Scale) also remain below direct sampling in the GT-free setting.Source: E Analysis of the cost and performance… · p. 23, Table 5 · p. 25
  • CoSPlay spends computation to improve both codes and tests: it prunes low-value codes, refreshes low-support tests, refines buggy codes with high-support non-trivial tests, and replaces zero-discrimination tests.Exploration-attack ideas target potential failure cases during UT generation, strengthening verification signals rather than merely increasing sample counts.Source: E Analysis of the cost and performance… · pp. 23, 25
  • Baseline comparisons use generous budgets: methods with built-in stopping run until convergence, while methods without it receive large sampling or search budgets to avoid under-exploration.The reported comparisons are against strong scaled configurations of existing TTS baselines; budget configurations are provided in Appendix D.Source: E Analysis of the cost and performance… · pp. 25–26

F Detailed data about generalization of our method

CoSPlay w/ Cluster improves average BoN across instruct and RLVR-tuned backbones, with gains depending on whether the initial Code-UT signal is useful but not saturated.

F Detailed data about generalization of our method

  • Averaged across evaluated backbones, CoSPlay w/ Cluster improves BoN from 37.8% to 45.3%, a +7.5% absolute gain.The evaluation spans models from 7B scale to DeepSeek-V3.2-685B.Source: F Detailed data about generalization… · p. 26
  • Seed-Coder-8B-Instruct BoN rises from 32.3% to 42.3%, while Absolute-Zero-7B rises from 24.0% to 30.2%, showing gains across instruction-tuned and RLVR-tuned backbones.The paper reports similar gains for other mid-sized and RLVR-tuned models, including DeepSeek-Coder-V2-Lite-Instruct-16B and AceCoder variants.Source: F Detailed data about generalization… · p. 26
  • DeepSeek-V3.2-685B average BoN improves from 65.7% to 68.2%, with CodeForces BoN increasing from 39.3% to 50.0%.The authors attribute the remaining opportunity to difficult tasks with plausible but incorrect candidate solutions.Source: F Detailed data about generalization… · p. 26
  • Self-play can hurt when performance is already near-optimal or the backbone is too weak to generate reliable Code-UT signals.On LiveBench, Gemini-2.0-Flash falls from 68.0% to 64.7% after CoSPlay and recovers only to 66.0% with Cluster; DeepSeek-V3.2-685B drops from 85.1% to 78.4%. On CodeForces, Absolute-Zero-7B with CoSPlay w/ Cluster remains below the 11.3% baseline BoN. The authors conclude that CoSPlay is most effective when the initial signal is useful but not saturated.Source: F Detailed data about generalization… · p. 26

G Dataset Overview · H Detailed Ablation Results

The evaluation draws on four competitive or contamination-controlled coding benchmarks, with dataset-specific sampled subsets for three of them; Table 6 defines measures for cross-model generalization.

G Dataset Overview

  • LiveBench contributes 128 coding problems for evaluating code reasoning and functional correctness under a frequently updated, objectively checkable benchmark [37].The benchmark spans multiple task domains and is designed to limit contamination.Source: G Dataset Overview · p. 26
  • LiveCodeBench v2 supplies 511 contest problems released from May 2023 to May 2024, supporting temporally controlled, contamination-aware evaluation [12].Problems come from LeetCode, AtCoder, and CodeForces and were validated by many participants.Source: G Dataset Overview · p. 26
  • CodeContests contributes 239 problems sampled from the CURE-provided dataset, which combines Codeforces, Description2Code, and APPS [18].Source: G Dataset Overview · p. 26
  • CodeForces evaluation uses 467 randomly sampled problems below difficulty level 2 from open-r1/codeforces, provided by CURE.The dataset includes custom checkers for multiple valid outputs and additional challenging tests.Source: G Dataset Overview · p. 26
  • Table 6 defines cross-model generalization measures for unit-test accuracy, code accuracy, UT signal quality, and final selected-code BoN accuracy.It compares these measures with each model’s standard BoN baseline; bold indicates the best performance, but the supplied caption gives no values.Source: Table 6 · p. 27

H.1 Ablation Variant Descriptions · H.2 Analysis · I Random UT Initialization Enhances UT Diversity

The ablations support CoSPlay’s joint refinement of code and unit-test quality, while random valid UT initialization helps preserve UT diversity across self-play rounds.

H.1 Ablation Variant Descriptions

  • The ablation set tests idea exploration, self-play stages, UT initialization and validation, UT and code resampling, and targeted UT selection against direct resampling.Variants also remove UT-side updates, disable self-consistency, combine removal of self-play and self-consistency as a lower bound, or retain zero- or full-pass UTs.Source: §H.1 · p. 27

H.2 Analysis

  • Ablations show that CoSPlay’s self-play depends on progressively refined UTs and targeted code refinement to improve both UT and code quality.Removing first-stage idea exploration also lowers BoN despite potentially higher raw code accuracy, consistent with exploration improving UT discrimination rather than naive code quality.Source: §H.2 · p. 28, Table 7 · p. 28
  • Removing Stage 1 lowers BoN despite potentially increasing raw code accuracy, indicating that idea exploration chiefly improves UT discrimination.The ablation removes solution-aware idea exploration and generates codes and UTs directly.Source: §H.1 · p. 27, §H.2 · p. 28
  • Removing random UT initialization, self-consistency, or UT regeneration hurts BoN; retaining all-fail or all-pass UTs causes severe or reduced performance, respectively.All-pass tests are trivial and non-discriminative, while all-fail tests provide unreliable UTs.Source: §H.2 · p. 28
  • Removing code refinement or 0%-pass code resampling degrades code accuracy and BoN, while replacing the targeted non-trivial Best UT with a random UT weakens refinement.The variants respectively omit code fixing, retain codes that every UT fails, or use a random UT for code fixing.Source: §H.1 · p. 27, §H.2 · p. 28

I Random UT Initialization Enhances UT Diversity

  • Random valid UT initialization yields higher UT rank than no random initialization across all self-play rounds and helps preserve broader UT diversity.Both variants drop after the first update, but the full method recovers better in later rounds; random UTs complement attack-oriented tests and prevent a narrow set of failure-focused cases.Source: I Random UT Initialization Enhances UT… · p. 28, Figure 12 · p. 28

J Detailed data of UT pass count distribution about Idea-Level Exploration vs. direct sample

At UT initialization, Exploration-Attack produces a more accurate and discriminative UT pool than direct sampling, reducing zero-pass tests and shifting UTs toward moderate pass counts.

J Detailed data of UT pass count distribution about Idea-Level Exploration vs. direct sample

  • Exploration-Attack reduces zero-pass UTs and shifts the initialization distribution toward roughly 2–10 code candidates passing, making UTs more accurate and discriminative.The w/o exp-atk variant has a pronounced concentration of zero-pass UTs across all four datasets; the full method produces a more uniform spread that differentiates among code solutions.Source: J Detailed data of UT pass count… · pp. 28–29, Figure 13 · p. 29

K Scalability Analysis of Random Input Size for Clustering

CoSPlay’s execution-consensus selection generally improves or stabilizes as it uses more random valid inputs for clustering, though gains are not strictly monotonic.

K Scalability Analysis of Random Input Size for Clustering

  • The analysis uses Qwen2.5-7B-Instruct with BoN (N = 16), varying the number of random valid clustering inputs k from 0 to 64.Source: K Scalability Analysis of Random Input… · p. 29
  • Increasing clustering inputs generally improves or stabilizes performance across four datasets, with especially visible gains on LiveBench and LiveCodeBench.More inputs provide a more reliable execution signature for grouping functionally equivalent candidates and separating spurious agreements.Source: K Scalability Analysis of Random Input… · p. 29
  • Performance gains are not strictly monotonic, but larger k does not cause a collapse and overall makes selection more stable.Random inputs can differ in discriminative strength; the results indicate the clustering stage can exploit additional test-time execution budget.Source: K Scalability Analysis of Random Input… · p. 29

L Variance analysis … New generated UT for CoSPlay-14B

This section examines how execution signals and Code–UT interactions support code refinement, UT evaluation, and candidate selection across self-play rounds and CoSPlay-7B/14B examples. It also analyzes how these patterns vary by dataset, task, and model scale.

L Variance analysis

  • Across most datasets and metrics, results show consistently small standard deviations over three independent runs, indicating stable improvements rather than random sampling variation.Table 8 reports mean accuracy and population standard deviation as percentages.Source: L Variance analysis · p. 29, Table 8 · p. 30

M Detailed Evolution of Cluster-Size Density for Correct and Incorrect Codes

  • Correct codes tend to occupy larger execution-signature clusters than wrong codes, especially on CodeContests and LiveCodeBench; greater overlap on CodeForces limits cluster size as a standalone correctness proxy.The separation is most useful when enough correct or near-correct candidates are generated to form stable clusters.Source: M Detailed Evolution of Cluster-Size… · p. 31, Figure 15 · p. 31

N Detailed Evolution of UT Pass-Count Density; O Detailed Evolution of Code Pass-Count Density

P Detailed Evolution of average true accuracy vs. cluster size

  • Across datasets and model scales, larger code clusters correspond to higher average true accuracy throughout self-play, supporting output consensus as a GT-free proxy for correctness.High-accuracy candidates increasingly concentrate in larger clusters, while small clusters are more likely to contain wrong or unstable solutions.Source: P Detailed Evolution of average true… · p. 40, Figure 39 · p. 40

Q Detailed Breakdown of the Relationship between UT Pass Count and True Accuracy; R Detailed Breakdown of the Relationship between Code Pass Count and True Accuracy

S Average evolution of the UT and code pass-count distributions across iterative self-play rounds

  • Across four benchmarks and both model scales, UT and code pass-count distributions shift toward higher counts during self-play, consistent with iterative refinement producing stronger, more mutually informative pools.For UTs, low-count density falls as overly hard, invalid, or spuriously coupled tests are removed or refreshed; for codes, more candidates pass a larger fraction of the evolved UT pool.Source: S Average evolution of the UT and code… · p. 45, Figure 51 · p. 45, Figure 52 · p. 45

T.1 Detailed Signal accuracy evolution during self-play rounds

  • Signal Accuracy rises across datasets and rounds for both model scales, with + Cluster consistently exceeding BoN-only selection; 14B is higher, while CodeForces remains most challenging.Signal Accuracy is the BoN accuracy of CoSPlay-generated UTs when selecting Qwen2.5-7B-Instruct-generated code candidates.Source: §T.1 · pp. 46–47, Figure 55 · p. 46

T.2 Detailed BoN evolution during self-play rounds; T.3 Detailed Code accuracy evolution during self-play rounds; T.4 Detailed UT accuracy evolution during self-play rounds

  • BoN and code accuracy improve across rounds, with the largest gains early; UT accuracy surges from Round 0 to Round 1 and then largely plateaus, while 14B generally outperforms 7B.CodeForces has the lowest BoN scores and code accuracy; UT accuracy shows its largest initial jump there, from the lowest to the highest accuracy across datasets.Source: Figure 56 · p. 46, Figure 57 · p. 46, Figure 58 · p. 46

Fixed Code from CoSPlay-7B

  • The balance-scale UT exposes a reciprocal-sum bug: the code outputs Right instead of Left, and refinement replaces reciprocal sums with total-weight comparisons.For A = 10, B = 10, C = 1, and D = 1, the correct totals are 20 and 2; the corrected code compares A + B with C + D.Source: Fixed Code from CoSPlay-7B · p. 49, Fixed Code from CoSPlay-14B · p. 52

Fixed Code from CoSPlay-14B

  • The monkey-seating UT exposes a one-row capacity error: with m = 10, a = 4, b = 4, and c = 11, the expected answer is 19, and refinement uses 2 * m - (a + b).The buggy calculation uses m - (a + b), treating the two-row classroom as one row; the corrected remaining-seat calculation allows all 11 no-preference monkeys to sit.Source: Fixed Code from CoSPlay-14B · p. 52

Erroneously coupled UT for CoSPlay-7B

New generated UT for CoSPlay-14B

U.2.3 Analysis for case study of UT-code coupling … Prompt for direct UT generation

CoSPlay’s case studies show how Step 2 breaks misleading agreement between incorrect code and self-generated tests by replacing suspicious tests, while the prompt suite structures code and test generation, regeneration, and repair. The examples show that this can restore useful discrimination even when a regenerated test is not yet oracle-correct.

U.2.3 Analysis for case study of UT-code coupling

  • The 7B wrong code passes a suspicious low-pass UT, while the 14B wrong/BoN-selected code passes a low-pass, non-unique UT; regeneration or resampling makes each code fail its replacement.These examples illustrate distinct suspicious UT patterns that can spuriously support wrong candidates.Source: §U.2.3 · p. 59
  • Replacing a suspicious low-but-nonzero-pass-rate UT while keeping the rest of the pool unchanged removes a false positive and restores a more discriminative execution-matrix signal.Figure 60 illustrates this intervention for CoSPlay-7B and CoSPlay-14B.Source: Figure 60 · p. 58, §U.2.3 · p. 60
  • In rectangle-overlap counting, flawed code and the initial UT both output 3 despite a true maximum of 4; resampling breaks this coupling, though the regenerated UT remains imperfect.Source: §U.2.3 · p. 60

Prompt for solution hints generation; Prompt for specific solution idea generation; Prompt for code generation from solution idea

Prompt for UT attack idea generation; Prompt for UT input generation from UT attack idea

Prompt for UT output generation from specific input; Prompt for generating non-coupling UT

Prompt for code fixing

  • The code-fixing prompt directs the model to diagnose runtime or logical failures, identify the root cause, and repair the general logic rather than hardcode a failed case.It requests an explanation of the bug and fully runnable refined code.Source: Prompt for code fixing · pp. 64–65

Prompt for random UT input generation; Prompt for direct code generation; Prompt for direct UT generation

Related papers

Loading 2605.23491v2…