PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

Jiacheng Xu, Feng Chen, Xiuneng Xu, Bo An

arXiv:2609.09135v1cs.LGcs.CL

TL;DR

Code-generation TTRL lacks canonical answers and trusted tests for constructing reliable rewards. The paper addresses this with output-free probes, PCR, and ERPO’s conservative rank-masked updates; ERPO substantially improves pass@1 and pass@k in in-domain and transfer settings, though evaluation remains limited to competitive-programming benchmarks.

  • Problem

    Code-generation TTRL lacks canonical answers, private test cases, and ground-truth outputs for constructing reliable rewards.

  • Method

    Probe-driven TTRL generates output-free probes, scores candidate-program behavioral agreement with PCR, and uses ERPO to suppress low-consensus candidates while controlling entropy.

  • Results

    ERPO substantially improves both pass@1 and pass@k in in-domain adaptation and zero-shot transfer across coding benchmarks.

  • Takeaways & Limitations

    The adapted model supports lower-cost single-pass responses and a stronger search performance frontier in latency-sensitive and quality-sensitive deployment scenarios.

  • Takeaways & Limitations

    Evaluation is limited to competitive-programming benchmarks and does not assess project-level or industrial-scale code generation.

Abstract

from arXiv · show

Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.

1 Introduction

Code-generation TTRL lacks canonical answers for self-voting, so rewards must compare program behavior. Probe-driven TTRL supplies behavioral rewards, while ERPO conservatively suppresses low-consensus candidates to improve both pass@1 and pass@k.

  • Code outputs are open-vocabulary programs, so exact-match voting cannot directly construct rewards for TTRL.
  • Probe-driven TTRL generates output-free, problem-specific probes and derives Probe Consensus Reward from candidate behavior without oracle outputs.
  • ERPO uses rank-masked negative PCR updates and an entropy ceiling to avoid reinforcing spurious majorities and stabilize sampling.
  • PCR is more reliable for identifying low-quality candidates than for certifying high-consensus programs as correct.
  • ERPO improves both pass@1 and pass@k, with gains transferring across coding benchmarks.

2 Preliminaries

Code-generation TTRL needs a surrogate reward because target-time inputs and oracle outputs are unavailable, and programs lack comparable canonical answers. The paper frames probe-based behavioral comparison as the route to usable policy updates without trusted tests.

  • RLVR uses verifier-derived scalar rewards to estimate advantages for sampled outputs.
  • TTRL adapts models on unlabeled test-time tasks and therefore must construct surrogate rewards when trusted verification is unavailable.
  • ERPO consists of probe-input generation, Probe Consensus Reward scoring, and a conservative policy update.
  • For code, behavioral comparison is required because semantically equivalent programs can have unrelated surface forms.
  • A reliable code reward depends on complete tests and trusted oracle outputs, which target-time TTRL lacks.

3 Method

The method constructs probe-based behavioral rewards for code programs without oracle tests, then uses PCR conservatively in ERPO to suppress low-consensus candidates while controlling entropy growth.

  • Probe-driven TTRL: Probe-driven TTRL generates distinct, output-free inputs from each problem statement and reuses them as shared execution points for sampled programs.The probes expose behavior without oracle outputs and provide inputs for reward construction during adaptation.
  • Probe Consensus Reward: PCR scores each candidate by the fraction of probes on which its valid output belongs to the probe-wise majority set.Failed executions receive no credit, and tied majority outputs are all credited.
  • Probe Consensus Reward: PCR measures behavioral agreement rather than correctness, so high consensus can reflect shared bugs or missed corner cases, while correct rare behavior can receive a low score.The method therefore treats PCR as a conservative training signal rather than a verifier.
  • Rank-masked optimization: ERPO ranks candidates by group-normalized PCR advantage, masks the top half, and retains lower-ranked signed advantages that usually reduce low-consensus program likelihood.High-PCR candidates are excluded from positive policy-gradient updates.
  • Entropy control: ERPO adds a fixed entropy ceiling that penalizes only violations, suppressing excessive entropy growth while leaving lower-entropy updates unpenalized.The combined objective is L_ERPO = L_PCR + L_H.

4 Experiments

Experiments show that ERPO improves both single-pass accuracy and the pass@k frontier, with gains transferring across coding benchmarks. PCR is useful mainly as a negative signal because low consensus strongly predicts failure, whereas high consensus does not certify correctness.

  • Evaluation protocol: ERPO adapts on unlabeled problem statements, while evaluation uses hidden test inputs and oracle outputs only post hoc to compute pass@k.The transfer setting adapts on LiveCodeBench and evaluates other benchmarks without target-benchmark updates.
  • Main results: ERPO improves both pass@1 and pass@16 on in-domain LiveCodeBench adaptation and transfers gains to CodeContests, CodeForces, and TACO.It is the only adaptation method reported to substantially improve both single-pass accuracy and repeated-sampling coverage.
  • PCR analysis: PCR has AUC 0.790 for Qwen3-4B and 0.750 for Qwen3-8B, but candidates with s ≤0.5 fail hidden tests more than 91% of the time.High PCR remains unreliable: s = 1.0 candidates are wrong 44.2% and 51.9% of the time for the two model sizes.
  • Optimization: ERPO uses PCR conservatively by masking high-ranked candidates and applying negative updates to lower-ranked, low-consensus samples.This avoids treating high-PCR candidates as positive correctness labels when groups may share spurious consensus.
  • Inference-time scaling: ERPO substantially improves pass@k at every evaluated k, whereas NSRPCR’s gains grow with k but remain limited at pass@1.NSRPCR’s pattern suggests sampling-diversity gains rather than improved single-pass accuracy.
  • Inference-time scaling: ERPO reaches 36.2% with one sample on LiveCodeBench, while both Best-of-n selectors remain below 30% even at n = 32.The comparison indicates that selectors do not realize much of the oracle pass@k headroom, whereas adaptation changes the generation distribution directly.
  • Hyperparameter ablation: With Hceil values {0.2, 0.5, 1.0}, all settings improve pass@1 and pass@16, while Hceil=0.5 best balances training efficiency and stability.Lower ceilings slow training; overly high ceilings accelerate early gains but degrade after their peak.

5 Related Work

Related work uses execution behavior to compare code programs and inference-time search to select among sampled solutions. This paper extends that direction to probe-driven test-time policy optimization, while addressing the risk that optimization reinforces incorrect consensus.

  • Test-time reinforcement learning: Code-generation TTRL cannot directly use answer-level exact-match voting because programs are open-vocabulary outputs without canonical surface forms.Behavioral comparison through execution is therefore used instead of program-text matching.
  • Execution-based methods: Prior code-generation methods use execution traces, functional consensus, generated tests, or shared inputs to compare and select candidate programs.These approaches provide execution-based signals for inference-time generation, selection, or training.
  • Motivation: Selection alone cannot improve the model’s generation capability and remains bounded by pass@k under the corresponding generation procedure.Test-time policy optimization can change the generation distribution, but may propagate incorrect consensus across updates.

6 Conclusion

The paper addresses two obstacles to code-generation TTRL: open-vocabulary programs lack directly comparable canonical answers, and pass@1-focused objectives can weaken the pass@k frontier. Probe-driven TTRL and ERPO address these issues by using output-free probes, PCR, conservative updates, and entropy control.

  • Contributions: Probe-driven TTRL constructs output-free probe inputs and uses Probe Consensus Reward to adapt code models without oracle outputs or directly comparable responses.Candidate programs are executed on generated probes, and behavioral consensus supplies the training signal.
  • Contributions: ERPO applies rank-masked updates and entropy-ceiling control to optimize the noisy PCR signal conservatively.The method avoids treating high PCR as a correctness certificate while controlling policy drift.
  • Results: ERPO substantially improves both pass@1 and pass@k, producing stronger single-pass responses and a better frontier for inference-time search.The adapted model is positioned as a stronger base policy for both latency-sensitive and quality-sensitive deployment scenarios.
  • Implications: The framework is presented as practically relevant for deployment because it supports higher-quality single-pass responses without relying only on costly inference-time search.It also improves the performance frontier available under repeated sampling.

Limitations

The evaluation is limited to competitive-programming-style coding benchmarks. Project-level and industrial-scale code generation remain unevaluated because they require substantially more infrastructure and computational resources.

  • Scope: The empirical evaluation covers competitive-programming-style coding benchmarks rather than project-level code generation.The authors state that broader executable code-generation tasks are possible in principle, but are not evaluated here.
  • Scope: Project-level and industrial-scale scenarios require more engineering infrastructure and computational resources for reliable execution, training, and evaluation.The paper leaves these settings for future work.

A Experimental Configuration

The experiments use Qwen3-4B and Qwen3-8B with shared RL infrastructure and sampling settings, while varying reward sources, losses, and method-specific hyperparameters. The study adapts on unlabeled coding problems, evaluates transfer, and implements probe generation and execution under explicit format and resource constraints.

  • Infrastructure: RL experiments use the verl framework with vLLM rollouts, an FSDP actor, and Qwen3-4B or Qwen3-8B backbones in non-thinking mode.Methods share optimization and sampling settings unless method-specific changes are noted.
  • Optimization: Training uses AdamW with a 5 × 10^-7 learning rate after 20 warmup steps, 16 prompts, 16 rollouts per prompt, and up to 150 global steps.Each step therefore contains 256 on-policy rollouts.
  • Sampling: Sampling uses top_p = 0.8, top_k = 20, max_response_length = 8192, and 16 validation samples for reported P@1 and P@16.The same sampling profile is shared by RL and validation rollouts.
  • Probe construction: Probe-based methods reuse precomputed probe sets, add 10 generated probes per problem, and compute PCR over their deduplicated union with example inputs.Probe generation iteratively uses the problem statement and prior probes while requiring valid, distinct raw inputs no longer than 5000 characters.
  • Reward and update variants: GRPOPub uses public-binary rewards, GRPOPCR directly uses PCR as a scalar reward, NSRPCR retains samples with PCR below 0.5, and ERPO masks the top half of each group.ERPO additionally uses λH = 0.5 and Hceil = 0.5 for entropy control.
  • Execution: Programs run in isolated subprocesses with timeouts and resource limits; compilation errors, runtime errors, and timeouts receive no credit for the corresponding probe.Network access is disabled, and failed executions are excluded from that probe’s majority computation.
  • Evaluation: The main transfer protocol adapts on unlabeled LiveCodeBench problems and evaluates zero-shot on CodeContests, CodeForces, and TACO.An additional experiment adapts Qwen3-4B on CodeContests and reports in-domain and zero-shot transfer results.

D Training Trajectory Analysis

Training trajectories show that ERPO sustains joint improvements in pass@1 and pass@16 while keeping entropy stable, avoiding the narrower distributions or sampling-diversity trade-offs seen in other methods. Transfer experiments likewise report ERPO as the most consistent method across in-domain and cross-benchmark metrics.

  • Training dynamics: GRPOPub improves pass@1 but keeps pass@16 near the Base reference, while direct PCR optimization reduces pass@16 and fails to improve pass@1.These trajectories are consistent with high PCR not reliably certifying correctness.
  • Training dynamics: ERPO sustains improvements in pass@1 and pass@16 while keeping entropy stable.Its rank mask avoids positive updates on high-PCR candidates, and the entropy ceiling limits upward entropy drift.
  • Inference-time scaling: ERPO leads at every reported sampling budget on Qwen3-8B and its advantage over Base grows with k.Its pass@1 also exceeds the Base model’s pass@16 with a single output.
  • Robustness: Three independent Qwen3-4B runs are evaluated in-domain on LiveCodeBench and zero-shot on CodeContests, CodeForces, and TACO using mean and standard deviation.The repeated-run setup assesses robustness across adaptation methods.

G Results on More Benchmarks

ERPO consistently improves both pass@1 and pass@16 across additional in-domain benchmarks and remains stable across a broad range of rank-mask percentiles and probe counts. Using ten generated probes provides a practical balance between PCR reliability, downstream performance, and execution cost.

  • Additional benchmark results: ERPO consistently improves both pass@1 and pass@16 on BigCodeBench and DS-1000.Both benchmarks use the same Qwen3-4B backbone, adaptation budget, and optimization settings as the main experiments.
  • Rank-mask sensitivity: Performance remains relatively stable across rank-masking ratios from 25% to 75%, with all three settings substantially improving over the corresponding Base models.Masking only 10% still improves over Base but performs worse than masking 25%–75%.
  • Probe-count sensitivity: PCR AUC improves as the number of generated probes increases, but the gain diminishes beyond Ngen = 10.Doubling the probe count from 10 to 20 increases AUC by only 0.009.
  • Probe-count sensitivity: Performance remains stable for Ngen ∈ {5, 10, 20}, and increasing Ngen from 10 to 20 does not yield consistent improvements despite added execution cost.These results support Ngen = 10 as a practical balance between reward reliability, downstream performance, and computational cost.

J Probe Examples and Hidden-Test Correctness

Probe examples show that disagreement can reveal incorrect programs, while unanimous probe agreement can miss narrow hidden-test failures. This asymmetry motivates using PCR as negative evidence rather than positive supervision.

  • Case A: Probe disagreement: In Case A, probe outputs form four clusters, and the largest cluster contains all seven candidates that pass the hidden tests.The other three clusters contain nine candidates, all of which fail.
  • Case A: Probe disagreement: In Case A, PCR perfectly separates passing and failing candidates across the ten generated probes.All seven passing candidates receive PCR = 1.0, while all nine failing candidates receive lower PCR.
  • Case B: Unanimous agreement: In Case B, all 16 candidates receive PCR = 1.0 because they agree on ten probes, yet two fail a hidden boundary input.The probes miss the narrow region around 2^29 where the incorrect candidates should output inf.
  • Implication for ERPO: The examples support ERPO’s asymmetric use of PCR: low consensus provides useful negative evidence, whereas high consensus is not treated as positive supervision.This design addresses the possibility that generated probes produce spurious consensus.
Loading 2609.09135v1…