PaperlayerView exact revision on arXiv

Source-linked AI summary

TIPCODER: Reinforcement Learning Boosted Test-time Instruction Proposer for Code Generation

Minyu Chen, Sihao Wu, Ling-I Wu, Song Qin, Jingyang Li, Lei Ning, Jianxin Xue, Guoqiang Li

arXiv:2609.03309v1cs.SE

TL;DR

Code generation remains sensitive to omissions and misleading interpretations in the original instruction, while fixed prompts may miss instance-specific guidance. TIPCODER proposes tips from debugging trajectories, generates base and tip-guided candidates for RM selection, and reports consistent improvements over sampling and prompt-optimization baselines across evaluated benchmarks and backbones.

  • Problem

    Code LLM outputs remain sensitive to omitted constraints, under-specified boundary cases, and misleading reasoning paths, while generic or dataset-level prompts can miss localized instance-specific failure modes.

    Source: §1 · p. 1

  • Method

    TIPCODER trains a Proposer on debugging-derived guidance with SFT and marginal-utility RL, then generates base and tip-guided code candidates for post-hoc Reward Model selection.

    Source: §1 · pp. 1–2, §2.3 · p. 4

  • Results

    TIPCODER consistently improves pass rates over base generation and stochastic sampling and outperforms self-hinting and generic context-optimization baselines under shared RM reranking; the trained Proposer transfers to four target backbones and is competitive with larger sampling budgets on evaluated Python benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Instance-level instruction proposing is a complementary test-time scaling direction to solution-space sampling, implemented as a black-box module without target-model parameters, gradients, or runtime execution feedback.

    Source: §1 · p. 2, §5 · p. 9

    TIPCODER adds inference cost for tip generation and a tip-guided candidate, and realized gains depend on Reward Model quality; its guidance may reflect its specific rollout and debugging-data pipeline.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Test-time scaling for code generation typically explores the solution space by sampling multiple programs from a fixed instruction. We study a complementary direction: instance-level instruction-space exploration. Our observation is that many coding failures stem from missing constraints, overlooked edge cases, or misleading reasoning paths induced by the original prompt. To address this, we propose TipCoder, a test-time instruction proposer that generates problem-specific auxiliary tips before code synthesis. TipCoder distills multi-turn debugging trajectories into proactive guidance and further optimizes the Proposer with reinforcement learning using a marginal-utility reward. At inference time, it generates both a base solution and a tip-guided solution, and applies a Reward Model for post-hoc selection. This exploration-selection design allows tips to expose additional candidate potential while reducing regressions from unnecessary guidance. Across the evaluated code-generation benchmarks and target Code LLMs, TipCoder provides a consistent instruction-level test-time scaling strategy, comparing favorably with stochastic sampling and generic prompt optimization baselines under a shared reward-model-based selection protocol.

1 Introduction

Code-generation outputs can remain sensitive to missing constraints, overlooked edge cases, and misleading interpretations in the original instruction. TIPCODER explores a complementary scaling axis by proposing instance-specific tips, training the Proposer from debugging trajectories, and selecting between base and tip-guided candidates; it reports consistent pass-rate improvements over sampling and prompt-optimization baselines under shared reranking.

1 Introduction

  • Fixed-prompt sampling can remain concentrated around a flawed interpretation, while instance-specific failures may hinge on localized constraints or edge cases that generic prompts miss.Examples include boundary cases, index conventions, library behavior, numerical constraints, and specification ambiguity.Source: §1 · p. 1
  • TIPCODER generates a problem-specific auxiliary tip before synthesis, then has the target Code LLM produce both a base solution and a tip-guided candidate.The tip changes the conditioning context to explore an alternative solution-space region rather than specifying the solution.Source: §1 · p. 1
  • The Proposer learns proactive tips from multi-turn debugging trajectories, then receives SFT initialization and RL optimization with a reward for marginal tip utility.The trajectories expose missing requirements or incorrect assumptions through failed solutions and subsequent repairs; the reward penalizes harmful or redundant perturbations.Source: §1 · pp. 1–2
  • TIPCODER consistently improves pass rates over base generation and stochastic sampling, and outperforms self-hinting and generic context-optimization baselines under the same RM reranking protocol.The authors attribute the gains to more effective instance-level instruction proposals rather than merely adding a candidate selector.Source: §1 · p. 2
  • TIPCODER uses RM-based reranking to approximate instance-wise selection between base and tip-guided candidates, and evaluates against sampling, self-hinting, and context-optimization baselines under the same protocol.The contribution frames instruction proposing as a complementary scaling axis to fixed-prompt sampling and dataset-level prompt optimization.Source: §1 · p. 2

2 Methodology

TIPCODER converts debugging traces into problem-specific tips, trains a proposer to improve downstream pass rates, and reranks base and tip-guided candidates.

2.1 Problem Formulation

  • The method treats a debugging-derived tip as a latent instruction c, learned from problem x to maximize expected downstream pass rate.The tip is intended to capture useful signals from multi-turn debugging without requiring a runtime debugging loop.Source: §2.1 · p. 2

2.2 Tip Supervision from Debugging Trajectories

  • Debugging-derived tips convert masked observations of repaired failures into proactive guidance, retaining only tips that pass the student’s unit tests.The pipeline yields 2,126 high-quality problem-tip pairs for supervised initialization.Source: §2.2 · pp. 3–4

2.3 Proposer Optimization

  • SFT initializes the proposer from verified tips, then RL optimizes downstream correctness using a marginal-utility reward relative to base generation.The reward discounts redundant tips on already-solved tasks and can be negative when a tip regresses relative to the base branch.Source: §2.3 · p. 4

2.4 Reward-Guided Reranking Inference

  • At inference, a black-box proposer creates base and tip-guided candidates, and Reward Model reranking selects between them to limit regressions from noisy tips.The method requires no target-model parameters, gradients, or execution feedback during inference.Source: §2.4 · p. 5

3 Experiments

Across four target backbones, TIPCODER-RL achieves the best average performance, while analyses show that instruction-guided candidates and post-hoc selection contribute separately to gains. Results also compare its performance and cost with Best-of-N sampling and examine which tip types repair or harm solutions.

3.1 Setup

  • Evaluation covers HumanEval+, MBPP+, and BigCodeBench-Instruct using Pass@1, with held-out tests unavailable to TIPCODER during inference and a shared Reward Model for selected candidates.The same trained Proposer is applied across four target Code LLM backbones without target-specific fine-tuning.Source: §3.1 · pp. 5–6

3.3 Disentangling Exploration from Selection

  • TIPCODER-RL often raises the oracle upper bound, exposing complementary correct candidates, while stronger code-oriented Reward Models convert more of that potential into final pass-rate gains.The oracle-to-selected-performance gap reflects remaining selection bottlenecks; raw adoption can choose harmful candidates.Source: §3.3 · pp. 6–7, Figure 2 · p. 8

3.2 Main Results

  • TIPCODER-RL achieves the best average performance on all four backbones, improving over each backbone’s strongest non-TIPCODER baseline by 1.81, 0.92, 0.99, and 0.47 points, respectively.Its average scores are 61.17 on DeepSeek-Coder-7B, 65.27 on DeepSeek-Coder-V2-Lite, 68.64 on Qwen2.5-Coder-7B, and 71.10 on Qwen3-Coder-30B-A3B.Source: §3.2 · p. 6, Table 1 · p. 7

3.4 Cost-Normalized Best-of-N Scaling

  • On HumanEval+, TIPCODER-RL reaches 88.21 Pass@1 with 2,812 tokens and 0.99 seconds versus Best-of-8’s 84.15, 4,610 tokens, and 1.94 seconds; it also exceeds Best-of-8 on MBPP+ with fewer tokens and falls between Best-of-4 and Best-of-8 on BigCodeBench-full.On BigCodeBench-full, TIPCODER-RL reaches 43.89, exceeding Best-of-4 at 42.54; Best-of-8 reaches 44.56.Source: §3.4 · p. 7

3.5 Systematic Analysis of Generated Tips

  • Across 6,504 evaluated instances, selected tips yield a net gain of 193 corrected instances, equivalent to 2.97 micro-averaged Pass@1 points.Semantic/constraint clarification accounts for 43.2% of positive changes and a net gain of 109 instances.Source: §3.5 · p. 7
  • Boundary-case, index/order/output-alignment, and algorithmic tips are net-positive, while numerical/formula/precision guidance is net-negative and type/parsing/API guidance is balanced.The examples also show that the selector is useful but imperfect.Source: §3.5 · p. 8

4 Related Work

Related work spans iterative refinement using feedback and prompt optimization as inference-time scaling. TIPCODER targets the underexplored code-generation setting with instance-level instruction search.

Iterative Refinement with Code LLMs

  • Iterative refinement methods use feedback to correct generations, with later code approaches incorporating execution signals or training refinement into model trajectories.TIPCODER instead distills foresight from iterative repair into a proactive tip for one inference pass without runtime execution.Source: §4 · p. 8

Prompt Optimization and Scaling

  • Prompt optimization treats instruction generation as a search problem, and prompt diversity can unlock varied reasoning paths.The cited work also relates instruction-space compute to gains comparable to increasing model size.Source: §4 · pp. 8–9
  • Systematic instruction-level scaling remains underexplored in code generation, where methods typically rely on stochastic solution sampling.TIPCODER introduces test-time instruction proposing for code generation.Source: §4 · p. 9

5 Conclusion

TIPCODER explores instruction space as a complement to solution-space sampling, using a proposer trained with SFT and RL. Across evaluated Python benchmarks, it transfers to four target backbones and is competitive with larger sampling budgets under reward-model selection.

5 Conclusion

  • TIPCODER is a black-box test-time framework that complements solution-space sampling by exploring instruction space with auxiliary instructions trained via SFT and RL.It requires no target-model parameters, gradients, or runtime execution feedback.Source: §5 · p. 9
  • Across evaluated Python benchmarks, the same trained Proposer transfers to four target backbones and is competitive with larger sampling budgets under RM-based selection.The conclusion identifies broader model families, programming languages, and repository-level tasks as future evaluation directions.Source: §5 · p. 9
  • Future work includes training with multiple rollout environments, improving post-hoc selection, and evaluating multilingual and repository-level code tasks.Source: §5 · p. 9

Limitations

TIPCODER's limitations concern inference cost, reward-model selection, and the training environment's influence on guidance.

Limitations

  • TIPCODER adds inference cost by generating an auxiliary tip and a tip-guided code candidate before reward-model selection.Source: Limitations · p. 9
  • Realized gains depend on post-hoc Reward Model quality, since an imperfect selector may fail to choose better candidates exposed by learned tips.Source: Limitations · p. 9
  • Although the Proposer transfers across tested target models, its guidance may reflect its specific rollout environment and debugging-data construction pipeline.Broader model families, languages, and repository-level tasks remain future work.Source: Limitations · p. 9

A Best-of-N Scaling Across Target Backbones

Figure 3 compares TIPCODER-RL with RM-reranked Best-of-N sampling on MBPP+ across four target Code LLM backbones. TIPCODER-RL remains competitive with the Best-of-4-to-8 range.

A Best-of-N Scaling Across Target Backbones

B Discussion about Post-hoc Verification Strategy

The paper favors post-hoc reward-model filtering over pre-hoc refusal to preserve diverse, adaptable tip generation while selecting candidates dynamically.

B Discussion about Post-hoc Verification Strategy

  • Post-hoc reward-model filtering is favored over pre-hoc refusal to maintain proposer diversity and adaptability.The authors say refusal training can produce reward-hacking, conservative policies that suppress high-value guidance.Source: B Discussion about Post-hoc… · p. 11
  • A proposer cannot accurately predict failure without access to the generator’s internal state, and problem difficulty depends on the model.The paper uses a specialized reward model to filter candidates instead.Source: B Discussion about Post-hoc… · p. 11
  • The authors describe TIPCODER’s filtering as a dynamic, model-adaptive scaling strategy that maximizes the available potential range without parameter intervention.Source: B Discussion about Post-hoc… · p. 11

C Hyperparameter Settings

The experiments use a two-stage SFT and RL training pipeline, with settings specified for optimization, reward stabilization, and computation.

C Hyperparameter Settings

  • The training pipeline has supervised fine-tuning followed by reinforcement learning, with Table 4 identified as the comprehensive hyperparameter list.Source: C Hyperparameter Settings · p. 11, Table 4 · p. 12
  • SFT fine-tunes Qwen3-4B with Adam, a 4096-token maximum sequence length, and a 3e-5 learning rate.Source: C Hyperparameter Settings · p. 11
  • RL uses GRPO through OpenRLHF, with a 1e-6 actor learning rate, 1e-3 initial KL coefficient, group-normalized advantages, reward normalization, and a 1e-5 length penalty.Source: C Hyperparameter Settings · p. 11
  • Experiments ran on four NVIDIA A800 GPUs, reserving one for reference-model inference and using the other three for policy optimization.Global batch sizes were configured to be divisible by three for balanced data parallelism.Source: C Hyperparameter Settings · p. 11

D Case Study

A case study shows tips surfacing a variable-length constraint that changes how the generator handles list inputs; the prompt templates describe offline tip construction and online inference.

D Case Study

  • The base model assumes equal list lengths and risks an index error, whereas TIPCODER’s variable-length and index-alignment tips lead the generator to use the minimum list length and handle the edge case correctly.The example illustrates that tips surface constraints rather than provide solution code.Source: D Case Study · p. 12

E Prompt Templates

  • Offline tip construction uses both incorrect-versus-correct solution contrasts and multi-turn debugging trajectories to extract execution-based reasoning logic.The templates are organized into offline data construction and online inference workflows.Source: E Prompt Templates · p. 12
  • The online inference template has three stages: a Proposer generates a tip, a Generator synthesizes code, and an Evaluator scores quality.The prompt templates are illustrated in Figures 5, 6, and 7.Source: E Prompt Templates · p. 12

F Use of AI Assistants

The paper reports using AI assistants for writing clarity and some experimental data collection, while its accompanying examples show generated tips and prompt-template workflows.

F Use of AI Assistants

  • AI assistants were used to improve writing clarity and automate parts of experimental data collection, with the authors taking responsibility for the content.Source: F Use of AI Assistants · p. 12

[User Instruction]

  • For the three-list task, TIPCODER’s generated tips warn about unequal or empty lists and index alignment, addressing ambiguity in the original instruction.The prompt asks for items identical at the same position across three lists, and the figure caption says the baseline misses variable list lengths while TIPCODER’s warning leads to a robust solution.Source: [User Instruction] · p. 13, Figure 4 · p. 13
  • The prompt-template figures cover incorrect-versus-correct solution comparison, trajectory-based distillation, and the three-step inference pipeline.The captions describe masking debugging logs to derive high-level optimization tips from execution feedback, and an evaluator scoring the problem-code pair.Source: Figure 5 · p. 14, Figure 6 · p. 14, Figure 7 · p. 15
Loading 2609.03309v1…