PaperlayerView exact revision on arXiv

Source-linked AI summary

ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning

Juyong Jiang, Jiasi Shen, Sunghun Kim, Kang Min Yoo, Jeonghoon Kim, Sungju Kim

arXiv:2603.05863v2cs.CLcs.LGcs.SE

TL;DR

Single-pass code generation struggles on complex tasks, while existing refinement often depends on external feedback. ReflexiCoder uses reinforcement learning to internalize a structured reflection-correction trajectory, achieving state-of-the-art open-source performance across seven benchmarks while reducing iterative-mode token use by approximately 40%.

  • Problem

    Single-pass code generation faces a performance ceiling on complex algorithmic tasks, and external-feedback refinement does not internalize intrinsic debugging capabilities.

  • Method

    ReflexiCoder applies RL to a structured reflection-correction trajectory, internalizing self-reflection and self-correction without external oracles or execution engines at inference time.

  • Results

    ReflexiCoder-8B achieves state-of-the-art performance among open-source models in the 1.5B to 14B range across seven benchmarks and uses approximately 40% fewer tokens in iterative mode.

  • Takeaways & Limitations

    Optimizing internal self-debugging capabilities via RL is presented as a scalable and effective direction for reliable code LLMs.

  • Takeaways & Limitations

    The approach primarily targets algorithmic correctness and local code issues within a single-file setting, not repository-level development, long-horizon refactoring, or evolving specifications.

Abstract

from arXiv · show

While Large Language Models (LLMs) have revolutionized code generation, standard ``System 1'' approaches that generate solutions in a single forward pass often hit a performance ceiling on complex algorithmic tasks. Existing iterative refinement strategies attempt to bridge this gap at inference time, yet they predominantly rely on external oracles, execution feedback, or computationally expensive prompt-response cycles. In this work, we propose ReflexiCoder, a novel reinforcement learning (RL) framework that internalizes the structured reasoning trajectory, encompassing initial generation, bug and optimization aware reflection, and self-correction, directly into the model's weights. Unlike prior methods, ReflexiCoder shifts the paradigm from external-dependent refinement to an intrinsic, fully autonomous self-reflection and self-correction capabilities at inference time. We utilize an RL-only training paradigm with granular reward functions to optimize the entire reflection-correction trajectory, teaching the model how to debug without reliance on ground-truth feedback or execution engines at inference time. Extensive experiments across seven benchmarks demonstrate that our ReflexiCoder-8B establishes a new state-of-the-art (SOTA) among leading open-source models in the 1.5B to 14B range, achieving 94.51% (87.20%) on HumanEval (Plus), 81.80% (78.57%) on MBPP (Plus), 35.00% on BigCodeBench, 52.21% on LiveCodeBench, and 37.34% on CodeForces in a single-attempt setting, rivaling or surpassing proprietary models like GPT-5.1. Notably, our framework is significantly more token-efficient than base models, reducing inference-time compute overhead by approximately 40% through disciplined, efficient reasoning and reflection patterns. The source code and data are available at https://github.com/juyongjiang/ReflexiCoder.

1 Introduction

ReflexiCoder addresses the limits of single-pass code generation and externally dependent refinement by training LLMs to perform intrinsic self-reflection and self-correction. Its RL-optimized reflection-correction trajectory improves benchmark performance while remaining token-efficient.

  • Motivation: Complex multi-step programming tasks often exceed the reliability of single-pass code generation, producing plausible-looking but functionally incorrect solutions.Existing refinement methods include re-ranking, external repairers, and feedback-guided refinement.
  • Motivation: External signals can prevent models from internalizing the ability to scrutinize and autonomously correct their own logic.Prior workflows rely on compilers, reflection or human oracles, or other environmental interaction.
  • Approach: ReflexiCoder internalizes a structured trajectory of reasoning, code generation, bug and optimization reflection, and correction directly into model weights.The framework targets intrinsic self-reflection and self-correction at inference time without ground-truth feedback or external execution engines.
  • Approach: RL optimizes the reflection-correction trajectory itself rather than only the single-pass generation policy, teaching the model how to debug autonomously.The training uses an RL-only paradigm with granular rewards for accurate error detection and successful repair.
  • Results: 94.51% on HumanEval, 35.00% on BigCodeBench, 52.21% on LiveCodeBench, and 37.34% on CodeForces establish ReflexiCoder-8B as state of the art among open-source models in its size range.The model surpasses or remains competitive with proprietary models such as GPT-5.1.
  • Results: Approximately 40% fewer tokens in iterative mode indicate that ReflexiCoder’s gains do not come from excessive inference-time compute overhead.The authors attribute this efficiency to disciplined reflection and nearly 50% fewer reasoning tokens.

2 Methodology

ReflexiCoder formalizes code generation as an RL-optimized, format-constrained self-debugging trajectory. Its composite reward balances progressive quality improvement, reflection efficiency, cycle regulation, and strict structural compliance.

  • Structured Reasoning-Reflection Process: The training pipeline represents each interaction as contiguous reflection-answer pairs within a globally specified, format-compliant trajectory.The trajectory set requires syntax extraction to match the target structural specification.
  • Format Compliance Constraints: Format violations gate the reward to zero, making valid reasoning, initial answers, and matched revision pairs prerequisites for quality optimization.Missing segments, incorrect ordering, or unmatched reflection-answer pairs break trajectory parsing.
  • Composite Reward: The reward combines cycle-count regulation, trajectory-level quality improvement, efficiency, and formatting constraints to optimize disciplined self-correction.The components jointly balance progressive refinement and economy of iterations.
  • Cycle Count Regulation: Cycle regulation preserves reflection freedom at shallow depths and uses a bounded oscillatory penalty to promote exploration beyond the no-penalty range.The sinusoidal term periodically nudges policies away from repetitive local optima and trajectory collapse.
  • Iterative Quality Improvement: The quality objective favors trajectories with nondecreasing solution scores and assigns greater weight to later improvement stages.Exponential time-weighting produces increasing weights across iterations, prioritizing later gains.
  • Iterative Quality Improvement: The improvement signal rewards quality gains, penalizes declines and suboptimal stagnation, and avoids penalizing unchanged scores near the maximum.A smooth tanh mapping bounds rewards, while stagnation bonuses and penalties distinguish near-optimal from improvable answers.
  • Efficiency Reward: The efficiency term rewards large average quality gains per reflection, encouraging maximal improvement with minimal iterations.It addresses fixed-cycle overfitting, task difficulty, noise sensitivity, and long-horizon credit assignment.
  • Optimization: Policy updates use GRPO with group-normalized advantage estimates instead of a value function to improve stability and reduce variance in large action spaces.The formulation is identified as the policy-optimization method for ReflexiCoder.

3 Experiments

Across seven code-generation benchmarks, ReflexiCoder-8B achieves strong performance under matched single-attempt budgets and improves further with iterative self-reflection, while using fewer tokens than reasoning baselines. Ablations and analyses attribute these gains to the structured generate-reflect-correct trajectory and granular reward design.

  • Main Results: ReflexiCoder-8B establishes a new state-of-the-art among open-source models in the 1.5B–14B range across seven code-generation benchmarks.The evaluation covers foundational and competitive-programming tasks, including HumanEval, MBPP, BigCodeBench, LiveCodeBench, and CodeForces.
  • Main Results: 14.46% and 13.64% are ReflexiCoder-8B (Single)’s absolute pass@1 improvements over Qwen3-8B on LiveCodeBench and CodeForces, respectively.These results are reported under the single-attempt setting, which uses the baseline-equivalent token budget.
  • Main Results: 54.12% on LiveCodeBench and 37.68% on CodeForces are ReflexiCoder-8B (Multiple)’s scores, exceeding GPT-5.1’s 48.03% and 34.70% on those benchmarks.On HumanEval+ and MBPP+, the Multiple variant reaches parity with or exceeds GPT-5.1 according to the reported comparisons.
  • Impact of Reasoning & Reflection: 18.64% and 5.22% are ReflexiCoder’s absolute average improvements over the Non-Reasoning and Vanilla Outcome-RL baselines across three reasoning-intensive benchmarks.Vanilla Outcome-RL uses binary pass/fail rewards without incentivizing the intermediate reflection-correction trajectory.

4 Conclusion

ReflexiCoder uses reinforcement learning to internalize self-reflection and self-correction without external oracles or environmental interaction at inference time. ReflexiCoder-8B achieves state-of-the-art performance among open-source models in the 1.5B–14B range while competing with proprietary models on complex reasoning benchmarks.

  • Conclusion: ReflexiCoder teaches LLMs intrinsic self-reflection and self-correction by formulating debugging as a trainable decision-making trajectory.The framework does not rely on external oracles or environmental interaction at inference time.
  • Conclusion: ReflexiCoder-8B achieves state-of-the-art performance among open-source models in the 1.5B–14B range and remains competitive with proprietary models such as GPT-5.1.The conclusion presents internal self-debugging optimized via RL as a scalable direction for code LLMs.

5 Limitations

ReflexiCoder’s iterative reflection and correction improves reliability but introduces token, latency, scope, and transfer limitations. The method is primarily evaluated for single-file algorithmic correctness on Python-centric benchmarks.

  • 5 Limitations: Multiple reflection and correction cycles can increase token usage and latency compared with single-pass generation.This trade-off may limit applicability in tight-latency settings, and the optimal reflection budget can vary with task difficulty.
  • 5 Limitations: The intrinsic debugging formulation targets algorithmic correctness and local code issues within a single-file setting.It does not explicitly address repository-level development, long-horizon refactoring, dependency management, or interactive debugging with evolving specifications.
  • 5 Limitations: Extending the trajectory formulation to multi-file contexts and richer tool interfaces remains future work.
  • 5 Limitations: The experiments instantiate ReflexiCoder on Qwen3 models and common Python-centric benchmarks.The passage states that transfer to other base models, programming languages, or domains where correctness cannot be captured by unit tests remains unclear.
  • Reward design: The reward structure encourages quality gains, penalizes declines and stagnation, and preserves already optimal quality.It balances continued refinement with stability once maximum quality is reached.
  • Policy optimization: Policy optimization embeds reflection-aware rewards into advantage computation while retaining GRPO’s stability advantages.The updates align with both code correctness and self-reflection efficiency.

D Implementation Details

The implementation compares ReflexiCoder with seven representative code models using official evaluation settings for open-weights baselines.

  • Implementation Details: Seven representative code models are used as comparison systems.They include Qwen2.5-Coder-7B-Instruct, Seed-Coder-8B-Instruct, DeepSeek-Coder-7B-Instruct, CodeGemma-7B-IT, CodeLlama-7B-Instruct, LeDex-RL-7B, and DeepCoder-14B-Preview.
  • Implementation Details: The six open-weights models are evaluated with the official chat templates and author-recommended generation settings.

D.2 Dataset Curation

The implementation uses curated open-source coding data, fixed reward settings, and a structured verification loop designed to preserve correctness during optimization.

  • D.2 Dataset Curation: The training dataset uses four DeepCoder subsets: TACO-Verified, LiveCodeBench, CodeForces, and LeetCode.Their stated sizes are 7,436, 599, 6,128, and 2,641 problems, respectively, with preprocessing for quality filtering and decontamination.
  • D.2 Dataset Curation: The dataset curation verifies no overlap with the LiveCodeBench v5 test split used by EvalChemy 1.The referenced test problems fall within August 1, 2024 to February 1, 2025.
  • D.2 Dataset Curation: Reward-related hyperparameters are kept fixed across tasks and benchmarks for reproducibility.Table 4 lists the reward-related settings used unless otherwise specified.
  • D.2 Dataset Curation: Sensitivity analysis reports stable final performance across a broad range of reward-weight settings.
  • D.4 System Prompt: The system prompt requires first-pass analysis, edge-case reflection, and correction or optimization only after functionality is confirmed.This verification loop is designed so efficiency improvements do not sacrifice correctness.

E Hyperparameter Analysis

ReflexiCoder is robust across a broad range of reward weights, with the strongest overall configuration emphasizing format and efficiency while avoiding excessive trajectory-quality weighting.

  • E Hyperparameter Analysis: Performance is robust across a wide range of reward-weight settings.The analysis varies trajectory quality φ, efficiency bonus ψ, and format constraint ξ.
  • E Hyperparameter Analysis: ⟨ξ, φ, ψ⟩= ⟨1.0, 0.5, 1.0⟩ yields the strongest and most consistent results across all benchmarks.The passage reports particularly large gains on LiveCodeBench and CodeForces.
  • E Hyperparameter Analysis: Overly large φ degrades performance, whereas increasing ψ generally improves results.These findings suggest that elaborate reflection can be counterproductive and that efficient multi-step self-correction should be explicitly encouraged.
  • E Hyperparameter Analysis: Larger models learn higher progressive-improvement and efficiency rewards more quickly and converge to a more optimal number of reflection cycles.Smaller models struggle to fully optimize the complex reward landscape.

F Training Dynamics and Reward Scaling

Reward components scale differently across model sizes: larger models more reliably learn positive iterative improvement and efficient deliberation, while cycle control and formatting follow distinct patterns.

  • Cycle Count Regulation: The 0.6B model reaches the highest final cycle-count regulation reward, while larger models rise quickly and plateau earlier.This pattern suggests smaller models rely on more cycles, whereas larger models learn stopping behavior sooner.
  • Progressive Improvement: Larger models learn stable positive iterative improvement, whereas the 0.6B model’s reflection reward becomes persistently negative after early training.Models at least 4B maintain positive Rtrajectory(τ), with 8B and 14B slightly higher and more stable.
  • Efficiency Reward: 4B, 8B, and 14B models converge to higher efficiency rewards than smaller models, indicating more concise and effective deliberation.The 0.6B model remains low, while 1.7B improves only to a moderate plateau.
  • Trajectory Format: Formatting compliance improves with scale, but the 0.6B model eventually achieves the highest trajectory-format reward.Larger models improve rapidly before saturating, suggesting format adherence is easier to learn than iterative quality.
  • Overall Dynamics: ReflexiCoder’s gains are attributed to structured self-correction, especially positive iterative improvement and efficient deliberation emerging at sufficient scale.The analysis distinguishes these learned policies from gains caused merely by longer rollouts or increased sampling.

G Interpretation of Reward Shaping

The reward landscape indicates that final code quality is highest when cycle regulation and iterative improvement are jointly strong. This supports optimizing self-correction as a structured multi-step trajectory rather than maximizing either component alone.

  • Reward Surface: The reward surface maps cycle-count regulation P(n) and iterative quality Rtrajectory to final code-quality reward z across model scales.Higher surface height and brighter colormap indicate better predicted code quality; observed samples are overlaid by training step.
  • Joint Reward Effects: Across 0.6B to 14B, the highest predicted code quality concentrates where both P(n) and Rtrajectory are strong.Quality gains are not explained by more iterations or better rewriting alone.
  • Joint Reward Effects: Increasing Rtrajectory without sufficient P(n) does not reliably produce high final-code reward.Unconstrained reflection can cause over-editing, oscillation, or verbose non-functional changes.
  • Cycle Regulation: Cycle-count regulation acts as a budgeting and credit-assignment aid by steering reflection toward depths more likely to yield correct code.The positive gradient along P(n) favors useful reflection rather than extended low-yield thinking.
  • Scaling: As model scale increases, the high-quality region broadens and attainable predicted reward increases, while the qualitative landscape remains consistent.This indicates larger models convert trajectory-level improvement signals into final correctness more consistently.
  • Interpretation: The shaping terms jointly encourage the model to learn when to stop reflecting and how to make corrections that improve code.The authors connect this loop to reduced reliance on external execution feedback and improved final functional correctness.

H Token Budget Discussion

ReflexiCoder addresses the token cost of iterative refinement by learning a specialized reflection policy while retaining strong performance under a single-attempt budget. Its efficiency reward favors short, high-quality trajectories.

  • Policy Conditioning: ReflexiCoder’s reflection behavior is conditioned on its RL system prompt and disappears without that prompt, reverting to single-pass base-model inference.The learned operating mode is therefore prompt-activated rather than hard-coded into every inference context.
  • Fair Comparison: Under identical single-attempt token budgets, ReflexiCoder consistently outperforms baseline models even without iterative reflection.This evaluation removes any advantage from extra inference tokens.
  • Trajectory Internalization: The reward design favors trajectories that produce a correct initial solution followed by at most one brief optimization step.This optimal-trajectory preference explains why the model can perform well in zero-reflection mode.
  • Efficiency: The efficiency reward penalizes redundant iterations, strengthening first-pass success and preserving competitiveness under constrained compute.Later reflections are encouraged to focus on readability or style once functionality is correct.

I Case Study

The case study illustrates ReflexiCoder’s structured loop: it generates code, reflects on correctness, repairs detected bugs, and may terminate after a verified optimization. The system prompt and output constraints make these stages explicit and bounded.

  • Case Study: The TACO case study follows a trajectory from an initial brute-force solution to iterative correction of a strict-increase bug.In Cycle 1, reflection detects that equal scaled values were incorrectly allowed, and the comparison is corrected from < to <=.
  • Case Study: The task counts length-(k+1) subarrays satisfying the chained scaled inequality 2^0 · a_i < 2^1 · a_i+1 < ··· < 2^k · a_i+k.The case study’s specification defines the target property over each qualifying subarray.
  • System Prompt: The system prompt requires a Reasoning → Code → Reflection → Iteration loop with correctness and efficiency as explicit goals.Generated code must be placed inside Markdown code blocks within the answer structure.
  • Reflection Loop: Detected bugs trigger a defined fix, corrected answer, and another reflection cycle.The protocol distinguishes logical, syntax, and edge-case errors as reasons to continue debugging.
  • Optimization: When functionality is confirmed, the model may apply one verified optimization and must terminate without reflecting again.The optimization must preserve correct output and avoid introducing new bugs.
  • Iteration Bound: The protocol permits at most five answer outputs, after which the model returns its best version regardless of remaining issues.This caps the number of correction attempts in the iterative process.
Loading 2603.05863v2…