PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs

Abhinav Anand, Sanjana Reddy Pachika, Shweta Verma, Mira Mezini

arXiv:2609.11956v1cs.LG

TL;DR

Code-LLM post-training with RL is costly because online sampling and verification require repeated generation and GPU-CPU communication. This paper studies fully offline RL using existing code-and-reward data, finding performance gains across models from 0.5B to 7B parameters while identifying instability and collapse boundaries.

  • Problem

    Online RL post-training for code LLMs requires costly repeated sampling and verification, motivating whether verifiable-feedback post-training can be performed without online sampling.

  • Method

    The paper applies offline RL to pre-existing code submissions and execution-derived rewards, analyzing model size, learning rate, training epochs, and stability.

  • Results

    Offline RL substantially improves code-generation performance across model families and sizes, but prolonged training can cause model collapse and instability linked to logit variance and logit gaps.

  • Takeaways & Limitations

    Offline RL is feasible as a compute- and data-efficient code-LLM post-training pathway, provided its instability and collapse modes are monitored.

  • Takeaways & Limitations

    The evaluation covers only two benchmarks and Python code, so the work does not establish performance across multiple programming languages.

Abstract

from arXiv · show

Post-training with reinforcement learning (RL) is a critical phase in the development of code-generating large language models (LLMs), as it ensures adherence to instructions and the production of functionally correct code. This process typically requires computationally intensive code sample generation from Transformer-based LLMs and substantial GPU-CPU communication for sequence verification. To address these computational challenges, this work examines whether RL-based post-training can be performed entirely offline by leveraging existing datasets rather than generating new samples. The findings indicate that, with only a few hours of training, zero-shot code generation performance of LLMs can be substantially improved without online sampling. Additionally, offline RL produces performance gains across models ranging from 0.5B to 7B parameters, although the extent of improvement varies among model families.

1 Introduction

Offline RL uses pre-existing code data to avoid costly online sampling and improves code-model performance, while outcomes depend on training settings and can become unstable.

  • Motivation: Online sampling is a major post-training bottleneck because Transformer inference and GPU-CPU verification make generation and evaluation slow and computationally demanding.These costs also discourage adding feedback signals beyond functional correctness.
  • Approach: The framework investigates whether verifiable-feedback post-training can operate without online sampling by optimizing on a pre-existing dataset.Experiments cover model families, sizes, learning rates, epochs, and training stability.
  • Stability: Performance is highly sensitive to learning rate and training duration: multiple epochs can help, but prolonged training can cause model collapse.The paper examines both optimization sensitivity and collapse during offline RL.
  • Stability: Offline-RL instability is primarily associated with logit variance rather than the advantage variance emphasized for online RL.The authors propose diagnostics for early stopping and identify increasing logit variance as a source of instability.
  • Scope: The analysis focuses on functional correctness feedback, while efficiency, security, and other feedback types remain future directions.Training and evaluation were conducted using a single GPU, with larger multi-GPU batch sizes identified as a possible improvement.

2 RL Framework

The paper replaces repeated online generation and verification with policy optimization on fixed code-and-reward data, using normalized policy gradients and an on-policy-motivated assumption.

  • Offline versus online RL: Offline RL learns directly from fixed code submissions and recorded rewards, avoiding repeated program generation and execution during training.Online code RL repeatedly generates candidates, executes them against tests, computes rewards, and updates the policy.
  • Evaluation: The policy is evaluated on unseen code-generation benchmarks after optimization on CodeNet submissions and execution-derived rewards.This tests generalization beyond the fixed training data.
  • Optimization: The method uses RLOO with GRPO-style group-wise advantage normalization to provide a stable offline policy-optimization procedure.Normalization reduces the effect of differing reward scales between groups.
  • Method rationale: The policy-gradient choice is motivated by the possibility that pretrained LLM policies remain close to human-generated reward datasets, making offline training effectively on-policy.The paper also favors direct policy learning because Q-learning policies are implicit and their inference instability is difficult to diagnose.

3 Study Methodology

The study trains six base code models with offline RL on filtered Python CodeNet data and evaluates them on MBPP and APPS across multiple difficulty levels and pass@k metrics.

  • Models: Six decoder-only models from Qwen2.5-Coder, DeepSeek-Coder, and CodeLlama, spanning 0.5B to 7B parameters, are trained directly with offline RL.The experiments use publicly available base models without supervised fine-tuning or instruction tuning.
  • Dataset: The training set contains 8,321 Python samples from 609 problems, including accepted and incorrect submissions with recorded execution status.At most 20 submissions are retained per problem to reduce imbalance.
  • Training setup: Training uses ten epochs, group size four, eight gradient-accumulation steps, and model-specific learning rates.Learning rates are varied to identify suitable optimization settings under GPU-memory constraints.
  • Rewards: Rewards encode execution status, with fully passing code assigned a reward of +1.0.The dataset includes both successful and unsuccessful attempts for policy optimization.
  • Batch construction: Groups contain one correct and one incorrect sample, while two remaining samples may have any status, reducing advantage variance.The grouping strategy uses the availability of multiple submissions for each prompt.
  • Evaluation: Evaluation covers Python generation on MBPP using pass@1 and on APPS using pass@1, pass@5, and pass@10 across introductory, interview, and competitive problems.MBPP contains simple Python problems, whereas APPS spans multiple difficulty levels.

4 Results and Discussion

Offline RL improves zero-shot code-generation performance across model families and sizes, while gains depend on model, learning rate, and training duration. Performance can collapse with prolonged training, so effective offline post-training requires careful stopping and avoids online sampling overhead.

  • 4.1 Performance: Offline RL substantially improves model performance across different model sizes, including Qwen 0.5B, Qwen 7B, DeepSeek 6.7B, and CodeLlama.Qwen 0.5B, Qwen 7B, and DeepSeek 6.7B improve by 48%, 15%, and 31%, respectively, while CodeLlama improves by 9%.
  • 4.1 Performance: Offline training improves performance on the more challenging APPS benchmark across pass@1, pass@5, and pass@10, including competition-level problems for Qwen 0.5B.The comparison includes CodeRL and PPOCoder baselines trained on APPS, whereas the proposed Qwen 0.5B model is trained on CodeNet and evaluated zero-shot on APPS.
  • 4.1 Performance: Offline RL gains vary across model families, with CodeLlama improving on introductory problems but not on interview- or competition-level problems.The disparity is hypothesized to reflect misalignment between pretraining-data composition and the post-training dataset.
  • 4.2 Efficiency: Learning-rate effects are model-dependent: lower rates may fail to improve performance, very high rates can degrade it, and intermediate rates support gains across multiple epochs.The reported effective range is 1 × 10−5 to 5 × 10−5, with the recommendation to stop before collapse.
  • 4.2 Efficiency: Offline RL eliminates computationally intensive model sampling and GPU-CPU verifier communication, while achieving improvements with minimal single-GPU training.This efficiency benefit follows from optimizing with pre-existing data rather than generating and verifying new samples online.
  • 4.3 Collapse: Extended training can cause model collapse after peak performance, with Qwen 0.5B reaching pass@1 of 0 by the seventh epoch after peaking at four epochs.Similar collapse is observed across model families and sizes, and different models collapse at different epochs.

5 Instability Analysis

Offline RL instability is driven primarily by logit variance rather than advantage variance, with performance gains occurring before excessive variance or negative logit gaps trigger collapse.

  • 5.2 Log Probability Variance Analysis: Intermediate learning rates can improve performance over multiple epochs, whereas very low rates stagnate and very high rates can cause rapid degradation.The optimal learning rate varies by model family and size.
  • 5.2 Log Probability Variance Analysis: Initial low variance followed by a rise is associated with optimal performance, while consistently low or initially high variance produces poor results.This pattern is reported across models, with DeepSeek 1.3B an exception because it already performs strongly on MBPP.
  • 5.1 Advantage Variance: Offline RL instability originates from logit variance, while advantage variance remains stable and approximately constant during training.The controlled positive-and-negative sample grouping helps keep advantage variance stable, distinguishing offline from online RL instability.
  • 5.3 Logit Gap: Monitoring logit variance and applying early stopping are proposed to prevent collapse, while comprehensive resampling remains future work.The authors identify variance monitoring and early stopping as diagnostics rather than fully verified stabilization methods.
  • 5.3 Logit Gap: Negative logit gaps typically grow faster than positive gaps; low proportional growth yields no improvement, while a high negative gap precedes collapse.A negative gap that starts low and later increases can support rapid improvement before eventual degradation.

6 Related Work

Prior work studies online or off-policy RL and offline RL alternatives, but this paper analyzes policy-gradient offline RL for code generation without sampling the trained model.

  • 6 Related Work: Unlike practical off-policy online approaches, this work trains code LLMs without sampling from the model during training.The study focuses on code, where correct outputs also require syntactic understanding.
  • 6 Related Work: Earlier offline RL studies did not comprehensively analyze model families, hyperparameters, or failure causes in offline settings.This work positions its contribution as the first direct policy-gradient analysis of offline RL without model sampling.

7 Conclusion

The paper evaluates complete offline post-training for coding LLMs using fixed datasets and rewards rather than model-generated samples, finding substantial gains after only a few hours on one GPU.

  • 7 Conclusion: Offline RL substantially improves coding LLM performance across model families and sizes after only a few hours of single-GPU training.The approach uses samples and rewards from an existing dataset instead of generating new samples with an LLM.
  • 7 Conclusion: Logit variance and the logit gap between model-favored tokens and offline-dataset tokens are identified as primary instability sources.Controlling these sources and incorporating model-generated samples may further improve performance, but the paper frames these as future directions.

Limitations

The study establishes offline RL feasibility but evaluates only two benchmarks and Python code, while proposed stabilization mechanisms remain incompletely verified.

  • Limitations: The evaluation covers only two benchmarks because the study targets offline RL feasibility rather than a new state-of-the-art code model.The authors consider the evaluation sufficient to show that offline RL can improve the model.
  • Limitations: Training uses code from only Python, so the study does not establish whether offline RL generalizes to multiple programming languages.This is an explicit scope boundary of the reported experiments.
  • Limitations: The paper identifies possible mechanisms for controlling instability but does not verify all their benefits, leaving stabilization experiments for future work.Offline RL remains unstable, although adapting online-training stabilization methods is suggested as a possible direction.

Ethical Considerations

The work frames offline RL as an efficient alternative to online RL, while acknowledging that the same efficiency could lower the computational barrier to harmful-content generation.

  • Ethical Considerations: Offline RL is studied as an efficient alternative to online RL for aligning LLMs to functional correctness of code.
  • Ethical Considerations: The methodology could also train LLMs to generate harmful contents using very few computational resources.
  • Ethical Considerations: GRPO is described as commonly used because it balances performance and resource requirements, while RLOO reduces policy-gradient variance without a separate value function.

E Logit Gap

The analysis examines logit gaps between model-preferred and dataset tokens, finding similar gap patterns across models and linking persistent near-zero gaps to absent performance improvement.

  • E Logit Gap: Figure 10 illustrates the gap between the model-preferred token and the dataset token, whose unnormalized output makes the gap unbounded.
  • E Logit Gap: Negative and positive logit gaps begin near zero across models, while negative gaps exceed positive gaps except for DeepSeek 6.7B.
  • E Logit Gap: If positive and negative gaps remain near zero throughout training, model performance does not improve.
  • E Logit Gap: Figure 8 evaluates pass@1, pass@5, and pass@10 on APPS across difficulty levels, with Qwen models showing the best improvement over their base models.
  • E Logit Gap: The MBPP results show that models reach peak performance at different epochs, and final performance after 10 epochs is usually below the peak.

G Effect of Different Data Sampling

The data-sampling study replaces CodeNet submissions while preserving programming problems and status distributions, finding further performance gains and consistent logit-gap behavior.

  • G Effect of Different Data Sampling: Replacement datasets keep the same programming problems while substituting CodeNet solutions with similar status distributions.
  • G Effect of Different Data Sampling: Qwen 0.5B improves on MBPP Pass@1 from 0 for the base model to 42 after initial offline training and 46 after new samples for the same problems.
  • G Effect of Different Data Sampling: Similar logit-gap trends across different samplings suggest consistent training behavior, while new samples again produce a near-zero gap associated with effectively on-policy training.
  • G Effect of Different Data Sampling: Tables 4–8 provide chronological MBPP Pass@1 evaluations across epochs for Qwen, DeepSeek, and CodeLlama configurations.
Loading 2609.11956v1…