PaperlayerView exact revision on arXiv

Source-linked AI summary

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning

Huimu Yu, Xing Wu, Haotian Xu, Debing Zhang, Songlin Hu

arXiv:2410.02229v2cs.AIcs.CL

TL;DR

Reasoning-focused reward-model finetuning is constrained by scarce, labor-intensive high-quality preference data, and existing public-source pretraining has limited reasoning preference pairs. CodePMP pretrains preference models on synthesized code pairs and reports improved reward-model and reasoning performance across mathematical and logical tasks.

  • Problem

    High-quality preference data is costly and labor-intensive to collect, while online reasoning-preference pairs are scarce for preference-model pretraining.

    Source: §1 · p. 1

  • Method

    CodePMP synthesizes chosen and rejected code responses from publicly available source code and uses the pairs to pretrain preference models before reward-model finetuning.

    Source: §1 · pp. 1–2

  • Results

    CodePMP significantly improves reward-model finetuning accuracy and Best-of-N reasoning performance across mathematical and logical tasks compared with direct reward-model finetuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that code-derived preference data can provide a scalable, cost-effective approach that reduces reliance on extensive preference annotation.

    Source: §1 · p. 2

    The evaluation focuses mainly on mathematical and logical reasoning, leaving effectiveness for other reasoning modalities less thoroughly examined.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have made significant progress in natural language understanding and generation, driven by scalable pretraining and advanced finetuning. However, enhancing reasoning abilities in LLMs, particularly via reinforcement learning from human feedback (RLHF), remains challenging due to the scarcity of high-quality preference data, which is labor-intensive to annotate and crucial for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that utilizes a large corpus of synthesized code-preference pairs from publicly available high-quality source code. CodePMP improves RM finetuning efficiency by pretraining preference models on large-scale synthesized code-preference pairs. We evaluate CodePMP on mathematical reasoning tasks (GSM8K, MATH) and logical reasoning tasks (ReClor, LogiQA2.0), consistently showing significant improvements in reasoning performance of LLMs and highlighting the importance of scalable preference model pretraining for efficient reward modeling.

1 Introduction

CodePMP addresses scarce reasoning-preference data with code-derived preference-model pretraining. The paper reports improved reward-model and generator reasoning performance across mathematical and logical tasks.

1 Introduction

  • CodePMP improves reward-model finetuning accuracy and Best-of-N performance over direct finetuning across mathematical and logical reasoning tasks.The reported comparison uses MetaMath-Mistral-7B and covers GSM8K, MATH, ReClor, and LogiQA2.0.Source: Figure 1 · p. 1, §1 · p. 2
  • High-quality preference data is expensive and labor-intensive to collect, constraining reward-model finetuning for reasoning tasks.Existing public-source preference pretraining is less effective for reasoning because online reasoning-preference pairs are scarce.Source: §1 · p. 1
  • CodePMP uses publicly available source code as a scalable source of preference pairs, whose logical and structured nature is described as suitable for reasoning tasks.The paper also notes reported correlation between code training and reasoning improvements.Source: §1 · pp. 1–2
  • CodePMP synthesizes millions of chosen and rejected code responses, using stronger CodeLLMs for chosen responses and weaker ones for rejected responses.The resulting pairs pretrain preference models with pairwise ranking objectives before reward-model finetuning.Source: §1 · p. 2
  • The authors conclude that code-derived preference data offers a scalable, cost-effective approach that reduces reliance on extensive preference annotation.They describe this as supporting more effective reward modeling for reasoning tasks.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe causal language modeling, reward modeling through relative output scores, and Best-of-N selection using those scores.

2 Preliminaries

  • Causal language models predict each next token from preceding tokens and train by minimizing negative log-likelihood.The text describes transformer architectures as standard for causal language modeling because they handle long-range dependencies effectively.Source: §2 · p. 2
  • Reward models score outputs and compare pairs, with pairwise ranking loss assigning higher scores to preferred outputs.The chosen and rejected outputs receive scores from the reward model, and the loss uses their score difference.Source: §2 · p. 3
  • Best-of-N sampling generates N candidate solutions and selects the candidate with the highest reward-model score.The text describes this approach as especially effective for mathematical problem-solving and logical inference.Source: §2 · p. 3

3 Code Preference Model Pretraining

CodePMP inserts preference-model pretraining between language-model pretraining and domain-specific finetuning, using code-derived pairs to train ranking ability while retaining language modeling.

3.1 Model Design

  • The pretraining phase uses a large, diverse code-preference dataset to teach generalizable patterns and ranking strategies before domain-specific finetuning.The paper positions this phase between basic language-model pretraining and finetuning on reasoning datasets.Source: §3.1 · p. 3
  • CodePMP pretraining combines reward modeling on code-preference pairs with language modeling on chosen code, using the summed RM and LM losses.The stated design aims to improve ranking ability without sacrificing general language-modeling performance.Source: §3.1 · p. 3
  • CodePMP-initialized reward models perform better on reasoning test sets in the reward-model accuracy comparison.The table caption characterizes this as better discrimination ability.Source: Table 1 · p. 3

3.2 Data Construction

  • The CodePMP dataset is built from more than 1.3 billion GitHub code files, summarized into prompts and used to generate chosen and rejected code responses.A more capable CodeLLM generates chosen responses and a less capable CodeLLM generates rejected responses.Source: §3.2 · p. 3

4 Experiments

The experiments evaluate CodePMP across mathematical and logical reasoning tasks using 1.5B and 7B reward models. CodePMP initialization improves reward-model accuracy and Best-of-N performance, while code-preference data scaling and smaller finetuning sets show further efficiency gains.

4.1.1 CodePMP Settings

  • CodePMP pairs DeepSeek-Coder-6.7B chosen responses with 1.3B rejected responses in a 28-million-file, 19-billion-token dataset, initialized from Qwen2-1.5B or Qwen2-7B.Source: §4.1.1 · p. 4

4.1.2 Reasoning Finetuning Settings

  • Mathematical RMs use MathShepherd-pair, while logical RMs use ReClor-pair and LogiQA2.0-pair, with each model evaluated on its corresponding holdout test set.Source: §4.1.2 · p. 4

4.2.2 BoN Accuracy Results

  • CodePMP-initialized models achieve higher Best-of-N accuracy across mathematical and logical tasks and model sizes, retaining an advantage through N=256.Non-CodePMP models degrade at higher N; the paper relates this pattern to the inflection point discussed by Chow et al. (2024) and suggests improved alignment and robustness.Source: Figure 3 · p. 4, §4.2.2 · p. 5

4.1.3 Evaluation Settings

  • Evaluation covers GSM8K and MATH for mathematics and ReClor and LogiQA2.0 for logic; logical tasks use four-option multiple-choice accuracy (Best-of-4).RM accuracy measures chosen-versus-rejected discrimination on holdout sets; Best-of-N accuracy measures correct solutions selected from N candidates.Source: §4.1.3 · p. 4

4.2.1 RM Accuracy Results

  • CodePMP initialization yields higher holdout RM accuracy for 1.5B and 7B models across mathematical and logical reasoning tasks.The authors report generalization improvements across different reasoning tasks.Source: §4.2.1 · p. 4

4.2.4 Scalability Analysis

  • Increasing code-preference pairs consistently improves Best-of-N accuracy across tasks and model sizes, with no observed diminishing returns.The authors infer that further scaling the code-preference data would likely yield additional performance gains.Source: §4.2.4 · p. 5

4.2.3 Sample Efficiency Analysis

  • 0.5k samples match baseline models’ 40k-sample performance, an 80× efficiency improvement that diminishes as datasets grow.The Figure 4 caption reports higher Best-of-N accuracy for CodePMP-initialized 7B models at equivalent sample sizes.Source: Figure 4 · p. 5, §4.2.3 · p. 5

5 Ablation Studies

The ablations examine CodePMP’s pair construction and loss, then test its generalization across backbones, generators, and broader reward-modeling benchmarks. Unless otherwise stated, experiments use the 1B model and report mathematical reasoning results due to resource and page limitations.

5.1 Impact of Pair Construction

  • GitHub-sourced pairs outperform web-crawled pairs, especially at larger N, while doubling the code-preference pairs improves Best-of-N accuracy without diminishing returns.Figure 5’s horizontal axis scales by 2, and dashed lines show baseline performance without CodePMP.Source: §5.1 · p. 5, Figure 5 · p. 6
  • Pairing positive samples from the 7B model with negative samples from the 1.5B model delivers the best performance across all test sets.The authors identify execution feedback as a direction for creating more accurate preference pairs.Source: §5.1 · p. 6

5.2 Impact of Loss Function

  • The combined RM and LM losses outperform either single-loss variant across Best-of-N settings, with particularly notable gains on MATH.The authors attribute the complementary roles to preference ranking from RM loss and preserved language capabilities from LM loss.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Cross-Architecture Generalization

  • CodePMP improves reasoning performance across Gemma2 and Llama3.2 model families and improves robustness at larger N, mitigating degradation in noninitialized models.The evaluation covers GSM8K, MATH, ReClor, and LogiQA-v2.Source: §5.3 · p. 6, Figure 7 · p. 7

5.4 Performance on Larger Backbone Model

  • CodePMP initialization consistently improves Qwen2-72B performance across mathematical and logical reasoning benchmarks, with larger gains at higher N on challenging MATH tasks.Only N = 4 was tested for ReClor and LogiQA-v2.Source: §5.4 · pp. 6–7, Table 3 · p. 7

5.5 Performance on More Powerful Generator

  • CodePMP’s benefits persist with Qwen2-Math-7B-Instruct and Qwen2.5-32B-Instruct generators, despite the latter being much larger than the preference-pair generators and reward model.Tables 4 and 5 report improvements on GSM8K and MATH with the specialized generator and consistent gains with the larger general-purpose generator.Source: Table 4 · p. 7, Table 5 · p. 7, §5.5 · p. 7

5.6 Performance on General RM Benchmarks

  • PMP-finetuned models consistently outperform models without PMP across RMBench tasks and model sizes, extending CodePMP’s reported gains beyond reasoning and coding.RMBench covers summarization, chat quality, and safety.Source: §5.6 · pp. 7–8, Table 6 · p. 8

6 Related Works

Prior work studies reward modeling, code training, and LLM reasoning through a range of methods, while annotation costs and limited datasets constrain some approaches. CodePMP combines these strands by generating preference data from code.

Reward Modeling

  • Reward-modeling work spans ranking, probability-based, critique-based, generative, and preference-pretraining approaches, while manual annotation costs and limited datasets constrain scalability.PMP uses large-scale pairwise ranking data for pretraining (Askell et al., 2021); CodePMP automates preference-data generation from code.Source: §6 · p. 8

Code Training

LLM Reasoning

7 Conclusion and Future Works

The paper introduces CodePMP, which uses synthetic code-preference pairs for preference-model pretraining. Experiments report improved sample efficiency and performance across reasoning tasks, and the authors propose code-based directions for process supervision and generative reward models.

Conclusion

  • CodePMP is a preference-model pretraining method that uses synthetic code-preference pairs to improve reasoning in large language models.The paper reports enhanced sample efficiency and performance across diverse reasoning tasks.Source: §7 · p. 8

Future Works

  • The authors propose CodePrMP, using compiler or interpreter verifiability for low-cost process supervision.Source: §7 · p. 8
  • The authors also propose GenPMP, which aims to improve generative reward models through code-based pretraining.Source: §7 · p. 8

Limitations

The stated limitations concern the source and construction of synthetic preference pairs, architectural coverage, and evaluation scope.

Limitations

  • Pair generation uses models with predetermined parameter sizes, potentially missing nuanced preference signals, and architectural differences remain incompletely explored.Source: Limitations · p. 9
  • Reliance on GitHub data may introduce biases tied to the composition of public repositories.Source: Limitations · p. 9
  • Evaluation focuses primarily on mathematical and logical reasoning, leaving commonsense and causal reasoning less thoroughly examined.Source: Limitations · p. 9

Ethics Statement

CodePMP’s supplementary experiments report gains in reasoning-task sample efficiency and performance over majority voting. The paper also discusses deployment, data, and annotation trade-offs and lists implementation hyperparameters.

Ethics Statement

  • The authors identify deployment and monitoring needs, intellectual-property concerns, and possible biases from source data or model preferences as ethical trade-offs.They report using publicly available code with permissive licenses and call for ongoing evaluation and refinement.Source: Ethics Statement · p. 9

A Hyperparameters

  • The appendix lists separate hyperparameters for CodePMP training, mathematical and logical reasoning RM finetuning, and Best-of-N generation.WSD denotes the warmup-stable-decay learning-rate scheduler, which the cited passage says reduces time for scaling-law experiments.Source: A Hyperparameters · p. 11

B CodePMP Dataset Statistics

  • The dataset contains 28 million files and 19 billion tokens, with chosen and rejected responses of similar average lengths.The reported averages are 194.5 and 189.9 tokens, respectively; the corpus includes Python, Jupyter Notebook, and other-language code.Source: B CodePMP Dataset Statistics · pp. 11–12, Table 11 · p. 12

C.1 Comparison with Majority Voting

  • CodePMP outperforms majority voting on both GSM8K and MATH, especially on the more complex MATH tasks.Table 12 presents the comparison under the same experimental setup.Source: §C.1 · p. 12, Table 12 · p. 12

C.2 Sample Efficiency Improvements on Reclor and LogiQA

  • With substantially fewer training samples, CodePMP-initialized reward models achieve comparable or better logical-reasoning performance than models trained from scratch with many more samples.Figures 8 and 9 compare RM accuracy and Best-of-N evaluation as sample size increases.Source: §C.2 · p. 12

C.3 Performance on Coding Tasks

On code tasks, CodePMP initialization improves reward-model accuracy and generally yields more stable, higher Pass@1 selection on HumanEval. The paper also describes diversity analyses of its synthetic data.

C.3 Performance on Coding Tasks

  • CodePMP initialization raises CodeUltraFeedback RM accuracy from 0.6841 to 0.758 for the 1.5B model and from 0.6912 to 0.7619 for the 7B model.Table 13 reports accuracy across model sizes.Source: §C.3 · p. 12
  • At N = 4, HumanEval Pass@1 is 0.7622 with CodePMP initialization versus 0.7073 without it; at N = 8, it is 0.7683 versus 0.6890.At N = 32, the corresponding scores are 0.7378 and 0.6585; without CodePMP, performance degrades at higher N values.Source: §C.3 · p. 13, Table 14 · p. 13

D Comprehensive Data Diversity Analysis

  • The paper describes diversity analyses intended to assess whether synthetic data retains diversity while capturing distributions in human-generated data.The analysis uses established methodologies for synthetic text data generation.Source: D Comprehensive Data Diversity Analysis · p. 13

D.1 N-gram Feature Distribution Analysis

The appendix analyzes whether CodePMP’s synthetic preference data resembles human data and whether its strong-versus-weak model comparisons produce consistent preferences. It also describes the data-generation pipeline and a constraint in evaluating logical reasoning with Best-of-N.

D.1 N-gram Feature Distribution Analysis

  • 97,653.69 for the Strong Model and 93,691.69 for the Weak Model are the synthetic-text n-gram density values, versus 134,538.40 for human text.The authors characterize synthetic n-gram distributions as more uniform and balanced across hash buckets.Source: §D.1 · p. 14, Table 15 · p. 14

D.2 Embedding Space Visualization

  • Both synthetic and human data have wide, dispersed embedding distributions with highly overlapping ranges, which the authors say indicates similarly broad semantic coverage.Figure 12 visualizes the distributions in a 2D space.Source: §D.2 · p. 14, Figure 12 · p. 14

D.3 KL Divergence Analysis

  • The authors report that synthetic-versus-human distribution differences fall within acceptable ranges relative to variation among internal human data.The comparison uses KL divergence; Table 16 presents the values.Source: §D.3 · p. 14, Table 16 · p. 14

D.4.1 Validation of Strong-Weak Model Ability Differences

  • Models from the same family with different parameter counts show consistent strong-over-weak advantages, while cross-family pairings show irregular gaps and occasional inversions.The authors say these cross-family inconsistencies could potentially add noise to preference signals.Source: §D.4 · p. 14, Figure 13 · p. 14, §D.4.1 · p. 15

D.4.2 External Evaluation of Preference Consistency

  • 75.12% of synthetic CodePMP preferences are consistent under GPT-4o evaluation, compared with 71.56% for CodeUltraFeedback.The authors interpret this as evidence that the synthetic preference distinction is sufficiently clear and consistent.Source: §D.4.2 · p. 15

D.5 Source Data Quality and Experimental Validation

  • Figure 6a shows that the synthesis strategy outperforms preference pairs constructed directly from source code, while the authors describe the synthetic data as diverse and distributionally representative.The paper attributes this diversity in part to its source data quality.Source: §D.5 · p. 15

E Detailed Implementation of CodePMP

  • CodePMP generates descriptions from repository code, then uses a strong CodeLLM for chosen responses and a weaker one for rejected responses, training with language-model and reward-model losses.The resulting preference pairs provide an initialization for reward models later finetuned on reasoning tasks, without expensive human annotation.Source: E Detailed Implementation of CodePMP · p. 15

F Logical Reasoning Evaluation Examples

  • For ReClor Best-of-4 evaluation, the reward model ranks the four original manually annotated options because added candidates make correct-answer identification unreliable.The authors report low consistency between GPT-4o annotations and manual inspection, and among GPT-4o’s own annotations.Source: F Logical Reasoning Evaluation Examples · p. 16, Table 17 · p. 17

Related papers

Loading 2410.02229v2…