PaperlayerView exact revision on arXiv

Source-linked AI summary

Prior Prompt Engineering for Reinforcement Fine-Tuning

Pittawat Taveekitworachai, Potsawee Manakul, Sarana Nutanong, Kunat Pipatanakul

arXiv:2505.14157v2cs.CLcs.AI

TL;DR

RFT research has left the design of its prior prompt relatively understudied, despite the prompt’s role in guiding desired behaviors during training. This paper translates five inference-time prompting strategies into pPE approaches and evaluates them after RFT. All pPE-trained models outperform corresponding iPE-only baselines, with null-example utilization producing the highest average gain and strongest GPQA Diamond improvement.

  • Problem

    Although RFT research has examined several pipeline components, systematic evidence on how different prior-prompt approaches shape behaviors internalized during training remains limited.

    Source: §1 · p. 1

  • Method

    The paper adapts five iPE strategies into pPE approaches, trains models with RFT, and evaluates benchmark performance and post-training behaviors.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    All pPE-trained models outperform corresponding iPE-only baselines, while null-example utilization achieves the highest average performance gain and greatest GPQA Diamond improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Different pPE strategies are associated with distinct behavioral styles, making pPE a previously understudied axis for RFT.

    Source: §1 · p. 2

    Experiments fix the training data domain to mathematics and the reinforcement-learning algorithm to GRPO, so generalization across domains, algorithms, reward schemes, and broader RFT settings remains to be studied.

    Source: Limitations · p. 9

Abstract

from arXiv · show

This paper investigates prior prompt engineering (pPE) in the context of reinforcement fine-tuning (RFT), where language models (LMs) are incentivized to exhibit behaviors that maximize performance through reward signals. While existing RFT research has primarily focused on algorithms, reward shaping, and data curation, the design of the prior prompt--the instructions prepended to queries during training to elicit behaviors such as step-by-step reasoning--remains underexplored. We investigate whether different pPE approaches can guide LMs to internalize distinct behaviors after RFT. Inspired by inference-time prompt engineering (iPE), we translate five representative iPE strategies--reasoning, planning, code-based reasoning, knowledge recall, and null-example utilization--into corresponding pPE approaches. We experiment with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and GPQA-Diamond). Our results show that all pPE-trained models surpass their iPE-prompted counterparts, with the null-example pPE approach achieving the largest average performance gain and the highest improvement on AIME2024 and GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by adapting a behavior-classification framework, we demonstrate that different pPE strategies instill distinct behavioral styles in the resulting models. These findings position pPE as a powerful yet understudied axis for RFT.

1 Introduction

The paper asks whether changing the instruction used during reinforcement fine-tuning (RFT) can guide models to internalize distinct behaviors, an understudied aspect of the RFT pipeline. It tests five prior-prompt approaches and finds that pPE-trained models outperform corresponding inference-time-prompted baselines while exhibiting distinct behavioral styles.

1 Introduction

  • RFT studies have examined objectives, algorithms, data, rewards, and model behaviors, but prompt design remains understudied.The paper focuses on the instruction prepended to training queries, rather than the task content.Source: §1 · p. 1
  • The prior prompt is the instruction component of an RFT prompt that guides desired behaviors, and the paper asks whether its design can shape behaviors internalized during training.Figure 2 distinguishes the highlighted prior prompt from task content.Source: §1 · p. 1, Figure 2 · p. 2
  • The study translates five iPE strategies—reasoning, planning, code-based reasoning, knowledge recall, and null-example utilization—into corresponding pPE approaches.The approaches are inspired by chain-of-thought, plan-and-solve, program-of-thought, generated-knowledge, and null-shot prompting.Source: §1 · p. 2, Figure 1 · p. 2
  • The experiments train five Qwen2.5 7B models with math-only RFT data and evaluate quantitative performance and behavioral differences against corresponding iPE-only baselines.Evaluation includes mathematical reasoning, coding, and question-answering benchmarks, alongside an adapted behavior-classification framework.Source: §1 · p. 2
  • All pPE-trained models surpass corresponding iPE-only baselines, while null-example utilization yields the highest average performance gain and the greatest GPQA Diamond improvement.The paper also reports distinct behavioral styles among models trained with different pPE strategies.Source: §1 · p. 2
  • Different pPE strategies produce distinct behaviors, and an updated classification approach quantifies both cognitive and elicited behaviors.Reported behavioral differences include response structure, verbosity, behavior types, and performance impact.Source: §1 · p. 2

2 Prior Prompt Engineering for Reinforcement Fine-Tuning

The section presents five prior-prompt approaches adapted from inference-time prompting and describes how they are tested in RFT. The evaluation combines benchmark performance with classification of post-training behavior.

2 Prior Prompt Engineering for Reinforcement Fine-Tuning

  • The study translates five representative iPE approaches into pPE formats, trains distinct models under a standard RFT setup, and compares them with iPE-prompted models.The models differ in pPE approach and format reward; quantitative and qualitative evaluations cover benchmarks and behavior categories.Source: §2 · p. 3
  • The evaluation uses math, coding, and question-answering benchmarks and classifies post-RFT behavior into four cognitive and five pPE-specific categories.The same evaluations are applied to the base model with iPE prompts for comparison.Source: §2 · p. 3
  • The reasoning approach maps chain-of-thought prompting to <think>, eliciting step-by-step reasoning before a final answer and serving as the RFT baseline.The paper identifies this as the most commonly used approach in RFT studies.Source: §2 · p. 3
  • Planning maps plan-and-solve prompting to <plan>, while code-based reasoning maps program-of-thought prompting to <code>.The expected behaviors are generating a plan before answering and generating code with comments to solve the task.Source: §2 · p. 3
  • Knowledge recall maps generated-knowledge prompting to <knowledge>, while null-example utilization maps null-shot prompting to <examples>.The expected behaviors are recalling or synthesizing relevant knowledge and generating or referencing illustrative examples.Source: §2 · p. 3
  • The five approaches are expected to differ not only in performance and behavior but also in training dynamics, including response length and per-step reward trajectories.Source: §2 · p. 3

3 Experimental Setup

The experiments compare prior-prompt variants under a shared reinforcement fine-tuning setup and evaluate both benchmark performance and model behavior. They also include smaller-scale experiments across model sizes and families.

3.1 Prior Prompts

  • The five prior-prompt variants modify the template’s instruction and corresponding tag; <think> is the standard RFT setup.The same templates are used for inference-time-prompted comparison models.Source: §3.1 · p. 3

3.2 Training

  • Training uses GRPO with prompts from STILLv3, a dataset of approximately 30K mathematical problems.The paper notes that math-only data supports verifiable rewards through checking generated answers against ground truth.Source: §3.2 · p. 4
  • The reward equally weights answer accuracy and output-format compliance, with the expected format updated to match each prior-prompt variant.The no-prior-prompt baseline omits format instructions and format reward, leaving accuracy reward with a maximum of 1.0.Source: §3.2 · p. 4
  • The main comparison uses Qwen2.5-7B with five variants sharing training settings, alongside inference-prompted baselines and a no-prior-prompt baseline.Qwen2.5-7B is chosen as a base model to mitigate confounding from instruction tuning; evaluation of its instruct variant is left for future work.Source: §3.2 · p. 4
  • Scaled experiments examine Qwen2.5-3B, Llama 3.1-8B, and Qwen2.5-Coder-7B using selected prompt variants.The comparisons use <think> and <plan> for Qwen2.5-3B and Llama, and <think> and <code> for Qwen2.5-Coder-7B.Source: §3.2 · p. 4

3.3 Evaluation

  • The study evaluates mathematical reasoning, coding, and question-answering benchmarks, alongside model behaviors and cognitive patterns.The benchmarks include AIME2024, AMC12 ’22–’23, MATH-500, HumanEval+ base and extra sets, and GPQA-Diamond; behavior analyses cover response length and classified behavior categories.Source: §3.3 · p. 5

4 Results and Findings

Across Qwen2.5-7B experiments, pPE affects benchmark performance, training dynamics, and post-RFT behavioral profiles. Null-example pPE has the largest average performance gain, while results across behaviors and model families vary by approach.

4.1 Performance Impact

  • The null-example pPE gains +6.98 points on average, exceeding reasoning pPE (<think>), despite null-example iPE performing worst.The null-example pPE also maintains strong HE+ performance, unlike its iPE counterpart.Source: Table 1 · p. 5, §4.1 · pp. 5–6
  • 13.33 to 26.67 on AIME and 55.60 to 70.40 on MATH under No PP show substantial gains from RFT and the dataset without prior prompts.On AMC, every pPE variant exceeds No PP; code-based reasoning reaches 46.99, and raises HE+ from 73.80 to 78.00.Source: §4.1 · p. 5

4.2 Behavioral Differences

  • Across pPE runs, reward rises sharply in the first 20 steps before stabilizing as response length recovers; No PP has lower rewards and longer, steadier outputs.The No PP reward difference is attributed to optimizing accuracy without a format reward.Source: §4.2 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6
  • Null-example pPE pairs the highest overall performance with some of the shortest average responses, while No PP produces the longest responses without the strongest performance.Reasoning pPE also reduces response length relative to its iPE counterpart while retaining strong performance gains.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • Backward chaining is prominent across setups, but pPE generally reduces its frequency and the overall presence of fundamental cognitive behaviors relative to iPE.These behavior ratios do not correlate well with final performance, though they distinguish behavioral profiles; the authors question whether the classification framework generalizes across pPE paradigms.Source: §4.2 · p. 7, Figure 6 · p. 8
  • Post-RFT behaviors are generally targeted to the pPE approach: planning pPE most increases planning, while code pPE uniquely boosts code-related behavior.Null-example pPE is an exception: it yields few knowledge-recall and null-example instances despite high GPQA gains; No PP also increases reasoning, planning, and knowledge recall relative to zero-shot.Source: §4.2 · p. 7, Figure 7 · p. 8
  • Qualitative examples show planning models listing and executing steps, reasoning models producing step-by-step reasoning, and knowledge-recall models giving relevant definitions and formulas.Qwen2.5-7B often prefers natural-language reasoning over code under code-based pPE; the qualitative analyses are preliminary, and mechanisms remain beyond the study’s scope.Source: §4.2 · pp. 7–8

4.3 Generalization

  • Across smaller or different model families, reasoning pPE is consistently robust, while non-reasoning approaches may be less successful and planning pPE can lead to reward hacking.The code-specialized Qwen2.5-Coder-7B exhibits code-based reasoning more effectively than Qwen2.5-7B; reasoning and code-based pPE improve performance over baseline, with reasoning highest in the reported generalization results.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5 Conclusions

The paper compares five pPE approaches and finds that pPE-trained models outperform their inference-time-prompted counterparts, with null-example utilization outperforming reasoning. Different pPE approaches also produce distinct behavioral patterns, and the study explores generalization across model sizes and families.

5 Conclusions

  • Null-example pPE is more effective than reasoning pPE for improving downstream task performance, while all five pPE approaches outperform their iPE counterparts.The five approaches are reasoning, planning, code-based reasoning, knowledge recall, and null-example utilization.Source: §5 · p. 9
  • Different pPE approaches induce distinct behavioral patterns; planning-trained models tend to list steps before solving problems.The paper also reports differences in performance impact, response structure, verbosity, and behavior types.Source: §5 · p. 9
  • The study explores whether pPE approaches generalize across model sizes and model families.Source: §5 · p. 9

Limitations

The experiments were constrained by computational resources and by fixed training conditions. The authors say broader experiments are needed to validate generalization claims.

Limitations

  • The study could not test larger models, larger datasets, or more training steps, leaving behavioral trends for larger models inconclusive.The authors note that emergent abilities at scale could lead to different outcomes and that assumed cross-size generalization remains unvalidated.Source: Limitations · p. 9
  • Training used a fixed mathematics domain, GRPO algorithm, and other configurations, so interactions with alternative domains, algorithms, and reward schemes remain untested.The authors call for further studies to confirm generalization in a broader RFT context.Source: Limitations · p. 9

Ethical Considerations

The paper discusses safety risks and the scope of its RFT setting alongside background on RFT and inference-time prompting. It describes pPE experiments, potential behavioral applications, and supplementary prompt and reward designs.

Ethical Considerations

  • RFT may unintentionally reinforce unsafe behaviors already latent in pretrained models, so the authors recommend alignment and safety measures, especially for user-facing or safety-critical applications.They identify malicious prompts and poisoned data as potential sources of unexpected behaviors in open-ended settings.Source: Ethical Considerations · p. 10

A.1 Reinforcement Fine-Tuning (RFT)

  • RFT, also called reinforcement learning with verifiable rewards, uses task-specific rule-based rewards for tasks with verifiable answers; its core components include the prior prompt.The paper notes that DeepSeek-R1-Zero popularized RFT with GRPO and a prior prompt to elicit reasoning.Source: §A.1 · p. 13

A.2 Inference-Time Prompt Engineering (iPE)

  • Inference-time prompt engineering elicits behaviors such as reasoning, planning, code generation, and knowledge recall; the paper tests whether moving such strategies into RFT changes behavior.Source: §A.2 · pp. 13–14

B.1 Domain generalization

  • Although training uses mathematics, performance improvements often extend to GPQA and HumanEval+, which the authors interpret as suggesting RFT may discover useful generation patterns rather than add new knowledge.The authors expect broader generalization with more diverse training data.Source: §B.1 · p. 14

B.2 pPE for RFT

  • Prompt wording and other prompt properties can influence RFT outcomes, and iPE insights do not directly transfer, motivating targeted study of prior prompts.Source: §B.2 · p. 14

B.3 Beyond Reasoning Models

  • The authors propose further pPE directions, including task-adaptive prompt selection, prompt optimization, and multi-tag structures for staged reasoning.They also suggest that additional reasoning styles could be transferred from inference-time prompting.Source: §B.3 · p. 14
  • Verifiable task accuracy can indirectly incentivize useful intermediate behaviors such as planning, which can be repurposed by stopping generation after the relevant tag.The paper describes repurposing such models as plan generators or code synthesizers without direct supervised signals.Source: §B.3 · pp. 14–15

D Additional Experimental Setup Details

  • Supplementary materials show prior-prompt examples and templates, describe training and evaluation setup, and specify a reward combining equally weighted accuracy and format components.The experimental templates include prompts for reasoning, planning, code-based reasoning, knowledge recall, and null-example utilization.Source: D Additional Experimental Setup Details · p. 15, §D.1 · p. 15, §D.2 · p. 16

D.3 Training Setup and Hyperparameters

The appendix describes the training script and hardware used, lists model licenses, and points to additional evaluation details.

D.3 Training Setup and Hyperparameters

  • The appendix lists licenses for the OpenRLHF and vLLM tools and the Qwen2.5-7B, Qwen2.5-3B, Llama 3.1-8B, and Qwen2.5-Coder-7B models.It states that all listed licenses permit research use.Source: §D.3 · pp. 16–17
  • Training used a script specifying the model, dataset, GRPO algorithm, reward configuration, and hyperparameters; runs used one node with 8 H100 GPUs and totaled 78 GPU-hours.Source: §D.3 · p. 16, Figure 15 · p. 17

D.4 Evaluation

  • The appendix provides additional details about the quantitative and qualitative evaluations described in the main evaluation section.Source: §D.4 · p. 17

D.4.1 Quantitative Analysis

The evaluation uses a fixed-seed pass@1 protocol across in-domain mathematical reasoning and out-of-domain coding and knowledge-question-answering benchmarks.

D.4.1 Quantitative Analysis

  • Evaluation uses one fixed random seed and pass@1 accuracy, with the training prior prompt prepended to elicit trained behaviors.The passage also notes that benchmark and training-set metadata appears in Table 4.Source: §D.4.1 · p. 17
  • AIME24, AMC12 ’22–’23, and MATH-500 serve as primary in-domain mathematical-reasoning evaluations [Hendrycks et al., 2021].Source: §D.4.1 · p. 17
  • HumanEval+ evaluates general coding ability out of domain, while GPQA-Diamond tests factual knowledge and complex reasoning [Liu et al., 2023; Rein et al., 2024].GPQA-Diamond also probes whether math-centric training can elicit knowledge-recall behaviors.Source: §D.4.1 · p. 17

D.4.2 Qualitative Analysis

The analysis examines whether prior prompts yield distinct behavior patterns after reinforcement fine-tuning, using behavior classification and response-length measures.

D.4.2 Qualitative Analysis

  • The study analyzes training dynamics and average generated-response token counts across prior-prompt approaches.Source: §D.4.2 · p. 17
  • Four reasoning behaviors—verification, backtracking, subgoal setting, and backward chaining—are assessed with an LLM-based classifier following Gandhi et al. (2025).They respectively concern checking intermediate results, trying alternatives, decomposing problems, and reasoning backward from an expected answer.Source: §D.4.2 · p. 17
  • The analysis also adapts the classification approach to assess five pPE-specific behaviors, including the five approaches represented by their classification prompts.The supplied figure captions describe prompts for step-by-step reasoning, planning, code-based reasoning, knowledge recall, and illustrative examples.Source: §D.4.2 · p. 18

E Additional Results

This section gathers additional generalization-study results, including training dynamics, behavior classifications, and an alternative visualization of main-experiment benchmark performance.

E Additional Results

iPE

  • Table 5 visualizes absolute accuracy changes relative to zero-shot Qwen2.5-7B, with columns labeled AIME, AMC, GPQA, MATH, HE+, and average.The supplied passages do not provide the table’s cell values.Source: Table 5 · p. 21, E Additional Results · p. 21

E.1 Training Dynamics and Average Response Length

The additional material documents training and response-length dynamics, behavior classifications, qualitative outputs, and the authors’ declaration of AI assistance.

E.1 Training Dynamics and Average Response Length

  • Figures 25–30 present reward and response-length dynamics during RFT for Qwen2.5-3B, Llama 3.1-8B, and Qwen2.5-Coder-7B; captions identify reward progression and average response-length evolution.Source: §E.1 · p. 21, Figure 25 · p. 21, Figure 26 · p. 21

E.2 Four Fundamental Cognitive Behaviors

  • Four fundamental behaviors are classified in generalization studies for three models, with figures and tables reporting behavior ratios and occurrence counts.The behaviors are backtracking, backward chaining, subgoal setting, and verification; Table 8 covers the generalization counts.Source: §E.2 · pp. 21, 23, Figure 31 · p. 22, Figure 32 · p. 22

E.3 Five Elicited Behaviors

  • Five elicited behaviors are classified across the three generalization-study models, with heatmaps and tables presenting activation patterns and occurrence counts.Table 10 reports counts for code, examples, knowledge, plan, and think; its caption explains bold and underlined column rankings.Source: §E.3 · pp. 23–24, Figure 34 · p. 23, Figure 35 · p. 23

F Qualitative Examples

G Declaration of AI Assistance

  • The authors report using ChatGPT for grammatical checking and LaTeX support, Cursor for trivial boilerplate code completion, and author review of all content and code.Source: G Declaration of AI Assistance · p. 24

Related papers

Loading 2505.14157v2…