Source-linked AI summary
OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning
Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, Jitao Sang
TL;DR
RFT for domain tasks faces a lack of reasoning-step data and limited quantities of domain-specific samples, while o1's generalization beyond mathematics and programming remains uncertain. OpenRFT augments questions, synthesizes reasoning-process data for SFT, and uses few-shot ICL within PRM-supervised RL to adapt generalist reasoning models to domain tasks. With only 100 domain-specific samples, OpenRFT increases performance by an average of 11% on eight SciKnowEval tasks.
Problem
RFT for domain tasks faces a lack of reasoning-step data and limited quantities of domain-specific samples, while o1's generalization beyond mathematics and programming remains uncertain.
Source: §1 · p. 1Method
OpenRFT augments questions, synthesizes reasoning-process data for SFT, and uses few-shot ICL within PRM-supervised RL to adapt generalist reasoning models to domain tasks.
Source: §1 · p. 2, §5 · p. 11Results
With only 100 domain-specific samples, OpenRFT increases performance by an average of 11% on eight SciKnowEval tasks.
Source: §1 · p. 2Takeaways & Limitations
The authors propose synchronous PRM updating and self-play with iteratively refined training data as a path for continued performance improvement.
Source: §5 · p. 11Generalization remains a fundamental challenge, especially for formats beyond multiple-choice questions that require suitable reward functions and action spaces.
Source: §5 · p. 11
Abstract
from arXiv · showhide
OpenAI's recent introduction of Reinforcement Fine-Tuning (RFT) showcases the potential of reasoning foundation model and offers a new paradigm for fine-tuning beyond simple pattern imitation. This technical report presents \emph{OpenRFT}, our attempt to fine-tune generalist reasoning models for domain-specific tasks under the same settings as RFT. OpenRFT addresses two key challenges of lacking reasoning step data and the limited quantity of training samples, by leveraging the domain-specific samples in three ways: question augmentation, synthesizing reasoning-process data, and few-shot ICL. The evaluation is conducted on SciKnowEval, where OpenRFT achieves notable performance gains with only $100$ domain-specific samples for each task. More experimental results will be updated continuously in later versions. Source codes, datasets, and models are disclosed at: https://github.com/ADaM-BJTU/OpenRFT
1 INTRODUCTION
OpenRFT explores fine-tuning generalist reasoning models for domain-specific tasks, addressing missing reasoning-step data and limited training samples. On SciKnowEval, it reports an average 11% performance increase using 100 domain-specific samples per task.
1 INTRODUCTION
- RFT's generalization to tasks beyond mathematics and programming remains uncertain, despite o1's strong reasoning abilities in those areas.The paper frames RFT as a way to create domain-specific reasoning models from dozens of high-quality question-answer pairs.Source: §1 · p. 1
- OpenRFT is an early implementation developed under the RFT demo settings and inspired by o1-replication efforts.The authors say stronger generalist reasoning models and corresponding PRMs are needed for RFT feasibility.Source: §1 · pp. 1–2
- OpenRFT addresses missing reasoning steps by synthesizing reasoning-process data for SFT and using a PRM to supervise reasoning during RL.The authors say PRM supervision can stabilize RL training.Source: §1 · pp. 1–2
- To address limited sample quantities, OpenRFT augments questions and uses domain-specific examples in few-shot ICL to guide policy exploration.Question rewriting and option shuffling increase data volume.Source: §1 · p. 2
- On eight SciKnowEval tasks corresponding to L3 reasoning across biology, chemistry, physics, and materials science, OpenRFT increased performance by an average of 11% using only 100 domain-specific samples.Source: §1 · p. 2
2 METHOD
OpenRFT uses limited domain-specific samples in augmentation, reasoning-process synthesis and SFT, and few-shot ICL within a PRM-supervised RL framework. Its method comprises six components, from data augmentation through RL-based exploration and self-improvement.
2 METHOD
- OpenRFT combines question augmentation, SFT-based imitation, and PRM-supervised RL exploration using few-shot ICL; the reported experiments use the policy model itself to synthesize reasoning data.The authors say this choice reflects the lack of a stronger reasoning model with consistent actions.Source: §2 · p. 2
2.1 DATA AUGMENTATION
- Question rewriting preserves the original meaning, then independently shuffled options create variations used alongside the original question in PPO.The task instruction requests five distinct meaning-preserving expressions.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2.1 REASONING PROCESS SYNTHESIS
- For each domain question, multi-sampling generates reasoning paths, and at least one path that infers the correct answer is selected to form reasoning-process training data.These question, reasoning-step, and answer records are aggregated into a dataset for SFT.Source: §2.2.1 · p. 4
2.2.2 SFT OF POLICY MODEL
- SFT initializes the policy model on complete synthesized solutions by maximizing the likelihood of their reasoning paths and answers given each question.The training balances mimicking synthesized paths with generalizing to new questions.Source: §2.2.2 · p. 4
2.3.1 FEW-SHOT ICL-BASED DOMAIN KNOWLEDGE EMBEDDING
- Few-shot ICL retrieves the top k most similar question-answer pairs for each sample and appends them to task instructions to enrich the prompt with domain knowledge.The method targets exploration inefficiency when only a few dozen samples are available.Source: §2.3.1 · p. 4
2.3.2 RL WITH PRM
- In RL, the final answer receives an outcome reward, reasoning steps receive PRM scores, and the rewards are combined to produce the sample's final reward.The resulting reward signal reflects both answer accuracy and reasoning-process scores.Source: §2.3.2 · pp. 4–5
3 EXPERIMENTS
Experiments evaluate OpenRFT on eight SciKnowEval L3 tasks using limited training data, compare variants of fine-tuning and reinforcement learning, and examine data scaling and teacher–student alignment. The reported results favor augmentation in the limited-data setting, while the discussion identifies action-space mismatch as a concern for synthesized reasoning data.
3.1 DATASETS
- The study samples 100 training and 100 testing examples from each of eight SciKnowEval L3 subsets, consisting of multiple-choice questions across biology, chemistry, physics, and materials.Source: §3.1 · p. 5
3.2 COMPARISON METHODS
- The comparison includes vanilla and SFT baselines, ReFT variants, and a full OpenRFT variant combining augmentation, self-distillation SFT, process-supervised RL, and ICL.The full variant applies the modules in that sequence.Source: §3.2 · pp. 5–6
3.3 EXPERIMENTAL SETUP
- Experiments use Skywork-o1-Open-Llama-3.1-8B as policy and Skywork-o1-Open-PRM-Qwen-2.5-7B as process reward model, with sixfold augmentation and retrieval of the three most similar examples for ICL.The provider has not stated whether the policy and reward model action spaces are aligned, which may affect RFT performance.Source: §3.3 · p. 6
3.4 RESULTS
- SFT+RL(PRM)+DA consistently improves over SFT+RL(PRM) across tasks and averages 11% higher performance than Vanilla, the best result among Skywork-o1-initialized methods.Adding few-shot ICL yields no further improvement and slightly decreases performance overall.Source: Table 1 · p. 7, §3.4 · p. 7
3.5 DISCUSSIONS
- On T8, all three methods improve as training data increases, while augmentation has its strongest effect with small datasets and diminishing benefit as data grows.The authors suggest augmentation errors, especially on complex molecular formulas, may contribute to that decline.Source: §3.5 · pp. 7–8, Figure 3 · p. 8
- SFT+ using QwQ-32B-Preview reasoning data significantly underperforms student-generated SFT and falls below Vanilla, which the authors attribute to teacher–student action-space mismatch.The authors use this comparison to emphasize aligning teacher and student action spaces.Source: Table 2 · p. 8, §3.5 · p. 8
4 RELATED WORK
The related work distinguishes direct-answer System-1 inference from multi-step System-2 reasoning, then reviews approaches to acquiring reasoning and fine-tuning foundation models. It positions RFT as distinct from prior RL fine-tuning because domain samples and a reasoning policy differ in task and action format.
4 RELATED WORK
- System-1 infers an answer directly from a question, whereas System-2 derives intermediate reasoning steps before producing an answer.The section frames its review around these two inference modes.Source: §4 · pp. 8–9
4.1 ACQUIRING SYSTEM-2 CAPABILITY FROM SYSTEM-1 MODEL
- System-2 capabilities can be added to a System-1 model through prompting, SFT on reasoning-step data, or RL that autonomously explores reasoning steps.ReFT combines an SFT warm-up with RL, but targets System-2 capability acquisition from a System-1 model; the paper says it cannot be directly applied to standard RFT.Source: §4.1 · p. 9
4.2 FINE-TUNING FOUNDATION MODELS
- Fine-tuning smaller models for specific tasks can improve domain performance and make inference more efficient, with o1-mini described as sometimes matching or surpassing o1.This is presented as a general motivation for domain-specific foundation-model fine-tuning.Source: §4.2 · p. 9
- System-2 fine-tuning uses a model’s existing reasoning to explore and learn through trial and error, while domain-specific reasoning steps can improve adaptation.The section contrasts this with SFT’s reliance on annotated data and overfitting risk, and notes that expert reasoning steps help determine the action space.Source: §4.2 · pp. 9–10
4.3 EMPLOYING REINFORCEMENT LEARNING FOR FINE-TUNING
- RL fine-tuning optimizes cumulative reward and generates experience through interaction, unlike SFT’s fixed labeled-data objective.The related-work discussion distinguishes RFT by using domain-specific samples as the reward-model source and a reasoning foundation model as the policy.Source: §4.3 · p. 10
- RFT faces inconsistent tasks and action modes: it adapts a foundation model to downstream domains, while its samples lack the reasoning steps produced by its multi-step policy.The paper says these differences prevent direct application of existing RL fine-tuning methods.Source: §4.3 · pp. 10–11
- Whether OpenAI’s formal RFT release will accept training data containing reasoning steps remains uncertain.The footnote leaves that release capability unresolved.Source: §4.3 · p. 10
5 CONCLUSION & FUTURE WORKS
OpenRFT uses limited domain-specific samples for question augmentation, reasoning-process synthesis, and few-shot ICL within an RL framework supervised by a general PRM. The paper identifies future work in domain knowledge embedding, data augmentation, and generalization, including reward and action design for open-ended tasks.
5 CONCLUSION & FUTURE WORKS
- OpenRFT uses question augmentation, synthesized reasoning-process data, and few-shot ICL within an RL framework supervised by a general PRM.The authors describe this as a simple implementation for fine-tuning generalist reasoning models on domain-specific tasks.Source: §5 · p. 11
- Future data work includes generating questions from unlabeled data and exploring harder samples through adversarial learning, because question sets of only dozens remain insufficient.The authors also identify domain knowledge embedding for both the policy model and PRM as a direction for further exploration.Source: §5 · p. 11
- Generalization remains a fundamental RFT challenge, especially when open-ended formats require suitable reward functions and action spaces for efficient search.The paper gives long-form technical reports as an example beyond multiple-choice questions.Source: §5 · p. 11
A APPENDIX
Figure 4 illustrates a chemical-equilibrium example in which an erroneous intermediate calculation coexists with a correct final answer.
A APPENDIX
- Figure 4 uses a chemical-equilibrium problem as its example.The figure focuses on a reasoning flaw in domain-specific data.Source: Figure 4 · p. 14
- An incorrect molar-mass calculation in Step 2 produces a wrong intermediate result despite the correct final answer.The red font highlights the reasoning errors.Source: Figure 4 · p. 14
- The passage says process supervision can effectively avoid the illustrated reasoning problems.Source: Figure 4 · p. 14