Source-linked AI summary
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, Benyou Wang
TL;DR
Medical reasoning has been less explored than mathematical reasoning, despite requiring careful, reliable deliberation and posing verification challenges. The paper constructs verifiable medical problems and uses verifier-guided search followed by reinforcement learning to train HuatuoGPT-o1. The resulting approach improves medical benchmark performance and outperforms comparable open-source baselines, with complex reasoning benefiting RL.
Problem
Medical reasoning is underexplored relative to mathematical reasoning, while its correctness is difficult to verify despite the need for reliable answers.
Method
The paper constructs verifiable medical problems and uses verifier-guided trajectory search for fine-tuning, followed by PPO-based reinforcement learning with verifier rewards.
Results
HuatuoGPT-o1 outperforms open-source general and medical-specific baselines; complex CoT improves performance by 4.3 points and produces larger RL gains than simple or no CoT.
Takeaways & Limitations
Complex reasoning improves medical problem-solving and benefits RL enhancements within the evaluated medical tasks.
Abstract
from arXiv · showhide
The breakthrough of OpenAI o1 highlights the potential of enhancing reasoning to improve LLM. Yet, most research in reasoning has focused on mathematical tasks, leaving domains like medicine underexplored. The medical domain, though distinct from mathematics, also demands robust reasoning to provide reliable answers, given the high standards of healthcare. However, verifying medical reasoning is challenging, unlike those in mathematics. To address this, we propose verifiable medical problems with a medical verifier to check the correctness of model outputs. This verifiable nature enables advancements in medical reasoning through a two-stage approach: (1) using the verifier to guide the search for a complex reasoning trajectory for fine-tuning LLMs, (2) applying reinforcement learning (RL) with verifier-based rewards to enhance complex reasoning further. Finally, we introduce HuatuoGPT-o1, a medical LLM capable of complex reasoning, which outperforms general and medical-specific baselines using only 40K verifiable problems. Experiments show complex reasoning improves medical problem-solving and benefits more from RL. We hope our approach inspires advancements in reasoning across medical and other specialized domains.
1 Introduction
Medical reasoning requires careful deliberation, but o1-like reasoning research has largely focused on mathematics and medical thought processes are difficult to verify. The paper addresses this gap with verifiable medical problems, verifier-guided reasoning search, and reinforcement learning.
- Research replicating o1-like advances has largely remained focused on mathematical reasoning, leaving specialized fields such as medicine underexplored.
- Medical decisions demand complex reasoning because reliable answers are especially important in this life-critical domain.
- 40K verifiable medical problems use open-ended questions with unique objective answers that an LLM verifier can check.
- Stage 1 searches for correct reasoning trajectories through verifier feedback and strategies including backtracking, exploration, verification, and correction before fine-tuning.
- Stage 2 applies PPO with sparse verifier rewards to further refine complex reasoning through reinforcement learning.
- Using only 40K data points, the method improves medical benchmarks by 8.5 points with an 8B model, while the 70B model outperforms open-source general and medical-specific baselines.
- The contributions include verifiable medical problems, two-stage training, HuatuoGPT-o1, and evidence that complex reasoning benefits medical problem-solving and RL.
2 Verifiable Medical Problems
The paper transforms challenging closed-set medical exam questions into open-ended problems with unique answers, then uses a verifier to assess model outputs against those answers.
- Verifiable medical problems are designed as open-formal tasks with unique, objective ground-truth answers, enabling outcome-based reasoning verification.
- The source pool contains 192K medical multiple-choice questions collected from MedQA-USMLE and MedMcQA.
- Closed-set questions can permit guessing and may lack a unique correct answer or sufficient complexity for reasoning.
- The processing pipeline filters easy, ambiguous, and unsuitable questions before reformatting selected items into open-ended problems.
- The resulting dataset contains 40K verifiable medical questions represented as problem–ground-truth-answer pairs.
- The verifier evaluates a generated Chain-of-Thought and result by checking the result against the ground-truth answer.
- An LLM-based verifier is used because medical aliases make exact matching impractical, and experiments report its reliability.
3 Methodology
The methodology uses verifier-guided search to construct complex medical reasoning trajectories for fine-tuning, then applies verifier-based reinforcement learning to refine reasoning further.
- Stage One: Learning Complex Reasoning: Stage 1 searches for correct reasoning trajectories by iteratively applying four strategies after verifier rejection: exploring new paths, backtracking, verification, and correction.Search continues until the answer is verified, with up to three iterations and three attempts per problem; failed problems are discarded.
- Stage One: Learning Complex Reasoning: Successful search trajectories are reformatted into coherent complex CoTs and formal responses for supervised fine-tuning.The reformatting uses smooth transitions to streamline reasoning and reduce token usage, teaching think-before-answering behavior.
- Stage Two: Enhance Complex Reasoning with RL: Stage 2 uses PPO reinforcement learning on remaining verifiable problems, with verifier-based rewards guiding policy updates.The policy samples a reasoning-response pair, computes its reward, and updates parameters.
- Stage Two: Enhance Complex Reasoning with RL: Correct answers receive reward 1, incorrect answers 0.1, and null responses 0; the total reward also includes KL divergence scaled by β.The KL term helps stabilize training with sparse rewards.
4 Experiments
Experiments evaluate HuatuoGPT-o1 on medical benchmarks, verifier reliability, ablations of reasoning and RL, and transfer to Chinese medical tasks. Results support complex reasoning, verifier-guided training, and the two-stage strategy.
- Experimental Setup: The evaluation covers MedQA, MedMCQA, PubMedQA, medical MMLU-Pro and GPQA tracks, with comparisons against general and medical-specific open-source LLMs.GPQA results were averaged over five runs because of its limited number of questions.
- Main Results: HuatuoGPT-o1 performs across all datasets; its 8B version improves 8 points over LLaMA-3.1-8B-Instruct, while the 70B version surpasses comparable open-source models including QwQ-32B.The two-stage strategy also improves performance over fine-tuning without RL.
- Ablation Study: Complex CoT improves performance by an average of 4.3 points, whereas direct response learning performs worst and simple CoT provides little benefit.The authors associate the gain with teaching models to refine answers through reflection.
- Ablation Study: Complex CoT yields a 3.6-point RL gain with an average of 712 tokens, exceeding simple CoT’s 2.6 points and no CoT’s 1.1 points.The comparison measures performance improvement from RL across reasoning strategies.
- Ablation Study: PPO performs best among the compared RL algorithms, followed by RLOO and DPO, under the same reward function.The paper attributes PPO’s advantage to its use of value models, while describing DPO as off-policy.
- Verifier Reliability: GPT-4o verifier accuracy is 96.5% in Stage 1 and 94.5% in Stage 2, compared with 70.5% and 74.5% for Exact Match.A fine-tuned 8B verifier also exceeds 90% accuracy, based on 20,000 scoring samples.
- Domain Compatibility: Applying the approach to 40,000 Chinese medical questions, HuatuoGPT-o1-7B-zh outperforms similarly sized Chinese LLMs.The result is presented as evidence of adaptability to a new domain.
5 Related Work
Prior work has explored LLM reasoning, medical-specific language models, and complex reasoning, but medical applications of o1-like methods remain underexplored. Existing approaches face challenges in scaling reasoning supervision and obtaining reliable verification or rewards for specialized domains.
- o1-like reasoning research has largely addressed mathematics, vision-language integration, and open-ended problem-solving rather than medical or other highly specialized fields.
- Medical LLMs commonly adapt generalist models through prompting or develop models specifically for medical tasks.
- Chain-of-Thought prompting and externally supervised model-generated paths improve reasoning, but expert-labeled paths remain costly and scalability challenges persist.
- Reward-model and oracle-based reinforcement learning methods can be slow, expensive, and constrained by supervision bottlenecks.
- Reflective reasoning methods have succeeded in general tasks but remain underexplored in medicine and struggle without reliable reward functions or verifiers.
6 Conclusion
The study advances medical LLM reasoning by constructing verifiable problems and a verifier, then using verifier-guided trajectory learning followed by reinforcement learning. HuatuoGPT-o1 achieves strong medical benchmark performance, and validation in Chinese medical contexts indicates adaptability beyond the initial setting.
- The study constructs medical verifiable problems and a medical verifier to support a two-stage training process.
- The first stage learns complex reasoning, while the second enhances it through reinforcement learning.
- HuatuoGPT-o1 is a medical LLM with thinks-before-it-answers behavior and outstanding medical benchmark performance.
- Additional validation in Chinese medical contexts shows the method’s adaptability to other fields.
A Ethical Statement
The authors state that HuatuoGPT-o1 may still generate hallucinations or inaccuracies. Because of these risks, they restrict its use in clinical and other industry applications where errors could have unintended consequences.
- HuatuoGPT-o1 may produce hallucinations or inaccuracies despite its complex reasoning capabilities.
- The authors prohibit clinical and other industry applications where inaccuracies could lead to unintended consequences.
- Users are expected to follow strict limitations to safeguard application safety and integrity.
B Constructing Medical Verifiable Problems
The paper constructs medical verifiable problems by filtering challenging, unambiguous exam questions, converting them into open-ended questions with unique answers, and using GPT-4o to verify model outputs. It then builds reasoning trajectories through iterative CoT refinement using multiple search strategies.
- Constructing Medical Verifiable Problems: Challenging questions are retained by removing items answered correctly by three small LLMs and discarding short questions.
- Constructing Medical Verifiable Problems: Questions with non-unique or ambiguous answers are excluded before reformatting selected multiple-choice questions into open-ended problems.
- Constructing Medical Verifiable Problems: Each reformatted problem targets the original tested point and includes a concise standard answer suitable for precise correctness matching.
- Verifier: GPT-4o compares a model response with the ground-truth answer and returns True for an accurate response or False otherwise.
- Searching Trajectories: GPT-4o first generates an initial Chain of Thought, then iteratively refines incorrect outputs through Backtracking, Exploring New Paths, Verification, and Correction strategies.
- Searching Trajectories: The reasoning format uses Inner Thinking, Final Conclusion, and Verification, returning to further reasoning when verification rejects the conclusion.
E Prompts for Constructing SFT Training Data
The procedure reformats successful reasoning trajectories into natural-language complex chain-of-thought and then generates a formal answer from that reasoning. The prompts emphasize stepwise, conversational, detailed reasoning without rigid formatting.
- Trajectory reformatting: Successful trajectories are reformatted into coherent natural-language reasoning processes called Complex CoT.The trajectory contains alternating reasoning and answer elements before being transformed into ˆe.
- Trajectory reformatting: The reformatting prompt requires step-by-step thoughts, natural transitions, and richer logical detail.Transitions may use conversational expressions such as “hmm,” “oh,” “also,” and “wait.”
- Output format: The prompt directs the model to return the revised natural thinking directly in JSON format.This output instruction follows the requested natural-thinking rewrite.
- Formal response generation: The resulting formal response ˆy is generated for question x using the complex CoT ˆe.The formal answer uses the conclusion of the complex reasoning process.
F Settings of other RL training
The study compares PPO with DPO and RLOO as alternative reinforcement-learning-related algorithms. DPO uses verifier-labeled correct and incorrect answer pairs, while RLOO retains PPO’s reward and parameters with rloo_k set to 2.
- Algorithm comparison: DPO and RLOO were compared with PPO as alternative RL-related training algorithms.The comparison included preference learning through DPO and a REINFORCE-style method through RLOO.
- DPO: DPO trains on verifier-identified pairs containing one correct and one incorrect answer.Questions without such pairs are discarded; verified correct answers provide positive examples and failed verifications provide negative examples.
- DPO: DPO training used a learning rate of 1e-6, batch size 128, and regularization parameter 1.These are the stated DPO hyperparameters.
- RLOO: RLOO used PPO’s reward function and parameters, adding rloo_k=2.The RLOO configuration otherwise matched PPO.
G Chinese Medical Model
HuatuoGPT-o1-7B-zh extends the training process to Chinese medical problems using Qwen2.5-7B-Instruct as its base. Evaluation covers Chinese medical benchmarks and selected medical tracks in CMMLU, with comparisons against general and medical Chinese models.
- Chinese model: HuatuoGPT-o1-7B-zh was built on Qwen2.5-7B-Instruct using Chinese medical verifiable problems.The Chinese training process followed the same process as the English HuatuoGPT-o1.
- Evaluation: Chinese evaluation used MedQA’s Chinese test set, CMB-Exam, and CMExam.These are the three Chinese medical benchmarks listed for assessment.
- Evaluation: CMMLU evaluation covered clinical knowledge, agronomy, college medicine, genetics, nutrition, and Traditional Chinese Medicine.These tracks were included as part of the medical section of the Chinese general benchmark.
- Comparison models: Comparisons included Qwen2.5, GLM-4, Yi, and the Chinese medical model HuatuoGPT-2-7B.The comparison set combines general Chinese models with one Chinese medical baseline.