Source-linked AI summary
Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization
Wataru Kawakami, Keita Suzuki, Junichiro Iwasawa
TL;DR
Clinical use of medical LLMs is challenged by factual and language-related limitations, as well as reasoning instability when explanations are required. The paper combines continued pretraining and reasoning preference optimization for Japanese medicine, achieving 0.868 on IgakuQA both with and without explanation prompting.
Problem
Medical LLM deployment faces factual inaccuracies, language-dependent performance, and degradation when reasoning explanations are required, despite clinicians’ need for rationales.
Source: §1 · p. 1Method
The paper adapts Qwen2.5-72B with CPT on Japanese medical data followed by RPO trained on preference comparisons of ground-truth and model-generated explanations.
Source: §1 · p. 2Results
0.868: Preferred-MedLLM-Qwen-72B scored this in standard 3-shot IgakuQA and maintained the same score when explanations were required.
Source: §4.2 · p. 5Takeaways & Limitations
The results support combining CPT for domain knowledge with RPO to maintain accuracy during explanation generation on IgakuQA.
Source: §5 · p. 7Evaluation primarily focused on multiple-choice question-answering benchmarks, leaving performance on broader clinical tasks unassessed.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Large Language Models (LLMs) show potential in medicine, yet clinical adoption is hindered by concerns over factual accuracy, language-specific limitations (e.g., Japanese), and critically, their reliability when required to generate reasoning explanations -- a prerequisite for trust. This paper introduces Preferred-MedLLM-Qwen-72B, a 72B-parameter model optimized for the Japanese medical domain to achieve both high accuracy and stable reasoning. We employ a two-stage fine-tuning process on the Qwen2.5-72B base model: first, Continued Pretraining (CPT) on a comprehensive Japanese medical corpus instills deep domain knowledge. Second, Reasoning Preference Optimization (RPO), a preference-based method, enhances the generation of reliable reasoning pathways while preserving high answer accuracy. Evaluations on the Japanese Medical Licensing Exam benchmark (IgakuQA) show Preferred-MedLLM-Qwen-72B achieves state-of-the-art performance (0.868 accuracy), surpassing strong proprietary models like GPT-4o (0.866). Crucially, unlike baseline or CPT-only models which exhibit significant accuracy degradation (up to 11.5\% and 3.8\% respectively on IgakuQA) when prompted for explanations, our model maintains its high accuracy (0.868) under such conditions. This highlights RPO's effectiveness in stabilizing reasoning generation. This work underscores the importance of optimizing for reliable explanations alongside accuracy. We release the Preferred-MedLLM-Qwen-72B model weights to foster research into trustworthy LLMs for specialized, high-stakes applications.
1 Introduction
The paper introduces a two-stage approach for Japanese medical LLMs that targets high accuracy and reliable explanations, addressing reasoning instability that complicates clinical trust.
1 Introduction
- Clinical adoption is challenged by factual inaccuracies, language-dependent performance, and degradation when models must provide reasoning explanations.Clinicians often need rationales for verification and trust.Source: §1 · p. 1
- The model combines CPT on a comprehensive Japanese medical corpus with RPO using curated comparisons of ground-truth and model-generated explanations.CPT targets domain knowledge; RPO targets stable, high-quality reasoning while maintaining accuracy.Source: §1 · p. 2
- Preferred-MedLLM-Qwen-72B achieves state-of-the-art IgakuQA accuracy and maintains high accuracy when explanations are required, unlike CPT-only models.The paper attributes the stability to RPO following CPT.Source: §1 · p. 2
- The authors release the model weights to facilitate research into trustworthy specialized LLMs.The paper emphasizes optimizing for explanation reliability alongside accuracy.Source: §1 · p. 2
2 Related Works
Prior medical LLM work includes English-focused foundation models and domain adaptation, while Japanese efforts leave open questions about large-model knowledge integration and reasoning stability.
2 Related Works
- Medical LLM research includes English foundation models, specialized-data training, prompt engineering, and methods for improving reasoning.The paper identifies hallucinations and inconsistent performance across languages and cultural contexts as persistent challenges.Source: §2 · pp. 2–3
- Continued pretraining adds domain knowledge, whereas instruction fine-tuning aligns models with specific instructions or interaction styles.These are presented as common strategies for adapting LLMs to specialized data and tasks.Source: §2 · p. 2
- Japanese medical LLM efforts have often used smaller models or focused on instruction following, leaving potential gaps in large-model knowledge integration and reasoning optimization.Source: §2 · pp. 2–3
- This work combines CPT with RPO to target Japanese medical QA and explanation stability, reporting state-of-the-art IgakuQA performance maintained under explanation prompting.RPO is described as targeting stability in the likelihood of the preferred reasoning path.Source: §2 · p. 3
3 Methods
The method adapts Qwen2.5-72B in two stages: CPT for Japanese medical knowledge, followed by RPO to improve the reliability of reasoning explanations.
3 Methods
- The method uses a two-stage fine-tuning process on a powerful base LLM to create Preferred-MedLLM-Qwen-72B, tailored specifically to the Japanese medical domain with enhanced reasoning capabilities.Source: §3 · p. 3
3.1 Stage 1: Continued Pretraining (CPT) for Domain Adaptation
- CPT uses Japanese medical texts, including JMLE explanations through 2017, to add domain knowledge while keeping evaluation questions from 2018–2022 outside the training materials.The stated cutoff is intended to avoid direct data leakage into IgakuQA.Source: §3.1 · p. 3
3.2 Stage 2: Reasoning Preference Optimization (RPO)
- RPO follows CPT to address explanation-related instability and train for accurate answers with coherent, reliable reasoning.Source: §3.2 · pp. 3–4
- RPO trains on ground-truth, correct model-generated, and incorrect responses ordered from highest to lowest preference.Its loss combines weighted chosen-response NLL with DPO preference pairs, including ground-truth over chosen and rejected responses.Source: §3.2 · p. 4
4 Results
Preferred-MedLLM-Qwen-72B combines CPT and RPO to achieve strong IgakuQA accuracy while maintaining it under explanation prompting. Ablations and additional Japanese medical QA benchmarks examine the stages’ contributions and broader performance.
4 Results
- The results section evaluates IgakuQA against baseline and proprietary models, then tests CPT and RPO contributions and generalization to other Japanese medical QA datasets.IgakuQA uses Japanese National Medical Licensing Exam questions from 2018 to 2022, with scores averaged across years and normalized by total points.Source: §4 · p. 4, §4.1 · p. 4
4.1 Performance on the IgakuQA Benchmark
- 0.868 accuracy in standard IgakuQA prompting slightly exceeds GPT-4o’s 0.866, and remains 0.868 when explanations are required.The evaluation averages scores across the 2018–2022 exams, normalized by total possible points.Source: §4.1 · pp. 4–5, Table 1 · p. 5
4.2 Ablation Study: Impact of CPT and RPO
- RPO alone scores 0.838 in 3-shot and 0.807 with explanations, while CPT alone improves standard-setting accuracy but drops when explanations are required.The baseline scores 0.802 and 0.710, respectively; the study compares CPT, RPO, and DPO applied to Qwen2.5-72B.Source: §4.2 · p. 5, Table 2 · p. 6
- 0.868 to 0.868 under explanation prompting for CPT+RPO contrasts with CPT+DPO’s 0.868 to 0.848 and CPT-only’s 0.867 to 0.834.The ablation attributes domain knowledge to CPT and reasoning-generation refinement to RPO; the comparison suggests RPO’s NLL loss component may offer superior stabilization.Source: §4.2 · pp. 5–6, Table 2 · p. 6
4.3 Performance on Other Japanese Medical Benchmarks
- Across other Japanese medical QA benchmarks, CPT+RPO achieves the highest average score, 0.716, versus 0.710 for CPT alone.In zero-shot evaluations, the model leads on Japanese medical-related MMMLU tasks and translated MedQA and MedMCQA.Source: §4.3 · p. 6, Table 3 · p. 6
5 Conclusion and Discussions
The paper presents Preferred-MedLLM-Qwen-72B as a Japanese medical model that combines CPT for domain knowledge with RPO for reasoning stability. It reports stable accuracy under explanation prompting, while noting evaluation and deployment boundaries.
5 Conclusion and Discussions
- CPT followed by RPO is reported to mitigate reasoning instability seen with CPT alone or CPT combined with standard DPO.The paper’s IgakuQA results report high accuracy both with and without explanation prompting.Source: §5 · p. 7
- The model’s evaluation primarily covers multiple-choice QA; broader clinical tasks and real-world validation remain future work.The authors also identify semi-automatically generated preference data, untested combinations with instruction tuning, and the computational cost of 72B models as limitations.Source: §5 · p. 7
- The conclusion frames combining CPT for knowledge and RPO for reasoning alignment as a pathway toward stable performance during explanation generation.The authors position this as relevant to specialized, high-stakes applications, including clinical practice.Source: §5 · p. 7
A Appendix
Appendix Table A.1 provides an example of human-generated explanatory data.
A Appendix
- Table A.1 presents an example of human-generated explanatory data.Source: Table A.1 · p. 13