PaperlayerView exact revision on arXiv

Source-linked AI summary

O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoning

Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, Xiaofan Zhang

arXiv:2501.06458v1cs.CL

TL;DR

Medical reasoning tasks are complex, and the application of inference-time scaling to medicine remains an open area of investigation. The paper combines inference-time scaling with journey-learning data distilled from o1 and evaluates it on medical benchmarks. It reports improved benchmark performance with 500 training samples, while finding that more difficult tasks call for longer reasoning.

  • Problem

    Medical reasoning involves complex tasks, while inference-time scaling has been validated in other domains but presents distinct opportunities and challenges in medicine.

    Source: §1 · p. 2

  • Method

    The study distills o1-generated LongStep and LongMonolog reasoning data for journey learning and evaluates inference-time scaling on medical benchmarks.

    Source: §2.2 · pp. 3–4, §2.1 · p. 3

  • Results

    6–11% accuracy gains across MedQA, Medbullets, and JAMA Clinical Challenges were achieved with 500 training samples.

    Source: §4 · p. 8

  • Takeaways & Limitations

    More difficult tasks are associated with longer reasoning outputs, supporting extended thought processes for challenging medical problems.

    Source: §1 · p. 2, §3.4 · p. 7

    Inference-time scaling requires sufficient model capability, and LongMonolog does not consistently outperform LongStep across benchmarks.

    Source: §3.3 · pp. 6–7

Abstract

from arXiv · show

Building upon our previous investigations of O1 replication (Part 1: Journey Learning [Qin et al., 2024] and Part 2: Distillation [Huang et al., 2024]), this work explores the potential of inference-time scaling in large language models (LLMs) for medical reasoning tasks, ranging from diagnostic decision-making to treatment planning. Through extensive experiments on medical benchmarks of varying complexity (MedQA, Medbullets, and JAMA Clinical Challenges), our investigation reveals several key insights: (1) Increasing inference time does lead to improved performance. With a modest training set of 500 samples, our model yields substantial performance improvements of 6%-11%. (2) Task complexity directly correlates with the required length of reasoning chains, confirming the necessity of extended thought processes for challenging problems. (3) The differential diagnoses generated by our model adhere to the principles of the hypothetico-deductive method, producing a list of potential conditions that may explain a patient's symptoms and systematically narrowing these possibilities by evaluating the evidence. These findings demonstrate the promising synergy between inference-time scaling and journey learning in advancing LLMs' real-world clinical reasoning capabilities.

1 Introduction

Medical reasoning spans diagnosis and treatment planning, where multifactorial conditions challenge conventional scaling. This work explores inference-time scaling and journey learning for medical tasks, highlighting longer reasoning for harder problems.

1 Introduction

  • Medical reasoning requires integrating clinical findings and evaluating multiple hypotheses, while traditional parameter- and data-scaling methods struggle with its complexity.The introduction frames inference-time scaling as allowing more processing for step-by-step problem-solving and iterative refinement.Source: §1 · p. 2
  • The study examines how inference-time computing performs across medical task complexities and how to synthesize supervised data for extended reasoning chains.It builds on journey learning and uses knowledge distillation from GPT-series models.Source: §1 · p. 2
  • The introduction identifies four findings: majority voting has limits on complex cases, scaling requires capable models, harder tasks need longer reasoning, and free-form responses support medical journey learning.The authors also report that removing multiple-choice options unlocks potential for more nuanced clinical reasoning.Source: §1 · p. 2
  • The authors release findings, distilled datasets, and experimental methodologies, while calling for continued scrutiny of distillation and rigorous evaluation in healthcare.They present these as contributions to collaborative research on inference-time scaling in clinical applications.Source: §1 · p. 2

2 Exploration Process

The exploration evaluates inference-time scaling on medical problems using public benchmarks and synthesized long-form reasoning data. It focuses on task difficulty, benchmark scope, and training examples that support journey learning.

2 Exploration Process

  • The exploration investigates inference-time scaling for complex, domain-specific medical challenges, with difficulty-related reasoning length as an initial observation.The authors note that difficulty increases alongside inference time and that the mechanisms behind its problem-solving effects remain underexplored.Source: §2 · pp. 2–3

2.1 Benchmark Overview

  • The benchmarks cover clinical cases and licensing exams; JAMA evaluation uses 646 cases, half of which are challenging for o1-mini.JAMA's 1,524-example archive spans 13 medical domains and runs from July 2013 to October 2023; Medbullets and MedQA draw on US medical board or USMLE exams.Source: §2.1 · p. 3

2.2 Journey Learning Data Synthesis

  • The training examples use two distilled formats: LongStep extracts o1's detailed solution steps, while LongMonolog expands summarized thoughts into reflective reasoning.The authors state that o1's internal thought processes are unavailable directly, so its summarized thoughts are expanded using prompts; the resulting data is preprocessed for quality and format.Source: §2.2 · p. 3
  • The 500-example training set draws 350 cases from MedQA and 150 from JAMA, excludes short reasoning cases, and uses multiple-choice options to guide hypotheses.The authors avoid in-house data because irrelevant clinical information can interfere with reasoning; mean example lengths are 729 for LongStep and 1,223 for LongMonolog.Source: §2.2 · pp. 3–4
  • Figure 2 presents weighted mean accuracy for Qwen2.5-72B-Instruct, LLama3.1-70B, and Qwen2.5-32B across three datasets and distinct strategies.The caption identifies the models, datasets, and strategy-based comparison but does not provide individual scores.Source: Figure 2 · p. 5

3 Experiments

Across medical benchmarks, the experiments report improved accuracy with inference-time scaling, while longer outputs accompany harder tasks. The section also examines model capacity, majority voting, and a free-response diagnostic setting.

3.1 Implementation Details

  • The experiments use three epochs of LoRA instruction tuning on Qwen2.5-32B, Qwen2.5-72B, and Llama3.1-70B, with 8 NVIDIA A800 GPUs, a 1 × 10−4 learning rate, and batch size 8.The authors selected these open-source base models in light of pilot-study performance and resource constraints.Source: §3.1 · p. 5

3.2 Main Results

  • Table 1 compares proprietary APIs, open-source baselines, and fine-tuned models; the authors report that their methods improve performance even with minimal training data.The comparison includes vanilla CoT, CoT SFT, LongStep, and LongMonolog settings.Source: §3.2 · p. 5

3.3 Does Inference-time Scaling Help?

  • Qwen2.5-72B gains +3.28%, +5.12%, +9.69%, and +11.36% across the respective reasoning strategies; longer journey-learning outputs average about 1,000 tokens and improve mean accuracy by about 10%.Vanilla CoT and CoT fine-tuning outputs range from 300 to 500 tokens and yield about a 5% increase in mean accuracy.Source: §3.3 · p. 5
  • LongStep and LongMonolog do not consistently outperform one another: Qwen2.5-32B reaches 56.34% accuracy on JAMA with LongStep versus 53.71% with LongMonolog.The authors suggest long thought processes can aid complex medical questions but require sufficient domain knowledge; redundant reflections sometimes accompany errors.Source: §3.3 · p. 6
  • Majority voting yields only a 0.32-point accuracy rise for Qwen2.5-72B-Vanilla on MedQA, from 74.31% to 74.63%, while LongStep and LongMonolog gain 1.26% and 1.50%, respectively.Qwen2.5-72B-Vanilla CoT peaks at 80.44% accuracy at higher token counts before declining to 79.81%.Source: §3.3 · p. 6

3.4 Harder Tasks, Longer Thoughts, More Inference Time

  • Harder tasks appear to benefit from longer outputs: Qwen2.5-72B averages 1,076 tokens on JAMA, 917 on Medbullets, and 873 on MedQA using detailed monolog.The authors rank task difficulty JAMA, Medbullets, then MedQA, based partly on overall performance across settings.Source: §3.4 · p. 7
  • Qwen2.5-32B sometimes produces longer think-aloud outputs on easier questions, while weaker models may become confused or reach incorrect conclusions during long monologs.The authors suggest the longer outputs can reflect unnecessary verbosity.Source: §3.4 · pp. 7–8

3.5 Generalizability and Future Directions

  • In a preliminary free-response study using 2024 JAMA Clinical Challenges, the journey-learning-trained model generated broader differential diagnoses and integrated contextual information toward concise conclusions.Training data were collected before October 2023 and included multiple-choice options, which were removed at evaluation.Source: §3.5 · p. 8

4 Conclusion

Inference-time scaling improved performance across MedQA, Medbullets, and JAMA Clinical Challenges, with reported gains of 6–11% using 500 training samples. The conclusion also points to longer reasoning for harder tasks and continued work toward clinical applications.

4 Conclusion

  • 6–11% accuracy gains across MedQA, Medbullets, and JAMA Clinical Challenges were achieved using 500 training samples.The conclusion also reports that majority voting had limited impact compared with long reasoning paradigms.Source: §4 · p. 8
  • Longer reasoning is associated with harder tasks, while shifting from multiple-choice to free-form responses showed potential for more nuanced medical journey learning.The conclusion describes this as fostering deeper clinical reasoning capabilities.Source: §4 · p. 8
  • The authors identify interpretability and effectiveness in real-world medical challenges as areas for continued exploration, alongside open resource sharing.Their long-term goal is systems that complement clinical expertise and address modern medicine’s growing complexity.Source: §4 · p. 8
  • A JAMA differential-diagnosis problem is paired with an incorrect Qwen2.5-72B response and a correct Qwen2.5-72B-LongMonolog response.Source: Figure 6 · p. 8, Figure 7 · p. 9, Figure 8 · p. 10

A Appendix

The appendix presents examples of JAMA problems and data synthesis, along with distilled data and model-generated thoughts. It also includes a generated failure case.

A Appendix

Related papers

Loading 2501.06458v1…