Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

1. Introduction

• Compact models are attractive for educational and health-informatics settings where model size and decoding costs constrain deployment.Rationale distillation uses a large teacher’s rationale as supervision for a smaller student.Source: §1 · p. 1
• Prior work improves rationale generation, selection, or encoding, but does not establish whether transfer comes from question-relevant content rather than length, style, or template.The cited approaches include separate label and explanation tasks, reasoning traces, multilingual or multi-teacher traces, retrieved knowledge, student-oriented selection, and latent reasoning.Source: §1 · p. 1
• Rationales can be irrelevant, support incorrect answers, or impose a model-specific format, risks especially consequential for compact open-weight students working outside English.The authors note that reasoning behavior may be anchored in another language.Source: §1 · p. 1
• Three unresolved issues are separating content benefit from long-target effects, accounting for teacher errors, and stabilizing comparisons when accuracy varies across data splits.Source: §1 · p. 1
• The study asks how to diagnose, select, and efficiently transfer reasoning supervision into a compact model, and why answer verification matters.Source: §1 · p. 1

2. Material and Methods

• Across ten paired splits, Stage 2 compared answer-only fine-tuning, step-by-step distillation, and single-target Distill-SFT on questions the baseline student missed but the teacher answered correctly.The study also evaluated ENEM to probe forgetting outside the residency-exam domain.Source: §2 · p. 2
• Only rationales associated with a correct teacher answer were used as training targets; this answer verification does not establish that every intermediate statement in a rationale is valid.Source: §2 · p. 2
• The coherent-versus-shuffled intervention held the question, student, prompt, and scoring rule fixed, estimating the effect of rationale relevance within this protocol.It does not identify the model’s internal causal mechanism or establish transportability beyond the evaluated setting.Source: §2 · p. 2
• A matched training ablation compared coherent and shuffled rationales on held-out transfer-subset questions, with delimiter sensitivity treated as a secondary control.The test set contained 164 questions per split.Source: §2 · p. 2
• Distill-SFT used one completion with the answer-verified rationale followed by the answer; step-by-step distillation used separate answer and rationale tasks, while Pure SFT used only the answer.Rationale tokens received weight α in Distill-SFT; both rationale-trained methods were evaluated under direct-answer and reasoning inference.Source: §2 · p. 3, Figure 1 · p. 3
• Stage 1 measured the frozen student’s probability for the correct A–E option as it read coherent or cross-question-shuffled rationale prefixes, stratified by teacher correctness.The explicit final-answer span was removed; the five answer-letter continuation logits were normalized over the restricted A–E set.Source: §2 · p. 3, Figure 1 · p. 3
• The loss balanced answer and rationale supervision with α values of 1.0, 0.3, and 0.1, weighting rationale tokens by α and answer tokens by 1 in the single-target method.Values below 0.1 were not advanced beyond preliminary runs because the rationale objective did not preserve a consistent reasoning signal.Source: §2 · p. 3
• Principal estimates used a two-level bootstrap with 10,000 replicates, paired split-level tests, and within-split exact McNemar tests.The examination questions were Brazilian Portuguese multiple-choice items; the authors note that these do not substitute for assessing clinical workflows or open-ended communication.Source: §2 · pp. 2–3

3.1. Rationales transmit the teacher’s answer direction

• 0.21 higher and 0.14 lower: coherent rationales changed gold-answer probability when the teacher was correct and wrong, respectively.The correctness-by-coherence interaction was 0.36 (95% cluster-bootstrap CI 0.33–0.39), showing that rationales carried the teacher’s answer direction.Source: §3.1 · p. 4
• Aligned targets outperformed shuffled targets; sensitivity checks preserved directional effects, while frozen-reader shifts did not distinguish techniques despite localized accuracy gains.The sensitivity analysis excluded overt option-wording cues but could not rule out semantic paraphrase.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.2. Efficient transfer, selective reasoning, and preserved capability

• 2.2 percentage points: Distill-SFT outperformed step-by-step on transfer subset B under reasoning, while step-by-step was stronger under direct inference.No differences were detected in aggregate ABC accuracy or difficult subset D under reasoning; neither rationale-trained method was globally dominant.Source: §3.2 · p. 4, Figure 3 · p. 5
• Step-by-step required reasoning decoding to reach its transfer-performance range; its direct mode remained competitive with pure SFT but did not reach that regime.At α = 1.0, reasoning improved step-by-step accuracy on subset B in all ten splits and on subset D in nine.Source: §3.2 · p. 4
• 44% fewer optimizer updates: Distill-SFT used 382 steps versus 678 for step-by-step, learning the rationale and answer in one target.The two methods had similar reasoning-output lengths, and no difference was detected on difficult subset D or in aggregate reasoning performance.Source: §3.2 · p. 4
• Down-weighting rationale loss improved direct accuracy on held-out items without a detected reasoning change; at α = 0.3, Distill-SFT also beat step-by-step on protected strata A and C under reasoning.At full rationale weight, hundreds of rationale tokens dominated the short answer and reduced direct performance on difficult items.Source: §3.2 · p. 5
• Reasoning recovered a net 40.5 correct answers per split against baseline, but gold-derived strata establish selective-reasoning potential rather than a deployable router.Distill-SFT was most useful when reasoning-enabled transfer was required, whereas step-by-step was stronger for direct prediction; reasoning also increased decoding latency.Source: §3.2 · p. 5
• On ENEM, answer-only SFT reduced reasoning accuracy from 0.85 to 0.75, while rationale-trained students retained accuracy near 0.82 and expressed reasoning in concise Brazilian Portuguese.ENEM was outside the adaptation data; rationale supervision mitigated but did not eliminate forgetting, without establishing mechanistic faithfulness.Source: §3.2 · p. 5, Figure 4 · p. 6

4. Discussion

• Across ten data splits, Distill-SFT improved transfer on baseline-missed questions under reasoning and used 44% fewer optimizer steps, while step-by-step remained stronger for direct answers.Rationale-based training also reduced the loss of reasoning ability on ENEM compared with answer-only training.Source: §4 · p. 6
• A localized final-answer gain without a detected difference in mean rationale influence means end-task accuracy alone cannot identify which traces are worth distilling.The authors frame auditing as a way to select question-relevant, answer-aligned, informative rationales for the target student.Source: §4 · p. 6
• Distill-SFT combines answer and rationale signals in one weighted completion, reducing training updates in the reasoning-enabled regime rather than ensuring uniform compute superiority.Step-by-step was stronger under direct inference, and its direct mode remained competitive with pure SFT.Source: §4 · p. 6
• Gains concentrated on transfer and difficult items, while direct answers remained preferable for knowledge already accessible to the student.The authors motivate a prospective gate that invokes reasoning when it is likely to correct a direct response.Source: §4 · p. 6
• The evaluation compared methods on the same questions across ten splits and used ENEM to assess retention beyond the residency-exam domain.These design choices aid interpretation but do not make an undetected difference proof of equivalence.Source: §4 · p. 6

4.1. Limitations

• The audit covers one model family, one language, and multiple-choice examinations; broader prospective validation is needed before clinical deployment.Answer verification checks only the final option, not the factuality, completeness, or clinical safety of intermediate rationale statements.Source: §4.1 · p. 6

• Question-relevant rationales moved a compact student toward the teacher’s answer, including when that answer was wrong.Source: §5 · pp. 6–7
• Distill-SFT used 44% fewer optimizer updates than step-by-step distillation and achieved higher accuracy on the target transfer subset under reasoning, while step-by-step was stronger under direct inference.Source: §5 · p. 7
• Prospective evaluation on clinical tasks is needed to determine whether these benefits extend beyond examination performance.Source: §5 · p. 7

• No ethics approval or informed consent was required because the study used no human participants, patient data, clinical records, or identifiable personal information.Source: Ethics statement · p. 7
• All experiments used publicly available examination questions and model-generated outputs.Source: Ethics statement · p. 7
• The ethics statement reports no use of patient or other identifiable personal information.Source: Ethics statement · p. 7

• The repository contains question-level study data, teacher rationales, canonical seed definitions, aggregate analysis tables, and analysis code.Source: Data availability · p. 7
• The released materials support auditing reported analyses and constructing comparable 85/15 seeded partitions.Source: Data availability · p. 7
• The examining bodies retain rights to the examination content, with provenance and use restrictions documented alongside the released data.Source: Data availability · p. 7

CRediT authorship contribution statement

• Martinelli is credited with methodology, software, formal analysis, investigation, data curation, visualization, and project administration.He also led the original draft.Source: CRediT authorship contribution statement · p. 7
• Papa and Pereira are both credited with conceptualization, validation, supervision, and writing–review and editing.Papa is additionally credited with resources and funding acquisition.Source: CRediT authorship contribution statement · p. 7
• Pereira is additionally credited with resources and funding acquisition.Source: CRediT authorship contribution statement · p. 7

Funding

• CAPEX funding from Hospital Israelita Albert Einstein supported a postdoctoral fellowship for TM and an academic visit by JPP.The hospital coordinated the study.Source: Funding · p. 7

Appendix A. Training and analysis

• Distill-SFT weighted rationale tokens by α and answer tokens by 1, with α values of 1.0, 0.3, and 0.1; step-by-step used α on its separate rationale task.The study compared these weights, pure SFT, and seven configurations across ten splits.Source: Appendix A · p. 7
• 0.390 and 0.391 coherent-rationale accuracies versus 0.237 and 0.285 for shuffled rationales; coherent-target delimiter formats showed no detected difference.The paired comparison gave p = 0.91; the authors caution this does not establish equivalence or general irrelevance of delimiters.Source: Appendix A · p. 8
• The paired t test across ten split-level accuracies was primary, with Wilcoxon tests as sensitivity analyses and item-level McNemar tests treated as diagnostics.Non-significant differences were not interpreted as equivalence; intervals were two-sided and unadjusted for multiplicity.Source: Appendix A · p. 8
• Generation confounders were limited to at most 0.45% truncation, at least 95.7% answer emission, and 0% native-English reasoning; none materially affected principal comparisons.Source: Appendix A · p. 8

Appendix B.1. Answer extraction

• 77 of 4,260 records received different answers from the two extraction parsers, and estimated teacher accuracy shifted from 74.9% to 74.3%.Training retained the original labels and partitions; neither parser was treated as a reference standard for ambiguous outputs.Source: Appendix B.1 · p. 8

Appendix B. Additional analyses

• The rationale-probability analysis scored five answer-letter continuations after removing the explicit final answer, comparing rationale prefixes with a fixed cross-question-shuffled control.It evaluated eight rationale fractions from 0 to 1; the teacher-answer check and initial probability diagnostic used all 4,260 records, with repeated stems sampled together in bootstrap checks.Source: Appendix B.1 · p. 8, Appendix B · p. 8

Appendix B.1. Answer extraction

• After jointly excluding option-wording cues, the effects remained +0.18 for teacher-correct and −0.12 for teacher-wrong items, with interaction 0.30 (95% CI 0.27 to 0.33).The post-hoc check reused stored probabilities and could not rule out paraphrases of the correct option.Source: Appendix B.1 · p. 9

Appendix B. Additional analyses

• The generated-rationale test compared each question’s own rationale with another question’s rationale using unchanged baseline Qwen3-4B parameters.It used outputs from distill-SFT and step-by-step at α = 1.0 on the same ten test splits of B and D; this measured answer influence, not factuality.Source: Appendix B.1 · p. 9

Appendix B.3. Transfer and retention

• At α = 0.3, distill-SFT had higher accuracy than step-by-step on both previously correct subsets A and C under reasoning; no lower accuracy was detected for distill-SFT in these strata.Source: Appendix B.3 · p. 10, Table B.5 · p. 10
• 61.0 correct answers on B and 47.9 on D were gained over baseline with reasoning at α = 1.0, while 47.4 on A and 21.0 on C were lost; net gain was 40.5 per split.Against pure-SFT direct answering, gains on B and D were offset by losses on A and C, so reasoning did not improve overall accuracy over pure SFT.Source: Appendix B.3 · p. 10
• Subset- or oracle-based selection requires knowing answer outcomes, so the reported selective-reasoning gains do not establish a prospective deployment policy.Table B.6 compares accuracy and latency against reasoning on every item.Source: Appendix B.3 · p. 10, Table B.6 · p. 10