PaperlayerView exact revision on arXiv

Source-linked AI summary

Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis

Aarav Singh, Animesh Pathak, Navyansh Singh

arXiv:2610.03176v1cs.CL

TL;DR

Rare-disease diagnosis motivates testing whether a smaller model can learn from reasoning traces generated by a teacher that sees the diagnosis. The paper filters copied ground-truth phrases from those traces and finds a narrow accuracy advantage for the filtered student over the teacher, concentrated in better-represented diseases.

  • Problem

    The paper studies distillation for diagnostic reasoning on underrepresented conditions, where smaller models are needed for resource-constrained clinical settings.

    Source: §1 · p. 1

  • Method

    A 1.5B student is fine-tuned on traces from an 8B teacher prompted with each case narrative and its ground-truth diagnosis, with a filtered variant removing explicit ground-truth phrases.

    Source: §1 · p. 1

  • Results

    StudentF gains +0.040 M3 over the teacher (p<0.001), while the unfiltered Student does not significantly outperform the teacher (p=0.129).

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The reported accuracy advantage is concentrated in better-represented diseases, and filtering—not hindsight distillation alone—drives the gain.

    Source: §1 · p. 2, §7 · p. 8

    The aggregate advantage depends on evaluation frequency: in Q1, with a median of 7 training cases, the teacher leads StudentF by 1.6 percentage points, and the models are statistically indistinguishable.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We study hindsight-guided distillation for rare disease diagnosis on ZebraMap: a 1.5B student is fine-tuned on chain-of-thought traces from a 8B teacher that observes the ground-truth diagnosis during generation. Absolute accuracy remains low for all models - the task is hard at this scale - but within this ceiling a filtered variant (StudentF) achieves a small, statistically significant accuracy advantage over the teacher (p < 0.001), concentrated in better-represented diseases. The unfiltered student does not significantly outperform the teacher (p = 0.129), establishing that contamination filtering - not hindsight distillation alone - drives the gain. The gap traces to an artifact we term GT hallucination. Label-visible generation causes the teacher to embed "ground truth is X" phrases in its reasoning chain; SFT copies the pattern. At inference, the unfiltered student reproduces the phrase in 33.9% of cases, with severe accuracy degradation when the hallucinated label is wrong. A regex filter removing these slots reduces contamination to near-zero, producing the observed gain - though the effect remains small. We precisely quantify this gain-cost tradeoff, document frequency-dependent knowledge transfer absent from the RL-trained teacher, and characterize a calibration gap that SFT does not close - identifying both as directions for future work.

1 Introduction

The paper studies hindsight-guided distillation for rare-disease diagnosis, where a label-visible teacher generates traces for a smaller student. It finds that filtering copied ground-truth phrases is necessary for a narrow accuracy gain, while exposing a contamination artifact and limits in the resulting knowledge transfer.

Study setup

  • The study evaluates hindsight-guided distillation on ZebraMap, a rare-disease resource, using an 8B teacher that sees the diagnosis and a 1.5B student trained to reproduce its traces.The student receives only the case narrative at inference.Source: §1 · p. 1

GT hallucination

  • 33.9% of unfiltered Student inferences contain a copied ground-truth phrase, and accuracy collapses by 18 percentage points when the leaked label is wrong.Removing explicit “ground truth is X” slots reduces contamination to near-zero, though the retrospective correct-label stratum shows a descriptive pattern consistent with a cost.Source: §1 · pp. 1–2

Main findings

  • StudentF significantly exceeds the teacher in Accuracy, while unfiltered Student does not, indicating that filtering—not hindsight distillation alone—drives the gain.The advantage is concentrated in better-represented diseases; static filtering also cannot distinguish correct from incorrect labels that would be reproduced at inference.Source: §1 · p. 2

Evaluation framework

  • The evaluation framework introduces three Accuracy variants and Mentioned as a search-space proxy, with Mentioned explicitly not treated as an accuracy measure.The contribution list identifies these measures as novel for this setting.Source: §1 · p. 2

Implications

  • The paper reports frequency-dependent accuracy gains and format imitation without epistemic calibration, motivating more refined training approaches as future work.The findings are described as consistent with established findings.Source: §1 · p. 2

2 Related Work

The paper situates its work among research on chain-of-thought distillation, label-conditioned rationale generation, rare-disease diagnosis, and training methods. It studies the complementary case where an external teacher sees the ground-truth label, with contamination filtering as a targeted trace-selection strategy.

Chain-of-thought distillation

  • Prior CoT-distillation studies examine trace supervision and its format effects without ground-truth label access during trace generation; this paper studies the complementary label-visible regime.The cited studies include findings that format alignment can account for gains and that stronger teachers do not always produce better students (Chen et al., 2025b).Source: §2 · p. 2

Label-conditioned rationale generation

  • Unlike STaR’s answer-conditioned rationale bootstrapping, this work uses a dedicated external teacher given the ground-truth label, quantifying resulting contamination in a clinical setting.STaR was introduced by Zelikman et al. (2022).Source: §2 · p. 2

Data-centric trace selection

  • Targeted regex filtering applies data-centric trace selection to remove contamination, eliminating most inference-time contamination while yielding a bounded net gain because correct labels can be useful.The discussion places this approach alongside broader trace-quality filtering work (Zhuang et al., 2026; Zhou et al., 2023).Source: §2 · pp. 2–3

LLMs for rare-disease diagnosis

  • The work complements rare-disease diagnosis research by quantifying hindsight-distillation contamination at lower inference cost than the teacher.The related work covers LLM clinical knowledge, rare-disease benchmarks, and specialist diagnostic tools.Source: §2 · p. 3

SFT vs. reward-based training

  • The results are presented as consistent with accounts of SFT producing format imitation and coarse-grained changes, unlike reward-based training’s finer selective optimization.The cited discussion includes Turpin et al. (2023), Yang et al. (2025), and Fu et al. (2026).Source: §2 · p. 3

3 ZebraMap

ZebraMap is a multimodal rare-disease resource built from PubMed case reports and organized into patient-level narratives, disease fields, and linked clinical images. The study defines its own split, with evaluation diseases represented in training at varying exposure levels.

Resource construction

  • ZebraMap comprises 1,727 rare diseases, 36,131 PubMed case reports, 69,146 patient-level cases, and 98,038 linked clinical images.Its case structuring combines rule-based section detection and demographic extraction, LLM-based field population, and a separate retrieval component for missing disease-level fields.Source: §3 · p. 3

Splits

  • The study creates its own split: 18,388 cases from 1,090 diseases for training and 2,000 cases across 250 diseases for evaluation.All evaluation diseases also occur in training, with exposure from 1 to 226 cases and a median of 17; the analysis therefore concerns varying training exposure, not zero-shot generalization.Source: §3 · p. 3

4 Methodology

The methodology generates label-visible teacher traces, filters explicit ground-truth phrases for one student variant, and fine-tunes two otherwise matched students. Evaluation uses narrative-only inference and defined measures for committed diagnosis accuracy, search-space mentions, and robustness to format non-compliance.

4.1 Overview

  • The four-stage pipeline generates traces with an 8B teacher, creates filtered and raw trace sets, fine-tunes a 1.5B student, and evaluates both students on narrative-only inputs.The evaluation cases are excluded from trace generation and fine-tuning.Source: §4.1 · pp. 3–4

4.2 Models

  • StudentF is trained on regex-filtered traces, while Student uses raw traces; both share the same architecture and training setup.The filter removes explicit “ground truth is X” slots; the teacher is evaluated zero-shot on narrative-only prompts.Source: §4.2 · p. 4

4.3 Hindsight Trace Generation

  • Label-visible generation leads the teacher to include “ground truth is X” in its reasoning, and SFT copies the pattern; filtering removes these slots from StudentF’s training traces.Both student variants see only the narrative at inference, without the label or the full ground-truth sentence.Source: §4.3 · p. 4

4.4 Evaluation Metrics

  • Accuracy (M3) scores only the committed diagnosis tag against a disease synonym list, whereas Mentioned (M1) is a search-space proxy and not an accuracy measure.The teacher’s upper-bound measure handles untagged responses, and format-compliant Accuracy is restricted to responses with a well-formed tag.Source: §4.4 · pp. 4–5
  • The study uses clustered permutation tests, clustered bootstrap confidence intervals, and Holm–Bonferroni correction within each comparison family.The reported settings are 250 disease clusters and 50,000 resamples for both tests and intervals.Source: §4.4 · p. 5

5 Results

StudentF significantly outperforms both the teacher and unfiltered Student on primary Accuracy, while contamination analysis links wrong GT phrases to a sharp accuracy drop. Filtering’s net benefit is positive, though static removal may also discard useful correct-label hints.

5.2 Quantifying Trace Contamination

  • 33.9% of Student inferences contain GT phrases; 89.1% of those name a wrong disease, and 74.5% of wrong labels are copied into <diagnosis>.Student Accuracy falls from 0.2050 when GT is absent to 0.0265 when GT is wrong, an 18 percentage-point collapse.Source: Figure 1 · p. 5, §5.2 · p. 5

5.1 Main Accuracy

  • Accuracy = 0.196 for StudentF, exceeding teacher (0.155, p<0.001) and unfiltered Student (0.172, p<0.05).StudentF also exceeds the teacher under every Accuracy variant, while Student alone does not significantly outperform the teacher (p=0.129).Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Quantifying Trace Contamination

  • StudentF retains only 3/2,000 contaminated traces and gains +0.024 Accuracy over Student overall (p<0.05).Its gain over Student on the GT-wrong stratum is 0.08 Accuracy, but StudentF scores 0.5000 versus Student’s 0.7703 on the GT-correct stratum; that comparison is descriptive, not causal.Source: §5.2 · p. 6
  • On the GT-absent stratum, StudentF (0.2194) significantly exceeds Teacher (0.1793, padj=0.009), while Student does not (padj=0.17).The clean-case advantage indicates that the filtering gain is not limited to contaminated cases.Source: §5.2 · p. 6
  • Static filtering cannot distinguish phrases that would be reproduced correctly from those that would be wrong, so whether selective filtering adds gains remains open.Training-time labels are always correct, but which labels a student would reproduce correctly at inference cannot be known in advance.Source: §5.2 · p. 6

6 Analysis

StudentF’s advantage over the teacher grows with disease training frequency, while its reasoning style and calibration remain distinct from the teacher’s. The analysis also finds that filtering reduces self-correction and uncertainty expressions, but does not establish adaptive calibration.

6.1 Training Frequency and Knowledge Transfer

  • StudentF Accuracy rises with training frequency (ρ=+0.242, p<0.01; OLS slope +0.058, p<0.05), while the teacher shows no frequency dependence (ρ=+0.029, p=0.65).The StudentF–Teacher gap grows from −0.016 in Q1 to +0.121 in Q4.Source: §6.1 · p. 7, Table 4 · p. 7
  • At Q1, with a median of 7 training cases, the teacher marginally leads StudentF and all three models are statistically indistinguishable.The authors conclude that SFT cannot create knowledge absent from training.Source: §6.1 · p. 7

6.2 Reasoning Style and Epistemic Calibration

  • StudentF self-corrects at 0.15 per response, below Teacher’s 0.28 and Student’s 0.83; it also expresses uncertainty 2.12 times versus 4.09 and 4.53.All three models think longer, self-correct more, and express more uncertainty when wrong, though absolute rates differ.Source: §6.2 · p. 7
  • Teacher makes confident assertions at 0.67 per response versus Student’s 0.31, a 2.1× difference; StudentF partially closes the gap at 0.51.The authors describe the pattern as consistent with RL training toward committed, evaluable answers.Source: §6.2 · p. 8
  • StudentF’s SC ratio CI [0.9646, 1.9537] straddles 1.0, so its greater self-correction when wrong is not statistically distinguishable from noise.Student and Teacher show more reliable miscalibration, with CIs [1.2969, 1.7884] and [1.0984, 2.0598], respectively.Source: §6.2 · p. 8, Table 5 · p. 8

7 Discussion

StudentF’s accuracy advantage over the 8B teacher is concentrated in better-represented diseases, while all models remain statistically indistinguishable on the least-represented stratum. Static filtering has a descriptive correct-label cost, SFT does not close the calibration gap, and ZebraMap results are bounded by disease-vocabulary leakage.

7 Discussion

  • StudentF, a 1.5B model, exceeds its 8B teacher by +0.040 M3 Accuracy (p<0.001), but at Q1 the teacher marginally leads by -0.016 with all three models statistically indistinguishable.The advantage is concentrated in better-represented diseases and may reflect knowledge transfer from well-represented training cases or format alignment.Source: §7 · p. 8
  • Static regex filtering cannot distinguish inference-time correct from incorrect label reproduction; the retrospective, unmatched-stratum analysis shows a descriptive offset, not a causal estimate.On the GT-correct stratum (n=74), Student reaches 0.7703 versus its uncontaminated baseline of 0.2050, while StudentF reaches 0.5000; filtering still yields a significant overall net gain (p<0.05).Source: §7 · p. 8
  • SFT transfers the teacher’s reasoning format and frequency-sensitive diagnostic knowledge, but does not produce epistemic calibration: confident assertions do not track correctness.StudentF’s SC ratio is 1.3365 without genuine uncertainty meaning.Source: §7 · p. 8
  • A reward signal penalizing wrong diagnoses is proposed as a way to improve diagnostic accuracy and discourage hallucinated GT phrases; reinforcement and verbal self-correction are suggested directions.The paper notes GRPO effects in general reasoning settings and Reflexion as a complementary approach.Source: §7 · p. 9
  • BM25 reaches 0.3935 label-classification accuracy versus the best model’s 0.196 M3 Accuracy, indicating disease-vocabulary leakage limits interpretation of absolute scores.The authors note that their disease-level split lacks the source-article grouping control used by a concurrent ZebraMap study, which may further inflate BM25’s upper bound.Source: §7 · p. 9

8 Conclusion

The study evaluates hindsight-guided rationale distillation for rare-disease diagnosis, training a 1.5B student on traces generated by an 8B teacher with access to ground-truth labels. Filtering the traces reduces GT contamination to near-zero and yields a narrow significant accuracy advantage over the teacher, while the unfiltered student does not significantly outperform it.

8 Conclusion

  • The study trains a 1.5B student with SFT on chain-of-thought traces generated by an 8B teacher given ground-truth label access.The training set comprises 18,388 hindsight-guided traces.Source: §8 · p. 9
  • StudentF achieves a narrow accuracy advantage over its 8B teacher on ZebraMap’s rare-disease diagnosis task (+0.040 M3, p<0.001), while the unfiltered Student does not significantly outperform the teacher (p=0.129).Filtering yields a significant net gain (p<0.05), though a retrospectively defined correct-label stratum suggests a partially offsetting descriptive cost.Source: §8 · p. 9
  • GT phrases appear in 33.9% of Student inferences, and when the leaked label is wrong the accuracy impact is an 18-percentage-point collapse.Regex filtering reduces contamination to 3/2,000 traces; the unfiltered Student does not significantly outperform the teacher.Source: §8 · p. 9

Limitations

The reported gains are qualified by a shared performance ceiling, frequency-dependent results, and limits in contamination measurement, filtering, and experimental comparisons.

Limitations

  • In Q1, where diseases have a median of 7 training cases, Teacher leads StudentF by 1.6 pp, so an evaluation set weighted toward low-frequency diseases could reduce or reverse the aggregate advantage.The aggregate advantage is concentrated in high-exposure diseases.Source: Limitations · p. 9
  • 46% of evaluation diseases (115/250) have max M3_posttag below 0.25 across all models, placing the gains atop a shared task-scale ceiling.The authors report that the shared difficulty profile suggests the task, not model-specific capacity, is the binding constraint at this scale.Source: Limitations · p. 9
  • The 33.9% Student contamination rate is a lower bound because the regex detector misses alternative phrasings of “ground truth is X.”The analysis uses a single family of regex patterns targeting that construction.Source: Limitations · p. 9
  • On the GT-correct stratum (n=74), StudentF Accuracy is 27 pp lower than Student, but this retrospective, unmatched comparison is descriptive rather than causal.The static filter cannot distinguish labels that would be reproduced correctly at inference; closing the gap would require an inference-time signal.Source: Limitations · p. 10
  • The measured accuracy gains cannot distinguish genuine knowledge transfer from format alignment, and the Base model lacks a matched pre-distillation CoT baseline.Student and StudentF use 4-bit NF4 while the Base reference uses full floating-point precision; cross-format effects between student NF4 and teacher Q4_K_M are unmeasured.Source: Limitations · p. 10
  • A single Student and StudentF were reported, and disease-cluster confidence intervals are conservative approximations because strict exchangeability does not hold.Variance across training runs is not quantified; diseases that are hard for one model tend to be hard for all.Source: Limitations · p. 10

A Error Taxonomy

The appendix specifies a first-match error taxonomy, tests Accuracy across bidirectional-match strictness settings, and defines the evaluation and style measures.

A Error Taxonomy

  • The priority cascade assigns exactly one error category by first match; GT-poisoned is Student-exclusive (22.5%), while uncategorized ICD-10 errors comprise a substantial share of wrong answers.The uncategorized categories cover errors where at least one of the predicted or true diseases lacks an ICD-10 code.Source: A Error Taxonomy · pp. 12–13, Figure 3 · p. 13

B Bidirectional Match Parameter Sweep

  • SF−T reaches 0.0400 at the exact/synonym default, and StudentF’s Accuracy advantage over Teacher is directionally consistent across all 22 tested values.At low mc = 0–4, any-length substring matches inflate all three models’ scores equally, compressing the absolute gap while leaving the ordering unchanged.Source: B Bidirectional Match Parameter Sweep · p. 13, Figure 4 · p. 14

C Metric and Style Feature Definitions

  • Accuracy (M3) requires the diagnosis tag to match the true disease name by bidirectional synonym-list substring match; Mentioned (M1) credits a ground-truth name anywhere in the response.Responses without a diagnosis tag score 0 on M3; M1 also credits mentions in rejected hypotheses.Source: C Metric and Style Feature Definitions · p. 13
  • Style measures count self-correction, explicit uncertainty, and confident assertions in the response; the SC ratio compares mean self-correction on incorrect versus correct responses.An SC ratio of 1.0 indicates equal self-correction regardless of correctness; higher values indicate elevated self-correction on wrong answers.Source: C Metric and Style Feature Definitions · pp. 13–14

D Reproducibility Details

The reproducibility materials include code, trained adapters, evaluation results, and filtered training datasets.

D Reproducibility Details

  • The authors make the code, trained LoRA adapters, evaluation result files, and filtered training datasets available.Source: D Reproducibility Details · p. 14
  • Table 6 is identified as the paper’s reproducibility settings table.Source: Table 6 · p. 15

Related papers

Loading 2610.03176v1…