PaperlayerView exact revision on arXiv

Source-linked AI summary

One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail

Zhaoyang Jiang, Zhizhong Fu, Yunsoo Kim, Zicheng Li, Xuanqi Peng, Fei Teng, Jiacong Mi, Honghan Wu

arXiv:2608.14683v1cs.LG

TL;DR

Selective prediction for ranked medical diagnoses uses one score for two distinct decisions: endorsing the leader and determining whether the candidate set contains an answer. This paper separates those checks and evaluates rank-based confidence signals across rare-disease diagnosis, retrieval, and entity linking, finding that feasibility depends on top-ranked accuracy and that margins help correctness but not candidate presence.

  • Problem

    Diagnostic systems commonly threshold one top score for both leader correctness and candidate-set usability, although these decisions may require different checks and signals.

  • Method

    The paper separates correctness from candidate presence, analyzes shared score components, and tests top scores, margins, and interventions across diagnosis, retrieval, and entity linking.

  • Results

    29.0% accuracy at 10% coverage versus 13.3% overall shows the phenotype-only Exomiser margin improves leader selection, while unlabelled scores cannot predict when switching signals will help.

  • Takeaways & Limitations

    Selective prediction must first verify that top-ranked accuracy makes the target feasible, then choose a confidence signal matched to leader correctness or candidate presence.

  • Takeaways & Limitations

    Reported thresholds are retrospective estimates rather than clinical guarantees, and new systems require labelled validation because scores alone cannot certify performance.

Abstract

from arXiv · show

Given a patient's clinical findings, a diagnostic system ranks possible diseases and must decide when to endorse its first prediction or defer it for review. This decision is usually made by thresholding the top score. Selective prediction over ranked outputs begins with two checks. First, the ranker must produce enough correct top-ranked predictions to make the target feasible. Across 2,000 patient records stratified by disease prevalence, eight small open-weight LLMs achieve at most 4.6% Recall@1 on ultra-rare diseases. At 10% coverage, even a perfect confidence ranking of their existing predictions therefore cannot reach 50% selective accuracy. More accurate models pass the same check, showing that the limit is regime-specific. Second, the confidence signal must match the decision being made. For fixed-candidate rankers, the top-two margin cancels components shared across candidates. On phenotype-only Exomiser, it selects 10% of cases at 29.0% accuracy, compared with 13.3% overall, while the top score provides no reliable gate. Yet that cancellation can remove information needed to detect whether the candidate list contains an answer. SciFact retrieval and biomedical entity linking confirm this distinction. Finally, we prove that unlabelled scores alone cannot determine whether switching to the margin will help.

1 Introduction

Selective prediction on rare-disease rankings requires separate feasibility and signal checks: confidence cannot create correct top predictions, and the best signal depends on whether the top candidate is correct or any correct candidate is present. Experiments show that low ultra-rare Recall@1 can make targets unreachable, while candidate gaps improve correctness selection but discard information needed for presence detection.

  • Motivation: Selective prediction endorses high-confidence top predictions and defers the rest, but correctness and candidate-list presence require different checks and signals.The top score is often thresholded for both decisions, although shared case-level information affects them differently.
  • Feasibility: A predictor correct on fraction p of cases and answering fraction c cannot exceed selective accuracy min(1, p/c), even with perfect confidence ranking.Recalibration or rescoring can reorder existing correct answers but cannot create new ones, making the ceiling especially important on distributional tails.
  • Decision-specific signals: Every zero-sum candidate contrast cancels shared case-level information, so the gap can help correctness while hurting presence when the candidate set contains no answer.The presence decision must therefore use information outside the candidate contest; Figure 1 summarizes this distinction and its formal treatment.
  • Feasibility: 4.6% ultra-rare Recall@1 across eight small open-weight LLMs caps selective accuracy at 46% at 10% coverage, while more accurate models make the ceiling non-binding.Two of five medical-specialised models clear a 50% target across their whole interval, and three frontier configurations reach 15.8–22.4%.
  • Decision-specific signals: 29.0% accuracy at 10% coverage versus a 13.3% base rate shows that the phenotype-only Exomiser gap selects correct leaders, whereas its top score provides no usable operating point.The gap removes case-level components shared across candidates and measures separation from the nearest alternative.

2 Related Work

Prior work separates selective prediction over fixed answers from deferral and adaptive retrieval, which change the underlying predictor. This section also situates score-based gating, long-tail rarity, and rare-disease diagnosis while motivating the paper’s focus on feasible operating points and ranked-output confidence.

  • Selective prediction and deferral: Calibration and uncertainty methods select among existing top-1 answers, whereas deferral, cascades, and adaptive retrieval route cases to another predictor and can change base accuracy.The oracle ceiling is used to distinguish these regimes.
  • Confidence for ranked outputs: Established ranked-output signals include the top score, leader gap, local-geometry trust, and input-specific nuisance subtraction.Examples of nuisance adjustment include query length, impostor cohorts, phenotype-query term counts, and cross-query normalization.
  • Confidence for ranked outputs: A top-two-margin retrieval gate under a monotone link cannot be chosen from unlabelled scores alone, motivating analysis of score components already emitted by the ranker.The section studies how the split in existing scores relates to the decision.
  • Long-tail knowledge: Long-tail research typically defines rarity by corpus frequency, while this work stratifies diseases by Orphanet prevalence.Matched-pair evidence leaves the per-disease gap essentially intact but is too limited to remove the exposure explanation.
  • Rare-disease diagnosis: Rare-disease systems rank diseases against curated profiles, while general-purpose LLMs trail phenotype prioritizers and newer systems add training, retrieval, or tools without prevalence-stratified or decontaminated evaluation.Related work also identifies retrieval coverage as determining whether an answer enters the candidate pool; this paper asks whether the predictor already places it first.

3 Selective Prediction over Ranked Outputs

Selective prediction over ranked outputs requires first testing whether the predictor has enough correct top-ranked answers to meet the target, then matching the confidence gate to whether leader correctness or candidate-set validity is being certified. Score decompositions explain why margins can help one decision while hurting another, but unlabelled score distributions alone cannot determine whether switching gates will improve performance.

  • Feasibility test: A feasibility test must precede confidence estimation: if the required correct answers do not exist among unchanged predictions, no recalibration or confidence score can reach the target.The target is only possible when the oracle ceiling permits it; remedies require a more accurate predictor, a larger candidate set, or a relaxed deployment target.
  • Decision-specific score decomposition: For fixed candidate sets, the top score retains case-level offsets, whereas the top-two margin cancels shared score components and compares candidates directly.More generally, every zero-sum contrast among candidates cancels the shared case-level offset.
  • Decision-specific score decomposition: Correct leader prediction implies candidate presence, but candidate presence does not imply correct ranking, so the appropriate gate depends on whether the decision certifies ycorr or ypres.A contrast may help leader correctness while hurting candidate-set presence; a case-side correction can help both, whereas the raw score helps presence only when nuisance variation is small.
  • Unidentifiability: Proposition 1 shows that no functional of the unlabelled candidate-score distribution can identify the gain from switching between the top score and margin.Two joint laws can share the same unlabelled scores and base accuracy while producing opposite signs for the gain.
  • Domain condition: The margin is meaningful only when one predictor scores a common candidate set with distinct top alternatives on a common scale.Free-form generation usually lacks a runner-up because its two highest-scoring samples are often identical strings.

4 Data and Experimental Setup

The study benchmarks rare-disease diagnosis on Phenopacket Store, stratifies cases by Orphanet prevalence, and evaluates deployable and medically specialized models. It also tests whether ranked scores support both top-answer correctness and candidate-list presence across disease and retrieval tasks.

  • Benchmark and labels: 10,374 real patient cases and 780 diagnoses define the benchmark, with rankers scoring all 8,553 OMIM entries having HPOA profiles.Each case contains observed HPO terms and a confirmed OMIM diagnosis; the fixed jointly scored candidate set supports margin analysis.
  • Rarity stratification: 4,780 cases are ultra-rare, 1,167 are less-rare, and 372 of 780 diseases have a documented Orphanet prevalence class.Cases with undocumented prevalence form a separate group, and the rarity collapse is checked under alternative prevalence rules and finer bands.
  • Model systems: Eight small open-weight models form the deployable regime, while five additional models represent medical specialization under the same prompt.The deployable models span the Qwen, Llama, Mistral, InternLM, and Yi families.
  • Decision tasks: The rare-disease benchmark always contains the gold diagnosis, whereas SciFact, BC5CDR, and MedMentions test whether a correct candidate is present.SciFact uses ColBERTv2 to rank 5,183 abstracts for 300 claims; masking creates inputs with no valid candidate in the transfer tasks.
  • Evaluation: Reported metrics include Recall@1/@5, selective accuracy by coverage, AUROC for top-1 correctness and candidate presence, and binned ECE for verbalized confidence.Generated disease names are linked to OMIM with SapBERT, while phenotype rankers return identifiers without a linker.

5 Results

Results show that selective accuracy on the rare-disease tail is first limited by top-ranked correctness and then by whether confidence measures correctness or answer-list adequacy. Margins can improve correctness gating, but their benefit is task- and score-construction-dependent and cannot be selected from unlabelled scores alone.

  • Feasibility: Recall@1 falls from ∼40% on a less-rare anchor to ≤5% on the ultra-rare tail across eight small open models.The best medical-specialised model reaches 7.6%, while DeepSeek-V4-Pro reaches 18.6% and reasoning reaches 22.4%.
  • Feasibility: At c=10%, the selective-accuracy ceiling is p/c, spanning 7% to 46% for the eight small models; Qwen-32B is capped at 43%.Thus targets above 46% are unreachable for those eleven models regardless of confidence discrimination, whereas frontier tail base rates of 15.8–22.4% make the target attainable.
  • Clinical prioritization: 29.0% accuracy at 10% coverage from Exomiser’s top-two margin exceeds its 13.3% ultra-rare base Recall@1, while the top score cannot form a reliable decile gate.The top-score decile lies inside one tie block, with accuracy varying over [0.0, 4.3]% depending on tie-breaking.
  • Clinical prioritization: 81.0% margin-gated accuracy exceeds 45.8% raw-score gating for the in-house retriever, while the delivered system reaches 74–81% on its most confident 10%.The corresponding bare LLM reaches only 2–18% at the coverage resolved by its own confidence.
  • Score construction: A label-free choice between raw score and margin is impossible: subtracting estimated shared score components lifts raw gating from 45.8% to 80.3%, but only with post-hoc calibration.Input-derived estimates reach at most 70.7%, and LIRICAL’s margin advantage swings from +0.056 to −0.281 under alternative monotone score mappings.
  • Cross-task validation: Margins improve top-1 correctness but can worsen answer-list detection: on SciFact, margin gating reaches 0.836 versus 0.764 for raw MaxSim, yet relevant-abstract survival is 0.647–0.696 versus 0.696–0.738.The same split appears in entity linking, where the margin is the worse NIL detector under SapBERT at every rate (−0.054, [−0.088, −0.023]).

6 Conclusion

Selective prediction first requires enough correct top-ranked predictions to make the target feasible, then requires a confidence signal aligned with the decision. Margin-based signals can improve leader correctness while discarding information about whether the candidate list contains an answer, so switching signals requires labelled validation.

  • Feasibility before gating: At 10% coverage, eight small open-weight LLMs on ultra-rare diseases cannot reach 50% selective accuracy, regardless of calibration.More accurate medical-specialised and frontier LLMs pass the same feasibility check.
  • Choosing the confidence signal: For fixed candidate sets with unnormalized scores, the top-two margin can better indicate whether the leader is correct by removing variation shared across candidates.The margin’s cancellation targets components common to the candidates rather than the leader alone.
  • Choosing the confidence signal: SciFact and entity-linking experiments show that removing shared variation can discard information needed to determine whether the candidate list contains an answer.Thus, a signal that helps assess leader correctness may not assess candidate-list adequacy.
  • Workflow: Selective prediction should first test whether base accuracy makes the target feasible, then choose a confidence signal for the decision being made.Unlabelled scores cannot reveal the gain from switching signals, so the final choice requires labelled validation.

Limitations

The reported thresholds are retrospective and not clinically certified, especially for ultra-rare diseases. Conclusions also depend on the delivery rule and on uncertain retriever accuracy under curation and prospective-input differences.

  • Clinical certification: Retrospective thresholds are not clinical guarantees: certifying 80% accuracy at 10% coverage would require about 26,600 ultra-rare cases.Disease clustering further reduces effective sample size, and conformal calibration misses its promised error rate for diseases absent from calibration data.
  • Delivery rule: At 50% accuracy and 10% coverage, all eight small models are ruled out at k=1, but only four at k=5.Passing the feasibility test means only that the target is not ruled out; it does not show that an available confidence score reaches it.
  • Retriever evaluation: Five-fold variation in base accuracy leaves the retriever’s absolute performance uncertain because leave-source-out does not remove broader curation effects.Benchmark phenotypes were curated after diagnosis, and feature removal is only a proxy for the difference from prospective inputs.

Ethics Statement … C.1 Prevalence Stratification

The paper documents ethical and equity risks in selective triage, then separates the algebraic invariance of margins from the empirical conditions under which they improve gating. Across controlled tests and prevalence analyses, score cancellation helps in some settings but cannot be justified from unlabelled scores alone or generalized beyond the tested rankers.

  • Ethics Statement: 71.0% accuracy on the answered decile means about three in ten answered patients receive a confidently wrong top-1, with potential testing, cascade-testing, and counselling consequences.The gate flags none of 217 patients with one or two recorded findings, versus 24.6% above fifteen terms; removing this dependence reduces performance from 81.0% to 70.7%.
  • Ethics Statement; A Terminology: The study uses secondary analysis of de-identified, already-published records containing structured HPO terms and gold labels, without recruiting or interacting with patients.The paper notes that original-report consent was outside the study and inconsistently recorded by aggregating resources.
  • B Margin Derivations and Mechanism Tests; B.1 Score Representation and Invariance: The margin removes a candidate-independent case-level score shift, whereas the top score moves with that shift and may reorder cases.For two candidates, the margin is proportional to the leader’s log-odds against its closest competitor, with the normalizer cancelling; the paper frames this as the algebraic basis for its gating argument.
  • B.2 A Deployed Instance, and the Limits of Our Evidence: The same ranking can produce different apparent margin advantages under different score units, so the paper reports measured properties without ordering the seven systems.LIRICAL’s margin advantage moves from +0.056 to −0.281 despite identical rankings, while related comparisons include gains of +0.178 and +0.077.
  • B.3 One Ranking, Fourteen Units: Under additive centering, the margin AUROC remains 0.796100 while raw-score AUROC falls from 0.822 to 0.790, confirming ranking preservation but score-domain sensitivity.Across injection arms, the margin AUROC stays 0.796 while the gain grows from +0.023 to +0.265 as between-input variance increases.
  • B.4 Mechanism Test on the Retriever: 81.0% accuracy at 10% coverage for the margin contrasts with 47.9% for a mean-centered top score, showing that shift invariance alone does not identify a useful gate.The normalized log-score ranges from 34.9% to 80.3% as T varies over [0.25, 50], with its best temperature T=3 selected after the fact.
  • B.5 Locating the Shared Score Component: R^2 = 0.92 of raw top-score variance is explained by input-derived quantities, but score-derived normalization performs differently: rank-10 gives 74.1%, rank-100 62.3%, and median subtraction 47.1%.The relevant shared level is attributed to the few genuinely competing candidates, rather than all 8,553 candidates or patient description alone.
  • B.6 Scope of the Argument; C Prevalence Strata and Averaging; C.1 Prevalence Stratification: The cancellation argument requires an approximately candidate-independent additive component on an evidence-additive scale, and the mechanism evidence comes from one retriever rather than every ranker.The paper treats cancellation as supported in its main setting, not as a theorem that margins must beat top scores universally; prevalence analyses separately show LLM accuracy collapsing toward a near-zero ultra-rare floor.

C.2 Per-Case and Per-Disease Averaging · D Curation Leakage and External Validation · D.1 Leave-Source-Out Decontamination

Per-case averaging exaggerates the apparent collapse because less-rare cases are concentrated in a few famous diseases, while the ultra-rare tail remains severely difficult under both averaging schemes. Decontamination identifies substantial source and curator leakage, bounds retriever accuracy rather than point-identifying it, and shows that residual gradients and triage advantages remain sensitive to controls and averaging.

  • C.2 Per-Case and Per-Disease Averaging: 34.7% of the less-rare bin is Neurofibromatosis-1, so per-case averaging overweights famous diseases and inflates the anchor and collapse ratio.The top five diseases account for 59.6% of the bin, whereas the ultra-rare bin contains 291 diseases and its largest disease is 9.7%.
  • C.2 Per-Case and Per-Disease Averaging: ∼2–6× is the per-disease LLM fall, while ultra-rare retriever Recall@1 remains ∼24% and small-model Recall@1 is 0.7–4.6% per case versus 1.3–4.9% per disease.The deployment recommendation rests on the persistent ultra-rare contrast, not on the less-rare anchor’s height.
  • C.2 Per-Case and Per-Disease Averaging: Prevalence is collinear with pretraining frequency and fame, so the analysis does not claim to separate prevalence effects from corpus exposure.Frequency-matched controls produce +0.169, i.e. 107%, with positive residuals for all 7 models, but only 13 matched pairs.
  • D Curation Leakage and External Validation: The two controls bound retriever accuracy from opposite sides: strict deletion is biased down, whereas leave-source-out is biased up, so the estimand is only partially identified.Strict control retains 67.2% of profile annotations and leaves only 6.8% of ultra-rare profiles empty.
  • D.1 Leave-Source-Out Decontamination: 74.5% of less-rare and 60.6% of ultra-rare cases share a source PMID with their gold disease’s HPOA profile, motivating leave-source-out decontamination.This means the retriever often matches a patient against a profile partly transcribed from the same publication.
  • D.1 Leave-Source-Out Decontamination: 98.7% of phenopackets came from one curator, and 51.3% of matching ultra-rare terms were biocurated by the patient-record author.The control removes 23.0% and leaves 26.6%, while surviving corroboration often comes from publications contributing same-disease cases to the corpus.
  • D.1 Leave-Source-Out Decontamination: 2,441/4,780 selected tail cases reduce retriever accuracy from 25.4% →9.7%, while the frontier model changes from 22.4→19.3%; thinner profiles and matched-term selection confound the gap.The gap is +34.8pp among cases matching ≥26 terms, which contain 63% of the tail, versus +32.3pp unstratified.
  • D.1 Leave-Source-Out Decontamination: 24.4→24.4% remains the tail Recall@1 after deleting globally private terms, while the decontaminated per-case gradient is −3.7pp with 95% CI [−14.5, +14.6].Macro averaging flips the gradient to +12.8pp, and absolute retriever percentages remain upper bounds because post-hoc HPO completeness is an unaddressed distribution shift.

D.2 Evaluation on External Rare-Disease Cohorts · E Feasibility, Calibration and Delivery Rules · E.1 Auditing the Feasibility Criterion

External rare-disease cohorts corroborate substantial curator-related leakage, while cohort heterogeneity limits the estimate’s universality. The feasibility audit is informative but often becomes unmeasurable or non-binding once actual confidence coverage and tie collapse are considered.

  • D.2 Evaluation on External Rare-Disease Cohorts: RareBench supplies an external test using five rare-disease cohorts, excluding LIRICAL because its cases come from the same Monarch/HPO ecosystem.Three included cohorts were curated by groups unrelated to Phenopacket Store.
  • D.2 Evaluation on External Rare-Disease Cohorts: 52.0% (n=244) on LIRICAL is twice the independent cohorts and close to the undecontaminated Phenopacket Store tail at 49.4%, consistent with leakage.The comparison is presented as corroboration rather than identification.
  • D.2 Evaluation on External Rare-Disease Cohorts: 19.4% among 216 disease-disjoint cases and 29.2% under per-disease averaging show that the estimate persists without shared diseases or case concentration.The disease-disjoint subset excludes cases whose disease appears in Phenopacket Store.
  • D.2 Evaluation on External Rare-Disease Cohorts: 0.0–53.8% between-cohort spread exceeds within-cohort sampling uncertainty, so ∼23% estimates this independent-cohort mixture rather than a universal constant.A population resembling HMS should expect much less, and HMS’s zero is partly structural because 3 cases lack any phenotypic HPOA annotation.
  • E.1 Auditing the Feasibility Criterion: At c=10%, 21 of 29 predictor–stratum cells have ceiling 1.0, leaving only 8 informative cells; at τ=50%, 7 of those 8 forbid.The audit spans LLMs, the retriever, Exomiser, and entity linkers across their respective confidence signals.
  • E.1 Auditing the Feasibility Criterion: Among 22 permitting cells, incumbent confidence reaches the target in 10, but only 15 are measurable because 7 signals are too tie-collapsed to resolve 10% coverage.The denominator excludes unmeasurable cells rather than treating them as failures.
  • E.1 Auditing the Feasibility Criterion: Exactly one of seven nominally forbidding cells remains measuredly forbidding at its signal-resolved coverage.Four are no longer forbidden at realized coverage, while two resolve no coverage at all.
  • E.1 Auditing the Feasibility Criterion: The criterion is non-vacuous: it forbids ultra-rare rare-disease cells but never entity-linking cells with base rates of 65–84%, without demonstrating seven independent rescues.The seven external-arm cells reuse the same retriever margin on the same ultra-rare stratum.

E.2 Confidence Calibration and Risk–Coverage … F.1 Exomiser in Phenotype-Only Mode

Selective accuracy is limited first by the base-accuracy ceiling and then by whether confidence matches the decision: margins can outperform top scores, but their operating levels and certification remain conditional. Across ranked outputs and phenotype-only Exomiser, Recall@k changes feasibility, while margin-based gating identifies useful tail subsets that raw scores do not.

  • E.2 Confidence Calibration and Risk–Coverage: AUROC ranges from 0.509 for Llama-8B to 0.905 for the 32B model, yet strong confidence ranking cannot overcome insufficient base accuracy.The smallest models are near chance, while larger models rank their own ultra-rare answers better; the ceiling can still make selective targets infeasible.
  • E.2 Confidence Calibration and Risk–Coverage: +67 to +71pp is the margin’s advantage for Llama-3.1-8B at 2–3% coverage, where p=4.6% and the ceiling no longer binds.Its confidence band is 10.7–11.1%, versus 77.8–82.1% for the margin, showing that a lifted ceiling alone is insufficient.
  • E.2 Confidence Calibration and Risk–Coverage: 74–83% accuracy at 10% coverage is achieved by retriever-first gating, versus 2–18% for the bare LLM under its own confidence.The absolute levels are not invariant over the leakage bracket, while the margin’s lift over each system’s base rate is 2.4–3.3×.
  • E.2 Confidence Calibration and Risk–Coverage: 11–44% of held-out-disease splits breach nominal risk under cluster-aware certification, compared with 13–23% under i.i.d. case splits.Split-conformal calibration provides a distribution-free finite-sample guarantee on held-out cases, but deployment commonly encounters diseases absent from available calibration data.
  • E.3 Recall at k: 2–8% ultra-rare Recall@5 preserves the small-model collapse, but the τ=50%, c=10% ceiling forbids all 8 models at k=1 and only 4 at k=5.Four models pass out of the forbidden region at k=5: Qwen2.5-14B, Qwen2.5-VL-32B, Llama-3.1-8B, and Mistral-7B-v0.3.
  • F.1 Exomiser in Phenotype-Only Mode: Exomiser gene-R@1 is 21.2% for less-rare, 13.3% for ultra-rare, and 6.0% for unknown cases.Its R@5 values are 42/28/13%, and the metric is gene-level rather than directly comparable to disease-level LLM or retriever results.
  • F.1 Exomiser in Phenotype-Only Mode: +28.5pp is the margin’s decile advantage over the raw score on 711 unsaturated phenotype-score cases, with disease-clustered variation of [+15.4, +42.3]pp.This shows the deployed tool’s margin-based triage is not merely an artifact of its 228-case top-score tie block.

F.2 Robustness to Phenotype Incompleteness · F.3 Characteristics of Answered Cases

Under simulated missing phenotypes, the tail-ranking conclusion and predictor-selection recommendation remain robust, while phenotype-only margin-gated answered cases succeed through broad converging evidence rather than a single giveaway finding. The answered decile spans many diseases and retains a substantial per-disease advantage over its base rate.

  • F.2 Robustness to Phenotype Incompleteness: Curated postdiagnosis HPO terms may overstate phenotype-only retrieval performance relative to prospective work-ups, so both predictors were tested after randomly or adversarially dropping findings.The adversarial setting drops highest-information-content findings first, removing the rare, distinctive terms most relied upon by phenotype-overlap retrieval.
  • F.2 Robustness to Phenotype Incompleteness: 25.6% Ultra-rare Recall@1 at ρ=0 reproduces the retriever’s clean tail result on the same 2,000-case sample.The full decontaminated set also reports 25.6%.
  • F.2 Robustness to Phenotype Incompleteness: The conclusion that no LLM surpasses the retriever on the tail survives realistic incompleteness, as the frontier LLM degrades at least as fast on the tail.This is decisive with reasoning off and supported by a partial n=127 check with reasoning on, so the predictor-selection recommendation stands.
  • F.2 Robustness to Phenotype Incompleteness: Degrading phenotypes only lowers the tail base rate, tightening the ceiling for achievable performance.
  • F.3 Characteristics of Answered Cases: On the full ultra-rare tail (n=4,780), cases in the answered top-10% (n=478) were selected by ranking the retriever’s decontaminated top1−top2 margin.The answered decile was compared with the deferred remainder.
  • F.3 Characteristics of Answered Cases: Answered cases are more richly phenotyped and decided by roughly twice as many converging findings, not by one disease-defining finding.The strongest finding has similar relative evidence share in both groups—2.7× versus 2.5× the uniform share—while raw differences of 5.3% versus 10.4% follow from more matched findings.
  • F.3 Characteristics of Answered Cases: 81.0% micro accuracy in the answered decile becomes 64.6% per-disease across 94 diseases, still 2.6× its own base.The disease-clustered 95% CI is [54.7, 73.5], and the top disease comprises 13.2% of the answered decile.

G Retriever–LLM Hybrids and Trained Fusion

Retriever–LLM hybrids are best gated by the retriever’s margin on rare-disease tails, while agreement is a genuine LLM-reliability signal that does not improve delivered-answer gating. Trained fusion fails to harvest the predictors’ complementarity, despite substantial oracle headroom.

  • Hybrid-gate performance: 0.74–0.77 ultra-rare AUROC for the retriever’s margin exceeds 0.54–0.56 for LLM confidence and 0.48–0.53 for agreement across four small configurations.On unknown-prevalence cases, the corresponding ranges are 0.73–0.74, 0.45–0.52, and 0.49–0.58; agreement leads only on less-rare disease.
  • Agreement as an LLM reliability signal: 0.919 versus 0.787 AUROC on the 32B tail shows agreement predicts LLM top-1 correctness beyond the retriever-solving-case difficulty proxy.Agreement is therefore non-circular reliability information, although the retriever’s margin remains the better deployment gate once the system defers.
  • Trained fusion: +0.2pp is the trained fusion’s frontier gain, with no configuration yielding a significant improvement over the heuristic hybrid.For small models, the classifier learns never to trust the LLM, reducing fusion exactly to the retriever.
  • Trained fusion: 39.2% versus 25.4% is the frontier oracle-versus-retriever performance gap, demonstrating real complementarity that trained fusion does not harvest.The fusion estimate is generous because it is fitted and evaluated on the same tail distribution without deployment shift.

H Audit of SapBERT-to-OMIM Linking · I Cross-Task Transfer and Boundary Conditions · I.1 Top-1 Correctness across Scorers

The audit finds SapBERT degradation on ultra-rare names, but arithmetic and corrected re-scoring show it cannot explain the small models’ collapse. Cross-task tests show that margin-based gating depends on scorer normalization and query-length effects.

  • H Audit of SapBERT-to-OMIM Linking: Held-out Orphanet synonyms resolve at 39.6% on ultra-rare diseases versus 49.9% on less-rare disease.The test avoids feeding the linker a candidate’s own name, which would score 100% by construction.
  • H Audit of SapBERT-to-OMIM Linking: Under corrected alias matching, small models reach 2–8% on the ultra-rare tail versus ≤5% strict, while aliases add almost nothing beyond top-5.The guard discards normalized forms claimed by multiple OMIM entries, preventing group-level scoring.
  • H Audit of SapBERT-to-OMIM Linking: The same generous rule adds exactly 0.0% for the retriever because, after disambiguation, it reduces to strict identifier matching.This asymmetry credits name-generating systems but cannot credit identifier-predicting systems.
  • H Audit of SapBERT-to-OMIM Linking: Inflating the best small model’s 4.6% strict tail Recall@1 by held-out linker resolution yields 11.6%, still below half the retriever’s 25.6%.The correction deliberately treats every linker miss as a discarded correct answer.
  • I Cross-Task Transfer and Boundary Conditions: For BC5CDR-to-MEDIC entity linking, the standard NIL rule thresholds top-1 rather than margin, and the two signals are indistinguishable at the 10% operating point.The recommendation is established for unnormalized accumulation scores, IC-weighted overlap, and Exomiser’s combined score, but is untested for normalized similarities.
  • I.1 Top-1 Correctness across Scorers: 0.793 vs. 0.743 AUROC: BM25 makes margin significantly better than raw score, with the same rare-concept ordering at 0.800 vs. 0.763.The comparison holds corpus and candidates fixed while changing only scorer normalization; the margin gain’s ∆95% CI is [+0.017, +0.082].
  • I.1 Top-1 Correctness across Scorers: On the authors’ encoder, top-score scale tracks query length at ρ = 0.97, while the deployed model shows ρ=+0.003 after fixed 32-token padding.Query-length normalization recovers about half of what the margin recovers on the authors’ encoder.

I.2 Candidate Presence · I.3 Boundary Conditions · I.4 Whole-Curve Evaluation

The paper separates correctness from candidate presence: the margin captures competition among candidates, whereas whole-set score levels detect whether an answer exists. This distinction holds across entity linking and passage retrieval, but depends on genuine ranked alternatives and does not make whole-curve metrics independent of AUROC.

  • I.2 Candidate Presence: Correctness depends on r1 −r2, while absent gold concepts are marked by low whole-candidate-set scores, b(x), which the margin cancels.The same score therefore supports two different decisions: whether the top candidate is correct and whether linking should occur at all.
  • I.2 Candidate Presence: 10.0/25.8/44.7% of 12,741 mentions become NIL when 10/25/50% of MEDIC concepts are masked.The masking protocol creates genuinely unlinkable mentions and evaluates both decisions in one run.
  • I.2 Candidate Presence: 0.636–0.657 AUROC makes BM25’s top-score-versus-field-mean contrast the worst statistic for candidate presence, below the case-side level at 0.767–0.775.The contrast cancels b(x), so it cannot compete with a statistic that reads case-side score level for presence detection.
  • I.2 Candidate Presence: 8.7/23.7/50.3% of 300 SciFact queries become unanswerable under 10/25/50% abstract masking, and the margin is worst for Qexists at every rate.The margin nevertheless beats raw score on Qcorrect by +0.043 to +0.080, reproducing the dissociation in passage retrieval.
  • I.3 Boundary Conditions: +0.072 AUROC occurs when SciFact’s top two abstracts are genuine alternatives, sharing 7.3% of content words versus 3.0% for random same-corpus pairs.The margin is informative only when the ranked candidates are genuinely different answers rather than near-duplicates.
  • I.3 Boundary Conditions: 0.409 AUROC results from the margin between the best and second-best of k=10 independent samples from a 7B model, showing that generated samples are not a ranked candidate set.The margin requires a ranked candidate set scored by one scorer; the base-accuracy criterion does not.
  • I.4 Whole-Curve Evaluation: 0.317 for the margin versus 0.343 for the raw score on the ultra-rare-tail retriever, while AUGRC and AUROC order the pair identically at fixed base accuracy.The paper verifies AUGRC = p(1−p)(1−AUROC)+1/2(1−p)^2 to within 10−5 and reports AUROC beside operating-point results.

J Clinical Grading of Residual Errors · K Table Notes · L Reproducibility

The paper grades residual diagnostic errors by clinical relatedness, while testing robustness, counselling implications, and difficulty confounding. It also documents table-scope caveats and releases the computational, software, data, and reproducibility details needed to interpret the results.

  • J Clinical Grading of Residual Errors: Wrong top-1 predictions are graded as same-test, same-class, unrelated, or unmeasurable rather than treated as clinically equivalent.Same-test requires a shared causative gene or Orphanet disorder; same-class requires a shared ICD-10 category.
  • J Clinical Grading of Residual Errors: 9.6% of the candidate space is unmeasurable because it lacks gene, Orphanet, and ICD-10 mappings, so it is not folded into unrelated.The paper treats calling these candidates unrelated as manufacturing harm.
  • J Clinical Grading of Residual Errors: Symmetric leave-source-out decontamination removes case-publication-only terms from the gold profile and every competitor.This prevents same-gene siblings from winning through provenance tied to the patient’s source publication.
  • K Table Notes: The table reports a globally answered decile of 200 cases, not the ultra-rare-stratified top-10% of n=478, and the LLM column is Qwen2.5-VL-7B at 1.7% tail Recall@1.The stricter same-publication control brackets the retriever tail at 5.2–25.6%, while its per-case 1.2× reverses to a per-disease rise.
  • J Clinical Grading of Residual Errors: +15.8pp is the reported positive shift, but robustness specifications yield +14.8pp, +14.4pp, and +10.1pp, with only the first two excluding zero.The paper therefore claims a direction rather than a resolved effect; symmetric tie exclusion produces the +14.8pp figure.
  • J Clinical Grading of Residual Errors: 7 of 15 same-test errors pair purely dominant and purely recessive entities, leaving counselling, cascade-testing, and recurrence-risk decisions wrong despite the assay finding the relevant target.Of the 15 errors, 4 share a causative gene and 11 share only an Orphanet disorder.
  • J Clinical Grading of Residual Errors: 81.1–83.9% versus 1.1–44.0% is the retriever-versus-LLM Recall@1 range inside margin-answered cases, favoring the retriever by +39 to +83pp across ten configurations.Both predictors improve in the selected band, but they do not converge; the advantage also holds for DeepSeek-V4-Pro with reasoning on: 44.0% vs. 83.3%.
  • L Reproducibility: The N=2000 evaluation sample is drawn once with seed 0 and reused across systems, while local runs use four RTX 6000 Ada GPUs and released software, prompts, pipeline, and derived outputs.The stack includes Python 3.11, PyTorch 2.11, CUDA 12.8, vLLM, and transformers; public resources are used under their published terms without redistributing source case-report text.
Loading 2608.14683v1…