SNOMED CT Concept Recommendation from Masked Clinical Context

1 Introduction

• Clinical coding maps free text to standardized identifiers that support interoperability, while SNOMED CT provides a large structured terminology for clinical information.The challenge resource pairs MIMIC-IV-Note discharge summaries with curated SNOMED CT annotations; its large, long-tailed target terminology often has few supervised examples.Source: §1 · pp. 1–2
• The study asks whether surrounding clinical context can recover a masked SNOMED CT concept and whether training-note retrieval improves ranking.The target mention is removed to prevent direct string matching and isolate information in the surrounding language.Source: §1 · p. 2
• The benchmark compares sparse TF-IDF prototypes, dense LSA, sparse-dense fusion, retrieved-note evidence, and retrieval-augmented ranking under a leakage-controlled historical split.It also reports vocabulary coverage and frequency-stratified results and uses note-clustered paired bootstrap resampling to compare retrieval augmentation with the strongest sparse baseline.Source: §1 · p. 2
• Here, “retrieval-augmented” means adding non-parametric evidence retrieved from training notes to the ranking function, not using a generative LLM.The archive lacks complete SNOMED CT descriptions, synonyms, hierarchy, and relations, so the study evaluates retrieval and ranking rather than terminology-level ontology-grounded RAG.Source: §1 · p. 2

2.1 Clinical coding, ontology annotation, and SNOMED CT entity linking

• The challenge resource supports SNOMED CT annotation, but masked-concept recommendation differs from conventional entity linking by ranking a missing concept rather than detecting its span.Earlier work also explored evaluation of ontology annotation systems and supervised Bi-GRU span labeling.Source: §2.1 · p. 3

2.2 Sparse and dense retrieval

• Sparse TF-IDF emphasizes discriminative local terms, whereas LSA projects sparse representations into a dense space intended to capture higher-order associations.Biomedical encoders such as BioClinicalBERT and SapBERT are identified as next baselines, but were not used to establish a challenge-data-only baseline.Source: §2.2 · p. 3

2.3 Retrieval augmentation and ontology-grounded clinical mapping

• Prior biomedical RAG and SNOMED CT mapping work motivates retrieval-plus-reranking, while this study isolates whether retrieved note evidence improves ranking before adding external terminology or generative models.Ontology-grounded RAG uses formal domain structure to guide retrieval.Source: §2.3 · p. 3

2.4 Relationship to our previous semantic analysis

• Earlier MIMIC-IV analysis characterized concept relationships; this study instead tests held-out recommendation by masking targets, ranking candidates, and measuring retrieval performance with uncertainty estimates.A clinically plausible relationship in exploratory analysis need not improve out-of-sample ranking.Source: §2.4 · p. 4

3.1 Source

• 75,491 annotation rows were assigned across 204 training notes and 68 test notes; no test note fitted the vectorizers, prototypes, projections, or retrieval indexes.Accepted quality-control additions were assigned to their note’s historical partition; repeated mentions remain in annotation counts but collapse into note-concept pairs.Source: §3.1 · p. 4, Table 1 · p. 4
• 208.5 unique concepts per test note on average versus 159.8 in training made test notes more concept-dense.The denser test-note distribution increases the difficulty of recovering one masked concept from a crowded context.Source: §3.1 · p. 4

3.2 Vocabulary overlap and the long tail

• 1,278 of 4,068 test concept types (31.42%) and 1,370 of 14,179 test pairs (9.66%) are unseen in training, beyond closed-vocabulary coverage.The training vocabulary is strongly long-tailed: 2,768 concepts (52.06%) occur in exactly one training note, and 3,533 (66.45%) occur in at most two.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

4.1 Instance construction

• Each unique note-concept pair becomes an instance; the first mention anchors a context window extending 350 characters on either side, clipped at note boundaries.All target-concept spans overlapping the window are replaced with [MASK], removing the target surface form and making context alone the basis for recovery.Source: §4.1 · p. 5

4.2 Candidate inventory and evaluation scope

• The primary framework compares sparse, dense, and retrieval-derived ranking signals over 5,317 training-observed concepts, evaluating only test pairs whose targets are in that inventory.Unseen targets are reported separately; counting them as failures lowers the strongest model’s Recall@10 from 33.43% to 30.20% and MRR from 0.2114 to 0.1910. A complete terminology release with descriptions, synonyms, and relations would enable candidates unseen in annotations.Source: Figure 2 · p. 6, §4.2 · p. 6

5.1 Popularity baseline

• The popularity baseline ranks each candidate by the number of training notes containing that concept, without using query text.Source: §5.1 · p. 6

5.2 Sparse TF-IDF concept prototypes

• Masked contexts are represented with word-level TF-IDF, and each concept prototype is the normalized mean of its target contexts’ vectors.Test queries are ranked by cosine similarity to prototypes; masking target strings makes this context retrieval rather than surface-form matching.Source: §5.2 · p. 6

5.3 Dense LSA embeddings

• LSA projects training TF-IDF vectors into 96 latent components, averages context embeddings into concept centroids, and ranks test contexts by cosine similarity.The projection explains 18.57% of TF-IDF variance; this is a data-derived dense baseline, not a pretrained biomedical encoder.Source: §5.3 · p. 7

5.4 Sparse-dense hybrid

• The sparse-dense hybrid combines min-max-normalized TF-IDF and LSA scores with weights of 0.75 and 0.25, selected using validation MRR.Source: §5.4 · p. 7

5.5 Training-note retrieval evidence

• Training-note evidence ranks each query against all 204 historical training notes and sums similarity-weighted concept evidence from the ten nearest notes.This provides non-parametric memory from retrieved documents without a generative model.Source: §5.5 · p. 7

5.6 Retrieval-augmented hybrid

• The retrieval-augmented condition combines 85% of the sparse-dense score with 15% retrieved-note evidence.Validation favored zero retrieval weight, so this condition tests whether a modest retrieval contribution improves historical test performance.Source: §5.6 · p. 7

5.7 Concept-graph diagnostic

• The concept-graph diagnostic scores candidates by summed similarity to other gold concepts in the test note, which are treated as known seeds.Because it assumes perfect knowledge of all other annotated concepts, it is an oracle diagnostic, not an end-to-end deployment condition; a 65% sparse-dense and 35% graph mixture tests its effect on ranking.Source: §5.7 · pp. 7–8

5.8 Implementation details and reproducibility

• Validation favored zero retrieval weight, so the 15% retrieved-note contribution is evaluated as a sensitivity condition rather than a test-set-optimized setting.Preprocessing, feature fitting, projections, retrieval indexes, and graph statistics are learned from the historical training partition only.Source: §5.8 · p. 8

• Each closed-vocabulary test instance ranks all 5,317 training-observed concepts, with Recall@1, Recall@5, Recall@10, MRR, and nDCG@10 reported.The evaluation also reports vocabulary coverage, frequency-stratified performance, median rank, and clustered uncertainty.Source: §6 · p. 8
• The 12,809 evaluable test pairs comprise 1,641 rare, 2,739 medium-frequency, and 8,429 frequent target instances, stratified by training-note frequency.Rare concepts occur in 1–2 training notes, medium-frequency concepts in 3–10, and frequent concepts in more than 10.Source: §6 · p. 8
• Uncertainty is estimated with 2,000 paired bootstrap resamples clustered by test note, preserving dependence among instances from the same discharge summary.All test instances remain untouched during feature fitting and parameter selection.Source: §6 · p. 8
• Sparse TF-IDF prototypes outperform the data-derived dense representation and retrieved-note evidence on Recall@10.Source: Figure 3 · p. 9

7.1 Overall recommendation performance

• 14.81% Recall@1, 33.43% Recall@10, 0.2114 MRR, and 0.2297 nDCG@10 make sparse TF-IDF prototypes the strongest overall model.The results cover 12,809 historical test note-concept pairs.Source: §7.1 · p. 9, Table 3 · p. 9

7.2 Does retrieval augmentation help?

• Retrieved-note evidence alone performs poorly at 8.49% Recall@10; adding 15% retrieval evidence to the sparse-dense hybrid changes Recall@10 from 31.97% to 31.99% and lowers MRR from 0.1983 to 0.1937.Source: §7.2 · p. 10
• The retrieval-augmented hybrid trails sparse TF-IDF by 0.0144 Recall@10 and 0.0177 MRR, with both paired 95% intervals excluding zero.The differences are retrieval-augmented minus sparse; negative values favor sparse TF-IDF.Source: §7.2 · p. 10, Table 4 · p. 10
• The oracle graph diagnostic reaches only 6.85% Recall@10; combining graph and text reaches 32.27%, still below sparse TF-IDF.The graph mixture's MRR difference from sparse TF-IDF is -0.0066, with a paired 95% interval of -0.0140 to 0.0009.Source: §7.2 · p. 10

7.3 Long-tail performance

• Recall@10 rises from 7.74% for targets in one or two training notes to 43.90% for targets in more than ten; median rank falls from 991 to 14.Figure 4 describes Recall@10 as increasing sharply with the number of training notes containing the target concept.Source: §7.3 · p. 10, Table 5 · p. 10, Figure 4 · p. 11

8.1 RQ1: Sparse local context is the strongest baseline

• 33.43% Recall@10 and 0.2114 MRR make sparse TF-IDF the strongest text-only approach, ahead of dense LSA at 17.10% Recall@10.Sparse-dense fusion does not recover the lost performance; the 96-dimensional LSA projection compresses distinct contexts while TF-IDF retains specific neighboring terms.Source: §8.1 · p. 11

8.2 RQ2: Whole-note retrieval does not improve ranking

• 31.99% Recall@10 and 0.1937 MRR for the retrieval-augmented hybrid trail sparse TF-IDF (33.43%, 0.2114), with paired intervals excluding zero.Whole-note retrieval may introduce concepts unrelated to a local masked context, motivating retrieval over smaller evidence units or terminology records.Source: §8.2 · pp. 11–12

8.3 RQ3: The terminology long tail dominates performance

• 7.74% versus 43.90% Recall@10 separates concepts seen in one or two training notes from those seen in more than ten; 9.66% of test pairs are unseen in training.The median rank shifts from 991 to 14 across those frequency groups, and unseen targets are impossible for a closed-vocabulary recommender.Source: §8.3 · p. 12

8.4 Implications for ontology-grounded RAG

• The proposed ontology-grounded system would index SNOMED CT names, synonyms, hierarchy, and relations, and jointly retrieve fine-grained note passages and terminology records.Ontology-grounded retrieval may be especially valuable for rare and unseen concepts because it introduces knowledge unavailable from the training notes alone.Source: §8.4 · p. 12

8.5 Scope of the retrieval-augmented evaluation

• The experiment evaluates retrieval augmentation at the ranking layer, not a complete generative RAG system, because the archive lacks the full SNOMED CT terminology release.The study therefore does not attribute its results to LLM generation or claim zero-shot ontology-grounded retrieval.Source: §8.5 · p. 12

• Masked-concept recovery is a constructed retrospective task, so its scores should not be compared directly with official span-detection or character-level challenge scores.Source: §9 · p. 13
• Using the first annotated mention as the sole anchor may miss other contexts for the same concept that are more or less informative.The study recommends evaluating section-aware sampling, multiple anchors, and multiple context-window sizes.Source: §9 · p. 13
• LSA is trained only on the available challenge contexts, limiting conclusions about neural semantic retrieval and leaving ClinicalBERT and SapBERT for direct evaluation [17] [18].Source: §9 · p. 13
• The retrieval corpus contains only 204 training discharge summaries and uses whole notes as evidence units; fine-grained passage retrieval may be stronger.Source: §9 · p. 13
• The training-only candidate inventory leaves 9.66% of test instances out of vocabulary, requiring a complete terminology index to evaluate unseen-concept recommendation.Source: §9 · p. 13
• The concept-graph condition assumes gold knowledge of all remaining concepts in a test note, making it an oracle diagnostic rather than a deployable end-to-end model.Source: §9 · p. 13
• Because the study uses MIMIC-IV discharge documentation from one source, generalization to other institutions, note types, specialties, or real-time workflows is uncertain.Source: §9 · p. 13

• 33.43% Recall@10 and 0.2114 MRR make sparse TF-IDF the best approach among the evaluated methods on 12,809 closed-vocabulary historical test pairs.Dense LSA, whole-note retrieval, sparse-dense fusion, and simple retrieval augmentation did not improve the baseline.Source: §10 · p. 13
• 7.74% Recall@10 for concepts in one or two training notes and 9.66% unseen test pairs mark the effects of terminology sparsity and coverage.Source: §10 · p. 13
• The proposed next step is a terminology index with biomedical and clinical encoders, finer-grained evidence retrieval, and a reranker constrained to valid concept identifiers.The benchmark and baselines provide a reference for testing whether ontology-grounded additions improve recommendation quality.Source: §10 · p. 13

Data and Code Availability

• The notes and annotations are not redistributed with the manuscript and require PhysioNet credentialing and a data use agreement.They are available through the SNOMED CT Entity Linking Challenge v1.2.1 resource.Source: Data and Code Availability · p. 13
• The reproducibility package includes analysis and figure-generation code, aggregate result tables, dataset statistics, and bootstrap summaries, but no raw clinical text.Its scripts expect local copies of train_notes.csv and train_annotations.csv.Source: Data and Code Availability · p. 14

Ethics Statement

• The study is a secondary analysis of deidentified data, involved no new patient contact, intervention, or prospective decision support, and its outputs are not clinical advice.Source: Ethics Statement · p. 14