Automated Chest CT Protocol Selection via Large Language Model Derived Text Embeddings from Imaging Request Text

Introduction

• The authors hypothesized that LLM-derived embeddings could improve the accuracy and consistency of automated chest CT protocol selection over traditional NLP baselines.Source: Introduction · pp. 2–3
• The study evaluates an LLM-based decision-support system against radiologist-selected protocols across common and rare chest CT classes.Source: Introduction · p. 3

Study Design

• The primary objective is to test whether LLM-derived embeddings improve automated protocol assignment over standard clinical assignments.Source: Study Design · p. 3
• Study design also includes an independent expert reader comparison and local PHI-preserving fine-tuning within a secure hospital environment.Source: Study Design · p. 3

Study Population

• Eligible patients had a clinical indication, a finalized scan-time protocol, and available procedure-name and HIS-comment fields.Source: Study Population · p. 3
• 285,123 imaging requests were retained and split into 228,099 training and 57,024 test requests.Source: Study Population · p. 3

Text Preprocessing

• Procedure names, indications, and HIS comments were combined, normalized, abbreviation-expanded, and stripped of identifiers and unrelated numeric codes.Source: Text Preprocessing · p. 4
• The radiologist-harmonized 18-class ontology excluded protocols with fewer than 100 occurrences amid a pronounced long-tail distribution.Source: Text Preprocessing · p. 4, Figure 1 · p. 4

• An open-weight LLaMA-3.1-70B model was adapted locally on institution-specific chest CT request text.Source: LLM Fine-Tuning and Embedding Extraction · p. 5
• Fine-tuning used parameter-efficient LoRA while keeping base-model weights frozen, with 4-bit NF4 quantization and bfloat16 where supported.Source: LLM Fine-Tuning and Embedding Extraction · p. 5
• Final-layer hidden states were mean-pooled across tokens into a fixed-length embedding for each input sequence.Source: LLM Fine-Tuning and Embedding Extraction · p. 5
• The resulting prompt-independent embeddings avoid variability from prompt formatting or instruction tuning.Source: LLM Fine-Tuning and Embedding Extraction · p. 5

Model Training and Class Imbalance Handling

• Multinomial logistic regression used L2 regularization, class-weighted cross-entropy, and repetition upsampling for protocols with fewer than 2,000 examples.Source: Model Training and Class Imbalance… · p. 5

Implementation Environment

• Training and inference ran on institutional high-performance computing infrastructure with an NVIDIA H100 GPU.Source: Implementation Environment · p. 5

Evaluation Metrics

• Evaluation included overall, weighted, and macro-averaged metrics and a separate analysis of the lowest-frequency protocol quartile.Source: Evaluation Metrics · p. 6
• Four experienced radiologists independently assigned protocols for 300 reader-study cases using clinical information, patient history, and imaging requirements.Source: Evaluation Metrics · p. 6
• Paired accuracy comparisons used McNemar's test, while per-class F1 comparisons used paired t-tests with significance at α = 0.05.Source: Evaluation Metrics · p. 6

Results

• 285,123 chest CT requests met the study criteria, with 228,099 (80.0%) assigned to training and 57,024 (20.0%) to testing.Source: Results · p. 6
• On the held-out test set, the model achieves high precision and recall, as reported for overall performance, without specifying outcomes across individual protocol classes.Table 1 reports overall performance on the held-out 20% test set.Source: Results · p. 6, Table 1 · p. 6
• The test set contained real, non-augmented data, unlike the upsampled training set, and was used for an unbiased performance evaluation.Source: Results · p. 6

F1 Score

• The reader-study analysis covered 16 protocol classes after consensus reassigned every case in two historically assigned classes and changed labels in four others.The two classes with no remaining reference-standard cases were Lung Cancer and Mediastinal Mass.Source: F1 Score · p. 7
• The clinical-assignment table reports per-class accuracy against radiologist consensus, with an overall accuracy of 83% (250/300).Source: F1 Score · p. 7, Table 2 · p. 7
• The LLM achieved 80% accuracy (241/300), compared with 83% (250/300) for human assignments against expert consensus.Source: F1 Score · p. 8
• The LLM table reports per-class accuracy against the four-radiologist expert consensus.Source: Table 3 · p. 8

Statistical Significance

• 80.33% of LLM predictions matched consensus, versus 83.33% for routine clinical assignments; the difference was not significant (χ² = 1.25, p = 0.263).Source: Statistical Significance · p. 8, Table 4 · p. 9
• The LLM matched consensus when routine assignment did not in 21 cases, while routine assignment matched consensus when the LLM did not in 30 cases.Source: Statistical Significance · p. 8
• LLM assignment entropy was 3.747, higher than the radiologists’ 3.449–3.555, indicating more even distribution across protocol classes and less concentration on common protocols.Figure 2 compares Shannon entropy for the LLM and four individual radiologists.Source: Figure 2 · p. 9, Statistical Significance · pp. 9–10

Discussion

• 80.3% model accuracy closely matched radiologists’ 83.3% against a four-reader consensus standard, with no statistically significant difference.The authors describe the decisions as largely indistinguishable from human experts.Source: Discussion · p. 10
• The LLM performed well on protocols with clear indications and exceeded radiologists on standard noncontrast chest and lung transplant follow-up protocols.The authors characterize these latter classes as broader or more heterogeneous in clinical presentation.Source: Discussion · p. 10
• The model underperformed on rare, high-complexity Trauma–Chest and combined PE + R + C protocols, each representing fewer than 1% of cases.The authors suggest scarcity limited fine-grained learning; models depend on sufficient examples to generalize.Source: Discussion · pp. 10–11
• The entropy analysis showed that the LLM distributed assignments more evenly than individual radiologists, suggesting a reduced tendency to over-select common protocols.Source: Discussion · p. 11
• The proposed decision-support tool is intended to streamline routine steps and help technologists prepare standard studies while radiologists focus on complex requests.The authors also identify resource-constrained settings, after-hours coverage, and training environments as contexts where support may be valuable.Source: Discussion · p. 11
• Fine-tuning within the institution’s secure computing environment keeps patient information within hospital infrastructure while adapting the model to local language and protocols.The authors connect this approach to jurisdictions that restrict transmitting imaging-request text to external services.Source: Discussion · p. 11
• The retrospective, single-center study requires external multi-institutional validation, and prospective workflow and patient-outcome effects remain unestablished.The authors also report limited performance on the rarest protocols, including 33% for PE + R + C and 63% for Trauma–Chest, reflecting few training examples.Source: Discussion · p. 11
• The authors conclude that LLM-based recommendation can achieve radiologist-level performance while improving consistency across a broad range of chest CT protocols.They describe reduced variability, efficiency, and reliability as potential benefits of decision support in practice.Source: Discussion · p. 12

4 PE (CTPA) 37,753 7.7

• Lung Transplant (LDCT + exp) is listed with 9,748 and 2.0.Source: §4 · p. 16
• ILD baseline (R + HighRes + exp + prone) is listed with 6,481 and 1.3.Source: §4 · p. 16
• Esophageal Cancer (R + C + oral) is listed with 5,129 and 1.0.Source: §4 · p. 16
• PE + R + C is listed with 2,850 and 0.58.Source: §4 · p. 16
• Lung Cancer (R + C, plus abdomen) is listed with 2,257 and 0.46.Source: §4 · p. 16
• Aorta – dissection (chest only) is listed with 2,103 and 0.43.Source: §4 · p. 16
• Aorta – dissection (chest + abdomen) is listed with 1,814 and 0.37.Source: §4 · p. 16

12 PE – ILD (LDCT + PE) 1,098 0.22

• PE + dissection (chest only) is listed with 830 and 0.17.Source: §12 · p. 16
• Trachea (R + thin section for 3D) is listed with 434 and 0.089.Source: §12 · p. 16