PaperlayerView exact revision on arXiv

Source-linked AI summary

ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders

Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Atilla Kiraly, Sahar Kazemzadeh, Zakkai Melamed, Jungyeon Park, Patricia Strachan, Yun Liu, Chuck Lau, Preeti Singh, Christina Chen, Mozziyar Etemadi, Sreenivasa Raju Kalidindi, Yossi Matias, Katherine Chou, Greg S. Corrado, Shravya Shetty, Daniel Tse, Shruthi Prabhakara, Daniel Golden, Rory Pilgrim, Krish Eswaran, Andrew Sellergren

arXiv:2308.01317v2cs.CVeess.IV

TL;DR

Medical-imaging AI often relies on costly expert-labeled data and narrow, vision-only models, although clinical workflows are multimodal. ELIXR grafts a language-aligned vision encoder onto a frozen LLM and, using paired CXR images and reports, reports strong performance across several CXR tasks.

  • Problem

    Medical-imaging AI faces costly expert-data curation and task-specific vision-only systems, despite clinical workflows that use multimodal inputs.

    Source: Introduction · p. 3

  • Method

    ELIXR adapts a language-aligned vision encoder to a frozen LLM and trains on paired CXR images and free-text radiology reports.

    Source: Introduction · p. 3, ELIXR system · p. 4

  • Results

    ELIXR achieved state-of-the-art performance in zero-shot classification, data-efficient classification, and semantic search, while also showing potential in VQA and report quality assurance.

    Source: Discussion · p. 23

  • Takeaways & Limitations

    The authors report that ELIXR improves data efficiency over supervised contrastive learning by two orders of magnitude, with potential to support models for long-tail diagnoses using less expert-curated data.

    Source: Discussion · p. 23

    Keeping the LLM and vision encoder frozen may come at the expense of overall performance.

    Source: Discussion · p. 24

Abstract

from arXiv · show

In this work, we present an approach, which we call Embeddings for Language/Image-aligned X-Rays, or ELIXR, that leverages a language-aligned image encoder combined or grafted onto a fixed LLM, PaLM 2, to perform a broad range of chest X-ray tasks. We train this lightweight adapter architecture using images paired with corresponding free-text radiology reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13 findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898 across five findings (atelectasis, cardiomegaly, consolidation, pleural effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images) training data), and semantic search (0.76 normalized discounted cumulative gain (NDCG) across nineteen queries, including perfect retrieval on twelve of them). Compared to existing data-efficient methods including supervised contrastive learning (SupCon), ELIXR required two orders of magnitude less data to reach similar performance. ELIXR also showed promise on CXR vision-language tasks, demonstrating overall accuracies of 58.7% and 62.5% on visual question answering and report quality assurance tasks, respectively. These results suggest that ELIXR is a robust and versatile approach to CXR AI.

Introduction

Medical imaging AI faces costly expert-data curation and narrow, often image-only systems, despite multimodal clinical workflows. ELIXR combines a vision encoder with a frozen LLM to support varied CXR tasks using paired images and reports.

Introduction

  • Vision-only systems are often limited to discrete tasks and image inputs, while clinical workflows draw on multimodal information; expert-annotated training data are costly.These are among the challenges identified as contributing to an implementation gap for medical-imaging AI.Source: Introduction · p. 3
  • ELIXR is a lightweight adapter that grafts a vision encoder onto a frozen LLM for a range of vision-language tasks relevant to medical imaging.The paper focuses its case study on chest X-rays because paired image-text data are widely available, while stating that the method applies to other image modalities.Source: Introduction · p. 3
  • The authors report state-of-the-art performance in zero-shot and data-efficient classification and semantic search, and demonstrate VQA and report quality assurance.They state that the model’s efficiency may support addressing long-tail diagnoses and developing more broadly useful diagnostic-workflow tools.Source: Introduction · pp. 3–4
  • The authors describe ELIXR as fast and resource-efficient to train compared with full LLM finetuning, with potential for rapid prototyping and adaptation to distribution shifts using little new data.They also note that alternative publicly available LLMs can be used.Source: Introduction · pp. 3–4
  • ELIXR is trained on paired CXR images and free-text radiology reports without expensive expert manual label curation.The authors describe this as using routinely collected data.Source: Introduction · p. 4

Methods

The methods describe a two-stage vision-language system, its datasets and task-specific evaluation procedures. The work uses ELIXR-C and ELIXR-B for classification and retrieval, and ELIXR-B with a fixed LLM for text-output tasks.

ELIXR system

  • ELIXR-C aligns CXR images and reports; ELIXR-B builds on it, training a Q-Former in two phases to align image and report representations and then generate report impressions through a frozen LLM.Both the vision encoder and PaLM 2-S remain frozen; the adapter components are trained. Phase 1 uses contrastive, generation, and matching objectives, while phase 2 uses language-modeling loss.Source: ELIXR system · pp. 4–5, Figure 1 · p. 6

Datasets

  • The study includes more than one million CXRs from six datasets; ELIXR-C training used IND1, US1, and MIMIC-CXR, while ELIXR-B training used MIMIC-CXR alone.The dataset set also includes CXR-14, CheXpert, and chest-only VQA-RAD.Source: Datasets · pp. 6–7

Evaluation

  • Five tasks were evaluated: zero-shot and data-efficient classification, semantic search, VQA, and report quality assurance; text-output tasks use ELIXR-B.Classification and semantic search use ELIXR-C or ELIXR-B phase 1 embeddings; VQA and quality assurance use ELIXR-B phase 2 with fixed PaLM 2-S.Source: Evaluation · p. 8, Table 2 · p. 8

Classification

  • Zero-shot classification scores positive and negative prompts against image embeddings for ELIXR-C or Q-Former image tokens for ELIXR-B; data-efficient classification trains a classifier on frozen image features.The data-efficient procedure uses a two-layer MLP and also evaluates linear classifiers at 1% and 10% training data; AUCs are averaged across 10 random training-set draws.Source: Classification · p. 9, Statistical analysis · p. 10

Semantic search

  • Semantic search uses 19 queries across four query types on the MIMIC-CXR test set, comparing ELIXR-C’s direct cosine-similarity retrieval with ELIXR-B’s candidate retrieval and reranking.A thoracic radiologist scores the top five results; precision@5 and NDCG@5 measure retrieval quality.Source: Semantic search · p. 10

Visual question answering

  • VQA is evaluated on MIMIC-CXR and chest-only VQA-RAD; ELIXR-B first generates an impression, then uses aligned tokens and the question with PaLM 2-S to produce an answer.A radiologist grades non-binary answers, while answers that can be mapped to yes/no are automatically scored; sensitivity, specificity, and accuracy are calculated from average grades.Source: Visual question answering · pp. 11–12

Report quality assurance

  • Report quality assurance tests altered impressions for laterality swaps, omitted findings, and erroneous additions, using selected MIMIC-CXR cases with an unambiguous primary finding.The evaluation includes controls and varies the alteration mix for edema and “No Finding” cases; impressions were included only when present and the primary finding was clear on the image.Source: Report quality assurance · pp. 12–13
  • For report checking, ELIXR-B generates image-based findings, then Med-PaLM 2 compares them with the altered impression to identify missing, added, or laterality-swapped findings.A thoracic radiologist scores outputs as correct only when the alteration is correctly described or an unaltered control is identified as correct and complete.Source: Report quality assurance · pp. 13–14

Results

ELIXR achieved strong zero-shot and data-efficient classification and semantic-search results, while also showing capability on VQA and report QA. The discussion identifies prompt sensitivity and evaluation scope as important qualifications.

Results

  • ELIXR-C and ELIXR-B surpassed CheXzero on zero-shot classification across 13 findings and matched fully supervised SupCon on five findings, despite CheXpert being held out.Figure 2 compares zero-shot AUCs across 11 findings and SupCon results across five.Source: Results · p. 14, Figure 2 · p. 15
  • ELIXR-B and ELIXR-C set new data-efficient classification state of the art on five CheXpert findings using 1% and 10% of training data, with performance on par with SupCon using roughly 100-fold less data.The comparison also reports that ELIXR outperformed fully fine-tuned ConVIRT; Figure 3 averages results over two datasets and seven findings.Source: Results · pp. 15–16, Figure 3 · p. 16, Table 3 · p. 17
  • ELIXR-B achieved perfect retrieval on 12 of 19 queries, including laterality-, severity-, and nuance-specific cases; both ELIXR-B and ELIXR-C performed worse on fracture- and pneumothorax-related queries than on other findings.Source: Results · p. 18
  • ELIXR-B achieved 58.7% accuracy on VQA-RAD, 54.8% on MIMIC-CXR VQA, and 62.5% on MIMIC-CXR report QA.The VQA-RAD evaluation used a CXR-only, more difficult split without training on the development set, so its results are not directly comparable to literature results.Source: Results · pp. 20, 23, Table 5 · p. 20, Table 6 · p. 22

Discussion

  • The frozen vision encoder and LLM reduce training compute because second-stage backpropagation updates only the much smaller Q-Former and MLP components.The authors describe retraining for new tasks or newer encoders and LLMs as relatively inexpensive.Source: Discussion · p. 23
  • ELIXR’s semantic search retrieved findings across laterality, severity, and nuanced wording, with potential uses in dataset curation, historical-image search, and education.The discussion reports perfect retrieval for subsets of findings-related, laterality-specific, severity-related, and nuanced queries.Source: Discussion · p. 24
  • VQA and report QA performance depended heavily on prompts and the LLM, and the authors expect more prompt engineering to be needed for greater QA specificity in clinical settings.The authors note that false-positive reports should be reduced before use in settings with lower error rates than the enriched simulated set.Source: Discussion · p. 24

Conclusion

  • ELIXR is presented as an initial step toward general-purpose X-ray AI, but its frozen components, prompt fragility, limited benchmarks, and complex ICU data constrain the evidence.Pneumothorax and fractures were challenging across tasks, only partly explained by low prevalence and noisy training references.Source: Discussion · pp. 24–25, Conclusion · p. 25

Supplementary materials

The supplementary materials include additional ELIXR training details.

Supplementary Text

The supplementary text describes the ELIXR-C encoder setup and the two ELIXR-B training phases, including their training and checkpoint-selection configurations.

ELIXR-C

  • ELIXR-C paired an EfficientNet-L2 vision encoder initialized from a SupCon checkpoint with a small T5 text encoder initialized from scratch.Reports were processed to prefer the impression section, then findings; input text was truncated at 128 tokens. Training used 128-dimensional projections, a fixed 0.07 temperature, and about 80,000 steps.Source: ELIXR-C · p. 31

ELIXR-B

  • ELIXR-B phase one spatially pooled precomputed ELIXR-C embeddings and trained a BLIP-2 Q-Former with 32 query tokens.Checkpoints were selected using validation performance and losses, with the choice depending on whether the target was classification/search or image-grounded text generation.Source: ELIXR-B · p. 31
  • In phase two, PaLM 2-S inference servers returned language-modeling gradients that were backpropagated through the Q-Former.Training used 32 server replicas, a batch size of 32, and Adam with a constant learning rate of 5e-4.Source: ELIXR-B · p. 31

Supplementary tables

The supplementary tables document prompts and evaluation procedures alongside classification and retrieval comparisons. They also specify rubrics for scoring visual question answering and report quality assurance.

Supplementary tables

  • Supplementary Table 1 lists the prompts used for zero-shot classification and states that the prompting strategy developed by CheXzero is adopted.Supplementary Table 8 reports AUCs with 95% confidence intervals across 13 findings in the CheXpert test set, excluding the “No Finding” label.Source: Table 1 · p. 32, Table 8 · p. 41
  • The semantic-search table lists queries and ELIXR performance using normalized discounted cumulative gain at five (NDCG@5).The listed query examples span findings, laterality, severity, and device-placement questions.Source: Table 2 · p. 33
  • The ground-truthing table pairs report-selection keywords with prompts that identify candidate cases for thoracic-radiologist confirmation.A separate question list includes queries about opacity, devices, fractures, and other findings.Source: Table 3 · p. 34, Supplementary tables · p. 37
  • The visual question answering materials specify findings-dependent questions and a scoring rubric for assessing answers against expected answers or the image and report.Questions cover presence, location, severity, size, or type, with additional findings-independent questions in some cases.Source: Table 4 · p. 36, Table 5 · p. 38
  • The report-quality-assurance materials describe case alterations and a radiologist’s rubric for scoring LLM output.The rubric distinguishes inaccurate, irrelevant, missing, or internally inconsistent answers from answers that cannot be compared with ground truth.Source: Supplementary tables · p. 38, Table 6 · p. 39, Table 7 · p. 40
  • The data-efficient classification tables compare ELIXR-B with ELIXR-C and SupCon across sample sizes for CheXpert and CXR-14.The SupCon comparison reports AUC differences with confidence intervals and p-values, and marks noninferiority at a 0.05 margin.Source: Table 9 · p. 43, Table 10 · p. 45

Supplementary figures

The supplementary figures show label-review outcomes and data-efficient classification performance across datasets, findings, and training-set sizes.

Supplementary figures

  • 1,568 MIMIC labels were flagged for review and 1,092 were modified; the figure distinguishes correctly identified label errors, correct labels, shared errors, and indeterminate cases by color.Green marks correctly identified errors or missing labels, blue correct MIMIC labels, red errors in both label sources, and gray indeterminate labels.Source: Figure 1 · p. 50
  • The label-review figure separates shared errors from errors or missing labels identified by Med-PaLM 2.Its color coding also marks correct MIMIC labels and indeterminate cases.Source: Figure 1 · p. 50
  • The data-efficient classification figure compares models by finding and training-set sample size on CheXpert and CXR-14.Significance-test results are provided in Supplementary Tables 9 and 10.Source: Figure 2 · p. 51
Loading 2308.01317v2…