Source-linked AI summary
An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT
Chong Ma, Zihao Wu, Jiaqi Wang, Shaochen Xu, Yaonai Wei, Fang Zeng, Zhengliang Liu, Xi Jiang, Lei Guo, Xiaoyan Cai, Shu Zhang, Tuo Zhang, Dajiang Zhu, Dinggang Shen, Tianming Liu, Xiang Li
TL;DR
Radiology impression generation is important, but specialized language and limited domain data complicate automatic summarization. ImpressionGPT builds dynamic contexts from similar reports and iteratively refines ChatGPT’s output. The authors report state-of-the-art results that surpass methods using large volumes of medical text for pretraining.
Problem
Radiology summarization is needed for clinical workflows, while existing methods and limited samples pose challenges for domain-specific processing.
Source: §I · p. 2Method
ImpressionGPT uses similar reports as dynamic context and iteratively updates ChatGPT’s response using automated evaluation and instructions.
Source: §I · pp. 2–3Results
ImpressionGPT reports state-of-the-art performance, surpassing methods that use large volumes of medical text for pretraining.
Source: §V · p. 10Takeaways & Limitations
The authors present the approach as a resource-efficient paradigm for adapting existing LLMs to domain-specific tasks.
Source: §V · p. 10The model is intended as a reference for clinical experts and does not replace their decision-making authority.
Source: §V · p. 11
Abstract
from arXiv · showhide
The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians, and it is typically written by radiologists based on the 'Findings' section. However, writing numerous impressions can be laborious and error-prone for radiologists. Although recent studies have achieved promising results in automatic impression generation using large-scale medical text data for pre-training and fine-tuning pre-trained language models, such models often require substantial amounts of medical text data and have poor generalization performance. While large language models (LLMs) like ChatGPT have shown strong generalization capabilities and performance, their performance in specific domains, such as radiology, remains under-investigated and potentially limited. To address this limitation, we propose ImpressionGPT, which leverages the in-context learning capability of LLMs by constructing dynamic contexts using domain-specific, individualized data. This dynamic prompt approach enables the model to learn contextual knowledge from semantically similar examples from existing data. Additionally, we design an iterative optimization algorithm that performs automatic evaluation on the generated impression results and composes the corresponding instruction prompts to further optimize the model. The proposed ImpressionGPT model achieves state-of-the-art performance on both MIMIC-CXR and OpenI datasets without requiring additional training data or fine-tuning the LLMs. This work presents a paradigm for localizing LLMs that can be applied in a wide range of similar application scenarios, bridging the gap between general-purpose LLMs and the specific language processing needs of various domains.
I. INTRODUCTION
Radiology impressions are clinically important, but specialized language and data constraints complicate automatic summarization. ImpressionGPT uses similar reports to build dynamic ChatGPT prompts and iteratively improves its generated impressions.
I. INTRODUCTION
- Traditional methods struggle with complex semantics, while neural and pretrained approaches have improved fluency and contextual relevance in summarization.The introduction frames this progress against growing demand for efficient summarization.Source: §I · p. 1
- Radiology reports support clinical decisions, and growing imaging volume motivates efficient processing and concise, informative report summaries.The introduction also connects automatic summaries to clinical workflows and communication among care teams.Source: §I · p. 2
- ImpressionGPT uses similarity search to select domain-specific examples and build dynamic prompts that provide contextual information for ChatGPT [16].The approach is presented as in-context learning with limited samples.Source: §I · p. 2
- An iterative prompt incorporates ChatGPT’s response and its evaluation to provide feedback and instructions for subsequent updates.The framework is reported to outperform current radiology summarization methods using only 5–20 samples for optimization.Source: §I · pp. 2–3
- The authors present the framework as a resource-efficient paradigm for adapting existing LLMs to domain-specific applications.They state that it does not require fine-tuning the LLM.Source: §I · p. 3
II. BACKGROUND AND RELATED WORKS
Prior summarization research spans extractive and abstractive approaches, while radiology impression generation has evolved from sequence models to pretrained and graph-based methods.
A. Text Summarization in Natural Language Processing
- Extractive methods select source sentences, but their summaries can be incoherent, have uncontrollable length, and depend heavily on the source text –, [19].Examples include rule-based filtering, graph ranking, and latent semantic analysis.Source: A. Text Summarization in Natural… · p. 3
- Abstractive summarization is an end-to-end generative task that understands source meaning and generates a new summary; it is more challenging than extractive summarization but aligns more with human writing habits,, [19]–.Source: A. Text Summarization in Natural… · p. 3
B. Radiology Report Summarization
- Radiology impression generation progressed from sequence-to-sequence models to pretrained biomedical or radiology models and graph-based approaches, –.The review notes that existing pretrained approaches still have poor generalization associated with low model complexity.Source: B. Radiology Report Summarization · p. 3
C. Large Language Model
LLMs offer broad task generalization, but specialized-domain performance and prompt design remain challenges. The section reviews manual and automated prompt strategies.
C. Large Language Model
- LLMs can generalize across downstream tasks without fine-tuning, yet ChatGPT’s performance in specialized domains remains unsatisfactory.The section motivates ImpressionGPT as an iterative optimization approach for radiology summarization.Source: C. Large Language Model · p. 4
D. Prompt Engineering
- Prompt engineering can guide outputs, but small prompt variations may substantially affect performance, making suitable prompts difficult to find.The section distinguishes manual prompts from automated template prompts.Source: D. Prompt Engineering · p. 4
1) Manual Prompt:
- Manual prompts specify what input information to focus on and how to perform the task, but creating them requires time and expertise,.They are used in medical settings where interpretability and specific output formats matter.Source: 1) Manual Prompt · p. 4, 2) Automated Template Prompt · p. 4
2) Automated Template Prompt:
- Automated prompt approaches include mining, paraphrasing, gradient-based search, prompt generation, continuous prompts, and static templates –.These prompt types can be used independently or in combination across language tasks.Source: 2) Automated Template Prompt · p. 4
III. METHOD
ImpressionGPT constructs a test-specific context from similar reports, then iteratively revises ChatGPT’s impression using evaluated good and bad responses.
III. METHOD
- The pipeline labels diseases in test-report findings, retrieves similar reports, constructs a dynamic prompt, and iteratively optimizes ChatGPT’s result.The figure depicts these stages as parts a–d.Source: §III · p. 4, Figure 2 · p. 5
- The iterative step compares generated impressions with examples in the dynamic prompt and feeds evaluated good and bad responses back with guidance for the next response.The next response is guided toward the former good response and away from the former bad response.Source: §III · p. 5
A. Dynamic Prompt Generation
- Dynamic prompts address fixed prompts’ limited prior knowledge for complex, domain-specific tasks by using similar examples from relevant corpora.The method’s dynamic prompt generation comprises similarity search and prompt design.Source: A. Dynamic Prompt Generation · p. 5
1) Similarity Search:
ImpressionGPT retrieves diagnostically similar reports to create a case-specific prompt, then iteratively evaluates and refines ChatGPT’s response. The prompts use findings–impression examples and feedback to steer subsequent responses.
1) Similarity Search:
- CheXpert labels encode 14 observations in report vectors, which are compared by Euclidean distance to retrieve similar reports for dynamic prompts.Each observation can be present, absent, uncertain, or unmentioned; unmentioned labels are represented as 2.Source: 1) Similarity Search · pp. 5–6, Figure 3 · p. 6
2) Prompt Design:
- The dynamic prompt combines a task description, similar-report examples pairing Findings with Impressions, and a final query containing the test Findings.The examples create a data-specific context by using reports with content similar to the target.Source: 1) Similarity Search · p. 6, 2) Prompt Design · p. 6, Figure 4 · p. 7
B. Iterative Optimization of Response
- Because a fixed dynamic prompt may produce an inappropriate one-off response, the method evaluates initial outputs and uses good- or bad-response feedback to construct an iterative prompt.The process repeats until the maximum iteration count is reached.Source: B. Iterative Optimization of Response · pp. 6–7
1) Response Evaluation:
Response evaluation averages Rouge-1 similarity against the impressions of the similar reports used in the dynamic prompt, then classifies the response against a predefined threshold.
1) Response Evaluation:
- The method averages Rouge-1 scores between a generated response and the impressions of similar reports, then labels it good above a predefined threshold and bad otherwise.The resulting feedback is incorporated into the iterative prompt for another ChatGPT response.Source: 1) Response Evaluation · p. 7
2) Prompt Design:
The iterative prompt adds evaluated positive and negative examples to the dynamic context, then gives ChatGPT rules for generating a revised impression. Good and bad responses are updated across iterations.
2) Prompt Design:
- The iterative prompt pairs good- and bad-response instructions with the corresponding ChatGPT responses, then asks the model to follow the good example and avoid the bad one.A length-limit instruction is included to avoid overly verbose outputs.Source: 2) Prompt Design · p. 7, Figure 5 · p. 8
- The prompt uses one good response and an indefinite number of bad responses, which can be appended while ChatGPT’s input limit is not exceeded.Good or bad responses are updated at each iteration.Source: 2) Prompt Design · p. 8
- ImpressionGPT combines semantically relevant dynamic context with evaluation-driven iterative updates to generate radiology report summaries.The authors report superior summarization results; this section does not provide the underlying benchmark comparisons.Source: 2) Prompt Design · p. 8
IV. EXPERIMENT AND ANALYSIS
The experiments evaluate ImpressionGPT on MIMIC-CXR and OpenI after filtering reports and reserving test sets. Evaluation uses Rouge overlap metrics and factual consistency measures.
A. Dataset and Evaluation Metrics
- The experiments use 1,606 MIMIC-CXR test reports and an OpenI split with 2,400 training and 576 test reports after report eligibility filtering.For MIMIC-CXR, the official split is used; OpenI has no official split, so the authors randomly divide the data.Source: A. Dataset and Evaluation Metrics · p. 8
- Table I reports dataset statistics for MIMIC-CXR with an official split and OpenI with a random split, including average Findings and Impression lengths.The reported lengths include sentence-based and word-based averages.Source: Table I · p. 8, A. Dataset and Evaluation Metrics · p. 8
- Table II presents an ablation study of ImpressionGPT on MIMIC-CXR and OpenI, but the supplied passage does not report its result values.The caption states that bold denotes the best result.Source: Table II · p. 9
- Evaluation reports Rouge-1, Rouge-2, and Rouge-L F1 scores for overlap with reference impressions, plus factual-consistency precision, recall, and F1 on MIMIC-CXR.The factual-consistency metric measures similarity using generic disease-related observations.Source: A. Dataset and Evaluation Metrics · pp. 8–9
B. Implementation Details
ImpressionGPT combines dynamic prompts and iterative optimization to improve radiology impression generation. Its results are reported against prior methods on MIMIC-CXR and OpenI, with the selected implementation settings established through ablations.
B. Implementation Details
- The selected settings are Ns=15, T=0.7, Gd=1, Bd=n (n>1), and I=17.Ns controls the number of similar reports in the dynamic prompt; T is the evaluation threshold, Gd and Bd set positive and negative feedback counts, and I is the iteration count.Source: B. Implementation Details · p. 9
C. Main Property Studies
- Rouge-1, Rouge-2, and Rouge-L rise by 11.85%, 9.27%, and 10.36% on MIMIC-CXR and 8.59%, 7.22%, and 8.45% on OpenI with Dynamic Prompt.BERTScore metrics also improve; adding iterative optimization to Fixed Prompt further improves results on both datasets, and on OpenI it exceeds Dynamic Prompt.Source: C. Main Property Studies · p. 9
D. Comparison with Other Methods
- ImpressionGPT outperforms the compared methods on all reported metrics except Rouge-2 on OpenI, where it trails Jinpeng et al. and WGSum (LSTM).The comparisons include pretrained and GPT-based models, Radiology-Llama2, and fine-tuned GPT-3; the framework does not fine-tune ChatGPT parameters.Source: D. Comparison with Other Methods · pp. 9–10, Table III · p. 10
- Semantically similar examples provide useful prompt context for domain-specific LLM tasks, while iterative interaction can further optimize generated outputs.The authors report that ImpressionGPT’s combination of dynamic prompts and iterative optimization improves on the compared methods without training ChatGPT parameters.Source: D. Comparison with Other Methods · p. 10
V. DISCUSSION AND CONCLUSION
The paper presents ImpressionGPT as a prompt-based approach to radiology report summarization, using relevant examples and iterative evaluation feedback. It reports state-of-the-art results and frames the system as clinical support, while identifying evaluation and future-development considerations.
V. DISCUSSION AND CONCLUSION
- ImpressionGPT uses corpus-retrieved relevant examples for dynamic prompts and iterative automated feedback with instructions based on good and bad responses.The authors report state-of-the-art results over methods using large volumes of medical text for pretraining.Source: §V · p. 10
- Rouge-1 was the selected evaluation criterion, and performance was sensitive to the threshold; the authors call for semantic evaluation criteria.They speculate that domain-specific wording differences make fine-grained Rouge-1 evaluation preferable for generated-result details.Source: §V · p. 10
- The model is intended as a reference for clinical experts, not a replacement for their decision-making authority.The paper reports testing on de-identified data and says automated evaluation reduces the possibility of biased or fabricated outputs.Source: §V · p. 11
- Future work includes adapting prompts with public and local data, addressing privacy and safety in multi-institution settings, and incorporating radiologist input.The authors also propose exploring knowledge graphs in prompt design.Source: §V · p. 11
APPENDIX A STUDY OF HYPER-PARAMETER SELECTION
The hyperparameter study used 500 MIMIC-CXR test samples to select settings, then applied the selected parameters to the complete test set for final results. The reported choices balance metric performance, input limits, and cost.
APPENDIX A STUDY OF HYPER-PARAMETER SELECTION
- The ablation selected parameters using 500 MIMIC-CXR test samples, then used the complete test set for final results.The study varied prompt example counts, evaluation threshold, good and bad response counts, and iteration times.Source: Appendix A · p. 13
- With 15 dynamic examples, Rouge-1, Rouge-2, and Rouge-L increase by 11.4%, 8.78%, and 10.0%, respectively, over Fixed Prompt.The study reports that 18 examples saturate results and may exceed the input token limit.Source: Table IV · p. 13, Appendix A · p. 13
- At T=0.7, iterative optimization improves Rouge-1, Rouge-2, and Rouge-L by 7.56%, 7.2%, and 8.66%, respectively, at about five times the cost.T is the response-evaluation metric threshold used in iterative optimization.Source: Table V · p. 13, Appendix A · p. 13
- One good response and multiple bad responses yield the optimal reported results; adding more good examples does not improve results and can exceed the text limit.The study states that bad examples can be appended while the input limit allows.Source: Table VI · p. 13, Appendix A · p. 13
- With 17 iterations, Rouge-1, Rouge-2, and Rouge-L improve by 1.84%, 1.73%, and 2.1%, respectively, over Table VI; more iterations add cost without improving performance.The resulting parameter set is Ns=15, T=0.7, Gd=1, Bd=n, and I=17.Source: Table VII · p. 13, Appendix A · p. 13
APPENDIX B EXAMPLES IN OUR ABLATION STUDY
The appendix illustrates prompt ablations on a radiology test case and examines how corpus size affects similarity search. Generated results improve with more iterations, while larger search corpora are associated with better performance.
APPENDIX B EXAMPLES IN OUR ABLATION STUDY
- Fixed-prompt scores are relatively lower, while adding iteration to the fixed prompt produces noticeable improvements.The authors suggest fixed examples may be insufficiently related to the test case; the fixed-plus-iteration results shown are from iteration 17.Source: Appendix B · p. 14
- The Table VIII test case presents Findings and Impression sections, with 14 similarity-ranked examples supplied as dynamic-prompt context.The examples are ordered from closest to furthest similarity to the test case.Source: Appendix B · pp. 13–14, Table VIII · p. 14
- Dynamic + Iteration results and Rouge-1 scores improve progressively across iterations 3, 5, 10, 15, and 17 (Table VIII).This method applies iterative optimization to the dynamic prompt.Source: Appendix B · p. 14, Table VIII · p. 14
APPENDIX C ABLATION STUDY ON CORPUS SIZE
- Performance improves gradually as the similarity-search corpus grows from 10,000 to 20,000 samples and then to the entire training set.The authors attribute the trend to larger corpora yielding more semantically similar reference examples.Source: Appendix C · pp. 14–15, Table IX · p. 15