Source-linked AI summary
Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T. Sio, Lisa A. McGee, Jonathan B. Ashman, Xiang Li, Tianming Liu, Jiajian Shen, Wei Liu
TL;DR
The paper addresses whether LLMs can be meaningfully evaluated on a highly specialized topic less represented by public test-preparation resources. It creates a 100-question radiation oncology physics exam and compares four LLMs with human groups, finding that GPT-4 performs strongly while remaining below a small team of medical physicists using majority voting.
Problem
Widely used exams may provide an inaccurate measure of LLM capability because their questions and preparation materials are extensively represented in available data.
Method
The study evaluates four LLMs on a newly created 100-question radiation oncology physics exam, compares them with medical physicists and non-experts, and tests GPT-4 prompting and deductive reasoning.
Results
GPT-4 outperformed individual medical physicists on average, improved when prompted to explain first, then answer, and scored 55% on the modified deductive-reasoning test.
Takeaways & Limitations
The findings suggest potential for LLMs to work alongside radiation oncology experts as highly knowledgeable assistants.
Takeaways & Limitations
The exam is not representative of detailed and nuanced daily clinical work, so its relative performance comparisons may misrepresent equivalency with individual medical physicists.
Abstract
from arXiv · showhide
We present the first study to investigate Large Language Models (LLMs) in answering radiation oncology physics questions. Because popular exams like AP Physics, LSAT, and GRE have large test-taker populations and ample test preparation resources in circulation, they may not allow for accurately assessing the true potential of LLMs. This paper proposes evaluating LLMs on a highly-specialized topic, radiation oncology physics, which may be more pertinent to scientific and medical communities in addition to being a valuable benchmark of LLMs. We developed an exam consisting of 100 radiation oncology physics questions based on our expertise at Mayo Clinic. Four LLMs, ChatGPT (GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ, were evaluated against medical physicists and non-experts. ChatGPT (GPT-4) outperformed all other LLMs as well as medical physicists, on average. The performance of ChatGPT (GPT-4) was further improved when prompted to explain first, then answer. ChatGPT (GPT-3.5 and GPT-4) showed a high level of consistency in its answer choices across a number of trials, whether correct or incorrect, a characteristic that was not observed in the human test groups. In evaluating ChatGPTs (GPT-4) deductive reasoning ability using a novel approach (substituting the correct answer with "None of the above choices is the correct answer."), ChatGPT (GPT-4) demonstrated surprising accuracy, suggesting the potential presence of an emergent ability. Finally, although ChatGPT (GPT-4) performed well overall, its intrinsic properties did not allow for further improvement when scoring based on a majority vote across trials. In contrast, a team of medical physicists were able to greatly outperform ChatGPT (GPT-4) using a majority vote. This study suggests a great potential for LLMs to work alongside radiation oncology experts as highly knowledgeable assistants.
1 Introduction
The paper argues that conventional, widely circulated exams may not accurately assess LLMs because of abundant training and preparation resources, motivating evaluation on radiation oncology physics. It introduces a new specialized exam to compare several LLMs with experts and non-experts.
- Motivation: Popular exams may overestimate LLM capability because extensive test-preparation resources and internet text could overlap with their training data.The paper cites large AP Physics and LSAT test-taking populations as indicators of abundant related resources.
- Motivation: Radiation oncology physics is proposed as a more obscure evaluation topic, with only 162 medical school graduates applying for radiation oncology residencies in 2022.Its specialized nature may make it more pertinent to scientific and medical communities.
- Study design: The study created a new 100-question multiple-choice examination to reduce concerns that evaluation questions were present in training data.The exam was developed from the authors’ expertise at Mayo Clinic.
- Study design: Four LLMs—ChatGPT (GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ—were compared with radiation oncology experts and non-experts.The study also examined ways to improve GPT-4 answers and assessed its deductive reasoning.
2 Related Work
Related work describes transformer-based language models, their scaling and alignment strategies, and their use in benchmark examinations. The paper extends this literature by evaluating LLMs in radiation oncology and medical physics, a highly specialized domain.
- 2.1 Large language models: Transformer-based pre-trained language models include autoregressive, masked-language, and encoder-decoder models, with newer LLMs developed at much larger scales.Examples include GPT, BERT, BART, T5, GPT-3, Bloom, PaLM, and OPT.
- 2.1 Large language models: Very large language models may reduce or eliminate the need for task-specific fine-tuning, unlike smaller models that often require continual pre-training and supervised fine-tuning.Their learned contextual and domain-specific representations support this potential.
- 2.1 Large language models: RLHF aligns LLM outputs with human preferences by training a reward model from annotators’ rankings of generated responses.ChatGPT is described as employing RLHF to follow prompts and generate comprehensive responses.
- 2.2 Language models and examination: This study is presented as the first evaluation of LLMs in radiation oncology and medical physics.The authors propose that it can inspire future evaluations in highly specialized medical branches.
- 2.2 Language models and examination: Prompts can adapt large pre-trained models to downstream tasks without fine-tuning, although determining an ideal prompt remains challenging.The study designs prompts and chain-of-thought prompts based on radiation oncology and medical physics expertise.
3 Methods
The methods use a newly created 100-question radiation oncology physics exam, repeated LLM trials, human comparison groups, consistency analyses, majority voting, and GPT-4 prompting and deductive-reasoning tests.
- 3.1 Comparison between LLM scores and human scores: A medical physicist created a 100-question multiple-choice radiation oncology physics exam aligned with ASTRO’s recommended resident-education curriculum.The exam covers topics including basic physics, radiation measurements, treatment planning, and imaging applications.
- 3.1 Comparison between LLM scores and human scores: Each LLM received the exam in five separate trials, except BLOOMZ, which was tested once, using batches of 20 questions and answer-only instructions.Trials began in new threads or after resets.
- 3.1 Comparison between LLM scores and human scores: LLM scores were compared with medical physicists and non-experts, including nine medical-physics professionals and six technically trained non-experts.Human participants completed the examination within a three-hour limit.
- 3.1 Comparison between LLM scores and human scores: Trial consistency was quantified using score standard deviation and average Pearson correlation between trials.An average correlation of 1 indicates identical score distributions, whereas 0 indicates purely random correspondence.
- 3.1 Comparison between LLM scores and human scores: Answer confidence was estimated by counting how often each question was answered correctly across trials and comparing results with random-guessing expectations.For five trials, random guessing was expected to yield approximately 1.25 correct answers on average.
- 3.1 Comparison between LLM scores and human scores: ChatGPT (GPT-3.5), ChatGPT (GPT-4), and Bard were also compared with human scores calculated by majority vote.This analysis evaluates whether aggregating repeated answers changes comparative performance.
- 3.2 Prompting strategy: GPT-4 was tested with prompts requiring it to explain its reasoning before selecting an answer, assessing whether step-wise generation improved accuracy.The initialization prompt matched Trial 1 while the answering instructions were modified.
- 3.3 Testing ChatGPT (GPT-4) on its deductive reasoning ability: For deductive reasoning, each question’s correct option was replaced with “None of the above choices is the correct answer,” then GPT-4 was tested with standard and explain-first prompts.The transformation requires rejecting the other answer choices rather than relying on direct question-answer context.
4 Results
GPT-4 outperformed the medical physicist group overall, while medical physicists led on math-based questions. LLMs were more consistent across trials, explain-first prompting improved GPT-4 scores, and majority voting benefited medical physicists far more than GPT models.
- GPT-4 was the only LLM to outperform the medical physicist group overall, although medical physicists performed better on math-based questions.
- LLMs showed lower scoring variability and higher answer correlation across trials than the human groups.This indicates greater consistency in both scores and question-level answers.
- GPT-4 answered 67% of questions correctly in each trial but also gave 14% incorrect answers in each trial.GPT-3.5 likewise showed repeated confidence or confusion across trials.
- Majority voting improved GPT-3.5 and GPT-4 scores by 1%, compared with 4% for Bard, 3% for non-experts, and 23% for medical physicists.The group answer was the most common answer choice for each question, with ties resolved randomly.
- Prompting GPT-4 to explain first, then answer, improved its overall score by 5%, with the largest gains in brachytherapy and math-based questions.The prompt allowed the answer to develop before the final response.
- Replacing the correct answer with “None of the above choices is the correct answer” reduced GPT-4 performance overall, but explain-first prompting raised math-question accuracy from 0% to 65%.The 65% result was only one question less accurate than the original explain-first trial.
5 Discussion
The discussion finds that GPT-4 performed strongly on this specialized exam, improved with explanation-first prompting, and showed deductive-reasoning accuracy, while expert groups retained an advantage when aggregating answers. The exam’s multiple-choice format limits how directly these comparisons represent detailed clinical work.
- GPT-4 outperformed the average medical physicist in nearly all subcategories and improved its answer accuracy when prompted to explain before answering.
- 55% overall accuracy was achieved on the deductive-reasoning test when correct answers were replaced with “None of the above choices is the correct answer.”The authors describe this result as potentially emergent and propose the method for future evaluation of reasoning across model sizes.
- A small group of medical physicists outperformed individual LLMs through majority voting, aided by lower confusion and complementary individual capabilities.The authors attribute the comparison partly to humans varying in knowledge and reasoning, while LLMs were confidently correct or incorrect across trials.
- The study suggests radiation oncology experts could use GPT-4 as a highly knowledgeable assistant, while future applications may integrate text, images, and dosimetric data.
- The exam may misrepresent equivalence between LLMs and individual medical physicists because multiple-choice questions do not represent detailed, nuanced daily clinical work.
6 Conclusion
LLMs, particularly ChatGPT (GPT-4), performed strongly on specialized radiation oncology physics questions and showed deductive reasoning capabilities. Human collaboration remained important, as a small group of medical physicists outperformed GPT-4.
- ChatGPT (GPT-4) excelled on radiation oncology physics questions and exhibited surprising deductive reasoning capabilities.
- A small group of medical physicists working together achieved better results than GPT-4, highlighting the value of diverse human knowledge and expertise.
- Combining LLM capabilities with human insight and experience may enhance knowledge and decision-making.
- The findings suggest a promising future for LLMs in specialized fields.
A Explain first, then answer responses from ChatGPT (GPT-4)
The section demonstrates ChatGPT (GPT-4)'s explain-first-then-answer responses across selected radiation oncology physics questions. The examples cover activity conversion, half-life, dosimetry, TG-43, HDR brachytherapy, and source calibration.
- Ten questions from the 100-question exam were selected to demonstrate ChatGPT (GPT-4)'s explain-first-then-answer prompting.
- 10 Ci of 192Ir corresponds to 370 GBq, with the listed answer identifying a decay rate of 37 GBq.
- The effective half-life equation combines physical and biological half-lives, and solving it yields the biological half-life.
- For 125I with an approximately 60-day half-life, the calculated initial dose rate is approximately 5.9 cGy/h.
- The examples identify 125I as having the greatest listed anisotropy correction and state that TG-43 considers source absorption and scattering.
- The examples describe radial-dose-function constancy for 192Ir below 5 cm, inverse-square dose estimation, and longer HDR dwell times at cylinder ends.
- The examples state that 103Pd has a lower permanent prostate implant prescription dose because its half-life is shorter and that source calibration is excluded from HDR daily QA.
B 100-question multiple-choice exam on radiation oncology physics
The exam presents multiple-choice questions spanning foundational physics, radiation interactions, dosimetry, imaging, linear accelerators, and treatment hardware. It also supplies a sequence of correct answer choices for the 100-question examination.
- The appendix provides a 100-question answer key as an ordered sequence of letter choices.
- Questions cover particle acceleration, nuclear mass, x-ray spectra, photon interactions, attenuation, KERMA, buildup effects, and photoelectric interactions.
- The exam includes questions on half-value layers, LINAC RF acceleration, multileaf-collimator tongue-and-groove design, electron-mode hardware, and flattening-filter-free beams.
- Additional questions address mass attenuation, energy-transfer and energy-absorption coefficients, and KERMA relative to absorbed dose for a 15 MV beam.
16. Which device below is the most suitable for measuring lateral beam profiles? (a) Diodes
This section presents answer options for measuring lateral beam profiles, measuring film transmission after stacking, and calculating biological half-life.
- The listed devices include plane parallel ionization chambers and Geiger-Muller counters as options for measuring lateral beam profiles.
- The section asks for the fraction of light transmitted through four stacked films, each with optical density 1.25.
18. Which of the following devices is not suitable for in-vivo dose measurements? (a) Diodes
The excerpt identifies thermoluminescent and optically stimulated luminescence dosimeters and introduces TG-51 charge-correction questions involving chamber measurements.
- Thermoluminescent dosimeters (TLDs) and optically stimulated luminescence dosimeters (OSLDs) appear as candidate dosimetry devices.
- The TG-51 question asks which correction factor is unnecessary when converting measured chamber charge to generated charge.
- Temperature-pressure and recombination corrections are listed among the possible factors for chamber-charge conversion.
- A separate question asks for the PTP factor for a farmer chamber under local conditions of 23°C and 720 mmHg.
21. How is the beam quality defined in TG-51? (a) Nominal beam energy
The supplied passages combine TG-51 calibration concepts with questions spanning dose calculation, treatment geometry, planning, imaging, and image-guided radiation therapy.
- TG-51 calibration: TG-51-based calibration places LINAC dose calibration in a water tank.
- Dose calculations: PDD and TMR are contrasted through their treatment of beam attenuation and divergence at depths beyond dmax.The supplied options state that PDD includes attenuation and divergence, whereas TMR is described in contrasting combinations across the listed choices.
- Dose calculations: The excerpt includes output, monitor-unit, wedge, field-matching, and craniospinal collimator-rotation calculations.
- Treatment planning: Treatment-planning topics include DVHs, fractionation, electron-beam dose behavior, SSD setup, effective SSD, electron range, and IMRT inverse planning.
- IMRT: IMRT planning and quality assurance are represented through fluence optimization, leaf sequencing, dose gradients, QA devices, and dosimetrist-set constraints.
- Imaging and IGRT: Imaging and guidance questions cover CT, MR, ultrasound, PET, DICOM, 4D CT, IGRT methods, respiratory motion, and prostate tracking.