Dual- versus Single-Suggestion AI Support for Radiographic Interpretation in Residents: Randomized Multireader Study

• Prior work examined single-AI presentation formats, while less-experienced physicians may benefit more from AI yet remain more susceptible to erroneous suggestions.Source: §1 · p. 2
• Whether dual-suggestion support mitigates erroneous AI suggestions during residents’ radiographic interpretation remains unclear.Source: §1 · p. 2
• The study compared dual- and single-suggestion support, emphasizing accuracy change and performance when the shared AI suggestion was incorrect, while also evaluating specialty and secondary outcomes.Secondary outcomes included interpretation time and confidence change.Source: §1 · p. 2

2.1 Study Design and Participants

• Residents with fewer than 3 years of clinical experience were stratified by specialty and randomized 1:1:1; incomplete assessments due to network interruption or physical discomfort were excluded.Figure 1 shows final analytic samples by specialty; testing used a smartphone or tablet.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 Case Selection and AI Output Generation

• From 218 chest and abdominal examinations, 60 balanced cases were selected; each AI model correctly diagnosed 39 of 60 despite differing case-level concordance with GPT-5.4.Five radiologists’ mean unaided accuracy on the cases was 54.3%.Source: §2.2 · p. 3

2.3 Study Procedure

• Residents recorded an unaided diagnosis and confidence before receiving GPT-5.4 alone or simultaneous GPT-5.4 plus Kimi-K2.6 or Gemini-3.6 Flash, then submitted final responses.Model identities were masked, and all groups interpreted the same cases in independently randomized order.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 Outcomes and Response Adjudication

• The primary outcome was AI-assisted minus unaided accuracy; other outcomes included accuracy by GPT-5.4 correctness, interpretation time, confidence change, and beneficial or harmful revisions.Responses were scored against a prespecified diagnostic lexicon, with automated matching reviewed by two masked radiologists.Source: §2.4 · p. 4

2.5 Statistical Analysis

• Welch ANOVA and Holm-adjusted Welch t tests compared support conditions, while HC3 robust linear models and Wald tests assessed specialty interactions.Sample size was based on feasible recruitment, without a formal a priori calculation.Source: §2.5 · p. 4

3.1 Participant Flow and Characteristics

• Of 132 residents allocated, 123 were analyzed; the analyzed group had a mean age of 24.1 years, and 65 were women.Participants included 60 radiology and 63 non-radiology residents recruited from three centers.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Diagnostic Accuracy before and after AI Assistance

• Across all 123 residents, mean accuracy increased from 46.3% to 61.5% after AI assistance, a mean change of 15.2 percentage points.Accuracy increased in each support condition: 13.9 percentage points in A, 15.0 in B, and 16.8 in C.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Accuracy Change, Interpretation Time, and Confidence Change

• Radiology residents improved more with dual than single suggestions: B–A was 6.69 percentage points and C–A was 7.87 percentage points; non-radiology accuracy change did not differ across conditions.Interpretation time and confidence change did not differ across support conditions within either specialty stratum.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4 Accuracy According to GPT-5.4 Suggestion Correctness

• When GPT-5.4 was incorrect, dual-suggestion groups had higher AI-assisted accuracy than group A in both specialties, with a significant specialty interaction in the dual-suggestion effect.The interaction difference was 10.44 percentage points (95% CI, 4.36–16.52; P < .001).Source: §3.4 · p. 6, Figure 5 · p. 7, §3.5 · p. 7

3.5 Revision Components and Specialty Interaction

• Dual-suggestion support increased beneficial revisions for radiology residents, while harmful revisions did not differ by support condition; neither revision type differed in non-radiology residents.The specialty interaction was significant for beneficial revisions, but not for harmful revisions.Source: §3.5 · p. 7, Figure 6 · p. 8

4 Discussion

• Dual-suggestion support was associated with higher accuracy when GPT-5.4 was incorrect in both specialties, while greater overall accuracy improvement was observed only among radiology residents.Interpretation time did not differ across support conditions in either specialty.Source: §4 · pp. 8–9
• The authors propose that an additional independent AI opinion may help limit the influence of erroneous suggestions; when both AI suggestions were incorrect, dual- and single-suggestion support did not differ significantly.The both-incorrect finding was exploratory and covered the model-specific subsets described in Supplementary Table S11.Source: §4 · p. 9
• The authors relate the findings to prior work on overreliance on erroneous AI advice and structured information sharing, while extending human–AI interaction research beyond single-AI presentation.The discussion describes the additional opinion as independent and notes no detectable increase in interpretation time.Source: §4 · p. 9
• Dual-suggestion effects differed by specialty; among non-radiology residents, accuracy was higher when GPT-5.4 was incorrect but lower when it was correct.The study also observed specialty interactions for accuracy change and beneficial revisions.Source: §4 · p. 9
• Process-level mechanisms were not directly assessed, and the frequency and direction of model disagreement may vary across pairings.The authors identify eye tracking and suggestion-weighting measures as possible approaches for future mechanism studies.Source: §4 · p. 9
• The 60-case set was purposively selected to balance disease composition and overall AI accuracy, so it may not reflect routine clinical case mix.The authors call for further evaluation in unselected clinical populations.Source: §4 · p. 9

Code Availability

• Inference code is available on GitHub; GPT-5.4 and Kimi-K2.6 outputs came through Microsoft Foundry, and Gemini-3.6 Flash outputs through Google Vertex AI.Microsoft Foundry is identified as formerly Azure AI Foundry.Source: Code Availability · p. 9

Supplementary Materials

• Supplementary tables provide participant-level descriptive results, prespecified support-condition comparisons, and additional analyses for the 123 analyzed participants.Participants were divided into radiology and non-radiology strata; conditional analyses used 39 GPT-correct and 21 GPT-incorrect cases, with accuracy change and transition outcomes defined in the supplement.Source: Supplementary Materials · p. 12

Supplementary Methods: Allocation Procedure

• Within specialty strata, participants were allocated 1:1:1 using a DeepSeek-generated list that was fixed before assessment and concealed from enrollment personnel until assignment.DeepSeek was used only for allocation, not as a diagnostic model.Source: Supplementary Methods: Allocation… · p. 13

Supplementary Statistical Methods

• The statistical methods specify Welch ANOVA and Holm-adjusted planned comparisons, with HC3-robust linear models for specialty interactions and exploratory analyses of cases where both suggestions were incorrect.The B–C comparisons were exploratory and unadjusted; tests were two-sided with P < .05 indicating significance.Source: Supplementary Statistical Methods · p. 13

Supplementary Appendix 1. Case Composition, Model Concordance, and Representative Fixed AI Suggestions

• The purposively selected set contained 30 chest and 30 abdominal examinations, and each of the three models correctly diagnosed 39 of 60 cases (65.0%).Models were matched on overall accuracy, not on which individual cases they diagnosed correctly.Source: Supplementary Appendix 1. Case… · p. 14, Figure S1 · p. 14
• In group B, 29 cases had both models correct, 10 had GPT correct and Kimi incorrect, 10 had GPT incorrect and Kimi correct, and 11 had both incorrect.Group C had 28 both-correct, 11 GPT-correct/second-model-incorrect, 11 GPT-incorrect/second-model-correct, and 10 both-incorrect cases.Source: Supplementary Appendix 1. Case… · p. 14
• The appendix’s first representative example shows both models correctly identifying bronchopneumonia in a chest radiograph.The example is labeled as a both-correct case.Source: Supplementary Appendix 1. Case… · p. 16
• In a GPT-correct/Kimi-incorrect example, GPT identified left lower-lobe pneumonia while Kimi suggested acute bronchitis.The case is labeled as Case 13.Source: Supplementary Appendix 1. Case… · p. 17
• In a GPT-incorrect/Kimi-correct example, GPT suggested bronchitis or small-airway inflammation while Kimi identified left pneumothorax.The case is labeled as Case 7.Source: Supplementary Appendix 1. Case… · p. 18
• The final representative case is labeled both-incorrect and shows GPT and Kimi both suggesting bronchitis.The reference diagnosis is pneumothorax.Source: Supplementary Appendix 1. Case… · p. 19

Scoring rules

• Responses counted as correct when the primary diagnosis matched the reference diagnosis or an accepted expression; qualifying language did not invalidate a match.The scoring rules were used for diagnoses before and after AI presentation.Source: Supplementary Appendix 2. Prespecified… · p. 20
• Negated diagnoses were rejected, while ambiguous, contradictory, or unclear responses required expert adjudication.Source: Supplementary Appendix 2. Prespecified… · p. 20
• Scoring focused on the reference diagnostic category, so omitted laterality, site, or severity did not invalidate a category-level match.Explicitly incorrect modifiers required expert adjudication.Source: Supplementary Appendix 2. Prespecified… · p. 20

Scoring and review

• Automated matching used a locked lexicon, and two masked senior radiologists reviewed classifications and resolved disagreements by consensus.The lexicon lists AI as artificial intelligence and GPT as generative pretrained transformer.Source: Supplementary Appendix 2. Prespecified… · p. 20

Accepted expressions and exclusions

• Pneumothorax expressions include unilateral or bilateral pneumothorax, pleural air, and tension pneumothorax when pneumothorax is present.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Pneumonia expressions include bronchopneumonia, lobar pneumonia, pulmonary infection, and infectious consolidation; nonspecific lung findings alone are listed as unacceptable examples.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Accepted lung-cancer terms include primary or bronchogenic cancer and specified malignant pulmonary neoplasms; a nodule, mass without stated malignancy, or metastasis is not enough.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Bronchitis expressions include acute or chronic bronchitis, tracheobronchitis, and bronchial infection or inflammation; increased lung markings alone are listed as insufficient.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Urinary-tract stone expressions include renal and ureteral calculi, while pelvic phleboliths and calcifications without a specified urinary source are examples not accepted alone.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Accepted bowel-obstruction expressions include mechanical, small- or large-bowel, complete, partial, high, low, and strangulated obstruction; bowel distention or air-fluid levels alone are insufficient.Source: Appendix Table A1. Accepted Diagnostic… · p. 21
• Perforation expressions include gastrointestinal perforation, pneumoperitoneum, and free intraperitoneal or subdiaphragmatic air; abdominal distention or peritonitis alone is not equivalent.Source: Appendix Table A1. Accepted Diagnostic… · p. 21

Supplementary outcomes

• The supplementary tables also cover participant characteristics, participant-level outcomes, diagnostic accuracy, accuracy change, interpretation time, confidence change, and diagnostic revisions.Source: Table S1 · p. 22, Table S2 · p. 22, Table S3 · p. 23

Pairwise accuracy contrasts

• Table S7 presents pairwise accuracy contrasts separately for radiology and non-radiology residents and for correct versus incorrect shared GPT suggestions.Source: Supplementary Tables · p. 26, Table S7 · pp. 26–27
• 20.14 percentage points (B–A) and 20.35 percentage points (C–A) were the AI-assisted accuracy differences among radiology residents in cases with an incorrect GPT suggestion; both Holm-adjusted P values were < .001.Source: Supplementary Tables · p. 27

Dual-suggestion effects and interactions

• The dual-suggestion effect is defined as the equally weighted contrast [(B+C)/2] − A, and specialty interaction is the radiology contrast minus the non-radiology contrast.Source: Supplementary Tables · pp. 29–30

Revision analyses

• Correctness strata use the shared GPT suggestion, and the second model's correctness could differ within those strata.Source: Supplementary Tables · p. 31