Source-linked AI summary
Improving O-RADS Risk Stratification from Ultrasound Reports: A Comparative Evaluation of Hybrid versus End-to-End LLM Reasoning Strategies
Xiaotong Tan, Chunli Qiu, Xin Liu, Qing Huang, Guangli Zhou, Bo Gao, Xiaoyan Song, Shuyan Wang, Xiuqin Wang, Wufeng Xue, Ruobing Huang, Dong Ni, Guowei Tao, Jun Cheng
TL;DR
Complex, guideline-driven medical classification remains challenging in unconstrained generative workflows and depends on accurate interpretation of complex rules. The study evaluates architectural decoupling, with hybrid classification achieving high accuracy, reliability, and interpretability while mitigating misclassification and overstaging.
Problem
Accurate interpretation of complex rules remains challenging for guideline-driven medical classification in unconstrained generative workflows.
Method
The study implemented architectural decoupling for complex guideline-driven classification, separating feature extraction from rule-based classification.
Results
99.2% accuracy was achieved by the hybrid Gemini 3.6 Flash approach, with weighted κ = 1.00 and performance exceeding original reports and end-to-end LLM strategies.
Takeaways & Limitations
Hybrid classification provides greater reliability and interpretability than end-to-end LLM classification for complex, guideline-driven medical tasks such as O-RADS.
Takeaways & Limitations
The study was conducted at a single site, and final performance remained dependent on the intrinsic capability of the underlying LLM.
Abstract
from arXiv · showhide
Background: Automating clinical guideline-based decision-making with large language models (LLMs) remains challenging because of reliability, hallucination, and limited interpretability. We compared the performance of LLMs and reasoning strategies for automated Ovarian-Adnexal Reporting and Data System (O-RADS) classification from free-text pelvic ultrasound reports. Methods: In this retrospective study, consecutive patients with ovarian masses who underwent pelvic ultrasound were included. Eight LLMs were tested with three reasoning strategies: implicit-knowledge end-to-end, rule-informed end-to-end, and a feature-based hybrid architecture that decoupled feature extraction from rule-based classification. The reference standard was O-RADS categorization established by expert consensus. Results: A total of 310 women with 390 ovarian masses were evaluated. The feature-based hybrid architecture using Gemini 3.6 Flash demonstrated the best performance, achieving an accuracy of 99.2% (387 of 390) and almost perfect agreement with the reference standard (weighted kappa = 1.00; 95% CI: 0.99-1.00). Its performance surpassed that of original clinical reports (accuracy, 87.7% [342 of 390]; weighted kappa = 0.94; 95% CI: 0.91-0.96) and end-to-end LLM strategies (accuracy range, 65.6% [256 of 390] to 95.9% [374 of 390]). For structured feature extraction, Gemini 3.6 Flash demonstrated higher overall accuracy than Claude Fable 5 (98.9% vs 97.8%; P < 0.001). The hybrid architecture reduced misclassification errors and mitigated the overstaging tendency observed in original reports. Conclusion: The feature-based hybrid LLM architecture that separates clinical feature extraction from deterministic guideline execution enables highly accurate, reliable, and interpretable automated O-RADS classification, providing a promising approach for standardized, guideline-based clinical decision-making.
3. Department of Ultrasound, Qilu Hospital of Shandong University, Jinan 250012, China
O-RADS ultrasound classification remains challenging because heterogeneous free-text reports require integrating multiple sonographic descriptors into complex rules. The study evaluates LLM strategies, including a hybrid approach that extracts structured features before deterministic rule execution.
- Background: Integrating lesion architecture, solid components, and vascularity into O-RADS remains challenging, particularly for less experienced practitioners.
- Background: Direct end-to-end LLM guideline classification is prone to instability and hallucinations, limiting reliability in clinical tasks.
- Methods: The feature-based hybrid framework converts unstructured reports into structured variables and maps them to final categories using deterministic rule-based algorithms.
- Methods: The hybrid method uses an LLM for structured extraction and a Python-based engine applying ACR O-RADS US decision logic.
- Methods: Extracted features are output alongside the final O-RADS category to support transparent interpretation.
- Study population: The independent test set contained 310 patients and 390 lesions after patient-level partitioning from the development data.
3. Results
Across evaluated models, hybrid reasoning produced the most consistent O-RADS classification performance, with Gemini 3.6 Flash achieving the highest accuracy and agreement. Hybrid reasoning also reduced misclassification, especially overstaging, although three Gemini errors remained.
- Implicit-knowledge end-to-end: 65.6% to 92.8% accuracy was achieved with implicit-knowledge end-to-end prediction, with substantial model variability for O-RADS 3.O-RADS 3 accuracy ranged from 34.4% to 86.9%.
- Rule-informed end-to-end: 82.6% to 95.9% accuracy was achieved with rule-informed end-to-end reasoning, and GPT-5.6 Sol reached 95.9%.All models except Claude Fable 5 surpassed the original-report baseline of 87.7%.
- Feature-based hybrid: 95.6% to 99.2% accuracy was achieved across hybrid models, exceeding the original-report baseline and outperforming both end-to-end strategies overall.
- Feature-based hybrid: 99.2% accuracy was achieved by Gemini 3.6 Flash, including 100.0% accuracy for O-RADS 4.The model correctly classified 387 of 390 lesions.
- Error analysis: Hybrid reasoning markedly reduced misclassifications for O-RADS 2 and O-RADS 3 lesions and reduced multi-category errors compared with end-to-end strategies and original reports.
- Agreement: Weighted κ was 1.00 for Gemini 3.6 Flash, compared with 0.94 for original clinical reports.The hybrid κ values for Gemini 3.6 Flash, GPT-5.6 Sol, and Grok 4.5 exceeded the original-report baseline.
- Feature extraction: Gemini 3.6 Flash extracted features with 98.9% overall accuracy versus 97.8% for Claude Fable 5.Gemini also achieved higher overall recall and precision than Claude Fable 5.
- Error analysis: Three Gemini 3.6 Flash hybrid errors comprised one feature hallucination and two feature-extraction omissions.The errors involved hallucinated physiologic cyst features, omitted irregular morphology, and missed ascites-related information.
4. Discussion
The study found that reasoning architecture strongly influenced automated ultrasound-based O-RADS classification, with the feature-based hybrid approach producing the most accurate and consistent results. Its performance exceeded original clinical reports and end-to-end strategies, while remaining dependent on model capability, report quality, and external validation.
- Performance: 99.2% accuracy was achieved by the Gemini 3.6 Flash feature-based hybrid method, corresponding to 387 of 390 lesions.Accuracy across feature-based hybrid methods ranged from 95.6% to 99.2%.
- Performance: Weighted κ = 1.00 (95% CI: 0.99–1.00) indicated almost perfect agreement with the reference standard, exceeding original reports and end-to-end LLM strategies.Original reports achieved 87.7% accuracy and weighted κ = 0.94.
- Reasoning architecture: End-to-end strategies remained unstable despite explicit guideline prompting, with accuracy ranging from 82.6% to 95.9%.The study attributes this pattern to limitations in operationalizing complex decision trees within unconstrained generative workflows.
- Reasoning architecture: Feature-based hybrid reasoning demonstrated greater consistency and significantly reduced performance variability compared with traditional end-to-end strategies.The hybrid architecture separates feature extraction from deterministic rule execution, mirroring the radiologists’ workflow.
- Model capability: Gemini 3.6 Flash outperformed Claude Fable 5 in overall feature extraction accuracy, 98.9% versus 97.8% (P < 0.001).The difference was especially pronounced for complex findings such as ascites or peritoneal nodules.
- Clinical implications: The hybrid method reduced misclassifications, mitigated the overstaging tendency of original reports, and provided transparent intermediate outputs for clinical oversight.The system is intended as decision support rather than a replacement for radiologists.
- Limitations: The study’s generalizability is constrained by its single-institution design, differences in reporting styles, and infrequent representation of some sonographic descriptors.External validation across diverse clinical settings is warranted, and feature-level estimates may have statistical uncertainty.
5. Conclusions
The study concludes that separating feature extraction from deterministic rule-based reasoning improves O-RADS classification accuracy, reliability, and interpretability. Prospective multicenter validation and multimodal integration remain priorities.
- Hybrid architectures that separate feature extraction from rule-based reasoning achieve higher O-RADS classification accuracy, reliability, and interpretability than end-to-end LLM approaches.
- The findings suggest that reasoning architecture aligned with clinical workflows may matter more than increasing model complexity alone.
- Prospective, multicenter validation across diverse populations is identified as a future research priority.
- Integrating ultrasound images with corresponding textual reports may further enhance clinical applicability and workflow efficiency.
Declarations
The study reports its terminology, ethics approval, data and code availability, funding, contributions, and conflict-of-interest disclosures.
- O-RADS denotes Ovarian-Adnexal Reporting and Data System, while LLM denotes large language model.
- The retrospective study used de-identified clinical data, with written informed consent waived by the ethics committee.
- The datasets are unavailable publicly because of patient privacy and institutional restrictions but may be obtained from the corresponding author on reasonable request.
- The computer code and analysis pipeline are publicly available on GitHub.
- The authors report no competing interests and acknowledge multiple institutional and national funding sources.
- The manuscript records contributions spanning writing, analysis, methodology, data curation, validation, supervision, funding, and review.
Tables
The tables present baseline population characteristics and accuracy comparisons across LLM reasoning strategies, with defined reporting conventions and strategy labels.
- Table 1 presents baseline characteristics of the study population.
- Continuous variables are reported as medians with IQRs, whereas categorical variables are reported as numbers with percentages.
- Papillary projections are reported categorically as presence or absence because zero values predominate.
- Table 2 reports lesion-level accuracy percentages for three LLM-based strategies in automated O-RADS classification.
- The implicit-knowledge paradigm relies on internal model parameters, rule-informed prompting incorporates ACR O-RADS guidelines, and the feature-based hybrid uses extract-then-reason processing.
- Bold formatting identifies the highest overall accuracy within each reasoning strategy.
Figures
The figures describe the experimental workflow, cohort construction, strategy-level accuracy, confusion matrices, and misclassification patterns. Across the visual analyses, the hybrid method is presented as reducing classification errors.
- Experimental workflow: Figure 1 compares implicit-knowledge end-to-end, rule-informed end-to-end, and feature-based hybrid strategies within the experimental workflow.
- Patient selection and dataset construction: Figure 2 depicts patient selection and dataset construction, ending with 319 patients and 404 lesions divided into development and test sets.
- Overall accuracy: Figure 3 presents overall accuracy for eight LLMs across the three strategies, with original clinical-report accuracy shown as the baseline.
- Confusion matrices: Figure 4 plots predicted O-RADS categories against the reference standard, with diagonal values indicating correct classifications and off-diagonal values indicating errors.
- Confusion matrices: The feature-based hybrid method showed the fewest off-diagonal misclassifications in the confusion-matrix comparison.
- Misclassification errors: Figure 5 compares error direction and severity across rule-informed end-to-end, feature-based hybrid, and original-report approaches.
Supplemental Material
Supplementary analyses compare O-RADS agreement, feature extraction, misclassification patterns, and category distributions across models and reasoning strategies.
- Pairwise agreement: 0.94 (0.91–0.96) was the weighted kappa for original reports versus the reference standard.Quadratic weighted kappa penalizes larger disagreements between O-RADS categories.
- Model comparisons: 1.00 (0.99–1.00) was the feature-based hybrid method’s weighted kappa in one model comparison, versus 0.95 (0.91–0.97) and 0.96 (0.95–0.98) for the end-to-end strategies.These values compare implicit-knowledge end-to-end, rule-informed end-to-end, and feature-based hybrid methods against the reference standard.
- Pairwise agreement: 0.99 (0.98–1.00) was the weighted kappa for the feature-based hybrid method versus the reference standard.The corresponding values were 0.98 (0.96–0.99) for rule-informed end-to-end and 0.95 (0.94–0.97) for implicit-knowledge end-to-end.
- Error analysis: Three hybrid-model misclassifications comprised one feature hallucination and two feature omissions.Feature hallucination means extracting unsupported features; omission means failing to identify key features present in the text.
- Category distributions: The hybrid-model distribution was shown alongside the reference standard, original reports, and rule-informed end-to-end predictions.O-RADS 0 indicates failure to detect or omission of target ovarian lesions described in reports.
Appendix S1. Prompt template for the hybrid method
The hybrid prompt extracts structured ovarian-adnexal imaging features from Chinese ultrasound text under explicit rules, then returns strict per-lesion JSON output.
- Prompt (English translation): The prompt instructs the model to extract structured imaging features of ovarian and adnexal lesions solely from explicit ultrasound descriptions.It prohibits inference, association, or supplementation with medical common sense.
- Prompt (English translation): Lesion location and maximum diameter are output for each lesion, with diameter represented as a numerical value.The template specifies ovarian or adnexal location labels and excludes explanatory text from the output.
- Prompt (English translation): Cyst architecture is encoded as binary unilocular, bilocular, or multilocular features according to explicit septation descriptions.The multilocular rule includes multiple, dense, or local septations.
- Prompt (English translation): Color Doppler flow is scored from 1 to 4 when described and assigned 0 when blood flow is not mentioned.The prompt maps absent, sparse, moderate, and abundant flow descriptions to increasing scores.
- Prompt (English translation): The prompt separately identifies physiologic, hemorrhagic, dermoid, endometrioma, paraovarian, peritoneal inclusion, and hydrosalpinx lesions.Each entity is marked when specified descriptors or the ultrasound impression support it.
- Prompt (English translation): Solid lesion status is defined by more than 80% solid area, while solid component status covers less than 80% and is evaluated only when solid lesion is 0.Certain hyperechoic or proteinaceous contents are explicitly excluded from solid-component classification.
- Prompt (English translation): Irregularity, acoustic shadowing, papillary projections, ascites, and peritoneal nodules are encoded only when explicitly described.Papillary projections require an exact count, and global ascites or peritoneal findings apply to all lesion objects.
- Prompt (English translation): Multiple lesions must be returned as a strict JSON array with one lesion object per element and no explanatory text.The prompt includes few-shot examples and a JSON template for lesion-level extraction.
Appendix S2. Representative ultrasound report examples
Representative examples clarify that inference used de-identified original Chinese ultrasound reports, while English translations were supplied for transparency.
- Data handling: All LLM feature extraction and reasoning experiments used the original Chinese ultrasound reports.The English translations were provided for reviewers’ and international readers’ reference.
- Data handling: Reports were de-identified before inference by removing patient personal information.This preprocessing step was applied before model evaluation.
- Data handling: Original O-RADS categories and baseline scores were redacted to prevent data leakage.The redaction was part of the report preprocessing before inference.
S2.1 Example 1 (O-RADS 2 Case)
The examples illustrate lesion-level feature extraction across mixed, cystic, solid, and confluent pelvic masses, including location, morphology, flow, and ascites descriptors.
- S2.1 Example 1 (O-RADS 2 Case): Example 1’s impression identifies the lesion as a mixed-echogenic left ovarian mass suspected to be a teratoma or dermoid cyst.The lesion’s structured descriptors include a solid component and absent obvious internal flow.
- S2.1 Example 1 (O-RADS 2 Case): Example 2 contains a 2.2 × 1.3 cm left ovarian cystic lesion with sparse internal blood flow.The same report describes a 2.6 × 2.0 cm predominantly solid right ovarian lesion with one papillary projection and no obvious internal flow.
- S2.1 Example 1 (O-RADS 2 Case): A neighboring 3.3 × 2.4 cm right ovarian cystic-solid mass has dense septations and a small amount of arterial blood flow.The impression lists cystic and cystic-solid right ovarian masses and a suspected corpus luteum cyst in the left ovary.
- S2.1 Example 1 (O-RADS 2 Case): Another example describes a predominantly solid hypoechoic mass posterior and right of the uterus, extending left of the cervix.The confluent lesions measure 9.5 × 5.4 × 4.4 cm overall, with irregular shape, ill-defined margins, and relatively abundant flow.
- S2.1 Example 1 (O-RADS 2 Case): The bilateral ovaries were not clearly visualized in the confluent posterior uterine mass example.The ultrasound impression localizes the mass posterior and right of the uterus and left of the cervix.