Source-linked AI summary
NeoRed: A Knowledge-Logic-Alignment Multimodal Large Language Model for Neonatal Respiratory Disease Diagnosis
Yinan Liu, Hongtai Xia, Haoran Xu, Jiankang Hong, Jingkuan Song, Ye Luo
TL;DR
Neonatal respiratory diagnosis remains challenging because existing multimodal models face neonatal–adult domain gaps and do not adequately integrate clinical context. NeoRed addresses this with neonatal datasets and a Knowledge-Logic-Alignment framework, outperforming mainstream MLLMs on neonatal benchmarks while generalizing to external validation.
Problem
Existing MLLMs face neonatal–adult domain gaps and insufficient integration of clinical context for multimodal neonatal respiratory diagnosis.
Method
NeoRed combines two real-world neonatal multimodal datasets with Knowledge-Logic-Alignment modules that inject expert priors, enforce diagnostic consistency, and align visual evidence with conclusions.
Results
NeoRed outperforms 8 mainstream MLLMs on NeoCXR and NeoCXR-EV, remaining best-performing under external disease-distribution shift.
Takeaways & Limitations
The results support neonatal-specific multimodal modeling with expert-informed alignment for neonatal respiratory diagnostic report generation.
Takeaways & Limitations
The diagnostic priors rely on disease–modality relevance determined by consensus between two neonatologists.
Abstract
from arXiv · showhide
Neonatal respiratory diseases are a major cause of neonatal morbidity and mortality, posing substantial challenges in clinical practice. Despite recent advances, existing Multimodal Large Language Models (MLLMs) face two key limitations in neonatal diagnosis: (1) domain gap arising from predominantly adult training data; (2) insufficient integration of multidimensional clinical context for accurate diagnosis. To address these challenges, we collect two real-world clinical datasets (NeoCXR and NeoCXR-EV) and propose NeoRed, to the best of our knowledge, the first MLLM tailored for neonatal respiratory disease, filling the gap in neonatal diagnostic reports generation. To enhance joint diagnosis from heterogeneous clinical context and chest X-rays, we design a novel Knowledge-Logic-Alignment (KLA) framework which constrains model behavior from three perspectives: 1) Knowledge Prior Injection (KPI) incorporates neonatologist-inspired diagnostic priors into multimodal representations, guiding disease-specific attention across modalities; 2) Diagnostic Logic Constraint (DLC) aligns the semantics of generated reports with multimodal diagnostic logic; and 3) Visual Semantic Alignment (VSA) establishes semantic correspondence between visual features and imaging conclusions. Extensive experiments demonstrate that NeoRed enables accurate neonatal diagnostic reports generation, achieving ROUGE-L of 53.29% and Clinical Efficacy F1 score of 65.19% on NeoCXR, outperforming existing MLLMs. NeoRed also preserves competitive report generation performance on adult benchmarks (MIMIC-CXR and IU-Xray). Datasets will be available upon application.
Introduction
Neonatal respiratory diseases require early, accurate diagnosis, but existing MLLMs face neonatal–adult domain gaps and insufficient integration of clinical context with chest X-rays. NeoRed addresses these limitations as a neonatal-focused diagnostic MLLM supported by new multimodal datasets and a Knowledge-Logic-Alignment framework.
- Neonatal respiratory diseases are major causes of newborn morbidity and mortality, making early and accurate diagnosis essential for timely intervention and improved outcomes.
- Existing MLLMs remain challenging to apply clinically because neonatal and adult populations differ, while models insufficiently prioritize clinical indicators when jointly interpreting CXRs and context.
- NeoRed is proposed as the first MLLM tailored to neonatal respiratory disease diagnosis, jointly using neonatal CXRs and clinical context to generate diagnostic reports.
- NeoRed incorporates a novel Knowledge-Logic-Alignment framework to enhance multimodal joint diagnosis from chest X-rays and clinical context.
- The work introduces NeoCXR and NeoCXR-EV, real-world multimodal neonatal report-generation datasets intended to address the domain gap and support future research through application-based access.
Related Work
General MLLMs have progressed from large-scale image–text pretraining to instruction-tuned visual reasoning, while medical MLLMs increasingly support radiology and visual-question-answering tasks. However, limited domain-specific knowledge and predominantly adult training data leave neonatal diagnostic modeling underdeveloped.
- General MLLMs: General MLLMs evolved from image–text-pretrained multimodal representations to instruction-tuned models with improved visual alignment and reasoning.Flamingo and BLIP established large-scale image–text pretraining, while LLaVA, Qwen-VL, InternVL, LLaVA-NeXT, and LLaVA-OneVision advanced language-space projection, instruction tuning, visual encoding, and multi-stage training.
- Research gap: Despite these advances, existing medical MLLMs remain unreliable for neonatal diagnosis because they lack domain-specific knowledge and are trained primarily on adult data.This limits their modeling of neonatal-specific disease patterns and diagnostic processes.
- Medical MLLMs: Medical MLLMs now address radiology report generation and medical visual question answering through general-purpose and task-specific models.BiomedGPT, LLaVA-Med, UMIT, HuatuoGPT-Vision, and Lingshu emphasize multimodal alignment and diagnostic reasoning, while LLaVA-Ultra, LLaVA-Rad, and RadFM target 2D and 3D radiological understanding.
NeoCXR and NeoCXR-EV Datasets
NeoCXR and NeoCXR-EV are two ethically approved, de-identified neonatal respiratory datasets collected from independent hospitals through a heterogeneous pipeline. They combine chest radiographs with structured clinical context organized around neonatal diagnostic logic.
- Dataset construction: A heterogeneous collection pipeline handles ready-to-use AP CXRs and metadata from Hospital A alongside multi-view CXRs and PDF clinical records from Hospital B.The pipeline is designed to accommodate distinct data formats across the two hospitals.
- Dataset construction: NeoCXR contains 6,278 samples from 2,466 patients, uses a patient-level 7:1:2 train-validation-internal-test split, and NeoCXR-EV contains 1,089 samples from 590 patients for external validation.These dataset summaries are provided in Table 1; both datasets received institutional ethics approval and were de-identified.
- Clinical context: Clinical factors are structured into developmental factors, perinatal risks, and physiological status according to neonatal disease progression and neonatologists’ diagnostic logic.Developmental factors include maturity and growth; perinatal risks capture maternal and obstetric conditions; physiological status represents immediate postnatal condition.
- Clinical context: Clinical factors are prevalent in both datasets, but their distributions differ notably between NeoCXR and NeoCXR-EV.Figure 3 reports the completeness percentages of each clinical factor across the datasets.
- Clinical context: Category-specific tokens explicitly delimit clinical inputs, while missing categories are replaced with “not provided” for consistent model input.The delimiters are <dev>...</dev>, <peri>...</peri>, and <phys>...</phys>.
Methodology
NeoRed combines neonatal chest X-rays with structured clinical context to generate reports containing imaging conclusions and disease diagnoses. Its KLA framework integrates diagnostic priors, diagnostic-logic constraints, and visual-semantic alignment through auxiliary training objectives.
- Overall architecture: NeoRed inputs a neonatal CXR and structured clinical context, concatenates visual and textual tokens, and autoregressively generates a structured report with imaging and disease diagnoses.The model optimizes report generation using autoregressive cross-entropy against the ground-truth report.
- Knowledge Prior Injection: KPI learns disease-specific modality dependencies by weighting developmental, perinatal, physiological, and CXR features with a neonatologist-initialized disease-modality prior matrix.The resulting prior, CXR, and clinical representations receive independent disease classification supervision through binary cross-entropy losses.
- Knowledge-Logic-Alignment: KLA comprises KPI, DLC, and VSA, respectively injecting diagnostic priors, constraining diagnostic consistency, and aligning image features with imaging conclusions.The framework is designed to emulate experienced neonatologists’ diagnostic logic and improve multimodal diagnosis.
- Diagnostic Logic Constraint: DLC injects diagnostic supervision into the BOS hidden state and uses global/local classification losses plus Jensen-Shannon divergence to enforce consistency between generated reports and diagnostic logic.The BOS state serves as a global diagnostic anchor because subsequent tokens can access it through self-attention.
- Visual Semantic Alignment: VSA applies bidirectional contrastive learning between pooled image-token representations and masked imaging-conclusion representations to encourage visually supported conclusions.The total objective combines language-modeling loss with KPI, DLC, and VSA auxiliary losses.
Experiments
Experiments evaluate the model on neonatal and adult chest-X-ray report-generation benchmarks, covering NeoCXR, NeoCXR-EV, MIMIC-CXR, and IU-Xray with specified test-set sample counts.
- Evaluation Benchmarks: The evaluation uses NeoCXR’s internal test set of 1,434 samples and the full NeoCXR-EV set of 1,089 samples.
- Evaluation Benchmarks: Adult evaluation uses the official MIMIC-CXR and IU-Xray test sets, retaining 2,737 and 3,193 samples after removing empty findings or impression sections.
- Evaluation Metrics: Generated reports are evaluated using natural-language metrics.
Results
NeoRed outperforms zero-shot generalist and medical MLLMs on neonatal benchmarks, remains competitive on adult report generation, and benefits from neonatal adaptation and the KLA framework. Ablations support the roles of KLA modules, expert priors, multimodal fusion, and clinical context.
- Performance on neonatal benchmark: NeoRed substantially outperforms Qwen3-VL-8B (24.25% F1) and HuatuoGPT-V-7B (9.73% F1) on NeoCXR, while remaining best on NeoCXR-EV under disease distribution shift.All pair-wise average-performance comparisons are statistically significant (p < 0.05).
- Performance on neonatal benchmark: Neonatal-domain fine-tuning yields average gains of 35.12% and 39.45% on NeoCXR and 6.45% and 21.12% on NeoCXR-EV for Qwen3-VL-8B and LLaVA-Rad-7B, respectively.With KLA, NeoRed further outperforms fine-tuned LLaVA-Rad on NeoCXR by 2.97% and Qwen3-VL on NeoCXR-EV by 4.52%.
- Generalization to adult benchmark: NeoRed retains adult-benchmark generalization, achieving 26.73% on MIMIC-CXR and 33.01% on IU-Xray, ranking second behind LLaVA-Rad-7B and Lingshu, respectively.LLaVA-Rad-7B is specifically optimized for adult chest X-rays.
- Ablation of KLA framework: Removing KPI or DLC mainly reduces clinical-efficacy metrics, whereas removing VSA primarily harms natural-language-generation performance, supporting complementary KLA contributions.Internal ablations show that all KPI and DLC components contribute, with Llc producing the largest overall impact among DLC components.
- Ablation of priors: Expert-consensus disease–modality priors outperform random, all-one, and all-zero initializations, while NeoRed achieves more balanced image–clinical attention than vision-dominated LLaVA-Rad during learning and diagnostic stages.Attention ratios are averaged across heads and tokens over 100 NeoCXR cases.
- Ablation of clinical context: Removing all clinical context produces the worst results, and removing any clinical category degrades performance, with developmental factors and perinatal risks most affecting clinical-efficacy metrics.The supplied passage also reports a diagnostic case comparison with and without clinical text, but its final outcome is truncated.
Conclusion
The study introduces NeoRed, the first MLLM tailored for neonatal respiratory disease diagnosis, alongside two real-world neonatal datasets. It structures clinical context and applies a Knowledge-Logic-Alignment framework to improve multimodal diagnosis.
- NeoRed is proposed as the first MLLM tailored for neonatal respiratory disease diagnosis, addressing a critical gap in the field.
- The study constructs two real-world neonatal datasets, NeoCXR and NeoCXR-EV, to support neonatal respiratory disease diagnosis.
- NeoRed organizes clinical indicators into developmental factors, perinatal risks, and physiological status, then applies a novel KLA framework for multimodal diagnosis.