Source-linked AI summary
GateSPINE: Gated Cross-View Fusion for Lumbar Spine MRI Report Generation
Hoang Nguyen Van, Cuong Vuong Tuan, Trang Mai Xuan, Bien Tran Van, Nam Tran Van, Thien Van Luong
TL;DR
MRI report generation must integrate multiple sequences and imaging planes, while existing approaches commonly represent a study as a single volume. GateSPINE fuses sagittal sequences and uses parallel 3D encoders with gated cross-view fusion; it improves clinical-efficacy micro-F1 over the strongest baseline across three datasets.
Problem
MRI examinations contain complementary sequences and imaging planes, but existing methods often treat studies as single volumes or use fixed combination rules.
Source: §1 · p. 2Method
GateSPINE merges sagittal T1 and T2 with training-free dynamic fusion, encodes sagittal and axial volumes in parallel, and gates cross-view features before report decoding.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5Results
Clinical-efficacy micro-F1 improves over the strongest baseline by 3.6 points on PhenikaaMec, 3.6 on SPIDER, and 26.9 on Lumbar.
Source: §1 · p. 3Takeaways & Limitations
Ablations show gated cross-view fusion outperforms other fusion strategies, while GateSPINE achieves the best clinical-efficacy performance on one private and two public datasets.
Source: §5 · pp. 10–11Global image representations can cause GateSPINE to miss subtle lesions or describe their anatomical locations inaccurately.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Automated report generation can ease the burden radiolo gists face when interpreting multi-sequence MRI studies. Unlike CT, MRI examinations comprise multiple sequences and imaging planes, each con tributing complementary diagnostic information. Existing methods en code a study as a single volume and combine multiple acquisitions by fixed rules. Findings visible in only one plane are thus diluted and of ten missed, lowering recall on clinical efficacy metrics, where a missed abnormality is most costly. We propose GateSPINE, a vision-language framework that fuses sagittal T1 and T2 volumes with a training-free operator, encodes the fused sagittal and axial volumes with two parallel 3D encoders, and decodes their combined representation into a report. Its core mechanism is a gated cross view fusion module that predicts, per feature channel and token, how much of each view to admit, so the more informative view dominates at each spatial location. We evaluate GateSPINE on three lumbar MRI datasets, comprising two public bench marks and a private cohort collected from Phenikaa University Hospital, using both natural language generation (NLG) and clinical efficacy (CE) metrics. GateSPINE achieves the highest CE F1 through improved re call on all three datasets; on SPIDER, which lacks an axial sequence, this reflects the sagittal fusion component rather than the gated cross-view mechanism, which is validated on the two cohorts with both imaging planes. GateSPINE also remains competitive on standard NLG metrics.
1 Introduction
Lumbar MRI report generation must integrate complementary sequences and imaging planes, but prior approaches do not learn to combine sagittal and axial evidence. GateSPINE uses sagittal fusion and adaptive cross-view gating, and reports improved clinical-efficacy results across three datasets.
1 Introduction
- Radiologists integrate multiple MRI sequences and imaging planes to produce comprehensive reports, a time-consuming task that has motivated automated report generation.MRI provides detailed visualization of soft tissues and neural structures.Source: §1 · p. 2
- Prior MRI methods either model the sagittal view alone, use single 2D slices, or rely on fixed combination rules, leaving sagittal-axial report generation unexplored.Axial images show findings such as lateral recess stenosis and facet hypertrophy that sagittal-only models can characterize only in part.Source: §1 · p. 2
- GateSPINE merges sagittal T1 and T2 with a training-free operator, encodes sagittal and axial volumes in parallel, then adaptively gates their features before report decoding.The gate learns how much of each view to use per feature channel and spatial position.Source: §1 · p. 2
- The stated contributions combine multi-sequence fusion, parallel 3D encoders, and adaptive cross-view fusion, with claimed gains in clinical-efficacy and language-generation performance.The evaluation covers a private PhenikaaMec cohort and the public SPIDER and Lumbar datasets.Source: §1 · pp. 2–3
- Clinical-efficacy micro-F1 improves over the strongest baseline by 3.6 points on PhenikaaMec, 3.6 on SPIDER, and 26.9 on Lumbar, with recall gains up to 38.0 points.SPIDER has no axial sequence, so its result reflects sagittal fusion rather than gated cross-view fusion.Source: §1 · p. 3
2 Related Work
Prior CT report-generation systems combine feature streams from a single volume, whereas MRI report generation must handle multiple sequences and imaging planes. Earlier MRI approaches combine same-plane contrasts or use structured or radiologist-authored findings; sagittal-axial image combination remains open.
2 Related Work
- CT report-generation systems combine multiple feature streams extracted from a single volume, so their fusion does not reconcile different anatomical views.Examples include CT2Rep, Dia-LLaMA, Med3DVLM, and generalist models [4].Source: CT report generation · p. 3
- SPINE combines sagittal T1 and T2 with segmentation-derived channels, but same-plane stacking does not incorporate axial evidence.Other MRI systems use structured gradings or radiologist-authored findings rather than images.Source: MRI report generation · p. 3
- Combining sagittal and axial evidence for MRI report generation remains an open problem, addressed here by within-plane sequence fusion and feature-level gating across planes.Source: MRI report generation · p. 3
3 Methods
GateSPINE keeps sagittal and axial MRI as separate streams, fusing sagittal T1 and T2 before encoding and learning how to combine the views. The fused features are projected into visual tokens for report generation.
3 Methods
- GateSPINE preserves sagittal and axial volumes as separate streams, merges sagittal T1 and T2, then uses parallel 3D encoders to extract view-specific features.A gated fusion module combines the two encoded views before report decoding.Source: §3 · pp. 3–4, Figure 1 · p. 4
3.1 Multi-Sequence Encoder
- A training-free, input-adaptive operator fuses sagittal T1 and T2 before encoding, while the axial volume forms its own stream.The operator uses reconstruction error to weight sagittal sources, and the axial branch needs no sequence fusion.Source: §3.1 · pp. 4–5
3.2 Gated Cross-View Fusion
- A learned gate mixes sagittal and axial features per token and channel, allowing their proportions to vary with content without increasing the token count.Concatenation would double the token count, while averaging fixes the mixing proportion across channels and locations.Source: §3.2 · p. 5
3.3 Vision–Language Projection and Decoding
- The projector compresses fused features into spatially structured visual tokens for the language model, while a trainable adapter maps them into its embedding space.The decoder is adapted with LoRA; the view encoders, gate, and adapter are trained with full gradients.Source: §3.3 · p. 6
3.4 Training and Inference
- Training uses autoregressive cross-entropy over report tokens, with gradients restricted to answer tokens; inference decodes greedily until the end token or token limit.Training samples prompts from clinical query templates.Source: §3.4 · p. 6
4 Experiments and Results
GateSPINE leads clinical results across the reported datasets, while the experiments find benefits from using all available sequences and gated fusion. Report examples still show errors in vertebral level and lesion severity.
4.1 Datasets and Evaluation Metrics
- PhenikaaMec and Lumbar include sagittal and axial views, whereas SPIDER has no axial sequence and tests the sagittal branch; clinical entities are scored with an LLM judge.The judge-validation process and report translation are left for future validation against radiologist annotations.Source: §4.1 · pp. 6–7
4.2 Implementation Details and Baselines
- Single-volume baselines receive the fused sagittal volume without the axial branch, so their comparison evaluates GateSPINE as a framework, while the ablation isolates fusion.The two-stage training uses a frozen LLM first and adds LoRA fine-tuning in the second stage.Source: §4.2 · p. 7
4.3 Evaluation Results
- GateSPINE leads clinical entity scores on PhenikaaMec and both metric groups on SPIDER and Lumbar; on PhenikaaMec, Micro-R is 67.35 vs. 52.05 for Dia-LLaMA.Dia-LLaMA scores higher on BLEU-4 and METEOR on PhenikaaMec.Source: §4.3 · pp. 7–8, Table 1 · p. 8
4.4 Ablation Studies and Analyses
- The full three-sequence setup is best on every metric, and adding axial input to fused sagittal T1 and T2 gives a large clinical-entity gain on both datasets.The strongest single sequence differs by dataset: sagittal T2 on PhenikaaMec and axial T2 on Lumbar.Source: §4.4 · p. 8, Table 2 · p. 9
- On both PhenikaaMec and Lumbar, simple gated fusion outperforms concatenation and dynamic fusion; MedGemma beats Phi-3 on every metric, with a larger gap in clinical efficacy.The authors attribute the gate's result to stable, content-dependent selection that is easier to train on small medical data.Source: §4.4 · p. 9, Table 3 · p. 10
- Across the three example datasets, GateSPINE recovers most major abnormalities with fewer hallucinated and missed findings than SPINE, but errors remain in vertebral level and lesion severity.The comparisons consider abnormality type, location, and severity.Source: §4.4 · pp. 9–10, Figure 3 · p. 10
5 Conclusions
GateSPINE combines sagittal and axial representations through gated cross-view fusion and achieves the best clinical-efficacy performance across the evaluated datasets while remaining competitive on language-generation metrics. The framework relies mainly on global image representations, which can lead to missed subtle lesions or inaccurate anatomical locations.
5 Conclusions
- GateSPINE achieves the best clinical-efficacy performance on one private and two public datasets while remaining competitive on language-generation metrics.Ablation studies also show that gated cross-view fusion outperforms existing fusion strategies.Source: §5 · pp. 10–11
- GateSPINE adaptively combines sagittal and axial representations to exploit complementary information across MRI views.The module learns how much information to preserve from each view for every feature channel.Source: §5 · p. 10
- Ablation studies show that gated cross-view fusion outperforms existing fusion strategies.Source: §5 · p. 11
- Reliance on global image representations can lead to missed subtle lesions or inaccurate anatomical locations.The paper identifies finer-grained anatomical information as a direction for improving localization accuracy and report quality.Source: §5 · p. 11