PaperlayerView exact revision on arXiv

Source-linked AI summary

LUX: A Lesion-Aware Graph-Conditioned Visual - Language Architecture for Explainable Endoscopic Captioning

Alexis Ivan Escamilla-Lopez, Gilberto Ochoa-Ruiz, Salvador Hinojosa, Sharib Ali

arXiv:2608.23853v1cs.CV

TL;DR

Endoscopic ulcerative-colitis assessment remains subjective, while existing vision-language systems lack explicit grounding in localized and relational pathological evidence. LUX addresses this with lesion-centric graph conditioning and outperforms competing models across captioning, classification, grounding, and hallucination suppression.

  • Problem

    Existing medical vision-language systems rely heavily on global representations and language priors, limiting explicit semantic, pathological, and relational grounding in endoscopic captioning.

  • Method

    LUX constructs lesion-centric graphs from activation maps and integrates their relational embeddings into T5 cross-attention for grounded caption generation.

  • Results

    LUX achieves the strongest joint performance across MES classification, captioning quality, and hallucination suppression, including CIDEr = 0.92 and hallucination suppression of 5.3%.

  • Takeaways & Limitations

    Lesion-aware graph conditioning improves clinical caption quality and reliability by aligning generated language with localized pathological evidence.

  • Takeaways & Limitations

    Attention-refined feature maps do not explicitly represent lesion entities or their relationships, motivating the graph-construction stage.

Abstract

from arXiv · show

The interpretation of endoscopic imagery in ulcerative colitis is complex and subjective, with variability in human assessment and subtle mucosal inflammation. Although deep learning has advanced automated analysis, most vision-language models rely on global visual embeddings that overlook the localized and relational nature of pathological evidence, limiting clinical reliability and interpretability. We introduce LUX (Lesion-aware Unified eXplainable captioning), a graph-conditioned vision-language architecture for explainable endoscopic image captioning. LUX constructs a lesion-centric scene graph from Grad-CAM and CBAM activation maps, representing pathological regions as nodes and encoding their spatial and clinical relationships. These graph embeddings are integrated into the cross-attention layers of a T5 decoder, enabling generated words to attend to specific lesion nodes rather than only to global image features. This provides direct alignment between linguistic content and pathological evidence, supporting token-level interpretability and relational reasoning. LUX outperforms strong baseline and state-of-the-art medical captioning models across BLEU, METEOR, ROUGE-L, and CIDEr, with particularly strong gains in CIDEr. It also reduces hallucinated clinical findings and improves lesion-level grounding through stronger correspondence between generated tokens and localized pathological regions.

1. Introduction

Ulcerative colitis endoscopic captioning remains difficult because conventional vision-language systems overlook localized pathological evidence, relations among lesions, and clinical grounding. LUX addresses these limitations with a lesion-aware graph-conditioned architecture that links generated language to lesion-level evidence through structured reasoning.

  • Introduction: Ulcerative colitis assessment is subjective because mucosal inflammation varies in severity and distribution, while clinicians interpret subtle vascular, textural, and color changes in heterogeneous images.
  • Introduction: Existing medical vision-language captioning systems rely on global visual embeddings, limiting lesion-level grounding and allowing conflated findings or unsupported pathological statements.
  • Introduction: LUX represents pathological regions as lesion nodes and their spatial and semantic relationships as a scene–lesion graph derived from Grad-CAM and CBAM activations.
  • Introduction: The framework treats endoscopic captioning as evidence-constrained multimodal reasoning, progressively localizing findings, modeling their relationships, and generating clinically structured descriptions linked to visual evidence.
  • Introduction: LUX integrates localized pathological evidence into caption generation through graph-conditioned decoding and dual cross-attention, enabling token-level grounding between language and lesion representations.
  • Introduction: LUX is designed to improve caption quality and lesion grounding while reducing hallucinated clinical findings compared with conventional medical captioning baselines.

2. Related Work

Medical image captioning has progressed toward transformer-based, multimodal, and grounded clinical narrative generation across several specialties. However, global representations and limited relational reasoning remain barriers to lesion-level explainability, motivating LUX’s graph-conditioned approach.

  • Medical image captioning evolved from CNN–RNN image-to-text systems toward transformer-based and multimodal frameworks producing more coherent, structured clinical descriptions.
  • Fine-grained, attention-guided, domain-specific captioning and multimodal pretraining have expanded medical captioning across pathology, ophthalmology, dermatology, and related imaging tasks.MedicalCLIP, MedCLIP, BLIP-based adaptations, retrieval-guided captioning, interpretable transformers, and hybrid alignment mechanisms target semantic consistency and visual–linguistic correspondence.
  • Prior ulcerative-colitis captioning introduced localized visual grounding but lacked explicit relational modeling and direct decoder-level lesion reasoning.More broadly, graph-based captioning typically uses relational structures in encoders or as auxiliary constraints, leaving decoder-level graph conditioning comparatively unexplored in gastrointestinal endoscopy.
  • Most systems rely on global visual embeddings that obscure individual pathological findings, weaken evidence correspondence, and can produce fluent but weakly grounded or hallucinated clinical narratives.These limitations are especially important in gastrointestinal endoscopy, where caption generation remains underexplored and computational work has focused mainly on classification or segmentation.
  • LUX addresses these gaps by unifying lesion localization, inter-lesion relational graph modeling, and evidence-grounded caption generation so clinical descriptions remain linked to pathological findings and their relationships.

3. Proposed Approach

LUX frames endoscopic captioning as evidence-grounded multimodal reasoning, progressing from salient visual cues to lesion entities, relational graphs, grounded language, and explainable output. Its dual cross-attention decoder integrates global visual and lesion-level relational representations to link clinical terms with pathological evidence.

  • Overall Architecture: LUX combines a hybrid visual encoder, scene–lesion graph construction, lesion-aware decoder, linguistic formatting module, and explainability component.The pipeline progressively constrains caption generation through lesion-centered representations rather than global visual features or linguistic priors alone.
  • Visual Encoder: CBAM-enhanced multi-scale visual encoding preserves mucosal detail while emphasizing clinically relevant regions for lesion localization and feature extraction.CBAM activations highlight intact vascular structures in remission and ulcerated or bleeding regions at higher MES grades.
  • Scene–Lesion Graph: Localized pathological evidence becomes lesion nodes connected by spatial and semantic relationships, with graph propagation capturing inflammatory organization and constraining downstream language generation.Grad-CAM-derived activations are aligned with expert lesion masks, and lesion-centric embeddings represent the graph nodes.
  • Lesion-Aware Captioning: Dual cross-attention enables the T5 decoder to combine global visual context with lesion-level relational information while maintaining token-to-lesion traceability.An alignment matrix quantifies each lesion node’s contribution to each generated token, linking linguistic elements to pathological evidence.
  • Formatting and Explainability: Deterministic formatting standardizes clinical terminology and evidence ordering, while Grad-CAM and token-level lesion attribution provide complementary explanations for generated descriptions.The resulting caption preserves lesion-grounded semantics in a standardized, clinically interpretable form.

4. Datasets and Clinical Context

LUX uses complementary UC-Caption and LIMUC datasets to combine expert-authored clinical narratives, MES severity labels, visual diversity, and lesion-level supervision. Their curation and clinical analysis preserve links between visible inflammatory findings, disease severity, and grounded caption learning.

  • Dataset integration: UC-Caption supplies paired image–text clinical narratives, while LIMUC provides large-scale visual diversity and MES supervision; a 400-image LIMUC subset adds lesion-level annotations.Both datasets use four MES categories from remission (0) to severe disease (3), and the lesion subset samples 100 images per category.
  • LIMUC: LIMUC contributes 11,276 images from 564 patients and 1,043 procedures, distributed across MES 0–3 with 6,105 MES 0, 3,052 MES 1, 1,254 MES 2, and 865 MES 3 images.The corpus is weighted toward remission and mild disease while retaining variability in vascularity, granularity, ulceration, bleeding, illumination, and image appearance.
  • Clinical context: Clinical analysis of UC-Caption shows that vascular pattern loss, friability, bleeding, and other inflammatory markers increase with MES, with Spearman correlations supporting their association.Mean feature trends and radar plots demonstrate progressive inflammatory changes across severity levels.

5. Experiments

The experiments evaluate whether lesion-aware graph conditioning improves caption quality, clinical grounding, localization, severity consistency, and robustness beyond conventional vision–language approaches. The protocol combines comprehensive metrics, patient-level dataset separation, progressive component comparisons, and domain-shift validation.

  • Evaluation Metrics: LUX is evaluated across linguistic quality, clinical grounding, lesion localization, and MES severity consistency using complementary captioning, grounding, spatial, and classification metrics.Captioning metrics include BLEU, ROUGE-L, METEOR, CIDEr, and SPICE; localization uses Dice, IoU, and center-of-mass error; MES evaluation uses accuracy, precision, recall, and quadratic weighted kappa.
  • Dataset Partitioning: Experiments use strict patient-level separation, with UC-Caption split 70%/15%/15% for training, validation, and testing and LIMUC split 80%/20% for pretraining and external evaluation.Fixed random seeds and version-controlled configurations support reproducibility, while LIMUC evaluation tests domain-shift generalization.
  • Baselines and Comparisons: The evaluation compares LUX with state-of-the-art and clinically relevant architectures spanning CNN–RNN, transformer, attention-refined, and lesion-aware relational approaches.The progressive baseline structure isolates contributions from CBAM, the lesion graph, dual cross-attention, and the alignment loss.
  • Experimental Protocol: A four-phase pipeline progressively pretrains lesion-aware visual and graph priors, fine-tunes supervised captioning, adds semi-supervised pseudo-captions, and performs external domain-shift validation.The phases couple MES classification and Grad-CAM localization with graph-conditioned generation, lesion–language alignment, and robustness evaluation.

6. Results

LUX achieves the strongest overall balance of classification accuracy, caption quality, and clinical reliability, while lesion-aware graph conditioning improves grounding and severity-consistent semantic stability. Across evaluations, LUX outperforms strong baselines and foundation vision-language models while reducing hallucination-related errors.

  • Progressive architecture results: Replacing recurrent decoding with T5 improves all metrics, while CBAM and lesion graph reasoning provide additional gains in localization and semantic performance.ResNet+T5 improves CIDEr by 40% and SPICE by 36%; CBAM increases CIDEr from 0.66 to 0.72 and SPICE from 0.15 to 0.18.
  • Clinical consistency: LUX maintains clinically consistent captions across all MES categories, adapting semantic specificity to severity while reducing hallucinations, severity inflation, and semantic drift.It emphasizes preserved vascular architecture and minimal inflammation in remission and mild disease, while describing friability, erosions, and ulceration in more severe cases.
  • Unified evaluation: Classification-oriented baselines achieve 77.8%–82.3% MES accuracy but produce templated captions, whereas captioning-oriented models improve language generation without matching specialized severity assessment.Classification baselines score below 0.20 BLEU-4 and show 18.5%–32.4% hallucination rates; captioning-oriented models reach BLEU-4 scores of 0.19–0.31.
  • Unified evaluation: Foundation vision-language models outperform earlier adapted captioning systems, but Qwen2-VL still trails LUX and retains clinically relevant hallucinations.Qwen2-VL reaches BLEU-4 0.36, CIDEr 0.83, SPICE 0.22, and a 9.4% hallucination rate, while foundation models lack explicit lesion-level evidence association.
  • Unified evaluation: LUX improves on Qwen2-VL across every reported caption metric and raises classification accuracy over ColitisAI, achieving the strongest overall balance.Table 4 reports accuracy improving from 82.3% to 84.7%, while BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE improve from 0.36 to 0.41, 0.26 to 0.29, 0.41 to 0.44, 0.83 to 0.92, and 0.22 to 0.25, respectively.

Model MES Acc. BLEU-4 METEOR ROUGE-L CIDEr SPICE Halluc. (%)

LUX achieves the strongest joint performance across MES classification and caption generation while reducing hallucinations. Ablations, qualitative analyses, and expert review further support its lesion-grounded, clinically specific, and interpretable captioning.

  • Unified evaluation: LUX achieves the strongest joint performance across MES classification and caption generation while substantially reducing hallucinated findings.Table 4 evaluates adapted classification, captioning, and foundation vision-language models under a unified UC captioning protocol.
  • Ablation study: CIDEr increases from 0.83/0.86 to 0.92 while hallucination rates fall below 6% when lesion-domain specialization and training diversity are combined.The complementary benefits improve robustness and semantic consistency for clinically grounded caption generation.
  • Qualitative analysis: LUX preserves lesion-specific correspondence, severity-consistent semantics, and multi-lesion relationships more reliably than baseline global-captioning strategies across MES stages.The qualitative analyses link generated terminology to Grad-CAM activations and localized mucosal abnormalities, especially as disease complexity increases.
  • Human evaluation: Expert reviewers scored LUX 4.4 ± 0.3 for clinical correctness, 4.6 ± 0.2 for lesion specificity, 4.7±0.1 for hallucination avoidance, and 4.2±0.4 for readability.Reviewers particularly noted improved descriptions of vascular attenuation, friability, erosions, ulceration, and bleeding, with less unsupported severity inflation.

7. Discussion and Conclusions

LUX improves caption quality and clinical reliability by grounding generated language in lesion-centered relational representations. The discussion identifies lesion-graph conditioning as the primary driver of clinical faithfulness, while outlining temporal and cross-domain extensions.

  • Why graph-conditioned grounding improves clinical captioning: CIDEr improves 16% relative to the strongest non-LUX baseline and SPICE improves 32%, confirming that lesion-aware graph conditioning enhances caption quality and clinical reliability.The architectural ablation further attributes the strongest benefit to the lesion graph rather than visual saliency refinement alone.
  • Why graph-conditioned grounding improves clinical captioning: Removing the lesion graph causes the largest component degradation, with ΔCIDEr = −0.16 and hallucination rate +8.1 pp, indicating that decoder-level relational reasoning drives clinical faithfulness.This degradation exceeds that from removing CBAM or graph-alignment supervision.
  • Task-specific grounding versus model scale: Domain-specific grounding and model scale are complementary: grounding constrains what is said, whereas scale constrains how well it is said.Large-scale multimodal systems can remain weakly anchored to localized pathological evidence, making visually unsupported statements clinically consequential in endoscopy.
  • Future directions: A primary future direction is temporal reasoning for video endoscopy, using spatiotemporal lesion graphs across consecutive frames to capture disease progression and support longitudinal treatment summaries.The discussion also proposes extending LUX's lesion-centric and relational design beyond ulcerative colitis.
  • Conclusions: LUX represents pathological regions as relational graph nodes from Grad-CAM and CBAM activations and conditions a T5 decoder through dual cross-attention over visual and graph representations.This design directly aligns generated clinical language with localized pathological evidence.

CRediT authorship contribution statement

The authors contributed across conceptualization, methodology, implementation, validation, supervision, project administration, data curation, visualization, and manuscript preparation. Alexis Iván López Escamilla led the broadest range of technical and analytical activities, while other contributors supported oversight, resources, validation, investigation, and writing.

  • Alexis Iván López Escamilla contributed to conceptualization, methodology, software, validation, formal analysis, investigation, data curation, visualization, and manuscript writing.
  • Gilberto Ochoa-Ruiz contributed to conceptualization, supervision, project administration, funding acquisition, resources, and manuscript writing.
  • Salvador Hinojosa contributed to validation, visualization, and manuscript revision.
  • Sharib Ali contributed to methodology, validation, investigation, data curation, and manuscript revision.

Ethics Statement

The study retrospectively used anonymized endoscopic images under institutional ethical and privacy safeguards, with institutional review board approval and adherence to the Declaration of Helsinki.

  • Ethics Statement: The study used anonymized images, retained no personally identifiable information, and followed institutional privacy requirements, review board approval, and the Declaration of Helsinki.Clinical image datasets were associated with retrospectively approved studies conducted under institutional oversight.

Appendix A. Preprocessing and Data Preparation

The preprocessing pipeline standardized endoscopic images, applied robustness-oriented visual augmentation, and expanded captions through lesion-guided semantic enrichment. LGSE produced clinically filtered, lesion-linked caption variants while increasing lexical and syntactic diversity.

  • Image Standardization and Enhancement: Images were resized to 512 × 512 pixels, ImageNet-normalized, and lesion masks were resized with nearest-neighbor interpolation.LAB histogram matching, CLAHE, and gamma correction were additionally used to reduce inter-device variability and enhance mucosal detail.
  • Visual Augmentation: Visual augmentations were applied online during LIMUC pretraining and UC-Caption fine-tuning to improve generalization and domain robustness.The strategy included brightness/contrast jitter, rotation, horizontal flipping, elastic deformation, and Gaussian noise.
  • Controlled Lexical Expansion: LGSE generated 2–3 clinically valid paraphrases per caption through controlled lexical substitutions filtered by Bio-ClinicalBERT similarity scores of 0.80 < s < 0.95.Substitutions used UCEIS descriptors and SNOMED-CT terminology while preserving grammatical and medical validity.
  • Lesion-Conditioned Recomposition: Lesion-conditioned recomposition generated captions from lesion, attribute, and region triplets while respecting anatomical and severity constraints.The process linked linguistic variation to Grad-CAM-derived lesion nodes to promote cross-modal grounding during decoder training.
  • Filtering, Balancing, and Encoding: Generated captions were backtranslated, filtered for ambiguous clinical expressions, manually reviewed by two gastroenterologists, balanced across MES categories, and dynamically tokenized to 64–96 tokens.T5-base biomedical models used constrained English →Spanish →English decoding with beam penalty λ= 0.3.
  • Impact on Caption Diversity: LGSE increased lexical entropy by 41.6% and syntactic diversity by 34.2% relative to the base UC-Caption dataset.The enriched corpus was approximately 3.8× larger and was intended to improve natural, descriptive, clinically precise generation.

Appendix B. Implementation Details and Reproducibility Specification

LUX combines a spatially preserved ResNet-50 with CBAM refinement, Grad-CAM-derived lesion graphs, and dual cross-attention fusion in a T5-based decoder. Its reproducibility specification details model dimensions, optimization settings, computational conditions, and research-only feasibility limitations.

  • Visual Encoding: LUX preserves 16 × 16 spatial feature maps by removing ResNet-50’s final conv5_x stride-2 downsampling operation for lesion localization.The backbone is ImageNet-pretrained and provides multi-scale representations.
  • Visual Encoding: CBAM modules follow each residual stage, combining channel attention with reduction ratio r=16 and 7 × 7 spatial attention.Spatial attention uses sigmoid normalization to emphasize diagnostically relevant responses.
  • Lesion Graph Reasoning: Grad-CAM responses generate 768-dimensional lesion nodes, while a two-layer graph convolutional network compresses intermediate representations to 512 dimensions before multimodal projection.The graph is formed through spatially weighted average pooling over fused feature representations.
  • Multimodal Fusion: Dual cross-attention separately attends to global visual embeddings and graph-derived lesion relationships, combining both pathways through a learned gating mechanism.The gating projection uses Xavier initialization with zero bias, initially centering γ_t around 0.5.
  • Computational Conditions and Limitations: Experiments used one NVIDIA A100 GPU with 40GB VRAM and mixed-precision PyTorch training, with sub-second individual-image inference but no claim of real-time clinical deployment.The reported computational measurements represent methodological feasibility under research conditions rather than operational guarantees.

Appendix C. Qualitative Analysis

The qualitative analysis compares captioning behavior across conventional, adapted, foundation, and LUX models. LUX explicitly connects localized evidence to severity grading, whereas other models commonly omit clinical cues, overstate findings, or provide weak lesion-level grounding.

  • Appendix C. Qualitative Analysis: Classification-based models typically produce generic inflammation descriptions with weak lesion specificity, incomplete cue coverage, and unsupported or ambiguous severity statements.Reported omissions include erosions versus ulcers, friability grading, vascular qualifiers, absence statements, and MES rationale.
  • Appendix C. Qualitative Analysis: Adapted classification models risk hallucinating severe findings and poorly differentiate mild from moderate cues, limiting borderline MES interpretability.They may introduce unsupported findings such as possible bleeding and provide weak lesion-level grounding.
  • Appendix C. Qualitative Analysis: Foundation and adapted captioning models improve fluency or clinical plausibility but often omit friability, bleeding, and grading cues, leaving MES linkage implicit.BLIP-Med and HistoCap add specificity or structure, yet grounding and explicit severity rationale remain incomplete.
  • Appendix C. Qualitative Analysis: LUX explicitly connects localized evidence to severity grading, minimizing hallucination and improving clinical traceability in the presented case.The appendix reports no major omissions for LUX in that case.
Loading 2608.23853v1…