PaperlayerView exact revision on arXiv

Source-linked AI summary

S-Chain: Structured Visual Chain-of-Thought For Medicine

Khai Le-Duc, Duy M. H. Nguyen, Phuong T. H. Trinh, Tien-Phat Nguyen, Nghiem T. Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, Van Trung Hoang, Khai-Nguyen Nguyen, Hy Nguyen, Chris Ngo, Anji Liu, Nhat Ho, Anne-Christin Hauschild, Khanh Xuan Nguyen, Thanh Nguyen-Tang, Pengtao Xie, Daniel Sonntag, James Zou, Mathias Niepert, Anh Totti Nguyen

arXiv:2510.22728v1cs.LGcs.CV

TL;DR

Large-scale expert datasets for medical chain-of-thought with explicit visual grounding remain scarce. S-Chain provides expert-annotated images with structured, visually grounded rationales, and its supervision improves predictive accuracy and interpretability while reducing hallucinations.

  • Problem

    Large-scale expert datasets for medical chain-of-thought that explicitly link reasoning steps to visual regions remain scarce, despite the need for expert-verified, multimodal, clinically valid annotations.

    Source: §1 · p. 2

  • Method

    S-Chain pairs 12,000 expert-verified medical images and ROI boxes with four-stage grounded rationales, spanning 700k multilingual QA pairs across 16 languages.

    Source: §1 · p. 2

  • Results

    SV-CoT improves predictive accuracy and interpretability and reduces hallucinations; on MedGemma, it boosts performance by up to +13.5 Accuracy and +14.6 F1.

    Source: §4.2.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The findings point to expert-grounded reasoning, complemented by retrieval, as a path toward clinically trustworthy reasoning.

    Source: §4.2.2 · p. 8

    Faithful CoT generation remains an open challenge, as models can produce reasoning only loosely aligned with localized evidence.

    Source: §5 · p. 10

Abstract

from arXiv · show

Faithful reasoning in medical vision-language models (VLMs) requires not only accurate predictions but also transparent alignment between textual rationales and visual evidence. While Chain-of-Thought (CoT) prompting has shown promise in medical visual question answering (VQA), no large-scale expert-level dataset has captured stepwise reasoning with precise visual grounding. We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT), explicitly linking visual regions to reasoning steps. The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability. Using S-Chain, we benchmark state-of-the-art medical VLMs (ExGra-Med, LLaVA-Med) and general-purpose VLMs (Qwen2.5-VL, InternVL2.5), showing that SV-CoT supervision significantly improves interpretability, grounding fidelity, and robustness. Beyond benchmarking, we study its synergy with retrieval-augmented generation, revealing how domain knowledge and visual grounding interact during autoregressive reasoning. Finally, we propose a new mechanism that strengthens the alignment between visual evidence and reasoning, improving both reliability and efficiency. S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical VLMs.

1 INTRODUCTION

S-Chain addresses the scarcity of expert-verified medical reasoning data with structured, visually grounded annotations. The paper also evaluates this supervision across VLM families and studies its interaction with retrieval and visual grounding.

Motivation

  • Expert-verified grounded CoT data remain scarce because medical annotations require clinical validity and visual ROI alignment, while synthetic rationales can lack structure and contain errors.The paper identifies hallucinations, redundancy, and clinically irrelevant content as limitations of existing auto-generated approaches.Source: §1 · p. 2

Dataset innovation

  • S-Chain pairs 12,000 medical images with expert-verified bounding boxes and four-stage grounded rationales, yielding over 700,000 QA pairs across 16 languages.The stages are object localization, image captioning, multiple-choice reasoning, and image classification.Source: §1 · p. 2

Extensive evaluation

  • The study compares specialized medical and general-purpose VLMs with baselines trained on GPT-4.1-generated rationales to assess expert-grounded supervision.The evaluation includes both model families and highlights gains from expert-grounded supervision.Source: §1 · pp. 2–3

Analytical insights

  • The analysis examines SV-CoT with retrieval-augmented generation and finds discrepancies between textual reasoning and referenced visual evidence during autoregressive training, motivating strategies to reinforce their alignment.The proposed strategies aim to couple grounded visual cues and CoT reasoning more tightly.Source: §1 · pp. 2–3

2 PROBLEM FORMULATION AND KEY CHALLENGES

Grounded medical VQA is framed as producing a diagnostic answer through stepwise reasoning linked to image regions; S-Chain addresses gaps in prior resources with expert-validated visual grounding at scale.

Problem formulation

  • SV-CoT requires models to localize relevant anatomy or abnormalities, link visual observations to clinical knowledge step by step, and generate an interpretable answer.The reasoning traces the final answer back to specific image regions, rather than providing only a prediction.Source: §2 · p. 3

Prior works

  • Medical VLMs have improved VQA through larger models and broader pretraining, but remain black boxes that lack grounded reasoning for verifying clinical markers and standardized scores.Examples of markers and scales include hippocampal shrinkage, sulcal widening, cortical thinning, and Scheltens, Pasquier, or Koedam scales.Source: §2 · p. 3
  • Existing medical CoT resources provide textual rationales without visual links, while visually grounded datasets use GPT-4.1-based synthetic rationales that risk hallucination and factual errors.In medicine, unreliable grounding boxes or generated explanations and diagnoses may lead to misleading conclusions or inappropriate clinical guidance.Source: §2 · pp. 3–4

S-Chain

  • S-Chain provides expert-validated SV-CoT for 12,000 medical images, with expert-drawn boxes, clinically verified rationales, 16 languages, and over 700,000 QA pairs.The annotation pipeline selects MRI slices, localizes regions of interest, records structured reasoning and standardized ratings, uses expert consensus for quality control, and validates translations.Source: §2 · p. 4, Figure 2 · p. 4

3 S-CHAIN DATASET

S-Chain structures medical VQA as clinically ordered, visually grounded reasoning, and provides expert annotations across multilingual MRI examples. Its training formulation teaches models to generate intermediate evidence-linked steps before the final diagnosis.

3.1 STRUCTURED VISUAL CHAIN-OF-THOUGHT DATA

  • Each example follows four clinical reasoning steps: ROI localization, lesion description, standardized lesion grading, and disease classification.Descriptions and grades connect localized visual findings, such as hippocampal shrinkage or sulcal widening, to a final diagnostic label.Source: §3.1 · pp. 4–5

3.2 DATA COLLECTION

  • S-Chain uses OASIS MRI volumes from 461 patients with volume-level labels for Non-Dementia, Mild-Dementia, and Moderate-Dementia.The data are publicly available under the Apache 2.0 license.Source: §3.2 · p. 5

3.3 DATA ANNOTATION PROCESS

  • Three trained doctors independently annotated representative MRI slices, localized and described ROIs, assigned standardized grades, and resolved labels by consensus.Certified linguists with basic medical training extended QA pairs into 16 languages; annotation required about 2,100 expert labor hours.Source: §3.3 · p. 5

3.4 DATA STATISTICS

  • 12,000 expert-annotated images and 700k QA pairs in 16 languages cover 64 patients, with non-overlapping train/test splits and a test set reflecting dementia-cohort proportions.The test set comprises 36% Non-Dementia, 27% Mild, and 36% Moderate cases.Source: §3.4 · p. 5, Table 2 · p. 5

3.5 LEARNING SV-COT VIA SUPERVISED FINE-TUNING

  • Autoregressive supervised fine-tuning conditions on the image and final disease question, training models to produce expert-annotated localization, description, grading, and diagnosis sequences.Teacher-forced cross-entropy trains intermediate reasoning traces before the final answer.Source: §3.5 · pp. 5–6

4 S-CHAIN IN ACTION: EXPERIMENTAL VALIDATION

Experiments find that expert-annotated S-Chain SV-CoT supervision outperforms answer-only and synthetic-CoT training, while retrieval adds complementary gains. Controlled analyses show that faithful CoTs and visually grounded ROIs improve diagnostic reasoning, and a proposed alignment regularizer yields further gains.

4.1 S-CHAIN VS. SYNTHETIC GROUNDING: THE VALUE OF EXPERT ANNOTATIONS

  • S-Chain supervision outperformed Q4-only training by 10–15% and GPT-4.1-generated synthetic CoTs by 4–5%, improved intermediate-step performance, and benefited general-purpose VLMs; proprietary APIs remained less reliable despite few-shot prompting.GPT-4.1 synthetic CoTs could inherit incomplete or inconsistent reasoning traces, including missing, misaligned, or absent bounding boxes.Source: §4.1 · p. 7

4.2.1 Retrieval Protocol.

  • The retrieval protocol used MIRIAD, pre-retrieving typically five passages with Q4 disease and imaging keywords and sharing them across related questions.The retrieved passages were concatenated with the image and question during training and inference.Source: §4.2.1 · p. 8

4.2.2 Observations.

  • MedRAG typically gained 1–5% Accuracy; SV-CoT gained up to +13.5 Accuracy and +14.6 F1 on MedGemma, while their combination gained up to +15.4 Accuracy and +15.7 F1 on ExGra-Med.The results indicate that SV-CoT was the larger performance driver, with retrieval supplying complementary knowledge.Source: Table 5 · p. 8, §4.2.2 · p. 8

4.3 FAITHFULNESS OF COT REASONING AND VISUAL GROUNDING

  • Ground-truth ROIs raised Q4 accuracy by ∼2%, while supplying correct CoTs with the ROIs pushed accuracy to 99%.For CoT grounding, visual prompting achieved 0.73 Acc versus 0.62 Acc with textual supervision, which often weakly attended to box coordinates.Source: §4.3 · p. 9, Figure 6 · p. 9
  • The proposed regularizer aligns CoT embeddings with ROI tokens, repels them from non-ROIs, and separates CoTs across diseases.With additional SFT, ExGra-Med improved from 60.4% to 62.5% in Accuracy and from 59.6% to 61.7% in F1; the optimal alignment method remains open.Source: §4.3 · pp. 9–10

5 DISCUSSION

The discussion finds that SV-CoTs improve medical reasoning through visual grounding, with further gains from combining them with MedRAG. It also identifies dataset limitations and the need for better-aligned CoT generation to support clinically trustworthy VLMs.

5 DISCUSSION

  • SV-CoTs improve predictive accuracy and interpretability while reducing hallucinations, and combining them with MedRAG brings further gains.The discussion attributes these benefits to linking reasoning steps to visual regions and describes grounded reasoning and external knowledge as complementary.Source: §5 · p. 10
  • S-Chain datasets have limited diagnostic coverage, overly linear reasoning, and no temporal or multi-expert dynamics, restricting evaluation in realistic settings.The discussion calls for addressing these gaps to test SV-CoTs in broader, more realistic settings.Source: §5 · p. 10
  • Faithful CoT generation remains an open challenge because model reasoning can be loosely aligned with localized evidence.Proposed directions include large-scale grounded supervision, cross-modal contrastive objectives, attention regularization, contrastive constraints, and faithful decoding.Source: §5 · p. 10

S-Chain Supplementary … A.2 SYSTEM PROMPTS

The supplementary material specifies fine-tuning configurations for six models and identifies system prompts for evaluation and CoT data generation, with prompt examples shown in figures.

A.1 DETAILED HYPER-PARAMETERS USAGE

  • ExGra-Med and LLaVA-Med were each trained for 3 epochs at 2e-5 with cosine scheduling, a 0.03 warm-up ratio, and batch size 32.Source: §A.1 · p. 19
  • MedGemma was fine-tuned for 3 epochs at 2e-5 with 0.01 weight decay, effective batch size 16, cosine annealing, and a 0.03 warm-up ratio.Source: §A.1 · p. 19
  • MedFlamingo used full-parameter SFT for 20 epochs on 10,000 VQA pairs, updating the language model and perceiver resampler while freezing input embeddings.The model combines a pre-trained ViT-L-14-336 vision encoder with MPT-7B; optimization used AdamW at 1e-4, cosine scheduling, and 10 warm-up steps.Source: §A.1 · p. 19
  • Qwen2.5-VL used full SFT with effective batch size 32: without CoT, 3 epochs at 1e-5; with CoT, 6 epochs at 5e-5.The without-CoT setting used cosine annealing and a 0.1 warm-up ratio.Source: §A.1 · p. 19
  • InternVL2.5 underwent full SFT for 3 epochs at 5e-5 with AdamW, cosine annealing, a 0.1 warm-up ratio, and effective batch size 32.Source: §A.1 · p. 19

A.2 SYSTEM PROMPTS

B FURTHER EXPERIMENT WITH RANDOM AND ABSENT BOUNDING BOXES · C EXTRA QUALITATIVE RESULTS

Experiments with ExGra-Med + SV-CoT show lower performance with shuffled bounding boxes and a further decline when boxes are removed.

B FURTHER EXPERIMENT WITH RANDOM AND ABSENT BOUNDING BOXES

  • With shuffled boxes, Accuracy fell from 60.4 to 55.4 and F1 from 59.6 to 54.3; without boxes, they fell to 44.4 and 41.8, respectively.The shuffled-box setting retained the original Q2–Q4 annotations; the no-box setting used only Q2–Q4 annotations.Source: B FURTHER EXPERIMENT WITH RANDOM AND… · p. 21

C.1 QUALITATIVE RESULTS OF GPT-GENERATED CHAIN-OF-THOUGHT

GPT-4.1-generated CoTs often fail to ground reasoning in visual evidence because their bounding boxes are missing, misaligned, or absent, undermining interpretability and diagnostic reliability.

C.1 QUALITATIVE RESULTS OF GPT-GENERATED CHAIN-OF-THOUGHT

  • GPT-4.1-generated CoTs frequently contain missing, misaligned, or entirely absent bounding boxes, breaking the link between reasoning steps and visual evidence.Figure 10 presents examples of this vision hallucination.Source: §C.1 · p. 21, Figure 10 · p. 21
  • These grounding errors undermine the interpretability and diagnostic reliability of synthetic CoTs.Source: §C.1 · p. 21
  • The examples highlight the limitations of relying on synthetic CoTs when their visual grounding is not faithful.Source: §C.1 · p. 21

C.2 QUALITATIVE RESULTS OF TRAINED MODELS USING S-CHAIN DATASET · D DETAILS OF DATASET CREATION

Qualitative examples show that fine-tuned ExGra-Med (7B) can localize disease regions and produce coherent reasoning and accurate predictions, while mislocalization can lead to flawed reasoning and incorrect decisions.

C.2 QUALITATIVE RESULTS OF TRAINED MODELS USING S-CHAIN DATASET

  • Successful ExGra-Med (7B) cases show accurate disease localization, coherent reasoning, and correct final predictions.Figure 11 presents these successful cases.Source: §C.2 · p. 22, Figure 11 · p. 22
  • Mislocalization of disease regions in failure cases can lead to flawed reasoning and incorrect final decisions.Figure 12 shows these failure cases.Source: §C.2 · p. 22, Figure 12 · p. 23

D.1 DATA ANNOTATION PROCESS · D.2 DATASET EXAMPLES

S-Chain’s annotation process links expert-selected imaging slices to localized visual evidence, diagnostic reasoning, and standardized grading, while the dataset examples show multilingual VQA conversations across three disease classes.

D.1 DATA ANNOTATION PROCESS

  • Experts selected four to five representative slices per target brain region, then boxed abnormalities in key regions to anchor reasoning and grading.The localized regions included the medial temporal lobe, parietal cortex, and posterior cingulate; experts wrote notes linking visible abnormalities to diagnostic criteria.Source: §D.1 · p. 23
  • Each ROI received hemisphere-specific scores on the Scheltens MTA, Pasquier GCA, and Koedam scales, justified with brief text.The scales were applied to coronal T1-weighted slices, axial FLAIR images, and multiple planes for posterior atrophy, respectively.Source: §D.1 · pp. 23–24
  • Annotations required agreement from at least two of three expert raters; non-consensus cases were excluded, leaving 100% inter-annotator agreement among retained labels.Source: §D.1 · p. 24
  • All English QA pairs were translated into 15 languages using automatic translation followed by Human-In-The-Loop review by certified professional linguists with basic medical training.Translators had at least C1 proficiency.Source: §D.1 · p. 24
  • Construction required approximately 2,100 expert-labor hours for 12,000 images and 700,000 QA pairs across 16 languages.The estimate includes about 600 hours for image annotation and roughly 100 hours of expert review per language subset.Source: §D.1 · p. 24

D.2 DATASET EXAMPLES

  • Dataset examples are multi-turn VQA conversations spanning 16 languages and three disease classes.Source: §D.2 · p. 25
  • Examples pair Non-Dementia with English, Arabic, French, and German; Mild-Dementia with Hindi, Indonesian, Japanese, and Korean; and Moderate-Dementia with Mandarin, Portuguese, Russian, and Spanish.Source: §D.2 · p. 25

D.3 S-CHAIN DATASET COMPARISON WITH OTHER GENERAL VISUAL COT … E.5.3 POSTERIOR ATROPHY (KOEDAM SCORE)

The merged sections present S-Chain’s scale and region-grounded reasoning, then motivate MRI-based Alzheimer’s assessment and describe the clinical, pathological, and imaging context for three visual rating scales.

D.3 S-CHAIN DATASET COMPARISON WITH OTHER GENERAL VISUAL COT

  • S-Chain contains 197k multilingual examples and uniquely links stepwise reasoning to explicit visual regions for evaluating interpretability and diagnostic accuracy.The total combines 172k train/validation examples and 25k test examples.Source: §D.3 · p. 25, Table 6 · p. 25

E.1.1 INTRODUCTION TO ALZHEIMER; E.1.2 RATIONALE

  • Because Alzheimer’s disease is progressive and early diagnosis matters, the sections motivate accessible MRI assessment as a less costly alternative to PET and a practical basis for monitoring.The rationale proposes standardized use of MTA, GCA, and Koedam scales to reduce inter-rater variability and support early detection and AI-assisted diagnosis.Source: §E.1.1 · p. 29, §E.1.2 · pp. 29–30

E.2.1 MOLECULAR PATHOLOGY AND PROTEIN AGGREGATION; E.2.2 NEUROINFLAMMATION AND MICROGLIAL DYSFUNCTION; E.2.3 GENETIC RISK FACTORS; E.2.4 ENVIRONMENTAL AND LIFESTYLE FACTORS; E.2.5 AGE AND COMORBIDITIES; E.3 PATHOPHYSIOLOGY

  • Alzheimer’s pathology links amyloid and tau accumulation with microglial inflammation, while genetic susceptibility, lifestyle exposures, aging, and comorbidities also shape risk and disease progression.The cited discussion identifies APOE-ε4 and TREM2 among genetic contributors and cardiovascular and lifestyle factors among modifiable risks.Source: §E.2.1 · p. 30, §E.2.2 · p. 30, §E.2.3 · p. 30

E.4 ALZHEIMER DIAGNOSIS

  • Alzheimer’s diagnosis combines clinical and cognitive assessment with laboratory evaluation and neuroimaging; MRI supports detection of structural changes and exclusion of alternative causes.The section describes history, mental-status and cognitive testing, and notes that PET-based biomarkers are limited by cost and availability.Source: §E.4 · pp. 31–32

E.5 MRI FINDINGS

  • MRI atrophy patterns help distinguish Alzheimer’s disease from other dementias, with MTA, GCA, and Koedam scores serving as the principal visual rating scales.The scales assess medial temporal, generalized cortical, and posterior atrophy, respectively.Source: §E.5 · p. 32

E.5.1 MEDIAL TEMPORAL LOBE ATROPHY (MTA)

  • The MTA scale rates hippocampal volume loss, choroid-fissure widening, and temporal-horn enlargement from 0 to 4, but pathological cutoffs vary with age, sex, and education.The study uses a dichotomized left/right score and considers the mean of both sides together because MTA cutoffs are inconsistently defined.Source: §E.5.1 · p. 32

E.5.2 GLOBAL CORTICAL ATROPHY (GCA)

  • GCA rates regional cortical thinning from 0 to 3; age-specific cutoffs help distinguish dementia-related atrophy from normal aging.Ventricular enlargement may indicate secondary atrophy but is less specific for differentiating dementia types.Source: §E.5.2 · pp. 33–34

E.5.3 POSTERIOR ATROPHY (KOEDAM SCORE)

  • The Koedam score rates posterior atrophy from 0 to 3 across MRI planes; its diagnostic performance is better in moderate and severe Alzheimer’s than in mild cases.It assesses parietal atrophy and can aid diagnosis in atypical or early-onset presentations; combining it with AD and GCA ratings supports structured MRI evaluation.Source: §E.5.3 · p. 34

E.6 METHOD … F.3 USAGE CONSIDERATIONS

The appendices describe a standardized MRI acquisition and expert annotation process, then set licensing, reuse, and cautious-use conditions for the dataset.

E.6.1 IMAGING ACQUISITION

  • MRI acquisition used whole-brain sagittal 3D T1-weighted MPRAGE with axial FLAIR and coronal T2 images, excluding scans with significant motion or unrelated abnormalities.The protocol specified typical MPRAGE parameters and visually inspected scans for quality.Source: §E.6.1 · p. 36

E.6.2 ANNOTATION PROTOCOL

  • Three trained physicians selected representative slices, annotated brain-region ROIs, applied standardized atrophy scales, and reached final scores by agreement of at least two raters.Regions were scored separately in both hemispheres with brief visual-feature explanations; the scales assessed hippocampal, global cortical, and posterior atrophy.Source: §E.6.2 · pp. 36–37

F.1.1 APACHE LICENSE 2.0

  • Apache 2.0 permits commercial use, modification, redistribution, proprietary integration, and combination with other code, while requiring attribution and disclaiming warranties and liability.It also grants a contributor patent license; attribution and license notices must remain, and redistribution under new branding is allowed.Source: §F.1.1 · p. 38

F.1.2 FAIR USE

  • Fair Use may support research reuse based on purpose, work type, amount used, and market effect, but remains context-dependent rather than guaranteed.The passage favors non-commercial transformative scholarship, factual data, limited or research-essential use, lack of market harm, and proper attribution.Source: §F.1.2 · p. 38

F.2 COMMUNITY USE AND RESEARCH APPROVAL

  • The dataset inherits Apache 2.0, permits use and redistribution subject to its terms, and bars redistribution of the original dataset unless substantially modified or extended.Redistributed versions must document changes, retain license and attribution notices, and reference the appendix; Apache terms also require preserving any NOTICE file and prohibit misrepresenting modified versions as the original.Source: §F.1.1 · p. 38, §F.2 · p. 39
  • Dataset users must cite the associated publications, link the official dataset page, acknowledge both datasets, and limit use to research and educational applications.Approved examples include academic studies, algorithm development and evaluation, and teaching or coursework.Source: §F.2 · p. 39

F.3 USAGE CONSIDERATIONS

  • Users should treat the medical dataset as potentially error-prone or biased, pilot-test derived work, and avoid clinical decisions without rigorous validation and qualified oversight.The authors also ask users to acknowledge limitations in derivative work and caution that misuse or overreliance is the user's responsibility.Source: §F.3 · p. 39
Loading 2510.22728v1…