Source-linked AI summary
Case2Flow: Bridging Patient Cases and Guideline Flowcharts through Multimodal Retrieval
Jiale Wei, Yufan Chen, Alexander Jaus, Zdravko Marinov, Julian Friedrich, Simon Reiß, Jens Kleesiek, Rainer Stiefelhagen
TL;DR
Medical guideline flowcharts encode actionable decision logic, but systems largely retrieve text rather than the specific visual artifact relevant to a patient case. Case2Flow and FlowAtlas address this retrieval problem, while CRISP refines late-interaction scoring without training. CRISP improves Recall@1 by up to +18.71 pp, although the paper reports important limits in synthetic data, closed-set evaluation, and clinical usability.
Problem
Existing guideline-support systems mainly retrieve textual passages or assume relevant context is available, leaving the specific decision artifact for a patient case difficult to locate.
Method
The paper defines Case2Flow, builds FlowAtlas from guideline flowcharts and aligned synthetic cases, and introduces training-free CRISP scoring for multimodal retrieval.
Results
CRISP improves Recall@1 by up to +18.71 pp, and blinded physician assessment on published case narratives provides preliminary feasibility evidence beyond synthetic queries.
Takeaways & Limitations
Case2Flow supports research on retrieving guideline decision artifacts, while physician assessment indicates preliminary transfer beyond synthetic queries.
Takeaways & Limitations
FlowAtlas uses synthetic cases, evaluation uses closed candidate pools, and top-1 charts were unusable in roughly half of assessed cases, precluding automated clinical decision-making without expert oversight.
Abstract
from arXiv · showhide
Medical guidelines encode rich, evidence-based decision logic, yet the specific decision artifact a clinician needs is hard to locate within a guideline, let alone across guidelines covering plausible diseases and treatments. While guideline passages have supported end-to-end question answering, flowcharts remain largely underused in decision support despite their ability to encode actionable clinical pathways. We therefore introduce Case2Flow, a task designed to retrieve the most relevant guideline flowchart for a given patient case from a collection of guideline documents. To support it, we construct FlowAtlas, a curated corpus of 202 flowcharts extracted from 2,080 medical guidelines, together with a pipeline that synthesises 1,911 aligned case-flowchart pairs. Our evaluation of multimodal retrieval methods reveals systematic failure modes, including overreliance on keywords and spurious token-patch matches induced by uninformative background regions in flowcharts. Motivated by this, we propose CRISP, a training-free scoring method that sharpens late-interaction retrieval by suppressing uninformative patches, discounting ambiguous token matches, and incorporating bidirectional query-image alignment. CRISP improves Recall@1 by up to 18.71 percentage points, while a blinded physician assessment on published case narratives provides preliminary feasibility evidence beyond synthetic queries.
1 Introduction
Case2Flow addresses the difficulty of locating actionable guideline flowcharts by introducing a benchmark, dataset, and training-free scoring improvement for multimodal retrieval.
- Motivation: Guideline growth increases the burden on clinicians navigating an expanding evidence base.Annual releases have surged by up to 29%.
- Motivation: Existing guideline question-answering and retrieval systems primarily operate on unstructured text, leaving visual decision artifacts underused.Flowcharts encode clinical pathways and decision-tree structures.
- Task: Case2Flow retrieves the most relevant guideline flowchart for a textual patient case from a large candidate pool.The case includes clinical information such as history, symptoms, and key findings.
- Evaluation and contribution: The study benchmarks six retrieval methods, analyzes failure modes, and introduces training-free CRISP scoring.CRISP improves Recall@1 by up to +18.71 pp.
2 Related Work
Related work has improved access to clinical knowledge through text retrieval and multimodal document retrieval, but locating the specific visual decision artifact for a patient case remains open.
- Clinical decision support: Clinical decision-support systems increasingly connect patient information with biomedical evidence and guideline knowledge, often through retrieval-augmented generation.These systems commonly retrieve textual passages or produce final answers.
- Multimodal document retrieval: Multimodal document-retrieval benchmarks evaluate natural-language retrieval of visually rich pages and layout elements.This line of work moves beyond OCR-only pipelines toward direct visual retrieval.
- Flowchart understanding: Medical flowchart research has mainly focused on extracting, formalising, executing, or following decision logic after the relevant source is known.Prior tasks include decision-tree construction, executable guidance, and workflow planning.
- Case2Flow distinction: Case2Flow instead retrieves the flowchart that best matches an unstructured patient case while preserving guideline logic in its visual form.It differs from executing known workflows or generating final clinical decisions.
3 Methodology
The methodology defines case-based flowchart retrieval, constructs FlowAtlas through graph-based extraction and case synthesis, and refines late-interaction scoring with CRISP.
- Case2Flow task: Case2Flow ranks candidate flowcharts for a patient case, with FlowAtlas using the chart containing the synthesised case’s decision path as the provenance target.This target is reproducible and does not claim that no other chart could be clinically appropriate.
- Flowchart corpus construction: The corpus pipeline collects guidelines, detects and verifies clinical flowcharts, manually removes false positives, and digitises retained charts as structured graphs.Nodes represent starts, decisions, actions, and terminals; labelled edges encode branch conditions.
- Case synthesis: DFS with cycle detection enumerates start-to-terminal paths, after which an LLM generates cases and focused questions using only information attached to each path.A second rewriting mode produces more natural clinical phrasing.
- Evaluation design: The evaluation retains Original and Rewritten cases to distinguish lexical-overlap accuracy from robustness to clinically equivalent paraphrases.It also evaluates six methods across per-source, mixed, and external settings.
- Failure modes: Late-interaction retrieval can spuriously match background patches and frequent tokens, causing MaxSim scores to reflect generic similarity rather than relevance.Examples of frequent tokens include “patient”, “history of”, and “mg”.
- CRISP scoring: CRISP combines patch weighting, adjusted forward scoring, and bidirectional coverage to suppress background, discount uniform token matches, and verify image-query alignment.It is training-free and operates at scoring time, replacing MaxSim in late-interaction retrieval models.
4 Experiments
Across increasingly difficult and externally sourced retrieval settings, multimodal late-interaction models generally outperform text pipelines, while CRISP improves retrieval most in harder pools and on weaker backbones. Ablations indicate that CRISP’s components interact, with bidirectional coverage providing the largest single contribution and residual errors often remaining within closely related guideline topics.
- Quantitative Results: Mixed-pool retrieval is substantially harder than per-source retrieval, with ColQwen3 falling to 78.44% and ColPali to 55.48%.The mixed pool contains all 257 flowcharts and is described as closest to deployment because the relevant provider is usually unknown.
- CRISP: CRISP improves ColPali Recall@1 by +15.50 pp on Mixed and +18.71 pp on NCCN, while improving ColQwen3 by +5.25 and +4.94 pp respectively.On Mixed, scores change from 55.48 to 70.98 for ColPali and 78.44 to 83.69 for ColQwen3; on NCCN, from 54.67 to 73.38 and 79.91 to 84.85.
- Baseline Comparison: Multimodal retrieval remains ahead of text-only alternatives, with ColQwen3 + CRISP exceeding the best text-only configuration by 13.4 to 15.0 pp.Graph-text reverses the relative weakness of OCR on NCCN versus Mixed, but does not close the multimodal gap.
- Ablation: Bidirectional coverage is the dominant single CRISP component, adding +10.18 pp R@1 on ColPali and +2.68 pp on ColQwen3.It accounts for 54% of the full R@1 lift on each backbone and larger shares of the MRR lift.
- Ablation: CRISP components are complementary but not independently monotone: ColQwen3’s single-component gains sum to −0.07 pp, whereas the full method reaches +4.94 pp R@1.On ColPali, the summed gain is +21.08 pp versus +18.71 pp achieved, reflecting sub-additivity.
- Behavioural Analysis: CRISP improves or preserves most ground-truth ranks, recovering 33.5% to 40.9% of cases previously below rank one while worsening only 4.0% to 6.3%.When CRISP helps, mean rank gains exceed mean losses on both backbones.
5 Conclusion
Case2Flow retrieves guideline flowcharts applicable to patient cases, supported by FlowAtlas and four guideline sources. CRISP addresses spurious background-patch matches and improves retrieval, while physician assessment offers preliminary evidence beyond synthetic queries.
- Case2Flow retrieves the guideline flowchart that applies to a patient case, and FlowAtlas contains 1,911 case-flowchart pairs across four guideline sources.
- Benchmarking six retrieval methods exposed spurious matches from uninformative background patches in late-interaction scoring.
- CRISP improves Recall@1 by up to +18.71 pp at a small scoring-time cost.
- A blinded physician assessment on published clinical case narratives provides preliminary feasibility evidence beyond synthetic queries.
- The study leaves headroom for future work on real clinical notes.
Limitations
The study’s evidence is constrained by synthetic cases, closed candidate pools, limited backbone and source coverage, and an oncology-heavy English corpus.
- FlowAtlas cases are synthetic and do not reproduce the noise, ambiguity, or incomplete information of real clinical notes.Clinical validation used one rater and 50 oncology-concentrated cases, so it provides feasibility evidence rather than an unconditional real-world accuracy estimate.
- The closed-set evaluation does not measure scaling to thousands of candidates or behavior when no relevant chart exists in the pool.CRISP’s coverage score could support abstention, but that use was not calibrated or evaluated.
- CRISP cannot recover clinical content absent from the underlying visual embeddings, such as dosage information rendered as a small footnote.
- CRISP is complementary to backbone improvements rather than a replacement, with no evidence for a pool-size threshold or adaptive switch.
- FlowAtlas covers four English-language providers but is not exhaustive and is concentrated in oncology, with 137 of 202 charts from ESMO and Onkopedia.National-specialty guidelines outside oncology and non-English guidelines are not represented.
Ethical Considerations
FlowAtlas uses public guideline documents and synthetic, non-identifying cases, while clinical validation relies on published narratives and a single clinician’s blinded rating.
- FlowAtlas is built from publicly accessible guideline documents and publicly released MedGUIDE flowcharts.
- Synthesised patient cases contain no real patient records or personally identifying information.
- Clinical validation uses published case narratives and redistributes case identifiers rather than article content.
- The blinded assessment was conducted by one clinician among the authors, without external annotators or human-subject data collection.The passage states that ethics board approval was therefore not required.
- FlowAtlas and CRISP are released for research on case-based guideline retrieval, not automated clinical decision-making without expert oversight.The top-1 chart was unusable in roughly half of the assessed cases.
B Per-Source Characteristics
FlowAtlas sources differ in layout, topic mix, and retrieval difficulty. The Mixed pool most closely approximates cross-provider deployment and produces the widest spread between methods.
- Source differences in layout style, caption convention, and topic mix affect retrieval difficulty.
- CDC is text-light and prescriptive, with a small candidate pool where most methods approach saturation.
- WHO and ESMO combine heterogeneous topics with flow-diagram and table-like layouts, producing ColPali’s largest within-pool drops.
- ColQwen3+CRISP reaches 96.85% within the oncology-specialised Onkopedia pool, whose decision paths extend to 20 nodes.
- CRISP improves ColPali by +18.71 pp R@1 on MedGUIDE, whose tabular layouts differ markedly from the FlowAtlas sources.The Mixed pool combines all five sources, requiring discrimination among layout styles; its method spread is widest.
C FlowAtlas Construction Pipeline
FlowAtlas combines guideline collection, flowchart detection, graph-based path enumeration, and synthetic case generation into a curated evaluation set. The pipeline produces 1,911 aligned case-flowchart pairs, with eligibility manually reviewed but downstream processing automated.
- 2,080 guideline PDFs from CDC, ESMO, Onkopedia, and WHO form the source collection, spanning broad population topics and oncology-specialised topics.
- Flowchart candidates are localized using caption keywords and visual layout classification, then verified for clinical flowchart content.
- Decision paths are enumerated by depth-first search with cycle detection, capped at 40 per chart to preserve case diversity.
- Synthetic cases are generated from path labels and rewritten into natural clinical phrasing with lay equivalents while preserving the same clinical situation.
- 1,911 aligned pairs remain after filtering empty, truncated, short, or byte-identical cases.
- Manual review determines chart eligibility, whereas graph checking, path enumeration, case synthesis, and post-generation filtering are automated; generated cases are not clinically adjudicated.
D Per-Source and Mixed-Pool Results
Across candidate pools and query modes, CRISP primarily improves the top of the retrieval ranking rather than lower-ranked recall. The results therefore frame rank-1 placement as the central retrieval challenge.
- CRISP improves Recall@1 and MRR more than Recall@5 or Recall@10 across the reported candidate pools and query modes.The reported pattern indicates sharper top-ranked retrieval rather than broad reshuffling of lower-ranked results.
- Above 97% Recall@10 for ColQwen3+CRISP on the Mixed pool in both query modes shows that shortlist retrieval is already strong for the strongest model.
E Visualising the CRISP Score
The visualizations decompose CRISP into patch weighting, adjusted forward matching, and reverse alignment to show how its score changes flowchart evidence. They illustrate suppression of background activations while preserving decision-node signals.
- Figure 5 isolates MaxSim, patch weights, adjusted forward matching, and complete CRISP on the same chart and case.
- MaxSim: MaxSim assigns strong activations to whitespace strips and connector arrows, producing scores comparable to decision boxes.
- Patch weights: Query-independent patch weights emphasize text-bearing decision and outcome nodes while assigning little weight to uniform whitespace.
- Adjusted forward: The adjusted-forward term credits each query token to its highest-weighted matching patch, yielding a sparse map focused on path-specific tokens.
- Complete CRISP: Complete CRISP combines the per-token forward term with the unweighted reverse term, preserving decision-node activations while suppressing whitespace.
F Statistical Reliability and Cost
CRISP’s Recall@1 improvements are statistically supported across Mixed and NCCN evaluations, while its scoring overhead is modest relative to backbone embedding generation. The tables and figure decomposition provide the reported evaluation views.
- Statistical reliability: 95% confidence intervals for ColQwen3 Recall@1 improvement exclude zero on original Mixed, rewritten Mixed, and NCCN, with paired McNemar tests yielding p < 10^-40 in all three.
- Cost: 101.2 versus 151.0 s for ColPali and 222.3 versus 280.0 s for ColQwen3 quantify CRISP’s scoring time against the corresponding baselines on 9,658 Mixed queries.The overhead is 1.49× for ColPali and 1.26× for ColQwen3, with peak GPU memory rising by under 0.2%.
- Evaluation views: Tables 7 and 8 report per-source Recall@k and MRR, while Table 9 reports the Mixed pool with 257 candidates and 9,658 queries per mode.
- Component analysis: Figure 5 expands the heatmap comparison by isolating CRISP components on the patch grid, with each panel normalized independently to its minimum and maximum.