MedQA-MM: Shortcuts Behind Medical Visual Reasoning

Problem framing

• A shortcut is an item-level, construct-irrelevant signal that enables above-chance answer selection without integrating the intended visual, clinical-text, and option-semantic evidence.Clinical context, devices, demographic facts, and annotations are not invalid by default; validity depends on whether a cue bypasses the route the score is meant to measure.Source: §1 · pp. 1–2
• The paper defines reasoning inflation as interpreting benchmark accuracy as direct evidence of image-grounded reasoning when credited decisions may follow shortcut routes.This does not imply models never use images; unaudited score movement is difficult to interpret because one benchmark number may reward multiple routes.Source: §1 · pp. 1–2

Audit-and-repair framework

• The audit connects option cueing, NLP artifacts, VQA language priors, and medical image-side leakage through a route-level evidence ladder.It canonicalizes six datasets, separates prompt-visible fields from source-only metadata, evaluates multiple modality settings, and applies traceable matched repairs that preserve the medical target and answer key.Source: §1 · p. 2

Results

• 62.63% full-input accuracy coexists with 53.96% text-only and 29.71% options-only accuracy, while matched repairs reduce accuracy by 6.58, 3.50, and 4.77 percentage points for three cue families.In MEDQA-MM, text-only and options-only accuracy fall to 5.21% and 12.33%; the subset is shortcut-mitigated, not shortcut-free.Source: §1 · p. 2

Contributions

• The contributions introduce the two-aha framing, unify shortcut risks across medical multimodal MCQs, demonstrate behavioral effects of matched repairs, and construct MEDQA-MM for shortcut mitigation and residual-risk reporting.The two routes may reach the same answer, but one uses a construct-relevant image finding while the other uses option form, non-visual context, visible text, markup, or other preserved cues.Source: §1 · pp. 1–2, Figure 1 · p. 1

Reasoning narratives, score validity, and MCQ cueing

• MCQ flaws such as option-length imbalance, grammatical mismatch, conspicuous or absolute wording, paired options, and implausible distractors can enable answers without target knowledge.This validity lens is applied to VLM benchmarks whose scores are often interpreted as image-grounded reasoning.Source: §2 · pp. 2–3

Dataset artifacts and modality shortcuts

• NLP and VQA studies show that annotation artifacts, syntactic heuristics, language priors, and answer-set regularities can inflate accuracy while masking shallow or nonvisual strategies (Gururangan et al., 2018; Agrawal et al., 2016).Recent VLM diagnostics therefore test visual indispensability, hallucination, and robustness to reduced or altered visual information.Source: §2 · p. 3

Medical multimodal evaluation and image-side shortcuts

• Medical evaluations must account for visible labels, arrows, overlays, devices, acquisition markers, and care-process artifacts, alongside site, demographic, device, and process signals that may replace target pathology (Geirhos et al., 2020).Because some clinical cues are legitimate, construct relevance is treated as item-specific rather than category-wide.Source: §2 · p. 3

Data-centric benchmark repair

• A data-centric workflow detects candidate cues, repairs them when safe, validates the medical target and one-best-answer structure, and documents residual uncertainty rather than merely listing flaws.This frames benchmark repair as part of evaluation design, especially when medical data scarcity makes blanket deletion costly (Kaushik et al., 2020).Source: §2 · p. 3

3.1 Construct and Evidence Ladder

• The evidence ladder escalates from screened candidates to ablation-supported, repair-supported, and expert-validated cues before final item disposition.Disposition considers cue status, repair fidelity, validation status, and release constraints.Source: §3.1 · p. 3

3.2 Shortcut Scope

• The audit screens text-side option and wording asymmetries alongside image-side text, annotations, context, devices, and acquisition artifacts, feeding ablations, repairs, and expert review.Automatic detections are candidates because the same surface cue may be clinically valid in one item but shortcut-like in another.Source: §3.2 · p. 3, §3.1 · p. 3

3.3 Option-Form Shortcuts

• 306 length-gap, 101 spatial/prepositional, and 101 absolute/conspicuous pairs receive minimal edits preserving the stem, image, label, order, and one-best-answer target.The convention ∆text = Accedited − Accoriginal makes negative values indicate reduced accuracy after cue removal.Source: §3.3 · p. 4

3.4 Partial-Input Probes

• Four prompt settings—full input, text-only, options-only, and Image+Options—probe whether full-input performance depends on image-grounded reasoning.Options-only and Question+Options guessers serve as diagnostic stress tests rather than deployable systems or leaderboard baselines.Source: §3.4 · pp. 4–5

3.5 Image-Channel and Natural-Cue Interventions

• Interventions produce 200 paired image-hint cases and 302 natural-cue cases, using edits or neutralization that preserve medical content while measuring performance loss and harmful reliance.All intervention metrics use modified minus original; negative values indicate lost performance, while the bad/cue ratio captures stricter harmful reliance.Source: §3.5 · p. 5

3.6 Low-Prevalence Image-Text and Device Audits

• Text-bearing images and devices or post-treatment materials are treated as shortcut risks only when they bypass the intended route by revealing diagnosis, stage, intervention, complication, or management.Confirmed unrepairable cases become exclusion candidates.Source: §3.6 · p. 5

3.7 Human Review and Shortcut-Mitigated Subset

• 142/150 option-form repairs, 148/150 improved or partially improved, 50/50 natural cues, and 45/46 image-hint edits passed sampled review; MEDQA-MM remains shortcut-mitigated, not shortcut-free.The subset combines image-availability checks, difficulty and visual-dependence scoring, shortcut filtering or repair, and removal of confirmed device/material risks.Source: §3.7 · p. 5

Datasets

• The canonical audit covers 7,706 examples across six medical multimodal MCQ datasets.Feature-specific analyses may have different counts after option normalization or source-variant expansion.Source: §4 · p. 5

Models and Prompt Settings

• The repair, modality, image-hint, and natural-cue panels contain 17, 16, 16, and 17 configurations, respectively, with 19 unique configurations overall.GPT-5.6 Sol is used only for reverse-control generation and error-analysis judgments, not as an evaluated panel member.Source: §4 · p. 5
• Matched option-form repairs use 20 repeated original/edited evaluations per model, while image-hint and natural-cue interventions use 10 repeated paired evaluations per model.The repeat counts differ because these experiments were run as separate evaluation batches.Source: §4 · p. 5

Metrics and Uncertainty

• Accuracy and accuracy change are reported in percentage points, with no-image random accuracy defined as the mean item-level chance rate, 1/K.Repeated generations are averaged within item×configuration×condition cells before uncertainty estimation; 95% confidence intervals use analysis-specific resampling units, and expert review uses categorical counts and percentages.Source: §4 · p. 6

Full-Input Accuracy Mixes Multiple Shortcut Routes

• Full-input accuracy remains route-ambiguous because correct answers can be supported by option form, question text plus options, or image-channel cues rather than the intended image-grounded path.The audits use options-only prediction and matched repairs, no-image and Question+Options probes, and image-side screening with paired interventions.Source: §5 · p. 6

Option-Form Shortcuts Create Answer Priors

• 57.78% options-only accuracy versus 20.00% item-level random baseline on MedThinkVQA is the largest audited option-form gap, while matched repairs reduce accuracy by 6.58, 3.50, and 4.77 pp.The reductions correspond to length-gap, absolute/conspicuous, and spatial/prepositional repairs, respectively.Source: §5 · p. 6, Figure 3 · p. 7

Destination-Specific Evidence for Length and Spatial Cues

• Across 306 length-gap pairs, edited-distractor selection rose 9.47 pp while untouched-distractor selection fell 2.87 pp, showing targeted prediction shifts.In the clean 46-pair spatial/prepositional subset, edited selection rose 2.63 pp and untouched selection fell 0.69 pp; excluding uniquely longest distractors left a +2.70 pp contrast across N = 39.Source: §5 · p. 6, Figure 4 · p. 7

Question Text and Options Create Strong No-Image Routes

• 62.63% full-input accuracy versus 53.96% text-only and 29.71% options-only accuracy shows that question text and options create strong no-image routes.The Question+Options guesser reaches 53.75% without images, although full input is usually best and Image+Options often exceeds options-only.Source: §5 · p. 6, Figure 5 · p. 8

Image-Channel Cues Are Common, But Reliance Is Model-Dependent

• 5,645 text residuals, 4,970 manual markups or overlays, 2,642 demographic/context features, and 955 device/material cues reveal prevalent image-side shortcut surfaces, not model reliance.Paired interventions show model-dependent harmful reliance: Qwen3.5-0.8B has ∆image = −3.20 pp with a 16.6% bad/cue ratio, while some models mention cues with little accuracy loss.Source: §5 · pp. 6–7, Figure 6 · p. 8

Visible Image-Text Leakage Is Rare But Direct

• 65 high-risk image-text shortcuts yielded only 15 directly repairable cases and 14 model-only datasetreadiness validations, making leakage a low-prevalence but direct data-quality risk.Device/material screening found 9 confirmed shortcut-risk cases and 48 borderline cases; unsafe image alteration favors exclusion or redesign over repair.Source: §5 · pp. 7–8

Shortcut Mitigation Weakens Non-Visual Routes

• 5.21% text-only and 12.33% options-only accuracy in MEDQA-MM show that shortcut mitigation substantially weakens non-visual routes, though the subset is not shortcut-free.Image+Options accuracy is 28.47% and full-input accuracy is 26.59%; the subset retains 22.6% of 4,432 complete candidate items.Source: §5 · p. 8

Targeted Error Analysis of MEDQA-MM

• Image+Options exceeds full input by 1.88 pp in targeted MEDQA-MM error analysis, while visual-understanding/reasoning failures total 36/70 and item/data issues comprise 29/70.Stem interference contributed in 32/300 cases, with 92% expert-panel agreement and Cohen’s κ = 0.703; these samples do not explain the aggregate gap.Source: §5 · p. 8, Table 2 · p. 9

• Full-input MCQ accuracy can combine visual evidence with option form, no-image clinical text, visible image text, annotations, devices, and natural context.Strong Text-only, Options-only, or Image+Options results show that accuracy may reflect benchmark-preserved structure as well as image-grounded reasoning.Source: §6 · p. 9
• Cue presence alone does not establish shortcut dependence; paired interventions and an evidence ladder connect cue removal or neutralization to correctness changes.Rationale judgments describe observable outputs rather than hidden mechanisms, while validation bounds the claim.Source: §6 · p. 9
• A device, annotation, or demographic fact can be target evidence in one item but an unintended route in another, so validity requires item-level judgment.Clinically necessary context should be retained and documented, whereas nonessential predictive context should be repaired, flagged, or excluded.Source: §6 · p. 9
• Benchmark repair should detect cues, minimally repair them when safe, evaluate matched pairs, validate changes, document residual risk, and discard unsafe repairs.Releases should report multiple modality ablations and retain provenance, cue flags, repair and validation status, and residual risks; MEDQA-MM is shortcut-mitigated under tested probes, not shortcut-free.Source: §6 · p. 9

• The paper unifies MCQ cueing, NLP/VQA artifacts, and medical imaging shortcut learning within one construct-validity framework.Source: §7 · p. 9
• Across six datasets, the study quantifies shortcut prevalence and no-image signal, showing why benchmark scores can overstate image-grounded reasoning.Source: §7 · p. 9
• Matched repairs show that three text shortcut families are behaviorally consequential, motivating MEDQA-MM, a harder shortcut-mitigated subset.Source: §7 · p. 9

• Behavioral claims rely on model judgment, rule screening, and sample-based expert validation rather than exhaustive clinician review, and image-text or device analyses remain curation-risk evidence.Validation cannot guarantee every automatically screened or repaired item.Source: Limitations · pp. 9–10
• GPT-generated edits may alter wording, fluency, plausibility, or difficulty beyond the targeted cue, even when repairs are minimal.Source: Limitations · p. 10
• MEDQA-MM is a shortcut-mitigated subset of existing datasets, requiring source-license, image-rights, item-level validation, and redistribution checks before public release.Source: Limitations · p. 10

Ethics Statement

• The work audits benchmark reliability rather than providing clinical advice or evaluating deployment, and any repaired-subset release must address licensing, consent, privacy, and clinician review.The authors did not upload raw datasets, private images, PHI, sensitive model outputs, or non-shareable files to ChatGPT.Source: Ethics Statement · p. 10
• Sociodemographic and visible-context analyses are treated as shortcut-risk audits, not as grounds for inferring protected identity from images.Edits that alter demographic or clinical context require clinician review.Source: Ethics Statement · p. 10

A Appendix Roadmap and Audit Inventory

• The appendix is organized by evidence route, with separate appendices covering text and options, image and context, MEDQA-MM construction, prompts, taxonomy, and representative cases.Table 3 maps main-text claims to appendix evidence, while Table 4 reconciles denominators for the canonical audit and feature-specific analyses.Source: A Appendix Roadmap and Audit Inventory · p. 13, Table 3 · p. 13
• The appendix evidence map separates prevalence screens, paired interventions, expert review, and release validation.These categories are summarized in Table 3 as a reference map from main-text claims to appendix evidence.Source: Table 3 · p. 13

A.1 Denominator Reconciliation

• The canonical audit denominator supports the main results, while differing totals occur only in feature-specific screens requiring option normalization or retained source variants.The 7,715-item option-valid screen supports longest-option prevalence, and the 7,722 source-variant audit supports the GPT shortcut screen.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 Dataset Inventory and Analysis Families

• Table 5 lists the canonical inventory for the six-dataset audit, while Table 1 summarizes the corresponding evidence families and intervention sizes.The final MEDQA-MM benchmark contains 1,000 examples after additional filtering and device/material cue removal.Source: §A.2 · p. 13, Table 5 · p. 13
• The final MEDQA-MM benchmark is a filtered subset distinct from the canonical six-dataset audit inventory.Its construction includes additional device/material cue removal, with further details provided in Appendix D.1.Source: Table 5 · p. 13

A.3 Detailed Experimental Settings

• 47.39% overall accuracy was obtained across 400,712 prediction rows, with 8,267 parse failures and a 2.06% residual failure rate.Broad audit summaries count parse failures as incorrect, while analysis-specific panels collectively contain 19 unique configurations.Source: §A.3 · p. 13, Table 6 · p. 13
• 62.63% Full input accuracy exceeded Text-only at 53.96%, Options-only at 29.71%, and Image+Options at 43.84% across 13 equally weighted open-model configurations.The panel pooled 7,706 questions across six datasets per configuration and excluded a duplicated setting-level aggregate.Source: §A.3 · p. 14

B.1 Diagnostic Guesser Models for Shortcut Analysis

• The options-only and Question+Options guessers isolate answer-choice artifacts and question-text routes without images, using item-disjoint splits, forced-choice scoring, and LoRA supervised fine-tuning.The options-only model receives no question text or image, whereas Question+Options receives question text and answer choices but no pixels, captions, rationales, or explanations.Source: §B.1 · p. 14, Table 7 · p. 14, Table 8 · p. 14

B.2.1 Length-Gap Repair Set

• 306 length-gap pairs were retained after rewriting one or two distractors, preserving option order and the correct answer while making a distractor at least 5 characters longer.The pairs were evaluated with 20 repeated original-versus-edited runs per condition.Source: Table 11 · p. 15, §B.2.1 · p. 15

B.2.2 Spatial/Prepositional Repair Set

• 101 spatial/prepositional repaired items remained from a semantically reviewed 174-item pool after medical, semantic, and evaluation-eligibility validation.The repaired-set composition is separately categorized, while downstream evaluation compares paired original and edited items without conditioning on those categories.Source: Table 12 · p. 15, Table 13 · p. 15

B.2.3 Absolute/Conspicuous Distractor Repair Set

• Absolute/conspicuous distractor repairs formed a separate evaluation family with construction stages and non-mutually-exclusive cue categories reported in Tables 14 and 15.The categories describe candidate cues and are not mutually exclusive.Source: §B.2.3 · p. 15, Table 14 · p. 15, Table 15 · p. 16

B.2.4 Cross-Dataset and Cross-Family Summary

• The cross-family analysis reports dataset-level options-only diagnostics and complete model-level accuracy changes for length-gap, absolute/conspicuous, and spatial/prepositional repairs.Modified-minus-original values are signed percentage-point changes, with negative values indicating that cue removal reduced accuracy.Source: §B.2.4 · p. 16, Table 16 · p. 16, Table 17 · p. 16

B.3 Directional and Control Analyses for Option-Form Repairs

• Directional controls show that length-gap edits, spatial distractor-only subsets, and absolute/conspicuous analyses require nested subsets and parser-specific interpretations rather than a single closed decomposition.Figure 7 reports increased selection of expanded distractors, reduced key selection in the Options-only reverse subset, and near-zero text-identical controls; Question+Options modality results separately distinguish options, text, image, and full-input gains.Source: §B.3 · p. 16, Figure 7 · p. 17, Table 18 · p. 17

B.7 Additional Observational Text Signals

• Observational screens identify lexical-clang and length-as-unique-longest-by-words associations, with canonical odds ratios of 1.97 and 1.33, respectively.These are observational screens rather than causal estimates and motivate controlled edits.Source: §B.5 · p. 21

B.5 Additional Text-or-Options Ablation Results

• Table 24 provides narrower item-level union diagnostics for MedXpertQA-MM, MMMU-HM, and MMMU-Pro-HM, defining text-or-options as cases solved by either setting.The analysis supplements, rather than replaces, the main modality-lattice analysis.Source: §B.5 · p. 21, Table 24 · p. 22

B.6 Controlled Editing Details

• The controlled-editing cohorts contain 306 length-gap, 101 spatial/prepositional, and 101 absolute/conspicuous pairs, each evaluated over 20 repeats per condition.Construction and validation follow the criteria in Appendix B.2.Source: §B.6 · p. 21

B.7 Additional Observational Text Signals

• The additional GPT-5.4 normal-setting audit examines lexical-clang, semantic, position, and length associations across 7,722 observed/source variants.This expands the observational text-signal audit beyond the specifically quantified examples.Source: §B.7 · p. 21

C.1 Image-Channel Audit Details

• Table 25 summarizes image- and context-side screening surfaces before paired interventions and clinical adjudication.The table functions as a pre-intervention audit of candidate image and context cues.Source: §C.1 · p. 22

C.2 Artificial Image-Hint Intervention Details

• The artificial image-hint repair funnel is reported in Table 26, with model- and family-level results in Tables 27 and 28.These tables organize the intervention results at progressively different levels of analysis.Source: §C.2 · p. 22
• Artificial-hint edits are retained only when they remove visible cues and remnants, preserve medically relevant content and question-answer semantics, avoid new artifacts, and pass safe-editability criteria.Cases that cannot be edited safely are discarded.Source: §C.2 · p. 22

C.3 Natural Clinical-Context Cue Details

• The natural-cue audit uses 302 paired cases spanning non-mutually-exclusive demographic, social, geographic, functional, reproductive, occupational, and socioeconomic contexts.Tables 29–31 report model-, family-, and category-level results.Source: §C.3 · p. 22
• Natural-cue reliance patterns are mixed: GPT-5.4 drops while GPT-5.4-mini improves, Qwen3.5 shows the strongest harmful reliance, and Qwen3-VL often mentions cues without harmful conversion.One MedGemma configuration has high cue mention but only a weak aggregate drop.Source: §C.3 · p. 22
• Natural-context counts are not a demographic-bias benchmark; cues are problematic only when nonessential to the intended image-grounded construct yet contribute to answer selection.The analysis targets shortcut-like use within a selected neutralization set.Source: §C.3 · p. 22

C.4 Visible Image-Text Leakage Details

• The visible-image-text audit screens answer-overlapping text inside evaluation pixels, excludes external text sources, and applies GPT-5.4 adjudication plus repairability review.The dedicated funnel is reported in Table 32, and its 65 cases are distinct from the 60 high-risk cases in Table 25.Source: §C.4 · p. 24, Table 32 · p. 24
• Only a small subset of visible-text cases is directly repairable; answer-revealing text should be masked, cropped, or relabeled when validity is preserved, otherwise the item should be removed or redesigned.Task-essential labels or text often cannot be edited safely without changing the item’s medical validity or structure.Source: §C.4 · p. 24

C.5 Device and Material Cue Details

• Confirmed device/material shortcuts are rare but can reveal treatment history instead of intended diagnostic evidence, so they are not treated as aggregate performance results.Because removing implants, tubes, coils, hardware, shunts, filters, or instruments can alter the image or create artifacts, unsafe cases should be excluded from the shortcut-mitigated benchmark.Source: §C.5 · p. 24

D.1 MEDQA-MM Benchmark Details

• MEDQA-MM contains 1,000 hard, image-dependent examples after confirmed device/material cases were removed, but remains shortcut-mitigated rather than shortcut-free.Full-input accuracy is summarized in Figure 8 by base-model series, averaging configuration-level accuracies when multiple modes were evaluated; Qwen3.5 and Qwen3.6 remain distinct series (Qwen Team, 2026a,b).Source: §D.1 · p. 25

D.2 MEDQA-MM Error-Analysis Protocol

• The error analysis sampled 300 incorrect predictions across GPT-5.4, MedGemma-27B, and Qwen3.5-122B-A10B, with 92% agreement and Cohen’s κ = 0.703 between an LLM screen and expert review.The analysis examined whether stem content overrode decisive image evidence, but neither sample identifies a single cause of the aggregate Image+Options–Full difference.Source: §D.2 · p. 25
• In a 70-error development split, experts labeled 29 item/data issues, 21 reasoning/integration failures, 15 visual failures, 3 response/scoring failures, and 2 shortcut/anchoring cases.“Visual or reasoning/integration” is the derived subtotal 15+21 = 36, not an additional or intersecting category; proportions describe the reviewed sample.Source: §D.2 · p. 25

D.3 Expert Annotation Protocol

• Expert validation reviewed 150 option-form repairs, 50 natural-cue neutralizations, and 46 image-hint edits for cue relevance, medical validity, answer preservation, and mitigation quality.The repair-validation pool comprised five medical experts: two senior clinicians and three medically trained annotators.Source: §D.3 · p. 25

D.4 Validation and Release Status

• Release documentation distinguishes source fields from new audit metadata and records validation requirements without implying that redistribution permission has been obtained.Public release still requires source- and item-level validation beyond sample-level expert annotation.Source: §D.4 · p. 27, Table 36 · p. 27, Table 37 · p. 27

E Prompt Templates

• The prompt-template appendix documents operational instructions for shortcut detection, adjudication, repair, quality control, visible image text, devices/materials, option wording, and SDOH audits.The templates are included because small wording changes affect repair admissibility and shortcut adjudication.Source: E Prompt Templates · p. 27

Visible Image Text Audit

• The visible-text audit separates rendered image text from stem, metadata, and answer-option leakage, distinguishing actionable shortcut text from legitimate task evidence.It records visible text and location/type, while treating labels or annotations as shortcuts only when they directly name or strongly cue the correct answer.Source: Visible Image Text Audit · p. 28

Treatment Device/Material Shortcut Audit

• The device/material audit classifies cues as intended, redundant, false-positive, borderline, or confirmed shortcuts according to whether weak options let them identify or strongly favor the correct answer.Stem-only device context is not an image shortcut, and ordinary clinical evidence is preserved as intended evidence unless the cue itself reveals the answer.Source: Treatment Device/Material Shortcut Audit · pp. 29–30

Option-Text Shortcut Repair

• The option-repair prompt makes minimal distractor edits while preserving the correct answer, question stem, option order, medical meaning, and intended difficulty.Repairs must avoid duplicates, new absolute shortcut terms, and changes to core clinical attributes; unsafe edits receive no_safe_edit or needs_human_review.Source: Option-Text Shortcut Repair · p. 30
• Option-text repairs minimally modify distractors or weak wording while preserving medical meaning, answer keys, option structure, and distractor wrongness.The protocol prioritizes safe distractor parallelization and leaves content-bearing diagnosis, anatomy, route, dose, threshold, and timing cues unchanged when editing would risk meaning.Source: Option-Text Shortcut Repair · pp. 32–34
• Length repairs make a distractor at least 5 visible characters longer than a uniquely longest correct option, changing at most two distractors or returning no_safe_edit.The question stem, correct option, option order, labels, and option count remain fixed; edits must preserve each distractor’s original wrongness and answer type.Source: Option-Text Shortcut Repair · p. 32
• Spatial and prepositional repairs scan all options, preferably parallelizing one or two distractors without inventing anatomy, locations, routes, targets, timing, or other unsupported content.Correct-option edits are reserved for weak, non-content-bearing wording; unsafe or trivial changes produce cannot_modify or require review.Source: Option-Text Shortcut Repair · pp. 33–34

Artificial Visual Hint Localization and Image Repair

• Artificial-hint repair localizes and removes only added arrows, circles, boxes, outlines, dots, overlays, or marker lines, preserving findings, anatomy, devices, measurements, labels, and natural image appearance.Masked editing removes all cue remnants while reconstructing only pixels occluded by the artificial mark and preserving unmasked pixels exactly.Source: Artificial Visual Hint Localization… · p. 35
• Image repairs remove only masked artificial cues while preserving anatomy, pathology, devices, labels, scale bars, and diagnostic patterns.Repairs use local texture and modality matching, add no new markup, and leave unmasked pixels unchanged.Source: Artificial Visual Hint Localization… · pp. 36–37
• Repaired candidates are judged against the original and mask for complete cue removal, medical-content preservation, and absence of new artifacts.A candidate passes only when it is usable for downstream model evaluation.Source: Artificial Visual Hint Localization… · p. 37

Sociodemographic/SDOH Shortcut Pipeline

• The SDOH audit defines shortcuts as explicit social or demographic cues unnecessary for the intended answer but associated with or changing the selected option or reasoning.The guardrail prohibits inferring sensitive attributes from images and separates natural auditing from synthetic injection stress testing.Source: Sociodemographic/SDOH Shortcut Pipeline · p. 38
• The pipeline uses regex-hit judging, regex-miss auditing, and clinical-relevance review before neutralization or injection variants are considered.It rejects cues necessary for answering or entangled with standard of care, requires clinician review for nontrivial judgments, and permits injection only when answer and care remain invariant.Source: Sociodemographic/SDOH Shortcut Pipeline · pp. 38–41
• The sociodemographic pipeline preserves clinical content and answer validity while neutralizing nonessential social descriptors, marking inseparable cues for exclusion and requiring clinician review.It prohibits adding clinical assumptions or inferring protected identity from images; safe edits use minimal neutralization rather than rewriting the question, and the process remains provisional without clinician review.Source: Sociodemographic/SDOH Shortcut Pipeline · p. 41

F Extended Shortcut and Cue Taxonomy for Medical Multimodal MCQs

• The extended taxonomy treats cues as candidate risk surfaces rather than automatic exclusions, distinguishing cue presence from model use and legitimate clinical evidence from construct-irrelevant leakage.It covers question text, demographic and clinical context, visible image text, annotations, devices, acquisition artifacts, metadata, and dataset-level regularities, while focusing experiments on measurable, repairable subsets.Source: F Extended Shortcut and Cue Taxonomy… · p. 42, Table 38 · p. 43

A. Length, conspicuousness, and formatting

• The taxonomy audits option-form, semantic, ordering, stem-side, and question-text cues, with repairs that preserve medical meaning while reducing test-wise answer leakage.Examples include length and specificity gaps, absolute wording, lexical overlap, grammatical mismatch, spatial phrasing, nested options, and text-only diagnostic giveaways.Source: A. Length, conspicuousness, and… · pp. 42–43

F.2 Image-Side, Multimodal, and Dataset-Level Cues

• Image audits target visible text, figure metadata, overlays, watermarks, screenshots, and exposed filenames or alt text that may disclose diagnoses, anatomy, modality, or answers.Recommended mitigations include OCR or manual screening, masking nonessential information, cropping diagnostic regions, and stripping metadata while preserving interpretability.Source: §F.2 · p. 44

B. Artificial visual hints and editing artifacts

• Artificial visual hints include arrows, shapes, heatmaps, calipers, enhancement, editing artifacts, devices, procedural materials, brands, and care-setting context.These cues should be removed, balanced, or explicitly treated as annotation- or procedure-context tasks when they provide localization, severity, treatment-state, or setting information.Source: B. Artificial visual hints and editing… · p. 44
• Acquisition and source artifacts include modality, projection, sequence, scanner style, lesion-centered crops, salient objects, panel order, and image-count patterns that can narrow answer families.Mitigations include option compatibility checks, stratified or source-held-out validation, larger fields of view, randomized ordering, and metadata-only baselines.Source: B. Artificial visual hints and editing… · pp. 44–45
• Dataset and interface cues include class imbalance, templates, near duplicates, public memorization, formatting asymmetry, rationale leakage, option count, and translation artifacts.The checklist recommends majority baselines, source-stratified splits, deduplication, contamination searches, standardized rendering, strict field whitelists, and language balancing.Source: B. Artificial visual hints and editing… · p. 45

F.3 Evidence Tiers and Recommended Handling

• The audit dictionary is a flexible decision framework rather than a blunt deletion list, because the same surface cue can change tier with evidence and repair validation.The broader taxonomy is intended as a reusable audit checklist, while headline experiments emphasize option-form repairs, non-visual probes, visual text, annotations, devices, clinical-context neutralization, and dataset construction.Source: §F.3 · p. 45
• Cue handling progresses from candidate cue and shortcut-risk cue to behaviorally useful cue, validated repair, unsafe repair, clinically relevant cue, or residual risk.The tiers distinguish cue presence from model use and validity, requiring expert judgment, behavioral evidence, repair validation, or explicit risk flags before stronger claims.Source: §F.3 · p. 45

G Representative Case Studies

• Representative cases illustrate the aggregate shortcut taxonomy, but cases lacking controlled post-edit evaluation remain qualitative rather than headline quantitative evidence.Table 39 summarizes the evidence status of the representative cases.Source: G Representative Case Studies · p. 46, Table 39 · p. 46

Case 1: Race/ethnicity cue shifts diagnosis

• A single added identity cue flipped the model from the image-grounded diagnosis to a race-linked distractor, despite no change to the image.The original answer was E, whereas adding “The patient self-identifies as Black” produced D, with the model invoking increased pretest probability.Source: Case 1: Race/ethnicity cue shifts… · p. 46

Case 2: SES/access cue shifts to tuberculosis

• An uninsured-status descriptor shifted the model from sarcoidosis to tuberculosis, replacing the visual-clinical rationale with socioeconomic access as a disease prior.The short original stem made the option list and added descriptor disproportionately influential.Source: Case 2: SES/access cue shifts to… · p. 47

Case 3: Language cue acts as a geography proxy

• A Spanish-interpreter cue caused a flip from echinococcosis to amebic liver abscess by acting as an unsupported geography or endemicity proxy.No travel, immigration, animal-exposure, or sanitation history was added.Source: Case 3: Language cue acts as a… · p. 47

Case 4: Orthopedic hardware identifies the keyed figure

• Orthopedic hardware alone selected the keyed radiograph after clinical context was removed, making treatment stage a shortcut for severe osteogenesis imperfecta.The Image+Options input changed the answer from E to B because Figure D contained a visible intramedullary rod.Source: Case 4: Orthopedic hardware identifies… · p. 48

Case 5: Endovascular material narrows diagnosis

• Visible endovascular material supplied a non-redundant vascular-lesion cue that could exclude competing diagnoses, but this remained qualitative evidence without controlled removal or re-evaluation.The audit therefore does not establish a controlled performance effect.Source: Case 5: Endovascular material narrows… · p. 48

Case 6: Brand text identifies the device

• Embedded brand text can identify a device more directly than morphology, converting visual recognition into text lookup, although the device itself is legitimate evidence.The image contained the IVC-filter brand “OPTEASE”; the Image+Options answer was correct, while the text-only ablation was wrong.Source: Case 6: Brand text identifies the device · p. 49

G.3 Text Inside Images

• An anatomy label and an embedded comparison table demonstrate distinct image-text leaks, with the table exactly naming the gold answer while the label entered an ultimately incorrect rationale.The anatomy-label repair removed the leak but also removed localization support; the table case was correct with Image+Options and wrong text-only.Source: Case 7: Anatomy label overlaps the… · p. 49, Case 8: Embedded table names the… · p. 50

G.4 Image-Hint Repair

• The image-hint repair workflow identifies artificial visual cues, masks their regions, generates repaired candidates, and selects one preserving medical content without the shortcut.Figure 9 presents this representative repair process.Source: §G.4 · p. 50, Figure 9 · p. 51