PaperlayerView exact revision on arXiv

Source-linked AI summary

When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models

Qizhen Lan, Yiling Ma, Ruiqi Chen, Mengchen Fan, Xi Xiao, Lin Zhao, Liang He, Baocheng Geng, Pulin Che, Tianyang Wang, Xiaoqian Jiang

arXiv:2610.23469v1cs.CV

TL;DR

Compact medical VLMs may answer correctly through language priors, while standard distillation does not test whether answer support depends on required visual evidence. ECDD distills the teacher’s margin change across matched original and evidence-absent inputs. Across evaluated settings it improves evidence-specific responses while retaining competitive accuracy; on DME, the positive-case target-minus-random response is 2.023 with ECDD versus 0.786 with matched KD.

  • Problem

    Original-image accuracy cannot establish whether medical VLM answers depend on the visual evidence required by the question, since language priors can support correct answers.

    Source: §1 · p. 1

  • Method

    ECDD pairs each image with a control matched to the evidence level and distills the teacher’s margin drop using a lower-bound objective alongside original-image supervision.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    2.023 with ECDD versus 0.786 with matched KD for the DME positive-case target-minus-random response; across evidence levels, ECDD improves evidence-specific responses while retaining competitive original-image accuracy.

    Source: §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Original-image accuracy and evidence dependency describe different aspects of behavior, and ECDD changes training without changing the student’s architecture, parameter count, or input at deployment.

    Source: §5 · p. 9

    The comparisons do not include ordinary KD applied to both original and evidence-removed inputs, leaving that matched-input comparison open.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Compact medical vision-language models (VLMs) can answer clinical questions correctly without reliably using the image, relying instead on question wording or learned priors. Standard knowledge distillation copies a teacher's answer preference on the original image, but it leaves unchecked whether that preference is conditional on the visual evidence required by the question. We study this missing target as medical visual evidence dependency: support for the correct answer should decrease when the matched evidence is absent. We introduce Evidence-Conditioned Dependency Distillation (ECDD), which pairs each training image with a control matched to the evidence level and distills the teacher's drop in answer margin across the pair. A lower bound objective transfers positive dependency by penalizing only shortfalls, while original-image supervision preserves task accuracy. The same formulation applies when the evidence is a queried clinical region, a surgical instrument, or whole-image visual content. Across three medical VQA settings, ECDD keeps original-image accuracy competitive with distillation and alignment baselines while making answer support more sensitive to evidence removal. Under the DME headline objective, for example, the positive-case margin response to target rather than random masking rises from 0.786 with KD to 2.023 with ECDD, while original-image accuracy differs by 0.11 points. These results suggest that medical VLM distillation should transfer both the teacher's answer preference and its dependence on visual evidence.

1. Introduction

Compact medical VLMs can answer correctly using language priors, so original-image accuracy alone does not show whether answers depend on the required visual evidence. ECDD distills how answer support changes across matched original and evidence-absent inputs, while retaining original-image supervision.

1. Introduction

  • VQA accuracy can reflect clinically plausible language priors, leaving nearly unchanged answer support even after the relevant visual evidence is removed.The proposed dependency property is that answer support should decrease when question-required evidence is removed; the matched control keeps the question, prompt, and candidate answers fixed.Source: §1 · p. 1
  • Existing answer distillation and alignment methods supervise predictions or representations on the observed image, not how answer support changes at an evidence-absent endpoint.The paper identifies a second gap: the dependency target must match whether the question depends on a whole image, a lesion region, or a task-specific visual object.Source: §1 · p. 2
  • ECDD transfers the teacher’s margin drop between an original image and a control matched to the question’s evidence level, using a lower-bound constraint while retaining original-image supervision.The formulation covers global image controls, question-specified region masks, and task-specific visual evidence masks.Source: §1 · p. 2
  • Across localized clinical evidence, task-specific visual objects, and global controls, ECDD improves evidence-dependency diagnostics while maintaining competitive original-image accuracy.The paper frames this as separating answer correctness from whether support remains conditional on the required evidence.Source: §1 · p. 2

2. Related Work

Medical VQA accuracy benchmarks do not establish whether answers depend on the visual evidence required by a question. ECDD draws on counterfactual evaluation but distills a matched, cross-condition margin change rather than a static answer distribution or generic debiasing weight.

2. Related Work

  • Medical VQA benchmarks measure answer accuracy, while evidence-removal controls matched to the queried level test dependence on a whole image, ROI, or instrument pixels.The described evaluation uses ROI masking for DME, instrument-pixel masking for RoboSurg-VQA, and global evidence-absent controls for PathVQA.Source: §2 · pp. 2–3
  • Counterfactual VQA protocols probe predictions when images are irrelevant, degraded, shuffled, or absent; ECDD instead transfers the teacher’s margin change across a question-matched medical-evidence contrast.Unlike post hoc probing or reweighting by unimodal bias, the target is cross-endpoint dependency rather than a static answer distribution or generic debiasing weight.Source: §2 · p. 3
  • Prior compact and medical VLM distillation methods transfer distributions, representations, alignment, or response structure under a given input condition; ECDD distills a cross-condition evidence-removal functional.That functional is the change in the gold-answer versus hardest-negative margin between the original image and a matched evidence-absent control.Source: §2 · p. 3

3. Method

ECDD extends compact medical VLM distillation from matching answers on the original image to transferring how answer support changes when question-relevant visual evidence is removed. It uses matched image-control pairs and a lower-bound dependency loss alongside original-image supervision.

3.1. Problem setting and candidate scoring

  • Closed-set medical VQA uses a fixed candidate set, and ECDD compares gold-answer support with the strongest wrong candidate rather than matching raw margin scales.The same comparison covers binary and multiple-choice questions; margin drops use a scaled lower bound because teacher and student score scales need not be calibrated.Source: §3.1 · p. 3

3.2. Evidence contrast matched to the queried level

  • Evidence controls preserve language inputs and remove only the visual evidence required by the question, with the control matched to the queried evidence level.Controls can target global visual content, a question-specified retinal region, or a task-specific object such as surgical-instrument pixels.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Evidence dependency margin

  • The dependency measure is the original-to-control drop in the margin between the gold answer and the strongest wrong answer.A positive drop means removing matched evidence lowers relative support for the gold answer; a value near zero indicates similar support at both endpoints.Source: §3.3 · p. 4

3.4. Why pointwise distillation misses dependency

  • Pointwise distillation observes only the original image, so it cannot specify the student’s response to evidence removal even when original-image observables match.Students can agree on the original-image observable yet have different masked-image margins and therefore different dependency.Source: §3.4 · pp. 4–5

3.5. Lower-bound ECDD loss

  • ECDD treats positive teacher dependency as a lower-bound target and penalizes only when the student falls short of the scaled target.It neither forces the student to copy every teacher fluctuation nor penalizes a student whose margin drop already exceeds the target.Source: §3.5 · p. 5

3.6. Overall objective and training instantiation

  • ECDD trains on the original image and an evidence-matched control, transferring the teacher’s margin drop while retaining original-image task learning and answer-distribution distillation.Teacher targets are cached before training; the student is scored on both endpoints during training, but inference uses only the original image.Source: §3 · p. 3, §3.6 · p. 5

4. Experiments and Results

Experiments test whether evidence-matched controls increase answer-support dependency while preserving original-image accuracy across lesion, instrument, and whole-image VQA. Comparisons use compact students, matched training baselines, Top-1 accuracy, and evidence-removal diagnostics.

4.2. Main Results

  • Across three evidence levels, ECDD improves mean dependency response over original-image KD with comparable or higher mean original accuracy across seeds 7/17/27.PathVQA uses a staged recipe, so its gains do not isolate the dependency loss from training schedule and reference regularization.Source: §4.2 · p. 6
  • On Qwen-3B DME lesion presence, target-minus-random response rises from 0.786 with matched KD to 2.023 with ECDD, while accuracy is 80.50% versus 80.39%.The accuracy difference is 0.11 points.Source: §4.2 · p. 6, Table 1 · p. 7
  • On RoboSurg-VQA, ECDD matches the highest original accuracy (73.00%) and leads on instrument gap (39.33), positive margin drop (1.811), and instrument-minus-random response (0.738).No baseline matches this combination.Source: §4.2 · pp. 6, 8, Table 2 · p. 7
  • On PathVQA, ECDD reaches 89.35% original accuracy, a 20.26-point VEA gap, and ∆mVEA = 3.863, improving strongest non-ECDD diagnostics by 14.0 points and 3.11 margin units.The PathVQA model uses a staged recipe with an additional training epoch and reference regularization.Source: Table 3 · p. 7, §4.2 · p. 8

4.3. Behavior under Question Priors

  • On the question-only-error subset, ECDD exceeds SFT, KD, and DBiLD by 4.51, 3.69, and 6.56 points, respectively.This result is for the complete staged model on unmodified images and does not isolate the dependency loss.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Ablation under the Headline Objective

  • Under the fixed DME headline objective, ECDD gives the largest specific response within 0.11 accuracy points of matched KD; changing its dependency target or endpoint lowers target-specific response.The ablations hold task objective, optimization rule, and training budget fixed.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Evidence Specificity and Intervention Transfer

  • Across teacher-dependency tertiles, ECDD’s target-ROI margin drop is 1.56/1.97/2.68 versus KD’s 0.50/0.77/1.24; its response also transfers to DME Telea inpainting.With Telea inpainting, ECDD retains KD-level original accuracy, increases target-specific response, and keeps matched-random response near zero.Source: §4.5 · p. 8, Figure 4 · p. 8, Table 6 · p. 8

4.6. Limitations

  • The controls measure dependency under specified interventions, not clinical robustness or causal grounding; teacher filtering also does not remove all teacher biases or intervention artifacts.RoboSurg-VQA has smaller dependency gains and substantial random-mask accuracy effects; PathVQA gains are confounded by its staged recipe, and ordinary KD on paired original and evidence-removed inputs remains untested.Source: §4.2 · p. 6, §4.6 · pp. 8–9

5. Conclusion

ECDD trains compact medical VLMs to transfer how answer support changes under matched evidence interventions, not only the teacher’s original-image answer preference. Across lesion, instrument, and global controls, it improves evidence-specific responses while retaining competitive accuracy.

5. Conclusion

  • Across lesion, instrument, and global controls, ECDD improves evidence-specific responses while retaining competitive accuracy.In localized tasks, target regions produce a larger response than matched random regions, concentrating the effect on question-relevant evidence.Source: §5 · p. 9
  • Accuracy and evidence dependency capture different behaviors: whether the answer is correct and how its support responds when relevant evidence is removed.The paired controls measure the latter behavior alongside original-image accuracy.Source: §5 · p. 9
  • ECDD changes training rather than deployment, retaining the same architecture, parameter count, and input as KD.The paper concludes that compact medical VLM distillation should transfer both answer preference and dependence on visual evidence.Source: §5 · p. 9

A. Dataset Details

The dataset experiments probe global visual evidence, question-specified lesions, and surgical instruments using controls matched to each evidence level. Localized ECDD uses region or object annotations, whose collection cost is not estimated.

A.1. PathVQA: Global Visual Evidence

  • PathVQA uses yes/no questions with fixed “Yes” and “No” candidates, retaining 9,751 training and 3,362 test examples; open-answer questions are excluded.Its visual-evidence-absent endpoint replaces the image with a same-size blank while keeping the question, candidates, and gold answer fixed.Source: §A.1 · p. 11

A.2. DME VQA: Region-Level Lesion Evidence

  • DME uses closed-form questions about hard exudates in a specified region, yielding 8,498 training and 918 test examples after candidate conversion.Each test item compares the original image, the question-specified ROI mask, and a same-area random mask to assess specificity to the queried region.Source: §A.2 · p. 11

A.3. RoboSurg-VQA: Scene-Local Instrument Evidence

  • RoboSurg-VQA evaluates a closed yes/no question about instrument occlusion at the frame center, using instrument segmentation and same-area random and instrument-noise controls.The experiment uses 1,800 training examples from Site1 videos 1–8 and 300 held-out examples from video10; video9 is omitted because its public preprocessed archive is incomplete for that split.Source: §A.3 · p. 11

A.4. Annotation and Intervention Requirements

  • Localized ECDD assumes relevant region or object annotations, and the experiments do not estimate the cost of collecting them in a new deployment setting.DME blur and Telea inpainting are held out from training, while random masks serve as specificity controls.Source: §A.4 · p. 11, Table A1 · p. 11

B. Metric Definitions

The metrics compare original-image task behavior with answer-support changes under evidence-removal controls. The accompanying training settings specify dataset-specific recipes, while paired DME bootstrap comparisons assess dependency gains alongside accuracy preservation.

B.1. Candidate and Endpoint Scoring

  • Candidate probabilities are normalized over the fixed candidate set, and original and control endpoints differ only in visual input.The scoring uses option-token logits at the first answer position and log probabilities as margins.Source: §B.1 · p. 11

B.2. Accuracy-Based Diagnostics

  • Control accuracy retains the original label as a behavioral diagnostic, and its gap from original-image accuracy is meaningful only when original accuracy is preserved.The control image is not assigned a new clinical ground truth.Source: §B.2 · p. 11

B.3. Margin-Based Diagnostics

  • Margin-based dependency is the original gold-answer margin minus its margin under the control; lower retained-margin ratios mean less original answer support survives intervention.The DME paired-bootstrap table reports ECDD-minus-DBiLD deltas, using accuracy for preservation and dependency diagnostics for the larger positive deltas.Source: §B.3 · p. 11, Table A2 · p. 12

B.4. Localized and Multi-Control Response

  • Localized response metrics contrast target-evidence margin drops with matched random or distractor controls, separating response to queried evidence from masking in general.For presence tasks, the positive-case average is computed on examples whose original answer indicates the evidence is present; PathVQA multi-control averages test specificity to a blank endpoint.Source: §B.4 · p. 12

B.5. Calibration and Paired Uncertainty

  • Calibration scores use original-image candidate probabilities rather than serving as evidence-removal metrics; paired DME bootstrap intervals are positive for both dependency diagnostics despite a small accuracy difference.The bootstrap compares ECDD with DBiLD on the same 918 test examples.Source: §B.5 · p. 12

C.1. Training Settings

  • Adapted models use LoRA with a frozen vision encoder, while PathVQA uses an SFT stage followed by an ECDD continuation with a reference KL term.That staged PathVQA recipe differs from the one-stage baselines, which do not receive the extra epoch or reference term.Source: §C.1 · p. 12

C.2. DME Training Configuration

  • The Qwen-3B DME headline model jointly optimizes task supervision, answer-level distillation, and dependency transfer; every example retains original-image task supervision.For localized presence questions, ECDD is active when the queried evidence is present and the teacher supplies a reliable positive dependency target.Source: §C.2 · pp. 12–13

C.3. Baseline Training and Computational Cost

The added baselines share the student and teacher backbones, candidate-scoring interface, original images, and cached teacher outputs. ECDD adds a paired control endpoint and dependency trace, increasing training cost but not inference overhead.

C.3. Baseline Training and Computational Cost

  • MedAlign, A3Tune, and Beta-KD use the same student and teacher backbones, candidate-scoring interface, original images, and cached teacher outputs as ECDD.ECDD adds the paired control endpoint and dependency trace.Source: §C.3 · p. 13
  • ECDD requires 529 ms/example and 12.29 GiB peak memory, versus 286 ms/example and 9.91 GiB for matched KD.The profile uses the same H100 and excludes one-time teacher-trace generation; ECDD requires 1.85× the student-training time.Source: §C.3 · p. 13
  • At inference, ECDD uses the same student architecture, parameter count, and input as KD, so it adds no inference overhead.The extra computation is incurred during student training.Source: §C.3 · p. 13

D. DME Mechanism Ablations under the Joint Objective

DME ablations support both the teacher-derived dependency lower bound and question-matched visual interventions. Additional tests show ECDD’s target-specific response across prediction distributions, dependency levels, and held-out removal operators.

D. DME Mechanism Ablations under the Joint Objective

E.1. Endpoint Prediction Distributions

  • On blank images, all three models predict “No” for all 918 questions, yielding 58.50% accuracy, equal to negative-class prevalence (537/918).For ECDD, target masking reduces predicted-Yes to 5.45%, while same-area random masking leaves it at 48.04%.Source: §E.1 · p. 13

E.2. Dependency-Stratified Response

  • Across low-, middle-, and high-dependency groups, ECDD’s margin drops rise from 1.563 to 1.971 to 2.680, while matched KD’s rise from 0.496 to 0.765 to 1.244.The groups are tertiles of the teacher’s target-ROI margin drop among 381 positive cases.Source: §E.2 · p. 13

E.3. Held-Out ROI Blur

  • With held-out ROI blur, ECDD’s target response is 1.215 versus 0.692 for matched KD, while their same-area random responses both round to 0.049.The target-specific separation persists when target pixels are blurred rather than zeroed.Source: §E.3 · p. 13

E.4. Held-Out Content-Aware Inpainting

  • Under held-out Telea inpainting, ECDD’s response exceeds KD by +0.423 (95% CI [0.378, 0.467]), while its random-region response remains near zero (−0.033).The evaluation uses 381 positive questions in 23 image clusters and a shifted same-shape, same-area random mask.Source: §E.4 · p. 13

F. Robustness across Training Seeds

The robustness section evaluates fixed SFT, KD, and ECDD recipes across three seeds and reports paired ECDD–KD gains. The supplied passages specify table scopes and metric definitions, but not the table’s result values.

F. Robustness across Training Seeds

  • The robustness check repeats fixed SFT, KD, and ECDD training recipes with seeds 7, 17, and 27.The DME KD task objective is λKD = 1.0, matching ECDD with λECDD = 0.Source: F. Robustness across Training Seeds · pp. 13–14
  • The appendix also identifies endpoint prediction distributions, positive-case margin drops under DME controls, and transfer to held-out content-aware ROI inpainting as separate table scopes.Table A6 uses frozen checkpoints under Telea; its random control uses a shifted mask with the same geometry and area.Source: Table A6 · p. 14, Table A4 · p. 14, Table A5 · p. 14
  • Table A7 reports mean ± sample standard deviation of seed-wise paired ECDD–KD gains in dependency response and original-image accuracy.Dependency response is target-minus-random positive-case margin drop for DME and RoboSurg-VQA, and original-to-VEA margin drop for PathVQA.Source: F. Robustness across Training Seeds · p. 14

G. Single-Factor Sensitivity

The sensitivity analysis reports a gradual increase in ROI-specific response as the dependency weight rises, with accuracy close to the selected setting until the largest values. A separate PathVQA evaluation compares models on errors defined by a question-only model.

G. Single-Factor Sensitivity

  • Raising λECDD from 0 to 0.01 increases the response from 0.786 to 1.368 without changing 80.50% accuracy; at 0.02, response reaches 2.023 with a 0.11-point accuracy difference.Across τ ∈ [4, 12], accuracy varies by 0.76 points and response approaches a plateau at τ = 8–12.Source: G. Single-Factor Sensitivity · p. 14
  • Across three seeds, ECDD–KD dependency-response gains are +1.213 ± 0.054 on DME, +3.661 ± 0.241 on PathVQA, and +0.125 ± 0.026 on RoboSurg-VQA.Corresponding original-accuracy differences are 0.00±0.29, +2.93±0.51, and +0.33±0.67 points.Source: Table A7 · p. 14

H.1. Question-Only Prior-Conflict Evaluation

  • On the 488-error PathVQA prior-conflict subset defined by a question-only model, ECDD improves over SFT, KD, and DBiLD by 4.51, 3.69, and 6.56 points.On the same subset, it improves over MedAlign, A3Tune, and Beta-KD by 6.15, 12.09, and 4.51 points.Source: §H.1 · p. 14

H.2. Question-Type Analysis

The question-type analysis uses fixed text-based diagnostic buckets, while the multiple-control analysis compares retained gold-answer margin across visual degradation controls. Morphology/histology shows the clearest reported change in the Q-only gap.

H.2. Question-Type Analysis

  • In morphology/histology, the same-model Q-only gap rises from 4.04 to 24.03 points; Presence and Other show smaller changes.The diagnostic buckets use fixed mutually exclusive text rules and are not official PathVQA labels.Source: §H.2 · pp. 14–15, Table A8 · p. 15

H.3. Multiple Visual Controls

  • Across gray, noise, blank, and blur controls, mean retained margin is 0.286 for ECDD and 0.745 for SFT.The diagnostic also reports that ECDD withdraws more gold-answer support than Base, SFT, and the 32B teacher along the gray/noise degradation path.Source: §H.3 · p. 15, Figure A2 · p. 15

Related papers

Loading 2610.23469v1…