PaperlayerView exact revision on arXiv

Source-linked AI summary

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Qiji Zhou, Junyi Hu, Peiyu Wang, Yue Zhang

arXiv:2610.12981v1cs.CVcs.AI

TL;DR

Visual interventions for multimodal LLMs may lose influence at some layers, and counterfactual subtraction can help or harm depending on the preference removed. This paper measures edit leverage across depth and evaluates calibrated selective routing, which improves attribute accuracy where universal subtraction is neutral or harmful.

  • Problem

    The paper asks where visual interventions can still change multimodal LLM answers and when those changes improve predictions.

    Source: §1 · pp. 1–2

  • Method

    The paper measures equal-RMS visual-residual edits across decoder layers and evaluates calibrated routing of counterfactual subtraction across question types.

    Source: §1 · p. 2, §3.1 · p. 3, §4.1 · p. 5

  • Results

    Routing improves full-set attribute accuracy by 1.56, 1.81, and 1.67 points across three checkpoints, while universal subtraction changes it by −0.42, −4.92, and −0.33 points.

    Source: §6.1 · p. 8

  • Takeaways & Limitations

    The findings support measuring a chosen edit’s influence at its intended depth and evaluating correction decisions on the target task before applying it.

    Source: §8 · p. 9

    The numerical results cover one boundary rule and one primary Q1 operator, and routing gains are established mainly for binary attribute claims.

    Source: §7 · p. 9

Abstract

from arXiv · show

Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the \textbf{Evidence Anchoring Boundary (EAB)}. With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are $13.78\times$--$26.10\times$ larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by $1.56$--$1.81$ percentage points across three checkpoints, while universal subtraction is neutral or harmful.

1 INTRODUCTION

The paper asks where visual interventions can still change an answer and when those changes improve it. It measures intervention leverage across decoder depth and evaluates selective subtraction across question types.

1 INTRODUCTION

  • The Evidence Anchoring Boundary marks the onset of a persistent low-effect region for an equal-RMS visual-residual edit across six checkpoints in three model families.The effect drop can precede stabilization of intermediate answer preferences.Source: §1 · p. 2
  • Routing improves full-set attribute accuracy by 1.56, 1.81, and 1.67 points, while universal subtraction changes it by −0.42, −4.92, and −0.33 points across three checkpoints.The router preserves the original prediction unless calibration supports an intervention for the question type.Source: §1 · p. 2
  • 13.78×–26.10× larger pre-EAB than post-EAB effects on disjoint queries show that the tested edit loses leverage across decoder depth.The boundary is selected on calibration queries and then frozen for the held-out comparison.Source: §1 · p. 2
  • Counterfactual subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer.The paper compares these outcomes across presence, attribute, and relation questions.Source: §1 · p. 2

2 RELATED WORK

Prior work studies training-free hallucination mitigation, visual intervention sites, intermediate predictions, and benchmark evaluation. This paper focuses on how a fixed visual edit’s influence changes across depth and when subtraction helps across question types.

2 RELATED WORK

  • Training-free methods modify visual inputs, intermediate representations, or decoding; examples compare original and distorted images or perturb instructions (Leng et al., 2024; Wang et al., 2024).Other approaches use masking, retrieval, counterfactual inputs, image-dependence measures, intermediate predictions, or decoding penalties (Liu et al., 2024; Chen et al., 2025; Li et al., 2026b; Favero et al., 2024; Wang et al., 2025a; Huang et al., 2024; Li et al., 2026a).Source: §2 · pp. 2–3
  • Visual-localization studies use attention, gradients, or decoder windows to identify intervention sites, whereas this work measures one fixed edit’s final-output effect across depth (Zhang et al., 2025; Chen et al., 2026a).The comparison distinguishes locating promising sites from testing how a specified operator’s influence persists.Source: §2 · p. 3
  • Logit lenses, tuned lenses, causal tracing, and activation patching study intermediate predictions or internal causal effects (Belrose et al., 2023; Meng et al., 2022).This work compares direct-edit effects with answer-readout stabilization; their separation means intermediate answer availability alone does not determine whether a particular edit remains effective.Source: §2 · p. 3
  • POPE, AMBER, HallusionBench, and CHAIR cover presence, attributes, relations, visually confounding questions, and object hallucination in generated descriptions (Li et al., 2023; Wang et al., 2023; Guan et al., 2024; Rohrbach et al., 2018).The paper separates claim-level comparisons, targeted generation, and transfer to distinguish binary decisions from generated answers.Source: §2 · p. 3

3 TASK AND EXPERIMENTAL SETUP

The setup uses binary visual claims to keep intervention effects and task accuracy distinct. It defines original and counterfactual answer scores, tests them across model and task combinations, and uses disjoint calibration and evaluation procedures.

3 TASK AND EXPERIMENTAL SETUP

  • Both experiments use binary visual claims: Q1 varies the layer of a fixed edit, while Q2 tests whether counterfactual subtraction improves the decision.The shared format keeps intervention effect and task accuracy distinct.Source: §3 · p. 3

3.1 TASK FORMULATION

  • A claim set contains claims associated with one image, each tested in the same Yes/No format for directly comparable answer scores.For attributes and relations, claims can be compared with alternatives about the same image.Source: §3.1 · p. 3
  • G(ci) subtracts the counterfactual Yes/No log-odds from the original log-odds, measuring the change associated with the visual-token intervention.The counterfactual preference may reflect the prompt, model knowledge, and information retained by the ablation; it is not assumed to isolate a pure language prior.Source: §3.1 · p. 3

3.2 MODELS, DATASETS, AND EVALUATION SCOPE

  • The evaluation uses LLaVA-1.5-7B/13B, Qwen2.5-VL-7B, and InternVL3.5-8B, with LLaVA-NeXT-34B and Qwen2.5-VL-32B added for Q1.Q1 uses 120 shared queries per checkpoint—48 calibration and 72 disjoint evaluation—while Q2 coverage varies by checkpoint.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 INTERVENTION AND METRICS

  • Q1 edits visual residuals at successive layers with equal RMS strength and measures final-output changes; Q2 mean-ablates visual-token representations to supply the preference removed by subtraction.Presence errors are assessed at matched original confidence, while attributes and relations use pair accuracy; the router is evaluated on the full attribute set without test-time access to the analysis-only hard-subset label.Source: §3.3 · p. 4

3.4 CALIBRATION, COMPARISONS, AND UNCERTAINTY

  • Calibration and evaluation use disjoint images, and five-fold image-disjoint cross-validation selects attribute routing rules without scoring an image with its own selected rule.Original scoring, universal subtraction, and routing are compared on the same images; paired image-cluster resampling preserves dependencies among claims from one image.Source: §3.4 · p. 4

4 Q1: WHERE CAN A VISUAL INTERVENTION MATTER?

The tested direct visual edit loses most of its influence after an operational boundary, even while intermediate answer preferences may continue to develop. This boundary is specific to the edit and rule studied, not a universal architectural transition.

4.1 A PERSISTENT DROP IN INTERVENTION EFFECT

  • EAB summarizes the effect trajectory of a fixed direct visual edit at a checkpoint, rather than a universal transition shared across interventions.Edits are matched for RMS strength; calibration selects the earliest qualifying layer after the effect peak, and the boundary is evaluated on disjoint queries.Source: §4.1 · pp. 4–5, §4 · p. 4
  • 13.78×–26.10× larger held-out effects occur before EAB than after it across six checkpoints under the frozen calibration rule.The boundary marks the onset of the edit’s operational low-effect region.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 SEPARATING INTERVENTION INFLUENCE FROM ANSWER STABILIZATION

  • Answer-readout reliability can continue to improve after EAB, so the tested visual edit can lose influence before answer preferences stabilize.The paper calls the readout stabilization point the Answer Commitment Point (ACP).Source: §4.2 · p. 5, Figure 2 · p. 5

5 Q2: WHEN SHOULD A VISUAL INTERVENTION BE USED?

Counterfactual subtraction can correct, preserve, or damage answer rankings depending on the question and the preference removed. The paper therefore develops calibration-based routing that defaults to the original score unless an intervention is supported for that question category.

5 Q2: WHEN SHOULD A VISUAL INTERVENTION BE USED?

  • Subtraction can correct a misleading preference, leave a ranking unchanged, or damage a correct answer by removing a task-relevant preference.Equal counterfactual scores leave the competing-claim difference unchanged.Source: §5 · p. 6, Figure 3 · p. 6

5.1 LIMITED BENEFIT FROM SUBTRACTION

  • Presence subtraction adds little consistent error discrimination beyond M, while inverse-spatial accuracy changes range from −2.3 to +0.4 points.The LLaVA-7B inverse-spatial decline is statistically resolved.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 WHEN THE REMOVED PREFERENCE IS MISLEADING OR TASK-RELEVANT

  • On hard attributes, subtraction improves pair accuracy by 13.44, 7.68, and 17.81 points, but full-set accuracy changes by −0.42, −4.92, and −0.33 points across checkpoints.For direct-contact relations, changes are −5.32 and −15.96 points on the two LLaVA checkpoints.Source: §5.2 · p. 6

5.3 CALIBRATED SELECTIVE ROUTING

  • Calibrated routing restricts eligible interventions by question category and uses calibration data to select an adjustment or retain M before evaluation.Evaluation labels and correctness are excluded; for presupposed questions, the query is neutralized and re-probed before scoring.Source: §5.3 · pp. 6–8, Figure 4 · p. 7
  • For attributes, calibration uses counterfactual-score spread, ranking disagreement, and sign consistency to select between preservation, partial subtraction, and subtraction.Presence defaults to M; relations retain M unless relation-specific calibration supports subtraction.Source: §5.3 · pp. 7–8

6 EVALUATION: BENEFITS, ABLATIONS, AND SCOPE

The evaluation tests attribute routing, presence-specific PEC, and whether these interventions remain useful beyond their calibrated settings. Selective routing improves attribute accuracy, while PEC’s gains do not consistently transfer across tasks.

6.1 ATTRIBUTE ROUTING AND TARGETED GENERATION

  • Routing improves full-set attribute accuracy by 1.56, 1.81, and 1.67 points on LLaVA-7B, LLaVA-13B, and InternVL, while universal subtraction changes it by −0.42, −4.92, and −0.33 points.All three routing gains have image-cluster confidence intervals above zero; targeted-generation hallucination-catch estimates increase by 2.96 and 5.07 points on the two LLaVA checkpoints, while false-positive changes are small and unresolved.Source: §6.1 · p. 8, Figure 5 · p. 8

6.2 PRESENCE-SPECIFIC INTERVENTION AND TRANSFER FAILURES

  • Frozen PEC improves POPE-adversarial accuracy by 3.30, 3.90, and 7.07 points, but its TextVQA and MME effects are negative or inconsistent across evaluated checkpoints.POPE-random changes are only +0.8/+0.8/−0.3 points, supporting restriction of PEC to its calibrated POPE-adversarial setting.Source: Figure 5 · p. 8, §6.2 · p. 9

6.3 SELECTION, CALIBRATION, AND FAILURE MODES

  • Table 3 summarizes controlled selection and task-specific calibration comparisons, which show that universal attribute subtraction loses routing gains and PEC can harm performance beyond its calibrated task.The comparisons support deciding where each correction is useful within its calibrated scope.Source: §6.3 · p. 9, Table 3 · p. 9

7 LIMITATIONS

The reported EAB and routing results have specific scope limits: they rely on one boundary rule and primary operator, and routing gains are established mainly for binary attributes.

7 LIMITATIONS

  • The numerical EAB results cover one boundary rule and one primary Q1 operator, so new models, prompts, operators, or task categories require recalibration.The 48/72 calibration/evaluation split supports the reported contrast, but a larger, more diverse sample is needed to estimate how precisely boundary location generalizes.Source: §7 · p. 9
  • Routing gains are established mainly for binary attribute claims; broader generation and faithful end-to-end comparisons with external decoding methods remain open.Source: §7 · p. 9
  • Additional counterfactual passes create compute cost, while wall-clock latency and FLOP overhead remain unmeasured.Source: §7 · p. 9

8 CONCLUSION

The paper finds that visual-edit influence can drop before answer-readout stabilization, and that calibrated selective routing improves attribute accuracy where universal subtraction is neutral or harmful. It argues for measuring operator influence and checking decisions on the target task before applying a visual correction.

8 CONCLUSION

  • The tested direct visual edit loses influence before answer-readout stabilization can occur, and calibrated selective routing improves attribute accuracy where universal subtraction is neutral or harmful.The paper recommends measuring the chosen operator’s influence at the intended depth and evaluating resulting decisions on the target task.Source: §8 · p. 9

AI USE STATEMENT

  • Generative AI assisted with manuscript drafting and revision and with writing and debugging some figure-generation code; authors reviewed the text and verified figures against experimental outputs.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study evaluates open-source MLLMs on public benchmarks, with no human-subject experiments, new sensitive-data collection, new dataset, or user-facing deployment.The authors restrict claims to evaluated models and tasks and emphasize calibrated, default-preserving use.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Appendix records map claims to evaluated model–task cells and document protocols, verification, extended outcomes, and paired significance tests to support reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL COVERAGE AND REPRODUCIBILITY

A.1 CHECKPOINT AND DATASET COVERAGE

The appendix details checkpoint and dataset coverage, intervention protocols, reproducibility records, and checkpoint-level EAB evidence. All six checkpoints show a held-out leverage drop, though absolute boundary locations vary across architectures.

A.1 CHECKPOINT AND DATASET COVERAGE

  • Table 4 lists the experiments used in Sections 4–6; Q1 uses 120 queries for all six checkpoints, while Q2 and deployment claims are limited to populated model–task cells.LLaVA-NeXT-34B and Qwen2.5-VL-32B are Q1-only scale replications.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 SPLITS, INTERVENTIONS, AND INFERENCE

  • EAB boundaries are fit on 48 calibration examples and frozen for 72 disjoint evaluation queries, with equal-RMS edits separating layer position from perturbation magnitude.For Q2, visual-token representations are mean-ablated with prompt and claim fixed; no evaluation image contributes to threshold selection.Source: §A.2 · p. 13

A.3 REPRODUCIBILITY CHECKLIST

  • The reproducibility checklist distinguishes environment control, experimental design, and artifact integrity, with frozen policies and records for verifying experiments and artifacts.Source: §A.3 · p. 13, Table 5 · p. 14

B CHECKPOINT-LEVEL EVIDENCE FOR EAB AND ACP

  • The checkpoint-level appendix expands the aggregate EAB result, separates depth changes, tests ACP as an irreversibility boundary, and distinguishes spatial localization from causal leverage.Source: B CHECKPOINT-LEVEL EVIDENCE FOR EAB… · p. 14

B.1 PER-CHECKPOINT EAB SUMMARY

  • Every checkpoint shows a large equal-RMS leverage drop on fully held-out queries, while absolute EAB locations vary across architectures.Table 6 reports calibration-frozen checkpoint boundaries and held-out effect ratios.Source: §B.1 · p. 14, Table 6 · p. 15

B.2 CHECKPOINT-SCALE COMPARISONS

The appendix examines how checkpoint scale, answer-readout stabilization, spatial localization, and boundary-rule choices qualify interpretation of the EAB. These audits distinguish the tested edit’s scope from broader architectural or correction claims.

B.2 CHECKPOINT-SCALE COMPARISONS

  • The Qwen2.5-VL 7B→32B audit shows a +22.64-layer EAB shift but an unresolved relative-depth shift; LLaVA-13B→NeXT-34B shifts later in relative depth.The latter comparison also changes model generation and language backbone, combining scale and architecture differences.Source: Table 7 · p. 15, §B.2 · p. 15

B.3 ACP AS AN ANSWER-READOUT COORDINATE

  • Readout reliability increases with depth, yet a controlled 96-triplet windowed edit corrects only two LLaVA-7B errors and none in three other models.ACP tracks readout stabilization; the intervention audit separately tests causal reach, limiting claims about a general correction window.Source: §B.3 · p. 15

B.4 SPATIAL LOCALIZATION AND INTERVENTION LEVERAGE

  • In one shared example per model, attention remains near the queried bowl at late depth after aggregate equal-RMS intervention effects have declined.The attention map locates attended regions; the intervention-effect measure tests leverage, and this example does not estimate prevalence.Source: §B.4 · p. 15, Figure 6 · p. 16

C EAB CONTROLS AND BOUNDARY INTERPRETATION

  • The appendix’s boundary interpretation depends on both the intervention operator and boundary rule; it does not imply a universal architectural phase transition.EAB characterizes the response to the tested direct visual-residual edit; equal RMS does not equate edit directions, targeted representations, or propagation paths.Source: C EAB CONTROLS AND BOUNDARY… · p. 16, §C.1 · p. 16

C.1 MATCHED PERTURBATION STRENGTH AND OPERATOR SCOPE

  • Equal-RMS matching controls perturbation norm across layers, but the six-checkpoint depth pattern applies to the specified Q1 edit and separately calibrated boundaries.The manuscript reports no quantitative cross-operator boundary agreement statistic; Q2 mean-ablation results do not establish shared EABs across intervention types.Source: §C.1 · p. 16

C.2 BOUNDARY LOCATION AND EFFECT-SIZE UNCERTAINTY

  • The frozen boundary rule uses a three-layer local mean and a remaining-suffix 75th percentile below 10% of the calibration peak; its held-out intervals do not quantify boundary-location uncertainty.The rule’s cutoff and persistence choices were not varied, and a smooth decay can satisfy it; distinguishing decay from a change point would require an unreported model comparison.Source: §C.2 · p. 16

D QUESTION-CATEGORY DIAGNOSTICS FOR Q2

  • The appendix separates attention-based localization and prompt-form diagnostics from the question of whether subtraction improves a decision.Its stated purpose is to test whether attention alone identifies presence errors or prompt form induces checkpoint-invariant correction.Source: D QUESTION-CATEGORY DIAGNOSTICS FOR Q2 · p. 16

D.1 AGGREGATE SPATIAL-ATTENTION DIAGNOSTIC

These diagnostics and extended evaluations distinguish attention and prompt effects from calibrated routing outcomes. They also show that frozen presence corrections do not transfer uniformly across tasks and checkpoints.

D.1 AGGREGATE SPATIAL-ATTENTION DIAGNOSTIC

  • Across the fixed 300-example-per-checkpoint attention audit, localization maps yield no stable correction rule for downstream routing across decision types and checkpoints.The study therefore excludes spatial attention from router inputs.Source: Figure 7 · p. 17, §D.1 · p. 17

D.2 PROMPT PRESUPPOSITION IS CHECKPOINT DEPENDENT

  • Presupposition changes false-Yes rates by +5.3 to +11.3 points in two LLaVA checkpoints and Qwen, but by −10.7 points in InternVL.Neutralization is strong for some checkpoints and nearly null for Qwen, motivating neutralization followed by re-probing.Source: Figure 8 · p. 17, §D.2 · p. 17

E ROUTER CALIBRATION AND ROUTING PROTOCOL

  • The router uses image-disjoint calibration, assigns question categories independently of model outputs, neutralizes presupposed presence wording, then re-probes before selecting an action.For attributes, each image receives a prediction from a rule selected without that image; PEC settings are fixed on calibration data before evaluation and transfer.Source: E ROUTER CALIBRATION AND ROUTING… · p. 17, §E.1 · p. 18, §E.2 · p. 18

E.4 INFERENCE COMPUTATION

  • For K claims, basic subtraction requires 2K score evaluations versus K for the original model, but the paper reports no wall-clock comparison.Actual runtime depends on batching, prefix reuse, visual-token count, and prompt re-probing; preserving the original score does not recover computation already spent on the counterfactual.Source: §E.4 · p. 18

F.1 DETAILED ATTRIBUTE-ROUTING OUTCOMES

  • Routed scoring improves full-set held-out attribute accuracy while global subtraction is neutral or harmful; Figure 9 also separates targeted-generation failure modes.The figure expands absolute pair accuracies and reports hallucination catch, false-positive rate, and residual hallucination for two checkpoints.Source: §F.1 · p. 18, Figure 9 · p. 19

F.2 PEC OPERATING RANGE AND OUT-OF-DOMAIN TRANSFER

  • Figure 10’s frozen PEC configuration loses TextVQA accuracy on three of four checkpoints, while full MME changes range from −9.9 to +2.5 points.On POPE-random, accuracy changes are +0.8, +0.8, and −0.3 points for LLaVA-7B, LLaVA-13B, and InternVL; transfer uses no task-specific retuning.Source: Figure 10 · p. 19, §F.2 · p. 19
  • Frozen MME category effects vary: LLaVA-7B loses most on OCR, counting, celebrity, and poster recognition but improves on numerical calculation and position.LLaVA-13B improves in several categories but loses on posters and landmarks; InternVL has substantial celebrity and code-reasoning losses.Source: §F.2 · p. 19, Figure 11 · p. 20

F.3 PAIRED SIGNIFICANCE AUDIT

The appendix reports image-cluster-aware paired audits, bounds secondary-baseline conclusions to matched scoring, and uses examples to illustrate intervention outcomes rather than estimate their prevalence.

F.3 PAIRED SIGNIFICANCE AUDIT

  • POPE accuracy gains were +3.30 points on LLaVA-1.5-7B, +3.90 on LLaVA-1.5-13B, and +7.07 on InternVL3.5-8B.The audit uses image-cluster-aware inference; the LLaVA-7B hallucination-catch increase had a positive bootstrap interval but remained unresolved by the paired test (p = .1314).Source: Table 8 · p. 20, §F.3 · pp. 20–21
  • TVQA presence subtraction ranges from −1.93 points on Qwen2.5-VL-7B to +0.07 on InternVL3.5-8B, while inverse-spatial effects also vary by checkpoint.Source: §F.3 · p. 21

F.4 SECONDARY BASELINE SCOPE

  • ICD and DeCo audits use matched one-token scoring formulas, so competitiveness under their full decoding control flow and stopping rules remains untested.Source: §F.4 · p. 21

G.1 ATTRIBUTE RESCUE, DAMAGE, AND PRESERVATION

  • Three cached LLaVA-7B AMBER pairs illustrate subtraction rescuing a mountain decision, damaging a correct desert decision, and preserving the sky ranking; they are not prevalence estimates.The removed preference explains each example’s outcome.Source: §G.1 · p. 21, Figure 12 · p. 22

G.2 PRESENCE LOCALIZATION CASES

  • The presence examples distinguish grounded evidence, borrowed evidence, and localization misses; PEC changes logits, not the displayed attention map, and uses no attention features.Source: §G.2 · p. 22, Figure 13 · p. 23
Loading 2610.12981v1…