PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Mitigating Evidence Misgrounding in Multimodal Knowledge Editing

Zhiwu Wang, Yichen Wu, Jinqian Yang, Yizhou Yang, Renzhen Wang, Qian Zhao, Deyu Meng

arXiv:2610.33075v1cs.CVcs.AI

TL;DR

Answer-level metrics can miss multimodal edits that produce the target answer yet remain sensitive to incidental image context. The paper introduces WATERBIRD-EDIT to test this under controlled background shifts and proposes VICE, a retrieval-based, frozen-backbone editing framework using visual demonstrations. VICE shows strong lifelong editing performance and substantially reduces sensitivity to controlled background shifts.

  • Problem

    Standard answer-level metrics can miss evidence misgrounding: an edit may produce the target answer while remaining sensitive to incidental context rather than intended visual evidence.

    Source: §1 · pp. 1–2

  • Method

    WATERBIRD-EDIT separates bird-habitat labels from background cues, while VICE stores edits in memory and applies them through visual in-context demonstrations designed to expose task-relevant evidence.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    VICE achieves strong lifelong editing performance across benchmarks and substantially reduces sensitivity to controlled background shifts on WATERBIRD-EDIT.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support explicitly exposing target visual evidence when evaluating and applying multimodal edits.

    Source: §6 · p. 9

    WATERBIRD-EDIT is limited to controlled bird-habitat editing, and VICE relies on generated image summaries and learned retrieval-based scope decisions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal knowledge editing aims to update visually grounded facts in Multimodal Large Language Models (MLLMs) while preserving unrelated text and visual behaviors. However, standard answer-level metrics can hide evidence misgrounding: an edit may produce the target answer on the original image yet remain sensitive to incidental context rather than the intended visual evidence. To examine this issue in a controlled setting, we construct WaterBird-Edit, an analysis set that decouples bird habitat labels from background context and measures whether edited predictions remain stable when the target evidence is preserved but the background changes. Motivated by this diagnosis, we propose VICE (Visual In-Context Editing), a retrieval-based framework that stores edits in external memory and applies them through visual in-context demonstrations designed to expose task-relevant evidence. Experiments on three standard benchmarks show that VICE achieves strong lifelong editing performance, while results on WaterBird-Edit show that it substantially reduces sensitivity to the controlled background shifts. Code and dataset are available in https://github.com/Wang-Z-W/VICE-main.

1 Introduction

Multimodal editing must update visually grounded facts while preserving unrelated behavior, but answer-level metrics can miss edits that remain sensitive to incidental image context. The paper diagnoses this mismatch with WATERBIRD-EDIT and proposes VICE, which applies edits through evidence-focused visual demonstrations and memory.

1 Introduction

  • Answer-level success can coexist with predictions that vary when target evidence is preserved but incidental image context changes.Standard reliability, generality, and locality metrics can therefore overestimate edit quality by overlooking context-sensitive image generalization.Source: §1 · pp. 1–2
  • VICE freezes the MLLM backbone and stores each edited fact with evidence-summary demonstrations that separate task-relevant visual cues from incidental context.At inference, memory retrieval selects a relevant edit and a learned scope head decides whether to apply it or bypass it.Source: §1 · p. 2
  • WATERBIRD-EDIT exposes evidence misgrounding by decoupling bird habitat from background context; VICE reduces sensitivity to controlled background shifts while achieving strong lifelong editing performance across four benchmarks.The benchmarks are EVQA, EIC, VLKEB, and WATERBIRD-EDIT.Source: §1 · p. 2

2 Diagnosing Evidence Misgrounding

The section defines evidence misgrounding as context-sensitive predictions despite corrected answers, then introduces WATERBIRD-EDIT to diagnose it under controlled target-background shifts. The diagnosis motivates VICE’s frozen-backbone, demonstration-based approach to applying edits.

2.1 WATERBIRD-EDIT Diagnostic Analysis

  • WATERBIRD-EDIT tests whether edited predictions follow bird habitat rather than correlated background, using four target-context groups and same-species image-rephrase pairs.The set is built on CUB-200-2011 and retains 500 images on which the model initially predicts habitat incorrectly.Source: §2.1 · pp. 2–3

2.2 Context Sensitivity After Editing

  • Background changes shift pwater for fixed habitat targets, including |∆bg| values of 0.19 for FT on water targets and 0.16 for LIVEEDIT on land targets.Masking the background raises FT I-Gen. from 85.2 to 87.8 and LIVEEDIT I-Gen. from 83.8 to 92.4.Source: Figure 2 · p. 3, §2.2 · p. 3

2.1 WATERBIRD-EDIT Diagnostic Analysis

  • After 500 edits, the analysis measures I-Gen., group I-Gen., PG, and normalized pwater; object-only variants preserve the bird while masking background.PG is I-Gen. minus the minimum group I-Gen.; larger gaps indicate target-context sensitivity.Source: §2.1 · p. 3

2.2 Context Sensitivity After Editing

  • FT and LIVEEDIT attain reasonable full-set I-Gen. scores of 85.2 and 83.8, but their worst groups yield large performance gaps (PG) of 24.3 and 16.1, showing sensitivity to incidental background context.Source: §2.2 · p. 3
  • The diagnostics show that parameter updates and hidden-state modulation can achieve answer-level editing while predictions remain background-sensitive, motivating explicit visual evidence in VICE demonstrations.VICE keeps the backbone frozen and uses demonstrations that represent task-relevant evidence and incidental cues.Source: §2.2 · pp. 3–4

3 Method: VICE

VICE adapts retrieval-based in-context editing by adding image evidence summaries to demonstrations and storing edits in external memory. At inference, it retrieves a candidate edit and uses scope verification to decide whether to apply it.

3 Method: VICE

  • VICE retrieves a candidate edit, verifies its scope, and applies it through visual in-context demonstrations while keeping the MLLM backbone frozen.Each edit is stored with evidence-summary demonstrations that distinguish task-relevant cues from incidental context.Source: §3 · p. 4

3.1 Preliminary: In-Context Editing

  • IKE uses retrieved demonstrations to teach when to copy, update, or retain answers; VICE follows its retrieval protocol and adds an Image Summary field.The added field lets demonstrations expose textualized visual evidence alongside the New Fact and Prompt.Source: §3.1 · p. 4

3.2 Evidence-Summary Demonstrations

  • A frozen summary model generates an image summary that highlights object evidence relevant to the prompt and separates it from incidental visual attributes.The generation prompt tells the model not to answer directly, keeping the summary evidence-focused rather than restating the edit target.Source: §3.2 · p. 5

3.3 Lifelong Visual In-Context Editing

  • For lifelong editing, VICE writes each edit and its retrieved demonstrations to sequence memory without modifying model parameters or hidden states.At inference, a retrieval encoder proposes the most similar memory entry, and a thresholded scope score determines whether its edit is applied.Source: §3.3 · p. 5
  • Table 3 presents lifelong-editing results for LLaVA-v1.5 and MiniGPT4 on EVQA, EIC, and VLKEB; “#Edits” represents new facts, and MEND, SERAC, and LiveEdit results are from Chen et al. (2025).Source: Table 3 · p. 6

4 Experiments

The experiments compare VICE with representative editing methods across three benchmarks and two backbones, then test controlled background robustness, component ablations, and threshold sensitivity.

4.1 Experimental Setup

  • VICE is evaluated on EVQA, EIC, and VLKEB with LLaVA-v1.5-7B and MiniGPT-4-vicuna-7B, against five representative editing baselines using reliability, locality, and generality metrics.The reported baselines include FT-VL, TP, MEND, SERAC, and LiveEdit.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 12.7 on EVQA, 3.9 on EIC, and 14.0 on VLKEB are VICE’s I-Gen. gains over LIVEEDIT for LLaVA-v1.5 at 1000 edits.For MiniGPT4, the corresponding reported gains are 11.7 on EVQA and 6.9 on VLKEB. VICE remains more stable as edits accumulate, although LIVEEDIT remains competitive on text-generality in some settings.Source: §4.2 · p. 7

4.3 Robustness on WATERBIRD-EDIT

  • VICE achieves overall I-Gen. of 93.4 and PG of 4.7 on WATERBIRD-EDIT, with object-only I-Gen. of 89.0 when the background is removed.Its group accuracies remain consistently high, unlike the sharp worst-group drops observed in some competing methods.Source: §4.3 · pp. 7–8

4.4 Ablation Study

  • Removing retrieved demonstrations causes the largest ablation drop; removing image summaries lowers reliability and both generality metrics, while removing the scope head mainly reduces image-locality.The ablations align demonstrations with conveying edits, summaries with visual grounding, and the scope head with locality.Source: Table 4 · p. 7, §4.4 · p. 8
  • Figure 4 shows a clearer in-scope versus out-of-scope similarity margin with the learned scope head than with raw CLIP-VL features.Raw features assign some locality queries similarities close to in-scope examples, making irrelevant edits more likely to trigger.Source: Figure 4 · p. 7, §4.4 · p. 8

4.5 Sensitivity to Similarity Threshold

  • On EVQA with LLaVA-v1.5, θ = 0.65 balances I-Gen. 96.9 and I-Loc. 97.2; lower thresholds apply edits too broadly, while very high thresholds reduce I-Gen.At θ = 0.50, I-Gen. is 98.4 and I-Loc. is 75.0; at θ = 0.85, I-Gen. is 65.3.Source: §4.5 · p. 8, Figure 5 · p. 8

5 Related Work

The related work spans parameter, module, and memory-based editing, while existing multimodal evaluations mainly assess answer correctness. VICE combines visual in-context demonstrations with a scope router to address the grounding gap.

Related approaches

Evaluation gap

  • Existing multimodal editing benchmarks largely evaluate answer correctness, not whether edited answers follow intended visual evidence.The paper’s diagnostic varies incidental background while holding the desired fact fixed.Source: §5 · pp. 8–9

VICE

  • VICE adapts visual in-context demonstrations to express edited facts and uses a scope head to decide when those edits should apply.The approach draws on visual demonstrations and task-intent image summaries in prior visual in-context learning work [Zhou et al., 2024].Source: §5 · p. 9

6 Conclusion

The paper studies whether edited predictions follow intended visual evidence, introducing WATERBIRD-EDIT and VICE to address evidence misgrounding. It reports strong lifelong editing and reduced background sensitivity, while noting scope and efficiency limitations.

6 Conclusion

  • WATERBIRD-EDIT isolates bird-habitat evidence from background cues, while frozen-backbone VICE stores edits in sequence memory and applies them through visual demonstrations with scope routing.Across lifelong benchmarks VICE remains strong; on WATERBIRD-EDIT it substantially reduces sensitivity to spurious background context.Source: §6 · p. 9

Limitations

  • WATERBIRD-EDIT is limited to controlled bird-habitat editing, and broader visual facts or open-ended scenes remain future extensions.VICE also relies on generated image summaries and learned retrieval-based scope decisions; ambiguous boundary cases may benefit from better summaries, retrieval calibration, and uncertainty handling.Source: Limitations · p. 9
  • VICE shifts editing burden from backbone updates to external memory, retrieval, and longer prompts, leaving memory compression, faster indexing, and compact prompts as improvement opportunities.These trade-offs accompany the framework’s frozen-MLLM and lifelong-editing design.Source: Limitations · p. 9

A WATERBIRD-EDIT Construction Details

WATERBIRD-EDIT pairs bird habitat labels with separately labeled backgrounds and tests whether edits follow bird evidence across changes in image and wording. Its evaluation also includes unrelated text and image queries to assess locality.

A WATERBIRD-EDIT Construction Details

  • The dataset assigns each image a bird habitat label and a water-or-land background label, forming target-context groups that are not included in the edit prompt.Background labels are produced by a pretrained vision-language model; the intended answer should follow bird identity rather than the scene.Source: A WATERBIRD-EDIT Construction Details · p. 13
  • Edit images are drawn from a CUB pool where LLaVA-v1.5-7B and MiniGPT4-vicuna-7B fail habitat classification, so requests correct pre-edit errors.The final annotation file contains 1,136 instances across 90 species, with 636 training and 500 test instances; the test stream is strongly target-context imbalanced.Source: A WATERBIRD-EDIT Construction Details · pp. 13–14
  • Generality probes vary the habitat question wording or use another image of the same species, while locality probes use unrelated NQ text and OK-VQA image questions.The image-rephrase probe is the main diagnostic for whether an edit follows the bird across background changes.Source: A WATERBIRD-EDIT Construction Details · p. 13

B Additional Implementation Details for VICE

VICE’s implementation generates and caches structured image summaries, then places summaries and edit facts into augmented prompts. A trained scope head supports routing and retrieval.

B Additional Implementation Details for VICE

  • VICE generates image summaries offline and caches them for reuse in evidence-summary demonstrations.The implementation uses LLaVA-v1.5-7B as the frozen summary model and greedy decoding with at most 128 new tokens.Source: B Additional Implementation Details… · p. 14
  • The summary prompt asks for target evidence, distractors, and their contrast, without directly answering the question.The required output has three lines labeled Target Evidence, Distractors, and Contrast.Source: B Additional Implementation Details… · p. 14
  • For accepted queries, the augmented prompt contains retrieved demonstrations, the retrieved edit, and the current query, using image-summary, new-fact, and prompt fields.The current query uses a textual placeholder directing attention to the real input image.Source: B Additional Implementation Details… · p. 14
  • The scope head is trained on fixed, normalized, concatenated CLIP text and image features, using retrieval loss for in-scope examples and margin loss for locality examples.Training uses AdamW for 80 epochs; the dataset-specific head is then frozen for memory retrieval and threshold routing.Source: B Additional Implementation Details… · p. 15

C Complete Experimental Results

The supplementary results section adds an intermediate lifelong-editing setting to the reported results.

C Complete Experimental Results

  • The full lifelong results include 1-, 100-, and 1000-edit settings, with an additional 10-edit setting for both LLaVA-v1.5 and MiniGPT4.These results complement the condensed main-text table.Source: C Complete Experimental Results · p. 15

D Additional Discussion

VICE’s summaries generally capture both target and contextual cues, and its retrieval and routing stages perform strongly on EVQA. The supplementary section also points to complete lifelong results for both evaluated model families.

D Additional Discussion

  • The audit of 500 WATERBIRD-EDIT summaries found consistent format compliance and that summaries usually captured target and contextual cues.The audit checks recognizable target-object and scene/context cues.Source: Table 6 · p. 15, D Additional Discussion · p. 15
  • At θ = 0.65 on EVQA with LLaVA-v1.5, VICE retrieves the intended edit for at least 97.3% of in-scope queries and exceeds 96.9% routing accuracy.Reliability and text-locality queries are routed without errors; image-locality queries have a 5.7% false-acceptance rate.Source: Table 7 · p. 15, D Additional Discussion · p. 15
  • The supplementary tables provide complete lifelong editing results for LLaVA-v1.5 and MiniGPT4 on EVQA, EIC, and VLKEB.The tables distinguish local-run results from entries copied from Chen et al. (2025).Source: Table 9 · p. 17, Table 10 · p. 17

Related papers

Loading 2610.33075v1…