PaperlayerView exact revision on arXiv

Source-linked AI summary

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

Yixin Wan, Tianle Zheng, Kai-Wei Chang

arXiv:2609.06245v1cs.CVcs.AIcs.CL

TL;DR

MLLMs can perform strongly on single-image understanding yet struggle to identify changes between similar images, and existing benchmarks lack challenging data and robust evaluation. VDiff-Bench tests this skill with multiple-choice image-pair questions; evaluation reveals category-specific brittleness, including weaker low-level than semantic performance in three 7–8B models.

  • Problem

    Current visual-difference benchmarks are limited by insufficient challenging data and accurate, robust evaluation metrics.

    Source: §1 · p. 2

  • Method

    VDiff-Bench is a 1,756-question, ten-category multiple-choice benchmark pairing each image pair’s true difference with plausible false choices and a no-difference option for deterministic scoring.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Across 11 MLLMs, category-specific brittleness cuts across source types: three 7–8B open models achieve 52.5–70.6% on semantic changes but 8.7–33.3% on low-level changes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluated models’ semantic–low-level gaps show that success on semantic changes does not guarantee low-level difference sensitivity.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

1 INTRODUCTION

VDIFF-Bench targets fine-grained image difference identification, a capability that remains brittle in MLLMs despite their strength on single-image tasks. Its challenging multiple-choice evaluation reveals category-specific failures, especially on subtle low-level changes.

Motivation

  • Fine-grained image difference identification requires comparing two views, detecting small localized or low-level changes, and avoiding hallucinated differences, despite MLLMs’ strength on single-image tasks.The capability could support image-editing evaluation and reward modeling by checking what changed, what stayed fixed, and whether changes match the intended transformation.Source: §1 · p. 2
  • Existing difference benchmarks can be too simple and have inaccurate ground truths, limiting their ability to probe modern MLLMs and motivating more challenging evaluation.For example, Park et al. (2019) used engine-synthesized scenes with simple changes, and a model could identify differences missing from the benchmark’s original ground truth.Source: §1 · p. 2

Benchmark contribution

  • VDIFF-Bench provides 1,756 multiple-choice questions across 10 change categories, pairing each image pair with a true difference and three distractors.The distractors include an explicit “no difference” option and plausible-but-false alternatives, which were human-verified and corrected; the benchmark supports deterministic scoring and category-, pair-, and response-level analyses.Source: §1 · pp. 2–3

Evaluation results

  • 88.8% and 82.8% low-level visual difference accuracy were achieved by Kimi K2.5 and Kimi K3, while Grok 4.3 reached 40.7%.Three open 7–8B models scored 52.5–70.6% on semantic changes but 8.7–33.3% on low-level changes, often selecting “no difference”; Grok 4.3 scored 5.3% on noise and 15.3% on texture. The results suggest scale alone does not ensure precise comparison.Source: §1 · p. 2

2 RELATED WORK

Prior MLLM evaluation has emphasized single-image tasks, while emerging paired-image benchmarks still face limits in change coverage and evaluation reliability. VDiff-Bench addresses this gap with controlled discrimination of selected image changes rather than free-form caption completeness.

2.1 GENERAL MLLM BENCHMARKS

2.2 IMAGE DIFFERENCE IDENTIFICATION AND CAPTIONING BENCHMARKS

  • Existing image-difference captioning benchmarks have limited domains or unreliable reference- and judge-based evaluation, whereas VDiff-Bench uses controlled discrimination with deterministic scoring.Spot-the-Diff and CLEVR-Change have limited domains and change inventories; OmniDiff uses BLEU-4 and ROUGE-l, while DiffCap-Bench relies on LLM judges vulnerable to robustness issues and bias.Source: §2.2 · p. 3

2.3 MLLMS FOR IMAGE EDITING EVALUATION

  • Fine-grained edit evaluation requires verifying requested changes and detecting unintended ones, but MLLM judges often hallucinate differences, motivating dedicated benchmarks.Earlier image-editing evaluation learned preference scores from human feedback; more recent methods widely use MLLM judges as image editing becomes more compositional and complex.Source: §2.3 · p. 3

3 THE VDIFF-BENCH BENCHMARK

VDIFF-BENCH is built by collecting and augmenting image pairs, generating and reviewing false-difference distractors, then combining them with ground-truth descriptions into multiple-choice questions.

3.1 TASK DEFINITION

  • Each question presents an ordered image pair with one supported change, two false alternatives, and a no-difference choice, which is always a distractor.Incorrect responses distinguish missed-change selections from competing-change selections.Source: §3.1 · p. 4

3.2 IMAGE-PAIR COLLECTION AND TAXONOMY

  • The taxonomy spans 10 semantic and low-level categories, including position, motion, color, appearance, substitution/size, OCR/text, noise/resolution, texture, and illumination.Whole-image color is also included among the low-level visual-trait changes.Source: §3.2 · p. 4, Table 1 · p. 4, §3.5 · p. 6

3.5 DATASET STATISTICS

  • 1,756 questions cover 10 difference categories across 1,543 distinct image pairs.The inventory spans semantic edits and low-level traits such as whole-image color, noise/resolution, texture, and illumination.Source: Table 1 · p. 4, §3.5 · p. 6

3.2.1 DATA SOURCES

  • Image pairs come from annotated resources, 196 curated find-the-difference puzzles, and unpaired images used for change augmentation.Sources include fixed-camera scenes, motion edits, and position, illumination, substitution, size, and OCR cases; augmentation inputs include FFHQ faces, scene-text images, and OmniEdit images.Source: §3.2.1 · pp. 4–5

3.2.2 DATA AUGMENTATION

  • Augmentation combines programmatic low-level edits, generated position and OCR edits, and expert-annotated puzzle differences reviewed by a second expert.Programmatic edits include noise, smoothing, RGB shifts, and illumination changes; edit prompts provide ground truth for generated examples, while reviewers check puzzle annotations for coverage and visual support.Source: §3.2.2 · p. 5

3.3 FALSE DIFFERENCE GENERATION

  • False alternatives are generated from ground-truth differences using nearby entities, reversed states or directions, and plausible attribute substitutions, then human-refined or discarded.Structured filters first remove duplicates and exact matches to the truth.Source: §3.3 · p. 6

3.4 MULTIPLE-CHOICE QUESTION CONSTRUCTION

  • The four choices are randomly shuffled with a fixed seed, approximately balancing correct-answer labels and no-difference positions across A–D.Each question combines the ground-truth caption, two false statements, and the fixed no-difference distractor.Source: §3.4 · p. 6

3.6 COMPARISON WITH EXISTING BENCHMARKS

  • VDIFF-BENCH unifies semantic, textual, and low-level changes across diverse image regimes and uses human-verified plausible alternatives with exact, judge-free choice scoring.The paper identifies two gaps in prior direct difference-captioning benchmarks: lack of joint coverage across change types and image regimes, and reliance on free-form captions assessed by reference metrics or an MLLM judge.Source: Table 2 · p. 5, §3.6 · p. 6

4 EXPERIMENTS

Overall VDI accuracy reaches 89.6%, but strong aggregate results can conceal sharply different category abilities and missed subtle changes. Low-level comparison remains a particular weakness for some models, even when semantic accuracy is high.

4.1 EXPERIMENTAL SETUP

  • The evaluation tests 11 MLLMs using answer-key choice accuracy overall and by change category and semantic, low-level, and OCR groups.Models receive images A and B, four labeled options, and an instruction to return one label; invalid outputs count as incorrect.Source: §4.1 · p. 6, Table 3 · p. 7

4.2 RESULTS BY CHANGE CATEGORY

  • 89.6% accuracy makes Gemini 3.1 Pro the overall leader, narrowly ahead of Kimi K2.5 at 89.2%, Gemini 3.5 Flash at 88.8%, and Doubao Seed 1.6 Vision at 87.7%.Overall accuracy spans 35.8%–89.6%, versus 25% uniform-guess and 33.3% informed-guess baselines.Source: §4.2 · p. 7, Table 3 · p. 7
  • Kimi K2.5 and Kimi K3 reach 88.8% and 82.8% low-level accuracy, while Grok 4.3 drops from 82.0% semantic-change accuracy to 40.7% low-level accuracy, underscoring that scale alone is insufficient.Grok 4.3 scores 5.3% on the noise difference category and 15.3% on texture difference; 7–8B open-weight models perform substantially worse on low-level than semantic changes.Source: §4.2 · p. 8
  • Category strengths differ: Grok 4.3 scores 96.4% on motion, Kimi K2.5 leads on regional and whole-image color, and appearance/disappearance peaks at just 79.6%.Appearance/disappearance reaches at least 90.8% in seven other categories; Kimi K2.5 also ties for the best substitution/size result.Source: §4.2 · p. 8

4.3 GROUP-LEVEL PERFORMANCE AND ERROR MODES

  • Grok and the three 7–8B models show semantic–low-level accuracy gaps of 37.2–53.8 percentage points, so semantic success does not reliably predict low-level sensitivity.For the four models with the largest gaps, 48.1–80.9% of low-level questions receive the no-difference answer despite every pair containing a real change, making missed changes more common than alternative-description errors.Source: §4.3 · p. 8

5 CONCLUSION

VDIFF-BENCH diagnoses fine-grained visual difference identification with 1,756 four-choice questions, and results show that MLLMs remain uneven at comparing images. Strong semantic comparison does not ensure low-level sensitivity, and neither source type nor model capacity alone explains the observed bottleneck.

5 CONCLUSION

  • Across 11 contemporary MLLMs, accuracy ranges from 35.8% to 89.6%, and no model dominates every change category.The results indicate that fine-grained comparative perception remains challenging for MLLMs.Source: §5 · p. 8
  • Grok and three 7–8B models show semantic–low-level gaps of 37.2–53.8 percentage points and frequently fail to register that any change occurred.The conclusion contrasts this with strong Kimi model performance, arguing against a simple open- versus closed-source explanation and noting that model capacity alone is not the only reason for the bottleneck.Source: §5 · p. 8
  • VDIFF-BENCH contains 1,756 four-choice questions from 1,543 image pairs across ten semantic, textual, and low-level change categories.Each ground-truth difference is paired with two plausible hard negatives and a no-difference distractor, enabling deterministic scoring and interpretable error analysis.Source: §5 · p. 8

A ADDITIONAL DETAILS ON DATASET CONSTRUCTION … A.3 AUGMENTATION WITH IMAGE EDITING DIFFERENCES

The appendix details benchmark source composition and describes programmatic and image-editing augmentations used to create fine-grained visual changes. These methods target low-level differences as well as OCR and object-position changes.

A.1 SOURCE COMPOSITION

  • Spot-the-Diff contributes 108 questions from 55 pairs, MotionEdit 137 from 67 pairs, and the kids-games source 102 from 12 pairs.Table 5 reports both questions and distinct image pairs because some pairs contain multiple annotated changes.Source: §A.1 · p. 12, Table 5 · p. 12

A.2 AUGMENTATION WITH PROGRAMMATIC VISUAL CHANGES

  • Gaussian noise is added independently per pixel and RGB channel to 150 FFHQ images, using zero-mean noise with standard deviation 15 in 8-bit space.Results are clipped to [0, 255], corrupting the image globally while preserving content and layout.Source: §A.2 · p. 12
  • The RGB-shift augmentation retains 147 examples after three flagged images are excluded, creating subtle global tone shifts while keeping images close to their originals.It removes average channel offset, scales the chromatic direction by 0.55, and caps the maximum absolute channel offset at 12 pixel values.Source: §A.2 · p. 12
  • Texture smoothing uses skin- and edge-aware processing on the 150-image FFHQ pool to suppress fine detail while preserving layout and object identities.The pipeline combines Gaussian blur, a soft skin mask, edge protection, residual detail, and adjustments to highlights, saturation, contrast, and gamma.Source: §A.2 · p. 12
  • Illumination changes use the first 50 sorted Country211 test images, applying gamma-plus-linear transforms to create brighter and darker examples.Brighter settings use γ ∼U(0.70, 0.85) and b ∼{5, . . . , 15}; darker settings use γ ∼U(1.15, 1.30) and b ∼{−15, . . . , −5}.Source: §A.2 · p. 12

A.3 AUGMENTATION WITH IMAGE EDITING DIFFERENCES

  • Image-editing-model augmentation adds examples to the OCR and object-position categories in the evaluation data.The appendix describes localized text edits and object relocations designed to preserve other scene content.Source: §A.3 · p. 13
  • 221 OCR examples combine 50 OmniDiff pairs with 171 custom text edits initialized from MLT19 and TextOCR images.Custom edits replace a character, digit, adjacent characters, or a same-script word; edited crops are pasted back with a feathered boundary and checked for text accuracy and scene preservation.Source: §A.3 · p. 13
  • For object-position changes, 150 OmniDiff swap-task seed images yield candidate edits from gpt-5.4-mini, with selected edits produced by gpt-image-2.The prompts constrain edits to relocating one visible object while preserving the scene, camera angle, lighting, style, identities, colors, and object counts.Source: §A.3 · p. 13

A.4 ADDITIONAL DETAILS ON FALSE DIFFERENCES GENERATION · B EVALUATION PROMPT · Evaluation Prompt

The paper generates visually plausible but false distractors conditioned on each image pair’s ground-truth changes, then evaluates models with a fixed four-choice prompt and controlled image-input settings.

A.4 ADDITIONAL DETAILS ON FALSE DIFFERENCES GENERATION

  • Gemini-2.5-pro generates semantically plausible, factually nonexistent differences as distractors for the final multiple-choice questions.Source: §A.4 · p. 14
  • For each pair, the generation prompt requests 2–6 false statements using object swaps, target-attribute changes, or plausible alternative changes to visible entities.Source: §A.4 · p. 14
  • Distractors must describe visible entities and avoid invented objects, indirect references to the true change, and complex relational wording.Source: §A.4 · p. 14
  • The prompt also solicits up to five potentially missing true differences for audit, and overlaps with displayed alternatives are flagged for visual adjudication.Source: §A.4 · p. 14

B EVALUATION PROMPT

Evaluation Prompt

  • Images are supplied directly in A–B order without difference masks; runs use temperature zero, with Qwen3-VL-8B Thinking reasoning disabled and no separate reasoning traces for other models.Source: Evaluation Prompt · p. 15

C ADDITIONAL RESULTS · C.1 NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES

Models frequently miss visual changes by selecting “no difference,” especially for low-level changes such as RGB, texture, and noise. This rate varies substantially across models.

C.1 NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES

  • Smaller models select the no-difference option at rates of 24.8–43.1%; for whole-image RGB, texture, and noise changes, missed-change selections exceed 70% in several cases.High no-difference-distractor rates indicate missed-change selections rather than confusion with another detailed description.Source: §C.1 · p. 15, Table 6 · p. 16
  • The findings show that models still fail to capture visual differences that are not straightforwardly observable.The reported failures are particularly frequent for low-level visual changes.Source: §C.1 · p. 15
  • Gemini 3.5 Flash, Gemini 3.1 Pro, GPT-5.4, and Gemini 2.5 Flash selected the no-difference option on 0.5%, 1.6%, 5.2%, and 12.1% of all questions, respectively.Table 6’s overall rate covers all 1,756 questions.Source: §C.1 · p. 15, Table 6 · p. 16

C.2 REPRESENTATIVE FAILURE CASES

Figure 6 shows representative cases where models select “no difference” for changes involving local texture, a global RGB shift, or small missing entities in a low-resolution frame. These examples illustrate distinct sources of difficulty and align with the repeated no-difference choice reported in aggregate behavior.

C.2 REPRESENTATIVE FAILURE CASES

  • On the smoothing case, Qwen, Gemini, LLaVA, and InternVL select “no difference,” while GPT-5.4 selects the keyed answer.Smoothing changes local texture without altering scene semantics.Source: Figure 6 · p. 16, §C.2 · p. 16
  • For a global RGB shift, Qwen, LLaVA, and InternVL select “no difference.”The shift distributes evidence across the entire image.Source: Figure 6 · p. 16, §C.2 · p. 16
  • When two people disappear, LLaVA and InternVL select “no difference.”The surveillance example requires locating small missing entities in a low-resolution frame.Source: Figure 6 · p. 16, §C.2 · p. 16

C.3 ADDITIONAL QUALITATIVE RESULTS … SUBSTITUTION / SIZE

The appendix provides qualitative VDiff-Bench examples across change categories, showing answer choices and representative model correctness rather than category-level prevalence. The supplied passages include examples for motion, texture, and substitution/size.

C.3 ADDITIONAL QUALITATIVE RESULTS

  • The appendix adds qualitative results to illustrate how different models perform on data from each benchmark category.Source: §C.3 · p. 16
  • Figure 7 presents qualitative VDIFF-BENCH results across all ten change categories in two parts; each panel shows the complete image pair, four choices with ground truth colored, and correctness for six representative MLLMs.Part 2 examples illustrate mixed model outcomes rather than category-level prevalence.Source: Figure 7 · pp. 17–18

MOTION

  • The motion examples appear in Figure 7’s first set of category panels, where model selections are marked correct or incorrect.Source: Figure 7 · p. 17

TEXTURE

  • The texture example offers choices describing no difference, lighter brows, smoother appearance, or sharper appearance.Source: TEXTURE · p. 18

SUBSTITUTION / SIZE

  • The substitution/size example contrasts a large weight becoming a small weight or black package, a left-side package becoming a small weight, and no difference.Source: SUBSTITUTION / SIZE · p. 18

Related papers

Loading 2609.06245v1…