PaperlayerView exact revision on arXiv

Source-linked AI summary

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

Yixin Wan, Tianle Zheng, Kai-Wei Chang

arXiv:2609.06245v1cs.CVcs.AIcs.CL

TL;DR

MLLMs often struggle to identify fine-grained changes between similar images despite strong single-image performance. VDIFF-Bench addresses this gap with a challenging, deterministic multiple-choice benchmark, revealing substantial and category-specific weaknesses in comparative visual understanding.

  • Problem

    Existing MLLM evaluations emphasize single-image understanding, while existing difference benchmarks lack sufficiently challenging data and robust evaluation of fine-grained comparative perception.

  • Method

    VDIFF-BENCH evaluates Image Difference Identification with 1,756 four-choice questions spanning 10 categories, using ground-truth differences, two hard negatives, and a no-difference distractor.

  • Results

    Across 11 MLLMs, performance is category-specific: three open 7–8B models score 52.5–70.6% on semantic changes but 8.7–33.3% on low-level changes, while Grok 4.3 reaches only 5.3% on noise and 15.3% on texture.

  • Takeaways & Limitations

    Strong semantic comparison does not guarantee low-level sensitivity, and no model dominates every change category.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

1 INTRODUCTION

Although MLLMs perform strongly on single-image understanding, they remain brittle at fine-grained Image Difference Identification (IDI), motivating VDIFF-BENCH as a challenging diagnostic benchmark.

  • MLLMs excel on single-image captioning and visual question answering but struggle to identify changes between similar images.
  • Fine-grained IDI requires comparative perception, sensitivity to localized or low-level differences, and restraint against hallucinating absent changes.
  • Existing benchmarks lack sufficiently challenging image-difference data and robust evaluation metrics.
  • VDIFF-BENCH contains 1,756 multiple-choice questions across 10 change categories, pairing each real difference with plausible false alternatives and a no-difference option.
  • Three open 7–8B models achieve 52.5–70.6% on semantic changes but only 8.7–33.3% on low-level changes, often selecting no difference.

2 RELATED WORK

Related work evaluates MLLMs on single-image understanding and increasingly on paired-image comparison, but existing difference-captioning benchmarks remain limited in domains and change inventories.

  • Prior MLLM evaluations largely focus on single-image tasks such as visual question answering, visual reasoning, and mathematical or diagrammatic reasoning.
  • Image difference captioning extends evaluation to paired images by asking models to identify changes between them.
  • Spot-the-Diff and CLEVR-Change test change localization and verbalization, but their domains and change inventories are limited.
  • Fine-grained visual comparison is also important for evaluating increasingly compositional and complex image-editing models.

3 THE VDIFF-BENCH BENCHMARK

VDIFF-BENCH combines diverse image-pair sources, synthetic and edited low-level changes, human annotations, and verified hard negatives into a deterministic ten-category benchmark.

  • Benchmark construction: The benchmark is constructed in three stages: collect or create image pairs, generate and verify distractors, then assemble randomized four-choice questions.
  • Taxonomy: VDIFF-BENCH spans semantic edits and low-level visual changes that require comparative perception of image traits.
  • Taxonomy: Its inventory contains 1,756 questions over 1,543 distinct image pairs.
  • Data sources: Sources include Spot-the-Diff, MotionEdit, OmniEdit, OmniDiff, online spot-the-difference puzzles, FFHQ faces, and scene-text datasets.
  • Data augmentation: Programmatic transformations add Gaussian noise, smoothing, RGB shifts, and illumination changes with transformation type and direction providing ground truth.
  • Data augmentation: Image editing augments under-represented position and OCR/text categories using generated or manually curated edit instructions.
  • Annotation: Human experts annotate visible differences in unannotated pairs and identify affected objects or regions using visual and spatial cues.
  • Distractor construction: False differences are generated from real annotations by moving changes to nearby entities, reversing states or directions, or substituting plausible attributes.

4 EXPERIMENTS

Across 11 MLLMs, VDI performance varies substantially by model and change category. Semantic comparison often exceeds low-level comparison, while aggregate accuracy can conceal distinct perceptual strengths.

  • Overall performance: Overall accuracy ranges from 35.8% to 89.6%, with Gemini 3.1 Pro highest at 89.6%.The 25% uniform-guess and 33.3% informed-guess baselines provide reference points.
  • Overall performance: No model dominates every category; the strongest category scores are distributed across Gemini, Kimi, and Doubao models.Similar aggregate accuracy can therefore conceal different perceptual strengths.
  • Results by change category: Kimi K2.5 and Kimi K3 reach 88.8% and 82.8% low-level accuracy, whereas Grok 4.3 reaches 40.7% low-level accuracy after 82.0% semantic accuracy.Grok 4.3 scores 5.3% on noise/resolution and 15.3% on texture.
  • Results by change category: Grok 4.3 reaches 96.4% on motion but only 5.3% on noise/resolution and 15.3% on texture.Appearance/disappearance is also difficult: its category maximum is 79.6%, versus at least 90.8% in seven other categories.
  • Group-level performance and error modes: Semantic–low-level gaps reach 37.2–53.8 percentage points for Grok and the three 7–8B models.The results show that semantic-edit success does not reliably predict sensitivity to visual-appearance changes.
  • Group-level performance and error modes: For the four models with the largest gaps, 48.1–80.9% of low-level questions receive the no-difference option despite every pair containing a real change.This exceeds selection of alternative descriptions, indicating that missed detection is often the dominant failure.

5 CONCLUSION

VDIFF-BENCH evaluates fine-grained visual difference identification with challenging multiple-choice questions and interpretable distractors. Results across 11 MLLMs show persistent semantic–low-level dissociations and no simple open-versus-closed-source explanation.

  • Benchmark and findings: VDIFF-BENCH contains 1,756 four-choice questions from 1,543 image pairs across ten semantic, textual, and low-level categories.Each question pairs the ground-truth difference with two plausible hard negatives and a no-difference distractor.
  • Benchmark and findings: Accuracy across 11 MLLMs ranges from 35.8% to 89.6%, with no model dominating every change category.The benchmark supports deterministic scoring and interpretable error analysis.
  • Benchmark and findings: Grok and the three 7–8B models show semantic–low-level gaps of 37.2–53.8 percentage points and frequently miss that any change occurred.Strong semantic comparison therefore does not guarantee low-level sensitivity.
  • Benchmark and findings: The Kimi models’ strong performance argues against a simple open-versus-closed-source explanation.The conclusion identifies model capacity as insufficient by itself to explain the bottleneck.

A ADDITIONAL DETAILS ON DATASET CONSTRUCTION

The benchmark combines existing sources with programmatic augmentations to cover diverse visual changes while preserving controlled image content and layout. Construction includes low-level transformations, OCR and position edits, and source-aware inventory accounting.

  • Dataset inventory: Table 5 distinguishes benchmark questions from distinct image pairs because some sources contain multiple annotated changes per pair.Spot-the-Diff contributes 108 questions from 55 pairs, MotionEdit 137 from 67, and the kids-games source 102 from 12.
  • Low-level augmentations: Gaussian noise is added independently per pixel and RGB channel with standard deviation 15 in 8-bit space, then clipped to [0, 255].This creates global low-level corruption while preserving image content and layout.
  • Low-level augmentations: Texture smoothing uses edge-aware Gaussian blurring, stronger smoothing on skin-like regions, protected edges, and mixed-back residual detail.The operation suppresses fine texture and produces a smoother appearance while preserving layout and object identities.
  • Low-level augmentations: Illumination changes are generated with a gamma-plus-linear transform using separate parameter ranges for brighter and darker examples.The augmentation extends the OmniDiff illumination subset with subtler changes.
  • OCR and position augmentations: The OCR subset contains 221 examples combining imported pairs with 171 custom localized text edits.Edits target readable, moderately sized text regions while controlling their spatial extent.
  • OCR and position augmentations: Position augmentation starts from 150 OmniDiff swap images and proposes five candidate edits that move one visible object without changing other scene properties.The position-edit prompt preserves scene, camera, lighting, style, identities, colors, and object counts.

A.4 ADDITIONAL DETAILS ON FALSE DIFFERENCES GENERATION

Hard-negative generation uses ground-truth-conditioned prompts to create factually incorrect but visually plausible alternatives. The final evaluation presents these alternatives alongside the true difference and a no-difference option under deterministic answer-only instructions.

  • False-difference generation: Gemini-2.5-pro generates false differences from each image pair’s ground-truth differences for use as distractor options.The candidates are designed to be semantically plausible but factually nonexistent.
  • False-difference generation: The generation prompt creates 2 to 6 incorrect statements using target swaps, source swaps, attribute changes, and alternative changes to visible entities.Each statement follows the formatting of the ground-truth differences.
  • False-difference generation: False differences must use subjects visible in at least one image and must not invent new objects.The prompt also requires them to remain factually incorrect and visually plausible.
  • False-difference generation: At least one and at most two generated false differences preserve the main changed object from the ground truth.Other instructions prohibit mentioning the ground truth or using complex relational references to former object locations.
  • Evaluation prompt: The evaluation prompt supplies images A and B with four options and requires exactly one choice label as output.Images are presented in A–B order, with no difference mask and temperature zero for all runs.

C.1 NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES

Models frequently select the no-difference distractor, especially for low-level visual changes such as whole-image color, texture, and noise.

  • C.1 NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES: Missed-change selections exceed 70% in several low-level visual-change cases, including whole-image RGB, texture, and noise differences.High no-difference rates indicate missed changes rather than confusion with another detailed description.
  • C.1 NO-DIFFERENCE SELECTION ON LOW-LEVEL CHANGES: The no-difference distractor rate ranges from 24.8–43.1% for smaller models across all questions.By contrast, several larger models select no difference on only 0.5–12.1% of all questions.

C.2 REPRESENTATIVE FAILURE CASES

Representative failures show that models miss changes caused by local smoothing, global RGB shifts, and the disappearance of small entities.

  • C.2 REPRESENTATIVE FAILURE CASES: Smoothing changes local texture without altering scene semantics, making the difference difficult to detect.Qwen, Gemini, LLaVA, and InternVL select no difference, while GPT-5.4 selects the keyed answer.
  • C.2 REPRESENTATIVE FAILURE CASES: A global RGB shift distributes evidence across the entire image, and several models consequently select no difference.Qwen, LLaVA, and InternVL choose the no-difference option for this example.
  • C.2 REPRESENTATIVE FAILURE CASES: The surveillance example requires locating small missing entities in a low-resolution frame, yet LLaVA and InternVL select no difference.This case illustrates the difficulty of detecting appearance or disappearance when the changed entities are small.

C.3 ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results present complete image pairs, all answer choices, and correctness markings across the benchmark’s ten change categories, illustrating mixed model outcomes.

  • C.3 ADDITIONAL QUALITATIVE RESULTS: Qualitative results cover all ten change categories using complete image pairs, four answer choices, and correctness indicators for six representative MLLMs.Green checks and red crosses mark correct and incorrect selections.
  • C.3 ADDITIONAL QUALITATIVE RESULTS: One example contrasts no difference with changes in brow color, smoothness, and sharpness between the two images.The answer choices include several nearby descriptions, requiring selection among subtle alternatives.
  • C.3 ADDITIONAL QUALITATIVE RESULTS: Another example contrasts substitutions involving weights and a black package across the two scale trays.The choices distinguish multiple object substitutions and a no-difference option.
  • C.3 ADDITIONAL QUALITATIVE RESULTS: The examples are selected to illustrate mixed model outcomes rather than category-level prevalence.The second figure part follows the same layout and symbols as the first.
Loading 2609.06245v1…