PaperlayerView exact revision on arXiv

Source-linked AI summary

Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning

Jiyoung Park, InJae Oh, Jung Uk Kim

arXiv:2609.01136v1cs.CV

TL;DR

Existing change captioning methods often use a unified pipeline even though different change types depend on distinct visual cues. MEDIC introduces softly routed, type-specialized memory experts with dedicated training objectives, and experiments report consistent improvements across datasets and change types. The framework remains dependent on change-type annotations, motivating future self-supervised alternatives.

  • Problem

    Most change captioning methods use a unified, type-agnostic pipeline despite color changes, object additions, and other changes relying on different visual cues.

  • Method

    MEDIC uses two-stage routing and type-specialized memory experts that dynamically retrieve relevant visual patterns, trained with routing, consistency, and disentanglement losses.

  • Results

    MEDIC consistently improves performance across all change types and datasets, including object-level and global-level changes.

  • Takeaways & Limitations

    Explicitly separating change types supports type-aware reasoning across diverse and challenging change-captioning scenarios.

  • Takeaways & Limitations

    MEDIC relies on the availability of change-type annotations when extending to unseen datasets.

Abstract

from arXiv · show

Change captioning is the task of generating natural language descriptions that explain the changes between a pair of images. Although different change types (e.g., color shifts, object additions) exhibit distinct visual cues and require specialized reasoning processes, existing methods often overlook these distinctions. To address this limitation, we propose Multi-Expert Diagnosis for Image Change (MEDIC), a novel framework that introduces change-type awareness by explicitly modeling change categories. MEDIC employs type-specialized memory experts that dynamically retrieve type-relevant visual patterns conditioned on the input. This design enables each expert to capture diverse variations within its change type while focusing on the most informative visual cues. By softly routing inputs across type-specialized experts and learning dedicated representations for each change category, MEDIC generates more precise and type-aware change descriptions. Extensive experiments demonstrate that the proposed MEDIC consistently outperforms existing methods across diverse and challenging datasets. The code is available at \href{https://github.com/VisualAIKHU/MEDIC}{GitHub}.

A Introduction

Change captioning methods commonly process heterogeneous changes through a single reasoning pipeline, despite distinct change types requiring different visual cues. MEDIC addresses this limitation with type-specialized memory experts, routing, and training objectives for type-aware descriptions.

  • Motivation: Change captioning generates natural-language descriptions of visual differences between image pairs for applications including surveillance and medical diagnostics.These applications require precise and reliable interpretation of scene changes.
  • Problem: Prior methods often use one reasoning process for color changes, object additions, and spatial movements despite their different visual evidence.Color changes depend on subtle intensity variations, whereas object additions require background separation and object-level reasoning.
  • Problem: Unified processing can misclassify a color change as an object addition, producing descriptions that omit the relevant changed object and context.The paper illustrates a red cylinder turning green being described as a green cylinder appearing.
  • MEDIC: MEDIC uses a change router followed by a type router to softly assign inputs to experts specialized for corresponding change categories.The framework explicitly models change captioning as a type-aware process inspired by mixture-of-experts.
  • MEDIC: Each expert is a memory network that retrieves input-dependent, type-relevant visual patterns while three losses promote correct routing, type-specific representations, and separation between expert representations.The losses are routing loss, expert consistency loss, and expert disentangle loss.
  • Results: MEDIC consistently outperforms existing state-of-the-art code-released methods across change types and datasets.The paper points to CIDEr comparisons across change types as evidence of this result.

B Related Works

Change captioning has evolved from differencing-based pipelines toward robust relation-aware modeling, but most methods still process heterogeneous change types through one type-agnostic reasoning path. MEDIC instead combines two-stage routing with type-specific memory experts and improves performance across datasets and change types.

  • Related change-captioning methods: Prior change-captioning methods improve robustness with semantic, relational, contrastive, syntactic, neighborhood, and cross-view cues, yet largely retain unified processing.These approaches include SRDRL, R3Net, SCORER, DIRL, SMART, NCT, I3N, MURAT, and CHEERS.
  • MEDIC architecture: Each change-type expert is a key-value memory network that retrieves type-specific visual cues from tokenized paired features through addressing.The retrieved token outputs are reassembled into spatial expert features for downstream caption generation.
  • MEDIC architecture: MEDIC uses a two-stage router that detects change presence, predicts the change type, and softly routes paired features to specialized experts or a no-change expert.The resulting expert features are aggregated by predicted type probabilities before transformer decoding.
  • Training objectives: The router is trained with cross-entropy objectives for change detection and, on changed samples, change-type prediction.The type-router target excludes the no-change category and uses the ground-truth change-type label.
  • Results: MEDIC consistently outperforms DIRL across all listed datasets and change types, including color, texture, add, drop, move, and no-change categories.The evaluation covers CLEVR-DC, CLEVR-Change, and Spot-the-Diff.

D.5 Ablation Studies

Ablations show that MEDIC’s type-specific memory experts, routing, and auxiliary losses jointly improve change captioning across architectures, change levels, and datasets. Visualizations and robustness tests further support discriminative, adaptive, and extensible representations.

  • Expert architecture: The memory-based expert outperforms an MLP-based expert across all metrics, supporting input-adaptive retrieval of type-specific visual cues.This comparison isolates expert architecture within the same type-specific framework.
  • Representation specialization: MEDIC produces well-separated t-SNE clusters across change types, whereas DIRL shows entangled clusters with unclear boundaries.The separation indicates more discriminative and type-specific expert representations.
  • Type-specific design: Type-specific experts outperform type-agnostic MoE designs across expert configurations, and gains are attributed to explicit specialized reasoning rather than type labels alone.A type-aware baseline using auxiliary change-type classification performs worse than MEDIC, according to the supplementary comparison.
  • Training losses: Removing the consistency, disentangle, or routing losses reduces performance, while combining all three achieves the best results.The results indicate complementary specialization effects and the value of explicit routing training.
  • Memory behavior: Increasing memory slots improves performance through 100 slots with marginal cost, while dynamic activation captures both shared type patterns and input-specific responses.Performance slightly drops beyond 100 slots but remains above the non-memory baseline.
  • Generalization and robustness: MEDIC improves across object-level and global-level changes, integrates with LVLM-based models, and remains robust to corrupted type labels.On CLEVR-DC, CIDEr remains 95.4, 94.3, and 94.0 with 10%, 20%, and 40% label corruption, respectively, versus DIRL’s 84.1.

C Robustness to Noisy Type Labels

MEDIC remains effective when change-type supervision is corrupted and can often produce accurate captions despite imperfect routing. Its soft expert aggregation helps preserve relevant visual evidence across competing type predictions, although extreme scene changes can still cause failures.

  • Noisy-label stress test: 95.4, 94.3, and 94.0 CIDEr under 10%, 20%, and 40% type-label noise, respectively, while remaining above DIRL’s 84.1 CIDEr.Performance decreases as label noise increases but retains a substantial improvement over DIRL even at 40% noise.
  • Noisy-label stress test: MEDIC’s routing and caption generation are jointly guided by image-pair features, caption supervision, and soft expert aggregation rather than discrete type labels alone.
  • Imperfect routing: Incorrect top-1 type predictions do not necessarily cause captioning failure because the ground-truth expert often receives a close second-highest routing probability.
  • Imperfect routing: Subtle modifications and image pairs containing multiple change types can spread routing probability across competing categories.
  • Failure cases: Extreme viewpoint variation or substantial scene reconfiguration can produce inaccurate captions through unreliable before-and-after object alignment.

E Type-Aware Baseline Analysis

Type supervision alone provides a modest gain, whereas MEDIC’s explicit type-specialized experts and routing yield a much larger improvement over the DIRL baseline.

  • Interpretation: MEDIC combines type-specialized expert architecture with routing-based expert selection to learn change-type-specific reasoning.
  • Baseline comparison: 87.2 CIDEr after adding auxiliary type supervision to DIRL, compared with 84.1 CIDEr for DIRL alone.
  • Baseline comparison: 99.8 CIDEr for DIRL+MEDIC, substantially exceeding the gain from auxiliary type classification alone.The comparison indicates that MEDIC’s improvement is not merely attributable to using change-type labels during training.

F Leave-One-Type-Out Evaluation for In-Domain Held-Out Change Types

Without a dedicated expert, MEDIC performs poorly on an excluded in-domain change type but can still generate captions through soft routing. Adding a new expert with lightweight fine-tuning substantially restores held-out-type performance without retraining the full model.

  • Held-out types: Held-out ‘Color’ CIDEr drops from 118.8 to 18.4 when that type is excluded during training.
  • Held-out types: Held-out ‘Add’ CIDEr drops from 86.1 to 13.1 when no dedicated Add expert is trained.
  • Held-out types: MEDIC still generates captions for held-out types by softly routing samples to the most relevant previously learned experts.
  • Incremental update: Adding a newly initialized held-out-type expert requires 0.2M additional parameters, or 0.82% of the 25.12M total parameters.
  • Incremental update: The incremental update substantially recovers held-out-type performance while avoiding catastrophic degradation on previously learned types.

G Taxonomy-Mismatch Stress Test

MEDIC remains functional when the change-type taxonomy is inconsistent: routing accuracy declines, but captioning degrades only moderately because the model can still learn from visual evidence. Caption quality is also stable under small routing perturbations but falls as routing noise grows.

  • Taxonomy mismatch: Removing the Color category and reassigning its samples creates ambiguous type supervision without removing the underlying color-change image pairs or captions.
  • Taxonomy mismatch: Taxonomy inconsistency reduces routing accuracy, but captioning performance degrades only moderately and many color changes remain correctly described.
  • Taxonomy mismatch: MEDIC remains functional under taxonomy-level ambiguity because its experts retrieve and integrate visual cues directly from the images.
  • Routing accuracy: Caption performance remains nearly unchanged for routing noise σ ≤1 but decreases steadily as noise increases.

I Ablation of Routing Strategy

The routing ablation contrasts MEDIC’s hierarchical two-stage routing with a simplified single-stage alternative. The comparison uses change-aware evaluation on CLEVR-DC with DIRL as the backbone.

  • Evaluation setup: The routing comparison is conducted on CLEVR-DC using DIRL as the backbone and change-aware evaluation criteria.The reported evaluation distinguishes changed from unchanged cases, change types, and content alignment.
  • Routing design: MEDIC’s two-stage router first detects whether a change exists, then predicts its specific type.This separates change detection from fine-grained type classification.
  • Routing design: The single-stage variant directly predicts among six categories, including the no-change class, without explicit decomposition.It replaces the hierarchical routing structure with one prediction step.

J Additional Evaluation Beyond Caption Similarity Metrics

The paper supplements standard caption similarity metrics with evaluations targeting change recognition, change-type recognition, and visual content alignment. DIRL+MEDIC improves over DIRL on all three measures.

  • Motivation: Standard caption metrics do not directly isolate change existence, change type, or detailed visual content.The paper therefore adds change-aware evaluations on CLEVR-DC using DIRL as the backbone.
  • Measures: The three complementary measures are change/no-change accuracy, change-type recognition, and CLIPScore-based content alignment.The first two use a rule-based parser, while Content Align. uses CLIPScore.
  • Results: 92.4 versus 83.5 in Chg/NC Acc. and 84.7 versus 77.8 in Type Recog. show DIRL+MEDIC outperforming DIRL.These metrics assess whether the caption identifies change occurrence and the correct change type.
  • Results: 27.2 versus 26.5 in CLIPScore shows improved content alignment for DIRL+MEDIC over DIRL.This complements the recognition gains with an image-caption alignment signal.

K Comparison with LVLM-Based Methods

The paper positions MEDIC against zero-shot and fine-tuned LVLM-based change captioning methods on Spot-the-Diff. Zero-shot LVLMs perform modestly, while fine-tuning substantially improves CIDEr; MEDIC remains competitive on METEOR.

  • Comparison setup: The comparison includes zero-shot LVLMs and a fine-tuned LVLM-based change captioning model.This evaluates MEDIC against both general-purpose zero-shot systems and task-adapted LVLMs.
  • Zero-shot LVLMs: 17.5–21.6 CIDEr is achieved by zero-shot BLIP-2, InstructBLIP, and VPG-C variants on Spot-the-Diff.The paper describes direct zero-shot application of general-purpose LVLMs as challenging for precise image-difference description.
  • Fine-tuned LVLMs: 51.4 CIDEr is achieved by fine-tuned BLIP2IDC, whereas DIRL+MEDIC obtains lower CIDEr but higher METEOR.The comparison highlights different outcomes across the two reported caption metrics.

L Qualitative Results

Qualitative examples indicate that MEDIC generally produces more accurate, type-consistent, and semantically detailed captions across datasets. Remaining failures center on subtle local changes, crowded multi-change scenes, and ambiguous global edits.

  • Qualitative findings: MEDIC generally produces more accurate and type-consistent captions across CLEVR-Change, CLEVR-DC, Spot-the-Diff, and Image-Editing-Request.The qualitative examples emphasize explicit change-type modeling and improved semantic accuracy.
  • Failure cases: Subtle localized changes can cause alignment failures and incorrect no-change predictions in CLEVR-Change.Small object movements under limited appearance variation are especially difficult.
  • Failure cases: Crowded scenes with multiple entities or simultaneous changes can yield incomplete captions that miss the underlying relation.Such captions may describe salient after-image objects without capturing movement or disappearance.
  • Failure cases: Ambiguous global edits can produce descriptions related to the target instruction without precisely matching the intended operation.Examples include contrast adjustment, color enhancement, and style-related appearance changes.
  • Future direction: The paper identifies subtle local changes, complex multi-object scenes, and ambiguous global edits as priorities for future work.These limitations span the three highlighted failure settings.

N Dynamic Slot Activation Analysis

The analysis shows that MEDIC’s memory-based experts dynamically activate different slots for inputs sharing a change type. These activations combine type-specific patterns with input-specific adaptations, supporting flexible retrieval rather than static slot assignments.

  • Visualization setup: Figures S.9 and S.10 visualize memory activation patterns for two CLEVR-DC samples per change type.The visualizations examine dynamic behavior across various change types.
  • Slot types: Type-specific slots activate consistently across samples sharing a change type, capturing generalizable patterns associated with that type.These slots are marked with green circles in the visualizations.
  • Slot types: Input-specific slots vary across examples, adapting retrieval to the particular input despite a shared change type.These slots are marked with blue circles and reflect input-dependent variation.
  • Dynamic retrieval: The experts dynamically retrieve relevant information by selecting different memory slots in an input-conditioned manner.This behavior demonstrates flexibility and semantic sensitivity without hard-coded priors or slot assignments.
  • Robustness to routing errors: Weighted aggregation can preserve accurate change captions even when the router’s top-1 predicted change type differs from ground truth.The model integrates complementary cues from all change-type experts in these cases.
Loading 2609.01136v1…