PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation

Abdelrahman Eldesokey, Aleksandar Cvejic, Bernard Ghanem, Peter Wonka

arXiv:2509.21989v1cs.CV

TL;DR

Subject-driven generation lacks annotated visual correspondences, while semantic features can miss appearance differences. The paper builds an automated correspondence dataset and contrastively disentangles diffusion features to create VSM; reported evaluations find stronger performance than existing metrics while also localizing inconsistencies.

  • Problem

    Annotated visual correspondences are lacking, and semantic features are insensitive to appearance changes, limiting visual-consistency evaluation.

    Source: §3 · p. 4

  • Method

    An automated pipeline creates annotated semantic and visual image pairs for contrastive disentanglement, whose representations define the VSM metric.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    VSM shows significantly higher correlation than feature-based metrics and VLM judgments in controlled evaluation, and consistently outperforms other metrics on real-world subject-driven generation.

    Source: §4.2 · p. 9, §4.3 · p. 9

  • Takeaways & Limitations

    The framework supports quantifying and localizing visual inconsistencies for more reliable and interpretable evaluation.

    Source: §6 · p. 10

    Disentanglement remains partial: visual features may retain semantic information, and matching across semantically different objects remains future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models, enabling visual correspondence in a manner analogous to the well-established semantic correspondence. While diffusion model backbones are known to encode semantically rich features, they must also contain visual features to support their image synthesis capabilities. However, isolating these visual features is challenging due to the absence of annotated datasets. To address this, we introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences based on existing subject-driven image generation datasets, and design a contrastive architecture to separate the two feature types. Leveraging the disentangled representations, we propose a new metric, Visual Semantic Matching (VSM), that quantifies visual inconsistencies in subject-driven image generation. Empirical results show that our approach outperforms global feature-based metrics such as CLIP, DINO, and vision--language models in quantifying visual inconsistencies while also enabling spatial localization of inconsistent regions. To our knowledge, this is the first method that supports both quantification and localization of inconsistencies in subject-driven generation, offering a valuable tool for advancing this task. Project Page:https://abdo-eldesokey.github.io/mind-the-glitch/

1 Introduction

Subject-driven generation aims to preserve a subject’s fine-grained appearance across images, but existing evaluation approaches struggle with subtle inconsistencies and localization. The paper proposes disentangled visual features and a metric that both quantifies and localizes inconsistencies.

  • Maintaining a subject’s visual consistency across generated scenes matters for narrative coherence and branding, but text prompts alone make specific content difficult to control.Source: §1 · p. 2
  • LPIPS and SSIM assume spatial alignment, while CLIP and DINO capture global semantics and can overlook subtle object-detail inconsistencies.Source: §1 · p. 2
  • VLM assessments remain global, with unclear judgment criteria and no localization of inconsistent subject regions.Source: §1 · p. 2
  • The proposed framework combines an automated correspondence dataset and contrastive feature disentanglement to build a metric that quantifies and localizes inconsistencies, outperforming commonly used feature-based metrics.Source: §1 · p. 2

2 Related Work

Related work covers diffusion-based semantic correspondence, subject-driven generation, and consistency evaluation. The paper builds on these threads by using visual features to measure appearance similarity and localize inconsistencies.

2 Related Work

  • The related-work section reviews semantic correspondence, subject-driven generation, and how consistency is evaluated in those methods.Source: §2 · p. 3

2.1 Semantic Correspondence using Diffusion Models

  • Diffusion decoder features and learned aggregation have supported semantic correspondence and downstream tasks, motivating representations that match visual appearance instead of semantics.Source: §2.1 · p. 3

2.2 Subject-Driven Image Generation

  • Subject-driven methods progressed from fine-tuning and learned subject embeddings to adapters, DiTs, and visual autoregressive models, with approaches targeting identity preservation and fidelity.Source: §2.2 · p. 3

2.3 Evaluating Subject-Driven Image Generation

  • Global image metrics and VLMs handle spatial variation but can miss fine detail or localization; controlled-inconsistency image pairs support contrastive visual-feature separation and spatially localized evaluation.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 Method

The method builds visual-correspondence data from subject-driven image pairs, separates semantic and visual features in a diffusion backbone, and uses the resulting representations to measure and localize visual consistency.

3.1 Data Generation Pipeline with Visual Correspondence

  • The data pipeline segments subjects, finds semantic correspondences, and uses matched points to identify localized regions for visual alteration.The pipeline uses Grounded-SAM for subject segmentation and CleanDIFT features from the diffusion model’s sixth decoder layer for semantic correspondences.Source: §3.1 · pp. 4–5
  • Matching-score skewness filters ambiguous matches: high skewness indicates a distinct match, while low skewness indicates diffuse correspondences.Flat-textured subjects can produce broad matches, prompting segmentation of the whole subject rather than a localized part.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Learning Disentangled Semantic and Visual Representations

  • Separate aggregation networks produce semantic and visual features from multiple decoder layers of the diffusion backbone.The networks use ResNet blocks and trainable layer weights to select features for semantic content or visual appearance.Source: §3.2 · pp. 5–6
  • Contrastive training aligns semantic features across correspondences, while visual features align outside inconsistent regions and separate within them.The combined objective weights the visual loss with a scaling factor that prioritizes the visual branch.Source: §3.2 · pp. 6–7

3.3 A Metric for Evaluating Subject-Driven Image Generation

  • VSM measures the proportion of semantically matched regions that are also visually consistent, and low visual similarity can localize inconsistent regions.It evaluates visual similarity at locations selected using semantic matches.Source: §3.3 · p. 7

4 Experiments

The experiments compare VSM with feature-based and vision-language metrics, first against a controlled oracle and then on generated images from recent subject-driven methods. They also examine learned feature aggregation and key design choices.

4.1 Implementation Details

  • The training pipeline uses consistent Subjects200k pairs, filters for match skewness, inconsistent-region area, and LPIPS, and produces 5,000 training and 500 validation pairs.Experiments use Stable Diffusion 2.1 features and train for 30 epochs on one A100 GPU; training takes 12 hours.Source: §4.1 · pp. 7–8

4.2 Evaluating the VSM Metric

  • In the controlled evaluation, VSM correlates more strongly with the oracle than CLIP, DINO, and ChatGPT-4o, while visual features differ in inpainted regions.The oracle is based on the ratio of the inconsistent region to the total object mask; semantic features align across semantically similar regions regardless of appearance.Source: §4.2 · pp. 8–9

4.3 Evaluating Subject-Driven Image Generation

  • The realistic evaluation generates 100 images per method for Diptych, DSD-Diffusion, and EasyControl, then manually annotates inconsistent regions to compute oracle scores.The study compares these evaluations with CLIP, DINO, and ChatGPT-4o.Source: §4.3 · p. 9, §4.2 · p. 9
  • VSM consistently outperforms the other metrics in controlled and realistic settings, with the highest agreement with the oracle in the score distributions.The qualitative examples also show VSM aligning most closely with the oracle.Source: §4.3 · p. 9, Table 1 · p. 9

4.4 Ablation Analysis

  • Visual features draw primarily from decoder layers 8 and 9, while semantic features draw from layers 8 and 10, so layer 8 contributes to both.The authors attribute differences from [22] to supervision: their model uses dense correspondences rather than sparse keypoints.Source: §4.4 · p. 10
  • The ablations find that VSM threshold Tv = 0.6 performs best, while α = 1 and skewness thresholds that are too high or low reduce correlation with the oracle.The authors link α = 1 to reduced emphasis on visual features and skewness extremes to restricted diversity or ambiguous matches.Source: §4.4 · p. 10, Table 2 · p. 10

5 Limitations and Future Work

The authors identify limits in feature disentanglement, training-data quality, spatial resolution, and the kinds of inconsistencies their framework targets.

5 Limitations and Future Work

  • Disentanglement is partial: visual features may still carry semantic information, and matching across semantically different objects remains future work.The authors describe full disentanglement as a promising direction.Source: §5 · p. 10
  • Learned-feature quality depends on the reference dataset; automatically validated Subjects200k pairs may be noisy, while diffusion-feature resolution limits fine-grained detection.Manual curation or improved filtering and higher-resolution extraction are proposed as possible remedies.Source: §5 · p. 10
  • The framework targets structural and appearance-level inconsistencies; broader variations such as artistic style or color may require further decomposition.Source: §5 · p. 10

6 Conclusion

The paper concludes that disentangled diffusion features support visual correspondence alongside semantic matching. The resulting metric quantifies and localizes subject-driven generation inconsistencies and outperforms existing metrics.

6 Conclusion

  • The framework separates diffusion features into semantic and visual components using an automated dataset pipeline and a contrastive architecture.Source: §6 · p. 10
  • The disentangled features enable visual correspondence across images and fine-grained analysis that complements semantic matching.Source: §6 · p. 10
  • The VSM metric quantifies and localizes inconsistencies in subject-driven generation, and the authors report that it outperforms existing metrics.Source: §6 · p. 10

A Additional Qualitative Examples

Additional examples show how semantic and visual correspondences behave under controlled inpainting and in evaluations of subject-driven generation methods.

A Additional Qualitative Examples

B Additional Ablation Analysis

The additional ablations examine semantic-branch design, aggregation dimensionality, visual-branch weighting, and training-set size. They identify trade-offs in model capacity and data quantity, with 384-dimensional aggregation and α = 10 supported as effective choices.

B Additional Ablation Analysis

  • Using zero-shot semantic features without a dedicated semantic aggregation network significantly degrades VSM performance; joint training aligns the semantic and visual representations spatially.That alignment is needed to compare the representations across subject regions and quantify and localize inconsistencies.Source: B Additional Ablation Analysis · p. 14
  • Two residual blocks per decoder layer slightly improve Spearman correlation but increase model size by about 25%, from 4.5M to 6M parameters.Table 3 reports test-set correlations with the oracle for the additional design choices.Source: B Additional Ablation Analysis · p. 14, Table 3 · p. 15
  • Increasing aggregated feature dimensionality from 384 to 512 significantly reduces performance, while reducing it to 256 causes a more gradual decline.The passage suggests redundant channels may add noisy features; it identifies q = 384 as optimal.Source: B Additional Ablation Analysis · p. 14
  • α = 10 gives better balance between semantic and visual branches than α = 1 or α = 20, which both reduce performance.The lower value insufficiently emphasizes visual-feature learning.Source: B Additional Ablation Analysis · p. 15
  • Training on 2,500 rather than 5,000 samples lowers performance by approximately 10%, while the authors judge 5,000 samples sufficient for effective learning.The authors infer diminishing performance gains as more data is added.Source: B Additional Ablation Analysis · p. 15

C Additional Details on Benchmarking Subject-Driven Generation Approaches

The benchmarking appendix evaluates recent subject-driven generation methods on a test set using a subject image and a prompt for a different environment. It documents the evaluated implementations and their inference settings.

C Additional Details on Benchmarking Subject-Driven Generation Approaches

  • The test set evaluation compares CLIP and DINO image-to-image similarity and DreamBench++ VLM judgments across methods given a subject image and a prompt for a different environment.The section identifies Diptych Prompting, DSD Diffusion, and EasyControl as evaluated methods.Source: C Additional Details on Benchmarking… · p. 15
  • Diptych Prompting uses FLUX-dev with the official template at 768×1536 input resolution, producing a 768×768 output; generation averages 92 seconds per sample.Its hyperparameters are kept at their official defaults.Source: C Additional Details on Benchmarking… · p. 15
  • DSD Diffusion uses FLUX with 28 steps and guidance scale 3.5, producing 512×512 outputs from 512×1024 inputs.Inference averages 8 seconds per sample, and Gemini prompt enhancement is disabled for comparison fairness.Source: C Additional Details on Benchmarking… · p. 15
  • EasyControl uses the FLUX subject-control variant at 768×768 resolution and full precision after reduced-precision runs produced black outputs for about 16% of samples.Inference averages 94 seconds per sample.Source: C Additional Details on Benchmarking… · pp. 15–16

D Additional Dataset Filtration Strategy

The appendix describes additional dataset filters and the setup for VLM evaluation. The filters constrain segmented regions and their cross-image consistency, while the evaluation prompts request a standardized score.

D Additional Dataset Filtration Strategy

  • Additional dataset heuristics require each segmented part to occupy 5%–60% of its object, with paired relative-size differences below 0.1.The pipeline also requires horizontal and vertical part-to-object aspect-ratio differences between paired images to remain within 20%.Source: D Additional Dataset Filtration Strategy · p. 16

E VLM Evaluation Prompts

  • The VLM evaluation follows DreamBench++ with GPT-4o at temperature 0 and modified prompts requesting scores from 0 to 100.The score range is chosen to align with other metrics and the user study.Source: E VLM Evaluation Prompts · p. 16

F Failure Cases

The appendix describes failure cases for VSM and provides additional qualitative examples of correspondence and metric evaluation. It also includes prompt material for evaluating semantic consistency with a VLM.

F Failure Cases

  • VSM struggles with incomplete or corrupted generations, major lighting changes absent from training, and under- or over-segmented subject regions.Missing parts can be hard to distinguish from occlusion, and corrupted outputs make subject degradation difficult to identify.Source: F Failure Cases · p. 16
  • Additional correspondence examples show semantic and visual matches alongside visual-feature heatmaps, where yellow marks the least consistent regions and dark red the most consistent.The heatmaps highlight visually inconsistent regions.Source: Figure 9 · p. 17, Figure 10 · p. 18
  • Additional VSM evaluation examples state that the metric can quantify and localize inconsistency and is more consistent with the oracle.The examples use dark red for greatest consistency and yellow for least consistency.Source: Figure 11 · p. 19, Figure 12 · p. 20
  • The VLM prompt evaluates shape, color, texture, and facial features when applicable, then assigns a score from 0 to 100 to a reference/generated image pair.The output format requests only the score.Source: F Failure Cases · p. 21
Loading 2509.21989v1…