PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmenting Visual Anomaly Detection with Automated Interpretability

Antonio De Santis, Arsenio Leo, Marco Brambilla

arXiv:2609.33818v1cs.CVcs.AI

TL;DR

Visual anomaly signals can mix defects with benign differences from normal data, making it difficult to distinguish relevant evidence from distracting variation. The paper uses MLLM-interpreted SAE features to guide interventions on PatchCore; these interventions improve performance across source data and distribution shifts.

  • Problem

    Differences from known-normal data can reflect either anomalies or benign visual variation, motivating the question of whether automated interpretations can distinguish them.

    Source: §1 · p. 1

  • Method

    The approach decomposes PatchCore residuals into sparse features, uses MLLM interpretations to label them, and intervenes on the features before rescoring edited representations.

    Source: §3 · p. 3, §6 · p. 9

  • Results

    0.8724 to 0.8857 macro-average AUROC on source data; interventions also improve performance under synthetic corruptions and real acquisition shifts across the evaluated categories.

    Source: §5 · pp. 7–8, §6 · p. 9

  • Takeaways & Limitations

    Across 43 categories from five benchmarks, interpreted sparse features provide an actionable interface for detector interventions, with labels aligned in aggregate with feature effects on normal and anomalous images.

    Source: §6 · p. 9

    The method requires anomalous calibration examples and calibration-AUROC-based amplification selection, may degrade performance in some cases, and lacks ground-truth annotations to verify individual feature interpretations.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Visual anomaly detectors identify deviations from known-normal data, but their anomaly signals may mix evidence of actual anomalies with benign visual variation. We investigate whether automated interpretability can augment visual anomaly detectors by identifying and intervening on different components of this signal. We decompose PatchCore nearest-normal residuals into sparse features using Sparse Autoencoders (SAEs), and provide high-activation and contrastive non-active examples to a Multimodal LLM, which describes each feature and labels it as anomaly, distractor, or uncertain. These labels guide interventions in the SAE hidden representation, where distractor features are suppressed and anomaly features amplified. The edited representation is then used to reconstruct patch embeddings, which are rescored with PatchCore. Across 40 categories from four benchmarks, applying both interventions jointly improves macro-average image-level AUROC from 0.8724 to 0.8857 on source data and from 0.8066 to 0.8210 under synthetic corruptions. On three additional RobustAD categories with real acquisition shifts, the same interventions improve AUROC from 0.8745 to 0.9056 on source data and from 0.6069 to 0.6599 under real acquisition shifts. Finally, individual feature interventions across all 43 categories show that the MLLM labels are aligned in aggregate with how features differently affect normal and anomalous images.

1 INTRODUCTION

Visual anomaly signals can combine evidence of actual defects with benign variation from known-normal examples. This study asks whether automated interpretations of sparse features can separate these components and guide detector interventions.

1 INTRODUCTION

  • Visual differences in background, lighting, viewpoint, surface appearance, or normal manufacturing variation can move embeddings away from normal references without indicating a defect.The resulting anomaly signal may mix actual anomaly evidence with variation that ideally should not affect detector decisions.Source: §1 · p. 1
  • The method decomposes PatchCore residuals into SAE features, which an MLLM describes and labels as anomaly, distractor, or uncertain using high-activation and contrastive examples.Uncertain features are left unchanged; distractor features are suppressed and anomaly features amplified before rescoring.Source: §1 · pp. 1–2
  • Across 40 categories from four benchmarks, interventions improve PatchCore on source data and persist under synthetic corruptions, with the largest improvements on three RobustAD categories with real acquisition shifts.Individual-feature interventions across 43 categories also show aggregate alignment between MLLM labels and effects on normal versus anomalous images.Source: §1 · p. 2

2 RELATED WORK

Prior work studies sparse feature discovery, automated semantic interpretation, and visual anomaly detection. This paper brings these threads together by using interpreted features to intervene on an anomaly detector and evaluates robustness under synthetic and real acquisition changes.

Mechanistic Interpretability

  • Sparse Autoencoders learn sparse dictionaries of features from activations, offering a way to recover features represented in superposition.Mechanistic interpretability seeks to understand neural-network components, while polysemantic neurons can respond to multiple unrelated concepts.Source: §2 · p. 2
  • Automated interpretability uses language or multimodal models to map discovered features to human-understandable concepts; this work uses those labels to guide feature suppression or amplification.Prior work also connects sparse features with editing internal representations.Source: §2 · p. 2

Visual Anomaly Detection

  • Visual anomaly methods model normality through feature distributions, stored representations, or representation discrepancies; prior interpretability work emphasizes local explanations and anomalous-region localization.In contrast, this work builds a global feature dictionary, semantically interprets it, and uses those interpretations to guide detector interventions.Source: §2 · p. 2
  • Synthetic corruption benchmarks apply controlled perturbations, but RobustAD examines real acquisition changes and prior work finds synthetic corruptions do not fully reflect them.The study evaluates interventions under both settings.Source: §2 · p. 3

3 METHODOLOGY

The method decomposes PatchCore’s deviations from known normality into sparse features, interprets them with an MLLM, then edits labeled features and rescores the reconstructed patch embeddings. It preserves information not captured by the SAE while applying category-level amplification selected on source calibration data.

3 METHODOLOGY

  • PatchCore’s nearest-normal embedding deviations are decomposed into SAE features, interpreted by an MLLM, edited, and rescored with the same frozen detector.A balanced calibration set of normal and anomalous images supplies the patch residuals used to train the SAE.Source: §3 · p. 3
  • The SAE uses tied encoder-decoder weights, no biases, unit-norm feature directions, and TopK sparsity to represent deviations from normality.The TopK operator retains the k largest activations; the SAE is trained to minimize mean squared reconstruction error.Source: §3 · p. 4
  • For each feature, the MLLM receives high-activation and contrastive zero-activation examples, visualizations, and category context to describe its visual pattern.Zero-activation examples are selected for embedding similarity to high-activation examples, and at most one patch per image or scene is used.Source: §3 · p. 4
  • The MLLM labels features as anomaly, distractor, or uncertain; features are automatically marked uncertain when either example group has fewer than two distinct examples.Anomaly denotes evidence of an object defect or violation of expected configuration; distractor denotes nuisance, artifact, or ordinary variation.Source: §3 · p. 4
  • Interventions zero-ablate distractor features, amplify anomaly features by α ≥1, or apply both operations jointly, while leaving uncertain features unchanged.Interventions apply at patches where the corresponding feature is active.Source: §3 · pp. 4–5
  • The method preserves SAE reconstruction error so edits change only the represented feature contribution, then selects α on source calibration AUROC and rescoring uses the unchanged normal memory bank.The selected category-level α is fixed for source and corruption evaluations; edited embeddings undergo a fresh nearest-neighbor search and the frozen PatchCore scoring rule.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

The study evaluates feature interventions on four benchmark families and RobustAD, using source calibration data to assess source performance and robustness to synthetic corruptions and acquisition shifts.

Datasets

  • Evaluation covers 40 categories across VisA, MVTec AD, MVTec LOCO AD, and MVTec AD 2, plus three RobustAD categories under real acquisition shifts.The four-benchmark calibration sets have a median of 20 images per category; RobustAD calibration sets have a median of 150.Source: §4 · p. 5

Implementation Details

  • PatchCore uses a frozen DINOv3 ViT-L/16 backbone, while each category has a separate 64-feature SAE with TopK k = 4.The reported tables cover image-level AUROC on source and synthetically corrupted data, and on RobustAD source and acquisition-shift data.Source: §4 · p. 5, Table 1 · p. 6, Table 2 · p. 7

Evaluation Protocol

  • Image-level AUROC is macro-averaged across categories, with RobustAD reported separately across its three categories.The comparisons include PatchCore, direct uses of calibration data, and AutoInterp-guided, calibration-guided, and random feature interventions.Source: §4 · p. 6

Robustness Evaluations

  • Synthetic robustness tests use brightness, contrast, defocus blur, and Gaussian noise at severities 1–5; RobustAD tests real acquisition shifts.Feature assignments and amplification factors are determined from source calibration data and held fixed for shifted evaluations.Source: §4 · pp. 6–7

5 RESULTS AND DISCUSSION

AutoInterp-guided interventions improve PatchCore’s source and robustness performance, while feature-level analyses find different score effects for anomaly and distractor labels. Controls and examples further characterize these findings.

RQ1: Performance on source data

  • 0.8857 macro-average AUROC is reached with joint AutoInterp-guided intervention, versus 0.8724 for original PatchCore across 40 source-data categories.The joint intervention improves performance on three of four dataset families; MVTec LOCO AD is the exception.Source: §5 · p. 7

Controls

  • Calibration-guided joint intervention reaches 0.8811 overall, while random assignments are weakest; AutoInterp remains stronger overall than calibration guidance.The calibration-guided approach is competitive under synthetic corruptions at 0.8190 AUROC versus 0.8210 for AutoInterp, but reaches 0.5638 on RobustAD shifts versus 0.6599 for AutoInterp.Source: §5 · pp. 7–8

RQ2: Robustness to corruptions and acquisition shifts

  • Under synthetic corruptions, joint intervention raises macro-average AUROC from 0.8066 to 0.8210; on RobustAD it raises shifted AUROC from 0.6069 to 0.6599.The gain under RobustAD acquisition shifts is 5.30 points, compared with a 3.11-point source-data gain; synthetic-corruption gains persist at every severity.Source: §5 · p. 8

Controls

  • The linear probe performs heterogeneously: it beats PatchCore on VisA under synthetic corruptions and on two RobustAD categories, but falls below it elsewhere.The authors suggest this pattern may reflect sensitivity to how well calibration examples represent the evaluation distribution.Source: §5 · p. 8

RQ3: Individual feature interventions by MLLM label

  • Across 43 categories, suppressing anomaly-labeled features reduces scores more on anomalous than normal images, while distractor-feature effects are similar across classes.Anomaly-feature class differences are significant under suppression and amplification; distractor-feature differences are not significant under either intervention.Source: §5 · pp. 8–9, Figure 2 · p. 8

Qualitative and distributional analyses

  • Feature-label separation is concentrated in distribution tails; examples show suppression reducing anomaly-image scores by 42.3% and normal-image scores by 29.32%.Figure 3 pairs feature descriptions and activation examples with feature maps and anomaly maps before and after intervention; its maps share a min-max color scale within each example.Source: §5 · p. 9, Figure 3 · p. 9

6 CONCLUSION AND LIMITATIONS

Across 43 categories from five benchmarks, the interventions improve PatchCore performance on source data and under distribution shifts, with particularly large gains under RobustAD’s real acquisition shifts. The authors identify calibration requirements and limits on per-category gains and feature-label fidelity.

6 CONCLUSION AND LIMITATIONS

  • Across 43 categories from five benchmarks, interventions improve PatchCore performance on source data and under distribution shift, with particularly large gains under RobustAD’s real acquisition shifts.At the feature level, MLLM labels align in aggregate with how features affect normal and anomalous images.Source: §6 · p. 9
  • The method requires anomalous examples in its calibration set and selects the amplification factor using calibration AUROC, so it is not purely normal-only.Interventions do not guarantee improvement for every category or model configuration and can degrade the original detector in some cases.Source: §6 · pp. 9–10
  • Feature descriptions and labels are hypotheses: MLLMs may misinterpret examples, SAE features may remain polysemantic, and no ground-truth annotations establish individual-feature fidelity.The observed consistency between feature behavior and assigned labels is an aggregate finding, not a guarantee for each interpretation.Source: §6 · p. 10

A APPENDIX OVERVIEW

  • The appendix covers SAE training and ablations, multiplier selection, MLLM and detector ablations, vision-backbone ablation, qualitative examples, and the annotation prompt.The paper also includes an AI use statement.Source: A APPENDIX OVERVIEW · p. 13

B SAE TRAINING, METRICS, AND ABLATIONS

The SAE reconstruction retains most original PatchCore AUROC, while tied SAEs reduce dead features and intervention benefits persist when reconstruction error is omitted. Results also show sensitivity to SAE configuration and the value of editing selected rather than random features.

SAE training, metrics, and ablations

  • 95.95% of original PatchCore AUROC is recovered on average across 40 RQ1 categories, and 98.74% on RobustAD, using SAE-reconstructed residuals without reconstruction error.Table 3 reports test-split reconstruction fidelity, macro-averaged across categories; Table 4 reports the RobustAD source test split.Source: B SAE TRAINING, METRICS, AND ABLATIONS · p. 13, Table 3 · p. 13, Table 4 · p. 13
  • Tied and untied SAEs achieve similar reconstruction MSE and recovered AUROC, but the tied SAE has 0.05 dead features on average versus 11.43 of 64.The tied SAE has 0% observed violations of the guarantee that individual feature removal does not increase raw nearest-neighbor patch distance.Source: B SAE TRAINING, METRICS, AND ABLATIONS · p. 14, Table 5 · p. 14
  • Across two tested categories, suppression and joint intervention improve over the original detector for all nine width-and-sparsity configurations, but gain magnitude varies non-monotonically.The authors note that more categories are needed to establish whether a single configuration is best.Source: B SAE TRAINING, METRICS, AND ABLATIONS · p. 14, Table 6 · p. 14
  • With reconstruction error omitted, joint intervention raises source AUROC from 0.8368 to 0.8551 and corrupted AUROC from 0.7760 to 0.8136.Retaining reconstruction error preserves residual information outside the SAE representation; the component AUROCs are not additive because rematching and image scoring are nonlinear.Source: B SAE TRAINING, METRICS, AND ABLATIONS · pp. 14–15, Table 7 · p. 15
  • Random feature interventions reduce performance relative to the unedited SAE-only representation, while AutoInterp remains higher under all three interventions on source data.This indicates that gains within the SAE representation depend on which features are edited, rather than on sparse intervention alone.Source: B SAE TRAINING, METRICS, AND ABLATIONS · p. 15, Table 8 · p. 15

C AMPLIFICATION MULTIPLIER SELECTION

The study compares fixed amplification strengths with a category-level factor selected using clean calibration AUROC. Calibration-selected strength yields the best source AUROC but not the best corrupted AUROC, while larger multipliers can reduce source performance.

Amplification multiplier selection

  • The study tests α values of 1, 1.5, 2, 3, 4, and 5; at α = 1, amplification-only leaves the detector unchanged and joint intervention becomes distractor suppression.In amplification-only, distractor and uncertain features remain unchanged; in the joint setting, distractor features are additionally suppressed.Source: C AMPLIFICATION MULTIPLIER SELECTION · p. 15
  • For joint intervention, 2× gives the highest source AUROC and 3× the highest corrupted AUROC among fixed multipliers; calibration-selected strength gives the highest source AUROC but not corrupted AUROC.For amplification-only, performance rises up to 3× on source and corrupted data, then declines at larger multipliers.Source: C AMPLIFICATION MULTIPLIER SELECTION · p. 16, Table 9 · p. 16
  • Selecting α independently on a small clean calibration set may overfit category-specific variation, while one fixed multiplier cannot account for differences in intervention sensitivity.The selected category-level value is held fixed during source and corruption evaluation.Source: C AMPLIFICATION MULTIPLIER SELECTION · p. 16

D MULTIMODAL LLM ABLATION

The MLLM ablation compares GPT-5.6 Sol with Gemma 4 31B-IT and Qwen3.5-27B on ten categories while holding the detector, SAE, interpretation protocol, and intervention procedure fixed. Joint intervention improves over the original detector on source and corrupted data for all three, despite model-dependent labels.

Multimodal LLM ablation

  • Joint intervention improves over the original detector on both source and corrupted data for all three tested MLLMs.The comparison uses a fixed subset of ten categories and keeps the detector, SAE, interpretation protocol, and downstream procedure unchanged.Source: D MULTIMODAL LLM ABLATION · p. 16, Table 10 · p. 16
  • Pairwise semantic-label agreement is 81.1% between GPT and Gemma, 69.5% between GPT and Qwen, and 67.8% between Gemma and Qwen.The annotators are GPT-5.6 Sol, Gemma 4 31B-IT, and Qwen3.5-27B.Source: D MULTIMODAL LLM ABLATION · p. 16

E ANOMALY DETECTOR ABLATION

On five categories, FRE interventions show different source and corruption patterns: suppression and joint intervention help on source data, while amplification helps under synthetic corruption. The authors describe this detector extension as preliminary and call for experiments on more categories.

E ANOMALY DETECTOR ABLATION

  • FRE uses a frozen DINOv3 ViT-L/16 backbone and a tied linear autoencoder with a 220-dimensional latent representation.The setup uses block 23 features at 256 × 256 input resolution; the autoencoder is trained for 440 epochs with mean squared reconstruction error.Source: E ANOMALY DETECTOR ABLATION · p. 17
  • The FRE residual is decomposed with a separate tied TopK SAE using 64 features and k = 4 per category.The same interpretation and intervention protocol is applied; uncertain features remain unchanged, while distractors are suppressed and anomaly features amplified.Source: E ANOMALY DETECTOR ABLATION · p. 17
  • FRE suppression and joint intervention improve source AUROC by 2.03 and 2.00 points, but their corrupted effects are -0.26 and -0.09 points, respectively.Amplification gains 0.26 source AUROC points and 1.14 corrupted AUROC points; PatchCore interventions yield positive average gains in both evaluations on the same five categories.Source: Table 11 · p. 17, E ANOMALY DETECTOR ABLATION · p. 17
  • The FRE ablation uses five categories and compares intervention deltas in AUROC percentage points against each original detector.Source: Table 11 · p. 17
  • The authors characterize the FRE evidence as preliminary and say more categories are needed for stronger conclusions.Source: E ANOMALY DETECTOR ABLATION · p. 17

F VISION BACKBONE ABLATION

Across five categories, joint intervention transfers positively to SwinV2 but has effects close to zero with WideResNet-50-2. The tested backbones use fixed intervention and corruption protocols, while their original performance differs substantially on Pushpins.

F VISION BACKBONE ABLATION

  • The study repeats the full pipeline with SwinV2-L and WideResNet-50-2 on the five detector-ablation categories, keeping the intervention and corruption protocols fixed.The backbones use their specified pretrained weights and layers; DINOv3-L/16 uses blocks 11 and 23.Source: F VISION BACKBONE ABLATION · p. 17
  • On Pushpins, DINOv3-L/16 reaches 0.7666 and 0.6994 AUROC on source and corrupted data, while WideResNet-50-2 reaches 0.7286 and 0.6807.SwinV2 has lower original AUROC than the other backbones, partly because PatchCore performs at approximately chance on Pushpins in both evaluations.Source: F VISION BACKBONE ABLATION · p. 17
  • Joint intervention improves source and corrupted AUROC by 2.86 and 2.88 points with DINOv3, and by 1.78 and 1.22 points with SwinV2; WideResNet-50-2 changes by -0.19 and -0.07 points.Source: Table 12 · p. 18, F VISION BACKBONE ABLATION · p. 18
  • The backbone ablation compares AUROC percentage-point deltas against each original detector across five categories.Source: Table 12 · p. 18

G ADDITIONAL QUALITATIVE EXAMPLES

This section presents additional qualitative examples of SAE feature labels and their feature and PatchCore maps before and after intervention, alongside prompts for identifying localized concepts and classifying their relevance to the inspection target using activation comparisons while avoiding judgments based on activation magnitude or visualization artifacts.

G ADDITIONAL QUALITATIVE EXAMPLES

  • Anomaly-labeled examples include connector-pin defects, solder bridging, and tangled zipper fibers, with test images, calibration examples, feature maps, and PatchCore maps shown before and after suppression or 2× amplification.The PatchCore maps for each example share a min-max color scale.Source: G ADDITIONAL QUALITATIVE EXAMPLES · pp. 18–19
  • Distractor-labeled examples include background specks and highlighted fibers, while another example is labeled uncertain for capsule surface defects or markings.The examples also include feature-map and PatchCore-map views for the original, suppressed, and amplified representations.Source: G ADDITIONAL QUALITATIVE EXAMPLES · pp. 18–19

H EXAMPLE MLLM ANNOTATION PROMPT

  • The annotation prompt asks the MLLM to find a visual concept in selected high-activation regions that is absent from selected zero-activation regions, comparing boxed locations rather than whole images.The prompt cautions that activation maps and boxes are approximate spatial cues because transformer tokens mix contextual information.Source: H EXAMPLE MLLM ANNOTATION PROMPT · p. 21
  • The prompt defines anomaly, distractor, and uncertain labels by relevance to the inspection target, distinguishing abnormalities from unrelated features or ordinary target variation.Uncertain is for mixed or unclear evidence; rarity alone is not a basis for classification.Source: H EXAMPLE MLLM ANNOTATION PROMPT · p. 24
  • The prompt requests a concise concept name and semantic type grounded in the localized visual attribute distinguishing activating examples, not the whole patch or weak incidental correlations.It also cautions against vague concept names and using activation magnitude alone to classify examples.Source: H EXAMPLE MLLM ANNOTATION PROMPT · p. 24
  • The prompt directs the MLLM to use the normal reference grid only to understand normal appearance and to ignore colors, textures, or boundaries introduced by the activation-map visualization.The response must be a JSON object with exactly two keys: concept name and semantic type.Source: H EXAMPLE MLLM ANNOTATION PROMPT · p. 24
Loading 2609.33818v1…