PaperlayerView exact revision on arXiv

Source-linked AI summary

TED:Text-Axis Evidence Decomposition for Prompted Anomaly Localization

JinYoung Kim, Geonho Kim, GiJeong Park, Geonu Lee, YoungJoon Yoo

arXiv:2609.39033v1cs.CVcs.AI

TL;DR

Adapted CLIP-based anomaly detectors can assign high scores to both defects and visually complex normal regions, leaving local evidence difficult to rank. TED compares ambiguous responses against source defect and hard-normal evidence without changing the backbone or prompts or using target-domain training. Across frozen backbones, adapted hosts, and transfer settings, it improves pixel-level localization.

  • Problem

    CLIP-based anomaly localization can leave true defects and visually complex normal regions difficult to distinguish, even when the normal-versus-anomaly response contains useful anomaly information.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Method

    TED re-ranks ambiguous local responses by comparing support from source defect patches and source hard false positives under the host’s text response.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across frozen multimodal backbones, adapted hosts, and cross-domain transfer settings, TED improves pixel-level localization.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results suggest that prompted anomaly localization is also a local evidence-decoding problem, with gains depending on remaining hard-FP competition and representative source evidence.

    Source: §5 · p. 10

    Correction quality can be less reliable when source banks lack defect types or normal structures relevant to the target domain.

    Source: E Limitations and Broader Impacts · p. 26

Abstract

from arXiv · show

CLIP is a powerful vision-language model, but it was not designed for fine-grained defect localization; CLIP-based anomaly detectors therefore adapt it with prompts or lightweight modules to increase defect sensitivity. We show that stronger sensitivity does not necessarily make local evidence reliable: under domain shift, adapted CLIP-AD models often assign high anomaly scores to both true defects and visually complex normal regions. The issue is not simply missing defect information, but a local scoring rule that decodes defect and hard-normal evidence, having the same anomaly evidence. We propose TED (Text-Axis Evidence Decomposition), a post-hoc scoring method that asks whether each ambiguous response is better supported by source defect patches or by source normal patches mistaken as anomalous. TED compares these supports under the host's normal-versus-anomaly text response, leaves the backbone and prompts unchanged, and requires no target-domain training. It works as a train-free score for raw VLM backbones or as a source-calibrated residual correction for adapted CLIP-AD hosts. Across frozen VLM backbones, TED substantially improves pixel-level localization over raw prompt similarity; across adapted hosts, it improves most pixel-level settings over P-AUROC, P-PRO, and P-AP. Gains are largest under stronger hard-FP competition, with mean localization gain increasing from +5.0 in low-competition regimes to about +10.9 in mid/high-competition regimes. These results suggest that recoverable defect evidence can already exist in pretrained multimodal representations, but reliable localization requires decoding it against hard-normal competitors. Code will be released at TED GitHub repository.

1 Introduction

Adaptation can increase defect sensitivity without making local scores reliably distinguish defects from visually complex normal regions. TED re-ranks ambiguous patches by comparing defect and hard-normal source evidence, while preserving the host and requiring no target-domain training.

1 Introduction

  • Adapted CLIP-AD models can score true defects and visually complex normal regions similarly, despite separating defects more readily from ordinary background.The local failure is hard-FP entanglement, not simply missing defect information.Source: §1 · pp. 1–2
  • TED compares each ambiguous patch with source defect and hard-FP examples under the host’s normal-versus-anomaly text response, boosting or suppressing its score according to support.It leaves the backbone and prompts unchanged and requires no target-domain training.Source: §1 · p. 2
  • TED provides a train-free score for raw VLM backbones and a source-calibrated residual for adapted CLIP-AD hosts, improving pixel-level localization most when hard-normal evidence competes with defects.Its post-hoc local rescoring is not tightly tied to a host architecture.Source: §1 · p. 2
  • The normal-versus-anomaly response remains useful, but it is insufficiently selective to reliably rank true defects above hard false positives.Simple false-positive suppression is insufficient because the same local response can support both defects and visually complex normal regions.Source: §1 · p. 2

2 Related Work

Prior anomaly-localization methods use feature statistics, patch distances, reconstruction, or feature differences, while prompted VLM methods use language-guided scoring and adaptation. TED instead retains the host response and changes how ambiguous local evidence is decoded.

2 Related Work

  • Feature-based, distance, flow, reconstruction, and teacher-student methods score local anomalies using distinct feature or discrepancy signals.Examples include SPADE [5], PaDiM [7], PatchCore, FastFlow [33], and CFlow-AD [12].Source: §2 · pp. 2–3
  • Prompted VLM methods use prompt ensembles or learned prompts and lightweight adaptation, but greater defect sensitivity does not necessarily separate hard normals from true defects.WinCLIP [16] uses normal and anomalous prompt ensembles; AnomalyCLIP [40] and AdaCLIP [3] are examples of prompt adaptation.Source: §2 · p. 3
  • TED differs from prompt learners and adapters by keeping the host response and decoding ambiguous local evidence through source defect and hard-FP support.This approach treats local scoring and ranking as part of localization quality, not only the representation.Source: §2 · p. 3
  • TED needs patch-level visual features and a normal-versus-anomaly text response, enabling its evidence comparison on frozen VLM backbones or adapted CLIP-AD hosts.The related work contrasts this lightweight local layer with multimodal-language-model approaches targeting reasoning, explanation, or end-to-end decisions.Source: §2 · p. 3

3 Proposed Method

TED addresses hard-FP entanglement by comparing ambiguous patch responses with source defects and hard false positives on the host’s normal-versus-anomaly text response. It offers a train-free local score and a bounded source-calibrated correction while leaving the backbone and prompts unchanged.

3.1 From Hard-FP Competition to Evidence Decomposition

  • Adapted hosts can separate defects from ordinary normal patches yet confuse defects with visually complex hard false positives, which may receive similarly high anomaly responses.TED therefore re-ranks ambiguous responses by whether source defect or source hard-FP evidence better supports them.Source: §3.1 · pp. 3–4

3.2 TED Overview

  • TED leaves the backbone, prompts, and host detector unchanged, boosting patches supported by source defects and suppressing those supported by source hard false positives.Neither mode uses target images, masks, labels, or target-score selection.Source: §3.2 · p. 4

3.3 Text-Axis Evidence Banks and Train-Free Score

  • T-TED compares query responses with source defect and hard-FP banks on the host text direction, using their support difference as a local score.Positive scores indicate more defect-like patches and negative scores more hard-FP-like patches; the score is averaged over host-selected layers and replaces raw prompt similarity.Source: §3.3 · p. 4
  • Figure 3 shows source-supported re-ranking separating defects from hard false positives while retaining strong separation from generic outside regions.Figure 4 contrasts raw prompt scoring, which can rank hard-normal patches close to or above defects, with TED’s source-supported re-ranking.Source: Figure 3 · p. 5, Figure 4 · p. 5

3.4 Source-Calibrated TED Residual

  • C-TED adds a bounded residual from the source defect-versus-hard-FP margin to the host score, trained on source pairs and frozen before target evaluation.The residual is trained to rank source defects above source hard-FPs while preserving pairs already separated by the host; no target images, masks, labels, scores, or metrics are used.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate TED on frozen VLMs, adapted CLIP-AD hosts, and recipe changes under a source-to-target protocol. Results show that local evidence decoding complements adaptation, with larger corrections when hard-normal responses compete strongly with defects.

4.1 Experimental Setting

  • The evaluation covers raw frozen VLMs, unchanged adapted CLIP-AD hosts with post-hoc C-TED, and layer or backbone recipe diagnostics.Source banks, calibration, and inference use no target masks, labels, scores, or metrics.Source: §4.1 · pp. 5–6

4.2 Main Results and Diagnostics

  • C-TED improves most pixel-level settings across adapted CLIP-AD hosts and gains more when defect and hard-normal responses overlap than when hosts already separate them.It complements host adaptation by correcting residual local-ranking errors, using source-domain evidence banks.Source: §4.2 · p. 7, Figure 6 · p. 7
  • +5.0 to +10.9: mean localization gain over ∆P-PRO and ∆P-AP rises from Low to Mid/High hard-FP regimes.Regimes are grouped using baseline hard-normal responses, not C-TED scores or target metrics; the grouping is diagnostic, not a tuning rule.Source: §4.2 · pp. 7–8
  • Recipe changes can shift host responses without clearly separating traced defect and hard-FP patches, while C-TED improves local PRO/AP after recipe retraining and bank reconstruction.Recipe search changes the features used; C-TED changes how ambiguous patches are compared with source evidence.Source: §4.2 · p. 8, Figure 7 · p. 8

4.3 Ablation and Control Studies

  • Hard-FP support and a bounded source-calibrated residual outperform host-score replacement or larger residual capacity as the correction strategy.Defect-only support leaves defect and hard-normal patches more entangled than adding hard-FP support.Source: §4.3 · p. 8, Table 3 · p. 9
  • Corrupting defect-versus-hard-FP supervision reduces or removes gains, while scalar source readouts are weaker than C-TED in completed settings.The controls support the role of evidence direction rather than source-bank access or residual capacity alone.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Across tested source-bank budgets, localization gains are nearly identical, supporting a fixed bank policy without target-domain score selection.The correction appears to saturate at a moderate retained-bank size.Source: §4.3 · p. 9

5 Conclusion

TED re-ranks ambiguous local responses by comparing source defect support with source hard-false-positive support, rather than replacing the host detector. It operates without target-domain training and uses a fixed source-evidence policy for raw backbones and adapted hosts.

5 Conclusion

  • TED provides a train-free score for raw VLM backbones and a source-calibrated residual correction for adapted CLIP-AD hosts, leaving visual backbones and prompts unchanged.It targets local ranking by contrasting source defect support with source hard-false-positive support.Source: §5 · pp. 9–10
  • Across frozen backbones, adapted hosts, and cross-domain transfer settings, TED improves pixel-level localization by reducing hard-FP competition.The conclusion identifies remaining hard-FP competition and representative source evidence as conditions associated with gains.Source: §5 · p. 10

A.1 Implementation Details

  • Source defect banks use anomalous patches overlapping source masks, while hard-FP banks use source normal patches scored highly anomalous by the host.For changed feature recipes, banks are rebuilt for the resulting representation.Source: §A.1 · p. 13
  • C-TED calibrates a bounded residual from source defect and hard-FP pairs, with parameters frozen before target evaluation.The objective raises source defect patches above hard-FP patches while preserving pairs already separated by the host.Source: §A.1 · p. 13

A.2 Fixed Evaluation Policy

  • The fixed source-calibration policy uses no target masks, labels, scores, or metrics for selecting banks or calibration settings.Class names are used only to instantiate fixed prompts when the baseline protocol requires them.Source: §A.2 · p. 13

A.3 Dataset, Metrics, and Compute

The appendix describes cross-domain industrial anomaly-localization datasets and reports image- and pixel-level metrics. It also includes source-supervision controls and fixed-compute implementation details.

A.3 Dataset, Metrics, and Compute

  • The evaluation uses MVTec AD, VisA, MPDD, and BTAD, with MVTec AD 2 as a held-out diagnostic target when completed.The datasets cover industrial objects, textures, metalpart defects, and pixel-level annotations.Source: §A.3 · p. 13
  • Pixel-level P-AUROC, P-AP, and P-PRO are primary because TED modifies local anomaly maps; I-AUROC is reported for transparency.Image-level AUROC is secondary, with official host global branches preserved when available.Source: §A.3 · p. 13
  • Correct C-TED gives positive reported changes of +1.03, +3.88, and +1.71, while label shuffle and label swap give negative changes.The supplied values do not identify their metric labels, so no metric-value pairing is asserted.Source: §A.3 · p. 14
  • A random subspace variant reports +0.90, +3.34, and +0.59, without metric labels in the supplied passage.The values are reported as a variant, not paired with specific metrics here.Source: §A.3 · p. 14
  • Experiments used NVIDIA RTX 6000 Ada GPUs with 48GB memory; most jobs used one GPU, and retained-bank peak memory was about 3.8–4.1GB for the named hosts.Banks are built once from the source split and reused at inference.Source: §A.3 · p. 14

B.1 Training-Control Sanity Checks

The appendix tests whether TED’s gains depend on source-label semantics, text-axis evidence decoding, recipe choices, and retained bank size. The controls and diagnostics support source evidence comparison while indicating that gains persist across recipe changes and moderate bank budgets.

B.1 Training-Control Sanity Checks

  • Correct C-TED outperforms label-swapped supervision on several reported transfer metrics, while a random residual subspace yields a smaller AP gain than correct C-TED.For AA-CLIP MVTec→BTAD, correct C-TED improves P-AUC/PRO/AP by +1.03/+3.88/+1.71, versus label swap at −6.40/−8.28/−10.45.Source: §B.1 · p. 14

B.2 Source-Supervision Semantics

  • Shuffling or swapping source labels degrades C-TED despite fixed source split, bank size, residual capacity, and calibration budget.This indicates that added source-only residual capacity alone does not explain the gain.Source: §B.2 · p. 14, Figure 8 · p. 15

B.3 Source Evidence in Frozen VLMs

  • Source-supported scoring improves defect-versus-hard-normal AUC over raw prompt scoring, and text-axis support is closest to TED.The text-axis variant retains the normal-versus-anomaly text coordinate.Source: §B.3 · p. 14, Figure 9 · p. 15

B.4 Layer-Recipe Stress Test

  • C-TED can improve localization after layer-recipe changes, although the effect varies by host and metric.For AdaCLIP's full recipe on MPDD, P-PRO rises from 30.48 to 44.00, while P-AP changes from 29.76 to 29.93.Source: §B.4 · pp. 14–15, Table 8 · p. 15

B.5 Source-Bank Budget and Efficiency

  • Increasing bank budgets from 64 to 512 keeps ∆Loc in narrow ranges for AA-CLIP and FAPrompt, supporting budget robustness rather than a precise scaling law.For AA-CLIP, ∆Loc changes from +2.7 to +2.9; for FAPrompt it remains within +8.6–+9.1.Source: §B.5 · p. 15, Table 9 · p. 16

B.6 Full-Feature Memory Replacement Control

Full-feature memory retrieval does not reproduce C-TED’s joint pixel-level improvements. The controls suggest that the evidence readout, not source-bank access alone, matters.

B.6 Full-Feature Memory Replacement Control

  • The extended controls hold source banks, hosts, targets, and aggregation fixed while changing only the evidence readout across 18 settings.The comparison includes 1-NN, prototype, and LogMeanExp in replacement and residual modes.Source: §B.6 · p. 16, Table 11 · p. 17
  • Most full-feature transfers degrade, and no tested Full-D readout improves all three pixel-level metrics on average, unlike C-TED.The prototype residual gains more P-PRO but reduces P-AUC and P-AP.Source: §B.6 · p. 16

B.7 Weak-Source and Bank-Budget Robustness

Across source-budget and readout controls, C-TED’s advantage is tied to comparing defect and hard-normal support against the host response, with the correction kept bounded.

B.7 Weak-Source and Bank-Budget Robustness

  • AA-CLIP retains positive localization gains at its smallest tested source-image budget, while FAPrompt gains remain similar when the bank budget falls from 64 to 8 entries per class.These are reduced-source, not zero-source, settings; representative defect and hard-FP evidence is still needed.Source: §B.7 · p. 17

B.8 Simple Source-Supervised Readout Controls

  • The MLP rank readout is weaker than C-TED in completed settings: replacement is brittle, and residual use is more stable but still underperforms.The comparison uses the same source defect and hard-FP banks.Source: §B.8 · p. 17, Table 13 · p. 18

B.9 Residual-Strength Boundary

  • Moderate residual strengths give small positive gains, but overly strong insertion can reduce localization quality.The diagnostic supports a bounded correction anchored to the host response, not a universal strength-sweep claim.Source: §B.9 · p. 18, Table 14 · p. 18

B.10 Host-Score-Only Calibration Control

  • Host-score-only calibration is not a reliable substitute: both AA-CLIP variants decrease AP, while C-TED improves PRO and AP.C-TED also gives the strongest AP change among compared variants for FAPrompt and AdaCLIP.Source: §B.10 · p. 18, Table 15 · p. 18

C.1 Raw VLM Failure and Recovery

  • On frozen CLIP, raw prompt maps respond to salient normal structures, while TED re-ranks local responses using source defect and hard-FP support.The figure describes raw scoring as confusing defect evidence with hard-normal saliency and TED as recovering a cleaner local ranking.Source: §C.1 · pp. 18–19, Figure 10 · p. 19

C.2 Failure-Conditioned Separability Robustness

The adapted-host results show that C-TED can improve local ranking, especially when hard-normal evidence competes with defects, but changes are host- and metric-dependent.

C.2 Failure-Conditioned Separability Robustness

  • Across 70 completed settings, TED improves anomalous-versus-hard-normal AUC, mean score margin, and localization metrics with positive setting-level bootstrap intervals.The analysis treats reduced hard-normal competition as mechanism evidence, not baseline severity as a universal predictor of gain.Source: §C.2 · p. 19, Figure 12 · p. 21

C.3 Qualitative Localization on an Adapted Host

  • Figure 11 presents selected qualitative C-TED examples for FAPrompt across four target datasets.Ground-truth defects are outlined in red, with a shared color scale for each Host/C-TED pair.Source: §C.3 · p. 19, Figure 11 · p. 20

D.1 Full Adapted-Host Results

  • Across adapted hosts, the clearest gains occur when hard-normal responses remain close to defects; already calibrated or saturated settings can show smaller or mixed changes.The detailed results are evidence for local evidence decoding, not uniform improvement across every host, backbone, and metric.Source: §D.1 · p. 19

D.1.4 AdaCLIP

  • AdaCLIP’s ViT-L/14 OpenAI setting on MVTec improves PRO from 44.32 to 49.01, although P-AUC or AP changes are mixed in some settings.The reported gains can be region-level without uniform improvement in threshold-free ranking or sparse-pixel AP.Source: §D.1.4 · pp. 21–22

D.1.1 AA-CLIP

  • For AA-CLIP, ViT-B/16+ on MVTec improves P-AUC/PRO/AP from 39.27/11.86/2.97 to 69.22/38.88/10.39.Weak baseline localization and competitive hard-normal responses are settings where C-TED can recover local ranking; a strong-baseline BTAD case has lower PRO/AP after correction.Source: §D.1.1 · p. 21, Table 17 · p. 22

D.1.2 FAPrompt

  • For FAPrompt, ViT-L/14-336 on BTAD improves P-AUC/PRO/AP from 90.52/60.32/21.54 to 94.99/70.95/45.01.The gains span ranking, region-level, and sparse-pixel localization; other settings show smaller gains.Source: §D.1.2 · p. 21, Table 18 · p. 23

D.1.5 BayesPFL

  • BayesPFL shows mixed pixel-level behavior: ViT-L/14 OpenAI on BTAD gains PRO from 64.28 to 65.57 while AP falls from 35.98 to 34.30.Image-level gains do not necessarily imply cleaner dense localization, making BayesPFL a boundary case for the method’s scope.Source: §D.1.5 · pp. 23–24

D.1.3 AdaptCLIP

  • For AdaptCLIP, ViT-L/14 OpenAI on BTAD changes P-AUC/PRO/AP from 93.80/65.41/41.15 to 94.09/65.81/41.58.These small gains are directionally consistent, while other entries are mixed, particularly for already strong or seed-unstable baselines.Source: §D.1.3 · p. 21, Table 19 · p. 24

D.2 Full Recipe-Retuning Results

C-TED remains useful after hosts are retrained with changed layer recipes, though its gains vary by host, recipe, and metric.

D.2 Full Recipe-Retuning Results

  • After bidirectional recipe retuning, FAPrompt improves AP across listed layer recipes; VisA→MVTec Full(B) rises from 22.35 to 30.94.AdaCLIP also shows PRO gains, including 34.83 to 46.27 for VisA→MVTec L-r and 54.13 to 65.18 for MVTec→VisA Full(B).Source: §D.2 · pp. 24–25
  • The results distinguish representation selection from evidence decoding: retraining or changing the layer recipe can shift host responses without necessarily removing hard-FP ranking errors.C-TED can still improve local ranking after retuning, but effects are not uniform across cells.Source: §D.2 · pp. 24–25

E Limitations and Broader Impacts

TED’s primary claim is pixel-level localization, and its correction depends on representative source evidence and the evaluated VLM backbones and CLIP-AD hosts. The authors also identify image-level and deployment reliability boundaries.

E Limitations and Broader Impacts

  • TED’s primary claim is pixel-level localization; image-level performance can still depend on the host’s global branch, pooling, or score aggregation.I-AUROC is reported for transparency, not as the method’s primary claim.Source: E Limitations and Broader Impacts · p. 25
  • If target defects or normal structures are absent from source banks, or source masks and mined hard-FP patches are unrepresentative, correction may be less reliable or provide limited benefit.Fixed-bank and source-supervision controls reduce some confounding explanations but do not remove the need for representative source evidence.Source: E Limitations and Broader Impacts · p. 26
  • Evaluation covers the VLM backbones and CLIP-AD hosts considered; backbones with weak patch features, unstable text responses, or incompatible feature scales need further validation.Source-calibrated residuals also require source-side calibration and bank storage, although moderate retained-bank sizes suffice in tested settings.Source: E Limitations and Broader Impacts · p. 26
  • Future work should examine source-bank construction, image-level aggregation, broader host families, failure cases, and monitoring under larger distribution shifts.Source: E Limitations and Broader Impacts · p. 27
  • In quality- or safety-critical inspection, missed defects and false alarms can harm product quality or safety, raise inspection costs, or reduce operator trust.The authors recommend human oversight, periodic target-domain validation, and monitoring for false positives and missed defects.Source: E Limitations and Broader Impacts · p. 27

Related papers

Loading 2609.39033v1…