PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Evolutionary Prompt Optimization with Cross-Category Transfer for Zero-Shot Anomaly Detection

Sisi Zhu, Changwei Yu, Renshuai Tao, Zhenliang Ni

arXiv:2608.29467v1cs.CV

TL;DR

ZSAD needs prompt rules that are both interpretable and transferable across unseen categories, while existing approaches often use opaque continuous representations and source-category selection. CoEvoAD performs discrete natural-language co-evolution with CCTO-based held-out transfer selection, achieving the highest reported mean scores among compared CLIP-based baselines while retaining important scope limitations.

  • Problem

    Existing ZSAD prompt methods struggle to provide interpretable, scalable natural-language rules and reliable cross-category validation signals.

  • Method

    CoEvoAD evolves separate normal and abnormal natural-language rule populations with a frozen scorer and selects them using held-out source categories as proxy unseen targets.

  • Results

    CoEvoAD attains the highest mean image AUROC/AP (95.0/96.2) and pixel AUROC/PRO (96.9/92.5) among compared CLIP-based ZSAD baselines.

  • Takeaways & Limitations

    Interpretable prompt search is feasible for ZSAD through role-separated co-evolution guided by held-out cross-category signals.

  • Takeaways & Limitations

    Benefits concentrate on pixel-level localization, while image-level gains are modest and not uniformly positive, and performance varies across target categories.

Abstract

from arXiv · show

Zero-shot anomaly detection (ZSAD) has gained significant attention for its practical value in industrial inspection. Recently, CLIP-based approaches have been widely adopted in ZSAD due to their strong vision-language generalization capabilities. However, existing methods commonly employ continuous prompt embeddings for prompt optimization and encode semantics in latent vectors, which lack interpretability and scalability. To this end, we propose CoEvoAD, a co-evolutionary framework for discrete prompt selection. CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm. Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language. Furthermore, we introduce a Cross-Category Transfer Objective (CCTO), which treats held-out source categories as proxies for unseen categories and scores prompt rules based on their estimated cross-category transferability, effectively improving cross-category generalization. Extensive experiments are conducted to validate the effectiveness of CoEvoAD, and the results show that it achieves state-of-the-art performance across multiple anomaly detection datasets. The code is available at https://github.com/rstao-bjtu/CoEvoAD.

1 Introduction

ZSAD reduces the need for target-category data, but existing prompt methods struggle to combine interpretable, scalable rules with reliable cross-category transfer. CoEvoAD addresses these gaps through discrete co-evolution and held-out source-category evaluation.

  • ZSAD uses visual comparisons with normal and abnormal text prompts to detect anomalies in unseen categories.
  • Existing methods encode prompt semantics in continuous representations, limiting interpretability and rule-level reuse.
  • Source-category performance alone provides limited evidence that selected prompts will generalize to unseen categories.
  • CoEvoAD searches discrete natural-language prompt rules with evolutionary generation, evaluation, and selection.
  • CCTO evaluates rules on held-out source categories treated as proxy unseen categories, without target-domain supervision.

2 Related Work

Related work spans conventional target-category anomaly detection, CLIP-based zero-shot detection, and prompt optimization. CoEvoAD is positioned within discrete prompt-rule search for cross-category ZSAD.

  • Traditional Anomaly Detection: Traditional anomaly detectors learn category-specific normal patterns from target-category images and therefore depend on target-category data.
  • Zero-Shot Anomaly Detection: ZSAD detects and localizes anomalies in categories unseen during training using normal and abnormal prompts.
  • Zero-Shot Anomaly Detection: CLIP-based ZSAD methods have progressed from handcrafted prompts toward object-agnostic, static-dynamic, and uncertainty-aware prompt representations.
  • Prompt Tuning and Discrete Prompt Optimization: Prompt optimization includes continuous context learning, black-box tuning, and discrete token-space search.

3 Method

CoEvoAD searches structured natural-language normal and abnormal rules with a frozen scorer, role-preserving evolution, and CCTO-based held-out transfer evaluation. Selected rules are rendered for unseen categories to produce image- and pixel-level anomaly outputs.

  • Discrete Prompt-Rule Co-Evolution: CoEvoAD keeps the prompt-conditioned scorer fixed and optimizes only natural-language rule strings.
  • Rule Representation: Each rule combines a role-compatible template, role-specific descriptors, and a class-name slot rendered into a CLIP text prompt.
  • Co-Evolution: Separate normal and abnormal populations evolve through role-preserving mutation and pair-level fitness, preventing role mixing while retaining semantic contrast.
  • Rule Transfer and Inference: For unseen targets, a fixed transfer policy selects a stored rule pair, renders it with the target class, and produces image-level scores and pixel-level maps.
  • Cross-Category Transfer Objective: CCTO renders a candidate rule on held-out source categories and aggregates the least transferable scores with a bottom-k operator.
  • Cross-Category Transfer Objective: Role-adjusted scores combine in-category performance with cross-category consistency for normal rules and a defect-discriminative margin for abnormal rules.

4 Experiments

CoEvoAD is evaluated under strict source-only cross-dataset transfer across six industrial anomaly-detection benchmarks, with image- and pixel-level comparisons, matched controls, robustness tests, ablations, qualitative analyses, and latency measurements. It achieves strong benchmark and matched-control performance while retaining interpretable rules and avoiding systematic inference overhead.

  • Experimental setup: The evaluation covers six industrial anomaly-detection benchmarks under strict source-only cross-dataset transfer, including VisA→MVTec-AD and MVTec-AD→VisA.BTAD, KSDD2, DAGM, and RSDD serve as external industrial targets without target-domain supervision.
  • Main results: CoEvoAD attains mean image AUROC/AP of 95.0/96.2 and pixel AUROC/PRO of 96.9/92.5, highest among compared CLIP-based ZSAD baselines.The largest image gain occurs on RSDD.
  • Matched comparison: CoEvoAD improves the matched class-name control in both primary transfer directions and yields consistent pixel-level AP and F1 gains on external industrial targets.Per-category confidence intervals exclude zero for VisA→MVTec-AD image AUROC and pixel F1, but include zero for MVTec-AD→VisA.
  • Robustness across categories: Bottom-k ∆ exceeds mean ∆ in all four Table 4 rows, indicating that average matched-control gains are not concentrated on easy high-baseline categories.The analysis tests robustness across per-category improvements.
  • Same-budget search controls: Random search and single-population variants do not consistently improve both transfer directions, while CoEvoAD provides the strongest overall balance across image AUROC and pixel AP.The crossover control obtains the highest Pixel AP on MVTec-AD→VisA.
  • Interpretability and localization: Selected rules remain editable, auditable, and transferable through a class-name slot, combining explicit category anchors with category-specific or generic defect descriptors.Qualitative results also show more concentrated responses around annotated defect regions on representative MVTec-AD and VisA categories.
  • Component analysis: CCTO produces consistent pixel-level gains over the class-name control across all four targets, while role-separated co-evolution alone has mixed image-level effects.The full pairing reaches +0.38 image AUROC under the matched protocol, and CCTO-augmented non-role-separated backbones do not consistently improve over their backbone-only variants.
  • Inference latency: Under one RTX 4090 batch-size-one protocol, evolved prompts differ from the class-name control by under 0.7 ms, while routing adds 0.19/0.27 ms per image.The reported latency comparison attributes no systematic overhead to the evolved rules.

5 Conclusion

CoEvoAD searches discrete natural-language normal and abnormal prompt rules with role-separated co-evolution and a Cross-Category Transfer Objective. Across six industrial benchmarks, it achieves the highest mean image- and pixel-level scores among compared CLIP-based ZSAD baselines while keeping the saved rules auditable.

  • Method: CoEvoAD searches discrete, interpretable normal and abnormal prompt rules in natural language and uses CCTO to favor rules generalizing across unseen target categories.The framework uses role-separated co-evolution guided by held-out cross-category signals.
  • Results: Across six industrial anomaly-detection benchmarks, CoEvoAD attains the highest mean image-level and pixel-level scores among compared CLIP-based ZSAD baselines.Matched-control gains are attributed to prompt-rule search.
  • Implication: The results support interpretable prompt search for ZSAD, with saved natural-language rules complementing continuous prompt embeddings.The rules remain auditable as natural-language strings.

Limitations

CoEvoAD’s benefits are concentrated in pixel-level localization, while image-level AUROC gains are modest and inconsistent. Performance also varies across target datasets and categories, and the method remains constrained by CLIP’s language representation of defects.

  • CoEvoAD’s gains concentrate on pixel-level localization rather than image-level AUROC.Matched-control image AUROC gains are modest and not uniformly positive.
  • CCTO gains vary across target datasets, and the MVTec-AD→VisA direction shows a worst-category pixel-AP regression on pcb4.
  • CoEvoAD inherits CLIP’s limitations for defects that are not well described by natural language.

Ethics Statement

The work uses public benchmark datasets without human-subject data or personally identifiable information. It is intended to assist, not replace, human inspection, and deployment requires oversight, validation, monitoring, and rule auditing.

  • The study uses public benchmark datasets and collects neither human-subject data nor personally identifiable information.
  • CoEvoAD is intended to assist defect screening rather than replace human inspection in safety-critical production.
  • Practical deployment should require human oversight, site-specific validation, continuous monitoring, and auditing of selected rules.The passages also caution against high-stakes deployment without further validation because of distribution shift and source-data or model bias.

A Experimental Protocol and Dataset Details

CoEvoAD is evaluated on six industrial anomaly-detection benchmarks under cross-dataset zero-shot protocols. The primary directions are VisA→MVTec-AD and MVTec-AD→VisA, with distinct source conventions for headline comparisons and matched ablations.

  • Datasets: Evaluation covers six industrial anomaly-detection benchmarks, including MVTec-AD, VisA, BTAD, KSDD2, RSDD, and DAGM.MVTec-AD and VisA provide multiple object, texture, electronics, and food categories, while the remaining datasets target industrial surface or product defects.
  • Cross-dataset zero-shot setting: The primary cross-dataset transfer directions are VisA→MVTec-AD and MVTec-AD→VisA.
  • Cross-dataset zero-shot setting: Headline comparisons use a VisA-trained checkpoint for external industrial targets, whereas matched-ablation tables use an MVTec-AD-trained checkpoint.
  • Cross-dataset zero-shot setting: Target-domain images, labels, masks, statistics, and metrics are excluded from training and validation.Target category names are used only as a frozen source-only transfer-policy key.

B Implementation Details

Implementation combines a fixed prompt-conditioned scorer with discrete evolutionary rule search, source-held-out CCTO scoring, and staged evaluation. Search cost scales superlinearly but below naive quadratic, runs offline without LLM calls, and CCTO’s benefit depends on the search backbone and matched-budget evidence.

  • Stage 1 training: The prompt-conditioned scorer is trained with classification and segmentation losses, defect-focused crop augmentation, and inter-role margin regularization.
  • Prompt-rule search: Rule search uses an evolutionary algorithm with separate normal and abnormal populations and role-preserving updates.
  • Search cost: Search cost is 2N(G+1)|Csrc| scoring slots plus 2Kr|Csrc| final re-ranking evaluations.
  • Scaling with source categories: Per-role candidate scoring grows linearly with source-category count, while total wall-clock is superlinear but below naive quadratic.The search is a one-time offline cost, uses no LLM calls in the loop, and parallelizes across source categories.
  • Matched-budget controls: Adding CCTO to non-role-separated search backbones does not consistently improve their corresponding backbone-only rows.
  • CCTO scoring: CCTO aggregates held-out source-category scores with bottom-k, using k=3, and combines them with in-category scores through role-specific weights.The shared-role configuration uses αccto=0.6, while the role-aware configuration uses λn=0.35 and λa=0.20.
  • Selection diagnostic: Within frozen survivor pools, CCTO has weak target-performance correlations and is therefore treated as a selection objective rather than a fine-grained post-hoc predictor.The MVTec-AD→VisA top-scored candidate ranks 53 of 200 and lies within 0.74 percentage points of the pool’s best target pixel AP.

E Complete Quantitative Results

CoEvoAD’s quantitative gains are strongest for pixel-level localization and vary by transfer direction, target category, and dataset type. Results are favorable on some external industrial and texture benchmarks but reveal clear transfer boundaries.

  • Cross-category transfer: VisA→MVTec-AD matched-control gains are statistically supported for image AUROC and pixel F1, whereas MVTec-AD→VisA gains include zero.The 95% confidence intervals exclude zero for VisA→MVTec-AD image AUROC [+0.10, +0.66] and pixel F1 [+0.06, +0.47], but include zero in the reverse direction.
  • Per-category results: Image AUROC improves on 8/15 VisA→MVTec-AD categories and 7/12 MVTec-AD→VisA categories, while declining on 2 and 5 categories respectively.The per-category breakdown shows that improvements are concentrated in subsets rather than uniform across target categories.
  • Multi-seed stability: Three-seed variability reaches 0.96 image-AUROC points on toothbrush, 1.73 on macaroni2, and 2.17 pixel-AP points on pipe_fryum.Per-category results should therefore be interpreted against the three-seed spread rather than a single run.
  • External industrial transfer: On five external industrial targets, the MVTec-source route gives consistent pixel AP and F1 gains while remaining image-neutral on average.The VisA-source route is image-neutral and slightly negative on pixel AP and F1, so it does not support a pixel-improvement claim.
  • Texture-oriented targets: On texture targets, CoEvoAD is strongest on the evaluated DAGM subset but trails Bayes-PFL by about two percentage points on DTD-Synthetic.The DTD-Synthetic gap is concentrated in two categories whose names have low semantic similarity to source object-defect classes.

F Additional Ablations

Additional ablations examine statistical significance, transfer routing, donor sensitivity, prompt-rule representations, and embedding structure. Together they show deterministic routing and bounded routing costs, while clarifying that evolved rules remain partly hybrid with learned context.

  • Paired category-level significance: MVTec-AD→VisA image-level gains are not distinguishable from category-level noise, unlike the VisA→MVTec-AD image-AUROC interval.The strongest evidence is direction-sensitive ablation and pixel-side robustness rather than a uniform image-level lift.
  • Test-time transfer policy: R1 and R3 coincide because semantic transfer covers every target category, so template fallback never activates once rule transfer is enabled.The MVTec-AD→VisA rows use one frozen rule set to keep the transfer routes matched within the ablation.
  • Per-class routing audit: All 27 target classes retain the same routed donor across three seeds, and the worst per-class image delta is −0.98 pp.On MVTec-AD→VisA, eight of twelve classes route to the same donor, pill, with semantic similarities of 0.756–0.826.
  • Donor-forced counterfactuals: The actual route beats uniform-random routing by +0.1–0.25 pp on all metrics, while adversarial assignments cost at most 0.54/2.26 pp image AUROC and 0.95/2.47 pp pixel AP.The oracle assignment uses target labels and is only an upper reference, not a zero-shot procedure.
  • Selected prompt rules: The selected rules are interpretable strings paired with a learned soft-context slot, so the overall prompt is hybrid rather than fully interpretable.The evolved rules remain saved and reloadable, while the learned latent context is denoted by “X.”
  • Transfer mapping and embeddings: Target categories are mapped to nearest source categories using text-embedding cosine similarity, with the target category name as the only target-side input.The embedding analysis separately reports within-role cosine similarity and cross-role distances for selected rules.

H Additional Qualitative Results and Failure Cases

Additional qualitative figures show CoEvoAD anomaly maps across primary and external targets, while failure analyses delimit where transfer is reliable. The main boundaries involve weak semantic overlap, category instability, and image-level performance ceilings.

  • Qualitative localization: Figures A2–A21 compare green ground-truth contours with CoEvoAD anomaly maps, whose warmer colors indicate higher anomaly scores.The maps cover MVTec-AD, VisA, BTAD, DAGM, KSDD2, and RSDD under the locked rule set.
  • Texture-only targets: On DTD-Synthetic, the blotchy and matted1 categories drive the aggregate gap because their abstract texture names lack clear source object-defect counterparts.Neither nearest-source transfer nor template fallback finds a strong source anchor for these categories.
  • External benchmarks: External industrial results are positive for the MVTec-source route but image-neutral or slightly negative for the VisA-source route on pixel metrics.Table 19 reports both source conventions across five external targets.
  • PCB-style targets: For MVTec-AD→VisA, pixel-AP behavior is dominated by pipe_fryum at +4.05 pp and pcb4 at −3.27 pp, while the category-level interval includes zero.This instability is consistent with the asymmetry between the two transfer directions.
  • Image-saturated categories: Categories already above 99 image AUROC leave no remaining headroom, producing ±0.00 pp changes under the current backbone.These ceiling effects are characterized as limits on improvement rather than failures of prompt search.
  • Per-category visual coverage: The qualitative appendix reports localization for individual categories including bottle, capsule, carpet, grid, hazelnut, leather, screw, tile, wood, zipper, candle, capsules, cashew, chewinggum, macaroni1, pipe_fryum, BTAD wood, DAGM fabric2, KSDD2 metal, and RSDD metal15.These figures provide category-level visual coverage but the supplied captions do not state comparative outcomes.
Loading 2608.29467v1…