PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)

Lars Heckler-Kram, Dorian Henning, Ashwin Vaidya, Jan-Hendrik Neudeck, Ulla Scheler, Anton Milan, Samet Akcay, Paula Ramos, Sebastian Höfer

arXiv:2610.04392v1cs.CV

TL;DR

Existing anomaly-detection benchmarks are saturated and may not reflect the restricted test access and application conditions of deployment. VAND 4.0 introduces a hidden-test benchmark across industrial manufacturing and retail logistics, assesses efficiency, and reports persistent challenges involving rare defects and resource use.

  • Problem

    Saturated benchmarks, unrestricted test access, and inconsistent evaluation protocols limit meaningful comparison and assessment of real-world anomaly-detection methods.

    Source: §1 · p. 1

  • Method

    VAND 4.0 evaluates visual anomaly-detection submissions on held-out industrial and retail test datasets and assesses computational efficiency with a dedicated metric.

    Source: §1 · pp. 1–2

  • Results

    Supervised retail detection is approaching saturation for common defects, while rare defects remain unsolved, off-the-shelf VLMs trail specialized models, and improving the performance-efficiency trade-off remains open.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Kaputt-Rare provides a retail anomaly-detection setting with approximately 1% defect prevalence, designed for learning from almost defect-free data.

    Source: §6 · p. 8

    Efficiency evaluation occurred post-hoc rather than as a live competition criterion, which limited participation.

    Source: §6 · p. 8

Abstract

from arXiv · show

Existing Anomaly Detection benchmarks are saturated and often unrealistic. As part of the VAND 4.0 Challenge, we introduce a hidden-test, application-driven benchmark across two deployment-critical domains: industrial manufacturing and retail logistics. In the Industrial Track (MVTec AD 2), the results reveal that unsupervised anomaly segmentation remains challenging: the best regular-setting method achieves only ~57\% pixel-level $SegF_1$, indicating substantial room for improvement. Zero-shot approaches trail by ~15 $SegF_1$ points, confirming that task-specific training on normal data remains essential for precise defect localization. Robustness to distribution shifts remains a key open challenge and DINOv3-backbones clearly dominate this track. In the Retail Track (Kaputt 2), the results reveal that (1) supervised defect detection is approaching saturation for common defect types; (2) the best off-the-shelf VLM approach trails specialized models by ~28 AP, confirming that currently VLMs cannot replace fine-tuned detectors, (3) reference images did not prove helpful for top-performing approaches. Performance collapses on rare defects (spillage ~53 AP, missing units ~27 AP), where the supervised ceiling is bounded by data availability. To drive future progress in this domain, we provide a new low-prevalence retail AD dataset (Kaputt-Rare). Across both tracks, computational efficiency is assessed as a first-class metric combining performance, throughput, memory, and power consumption. We introduce a novel metric for measuring efficiency and reveal that that top-performing methods rely on heavy architectures while efficiency is largely neglected. Overall, we conclude that the community needs (a) more efficiency-aware method development, and (b) true anomaly detection approaches for rare defects and shifting conditions. https://sites.google.com/view/vand4-cvpr2026/challenge

1. Introduction

Existing anomaly-detection benchmarks are saturated, expose test data, and vary in evaluation protocols, limiting comparison and real-world relevance. The paper introduces a unified, application-driven benchmark for manufacturing and retail logistics, with held-out evaluation, efficiency assessment, and a low-prevalence retail dataset.

Motivation

  • Existing industrial benchmarks have reached up to 99.9% AU-ROC, while unrestricted test access and inconsistent protocols complicate comparison and real-world evaluation.The introduction identifies robustness to uncontrolled settings as an unresolved challenge.Source: §1 · p. 1

Benchmark contribution

  • The benchmark evaluates visual anomaly detection in manufacturing and retail logistics using held-out test datasets and objective criteria.The VAND 4.0 Challenge also analyzes submissions and open challenges.Source: §1 · pp. 1–2

Efficiency

  • The benchmark treats algorithmic efficiency as a key criterion and introduces a metric to measure this deployment-relevant property.The authors call for more efficiency-focused method development.Source: §1 · p. 2

Kaputt-Rare

  • Kaputt-Rare provides a retail logistics dataset with approximately 1% defect prevalence for methods learning from almost defect-free data.It is intended as a more challenging and realistic setting than Kaputt.Source: §1 · p. 2

2. Related Work

The related work reviews anomaly-detection paradigms, benchmark limitations, and the evolution of VAND challenges. It motivates evaluation on restricted test data and recognition of efficiency alongside detection performance.

2.1. Anomaly Detection Methods

  • Supervised methods use labeled defects; unsupervised methods use only normal references, while few-shot and zero-shot methods require minimal or no target-domain references.Supervised results can provide an upper-bound baseline, although defect annotation is costly.Source: §2.1 · p. 2
  • Unsupervised industrial methods include memory-bank, reconstruction, and student-teacher approaches that learn normal-data representations and flag deviations at test time.Some approaches introduce synthetic defects during training.Source: §2.1 · p. 2
  • VLMs use language and vision representations to detect anomalies with potentially no target-domain references, while MLLMs are appealing for retail variability but costly for real-time deployment.The cited VLM opportunity is linked to CLIP, and the passage notes prior work.Source: §2.1 · p. 2

2.2. Anomaly Detection Datasets

  • Saturated benchmarks and unrestricted defect-test access can reduce comparability and compromise evaluation of generalization to unseen defects.The challenge therefore uses MVTec AD 2 and Kaputt 2 with restricted test access.Source: §2.2 · pp. 2–3

2.3. Previous VAND Challenges

  • VAND challenges expanded from few-shot industrial detection to VLM evaluation, robustness under changing conditions, retail logistics, and explicit efficiency evaluation.The challenge series began in 2023 and added retail logistics for CVPR 2026.Source: §2.3 · p. 3

3. VAND4.0 Challenge and Benchmark Design

VAND 4.0 benchmarks industrial and retail anomaly detection under application-oriented settings, with performance and computational efficiency evaluated across restricted-access datasets. Its metrics assess localization, image-level detection, robustness, and resource use.

3. VAND4.0 Challenge and Benchmark Design

  • The challenge has industrial and retail tracks, and treats model efficiency as an evaluation criterion alongside detection performance.Award categories include runtime, memory, and power consumption.Source: §3 · p. 3, §3.1 · p. 3

3.1. Tracks, Settings and Award Categories

  • The industrial track evaluates segmentation on MVTec AD 2, with regular training restricted to normal images and zero-shot training barred from using that dataset.The dataset supports assessment of robustness to varying lighting conditions.Source: §3.1 · p. 3
  • The retail track uses Kaputt 2 for image-level detection, allowing supervised or anomaly-detection methods in the regular setting and off-the-shelf VLMs or MLLMs without core-model weight modification.The off-the-shelf setting targets zero-shot or few-shot evaluation.Source: §3.1 · p. 3

3.2. Datasets

  • MVTec AD 2 contains more than 8,000 high-resolution images across eight categories, with normal-only training and test sets spanning seen and varied lighting.Kaputt 2 is a held-out set of 10,000 RGB images with approximately 29% defect prevalence and hidden labels.Source: §3.2 · pp. 3–4

3.3. Metrics

  • Industrial evaluation uses pixel-level SegF1, requires threshold selection, and ranks submissions by average performance rank across seen and mixed-lighting test sets.When final ranks tie, the smaller absolute difference between SegF1,priv and SegF1,priv mix breaks the tie.Source: §3.3 · p. 4
  • Retail performance is measured by APany across ranked defect probabilities, while the efficiency score combines task performance, normalized throughput, memory, and power.The efficiency score uses a geometric mean; throughput normalization has diminishing returns, and resource use is preferred to be lower.Source: §3.3 · p. 4

3.4. Submission Process and Evaluation Protocol

  • Test access is restricted through limited submission budgets, and leaderboard eligibility requires reproducible code, model weights or download instructions, and a public technical report.Efficiency was measured after the challenge deadline under unified hardware.Source: §3.4 · p. 5

4. Results and Analysis

Industrial results show DINOv3-based methods leading while robustness to lighting shifts remains difficult; retail results show near-saturation for common defects but persistent failures on rare types. Across both tracks, efficiency remains a challenge for high-performing methods.

4.1. Industrial Track

  • More than 2.5 points on SegF1,priv and nearly 3 points on SegF1,priv mix separated SuperADD from the previous winner; both leading regular-setting methods used DINOv3 backbones.SuperADD also advanced normal-feature subsampling, while ISVL++ used synthetic anomalies to train a segmentation head and estimate thresholds.Source: §4.1 · p. 5
  • Industrial submissions often combined morphology, connected-component filtering, or SAM refinement with segmentation; tiling and multi-scale processing address subtle defects in high-resolution images but can harm efficiency.The regular-setting leaderboard received 9 complete submissions and the zero-shot setting received 5.Source: §4.1 · pp. 5–6
  • Most regular-setting industrial methods showed a substantial SegF1,priv versus SegF1,priv mix difference under lighting shifts, while zero-shot gaps were usually below 1.5 percentage points.The passage identifies robustness mechanisms as an open question, including whether target-domain normal samples can improve it.Source: §4.1 · p. 6

4.2. Retail Track

  • Top regular-setting retail submissions scored 89.27–90.84 APany, with the top 30 spanning less than 1.6 points; the winner's 56-view ensemble cost roughly 3–5× a single model.The leading methods used supervised Kaputt training and DINOv3 patch representations; reference images were rarely used.Source: §4.2 · p. 6
  • The best off-the-shelf VLM scored 61.88 APany, approximately 29 AP points below the best supervised model; prompt engineering improved a naive baseline by 19.6 AP.Multi-prompt diversity outperformed multi-scale or multi-model diversity among the reported approaches.Source: §4.2 · p. 6
  • Common retail defects appear close to saturation: the top 30 were within 1.6 APany, while further gains relied on computationally expensive ensembles.The analysis attributes this pattern to sufficient labeled data for common, visually salient defects.Source: §4.2 · pp. 6–7
  • Spillage reached only ~53 APany and missing unit ~27 APany, with rare-defect failures linked to scarce training examples; reference images did not close this gap.Kaputt had only 705 spillage instances, and top-performing methods did not use reference images as a primary signal.Source: §4.2 · p. 7

4.3. Efficiency

  • Efficiency scores combine performance, throughput, memory, and power; SPR led the industrial efficiency ranking at 0.1937 despite the lowest performance score among considered methods.The only eligible retail method scored 0.3473; overall, submissions prioritized performance over resource-efficient inference.Source: Table 5 · p. 7, §4.3 · p. 7

5. Discussion

The benchmark compares anomaly detection in manufacturing and retail, where general-purpose approaches remain behind task-specific methods. Rare defects, lighting shifts, and resource demands remain central challenges in these settings.

5. Discussion

  • The industrial zero-shot gap was ~15 SegF1 below regular, and the retail off-the-shelf VLM gap was ~29 APany below supervised; general-purpose models did not match task-specific fine-tuning.Prompt engineering and transfer learning from related domains substantially narrowed these gaps.Source: §5 · pp. 7–8
  • Rare retail defects remain difficult for supervised approaches without additional data, while industrial methods show degradation under unseen lighting conditions.These challenges reflect the rarity of defects and variability in image-acquisition conditions.Source: §5 · p. 8
  • The strongest-performing methods relied on expensive retail ensembles or heavy industrial backbones, so deployment calls for attention to the efficiency–performance trade-off.The discussion identifies lightweight networks with comparable detection performance as a potential direction for further work.Source: §5 · p. 8

6. Conclusion

The challenge finds persistent gaps in rare-defect detection and performance-efficiency trade-offs, while identifying design choices that limited participation and comparability. Kaputt-Rare provides a low-prevalence setting and initial baselines for retail anomaly detection.

6. Conclusion

  • Across the two tracks, retail detection nears saturation for common defects while rare defects remain unsolved; off-the-shelf VLMs trail specialized models, and improving performance-efficiency trade-offs remains open.The efficiency challenge is especially prominent for small, subtle defects in large images.Source: §6 · p. 8
  • Post-hoc efficiency evaluation limited participation; making efficiency constraints part of submissions could incentivize more resource-aware methods.Source: §6 · p. 8
  • Retail rankings combined scenarios with and without reference images, while zero-shot rules need clearer boundaries for adaptation, normalization, cross-image comparison, and pretraining.Source: §6 · p. 8
  • Comparability could improve by fixing the feature extractor or requiring backbone ablations to isolate each method’s contribution.Source: §6 · p. 8
  • Kaputt-Rare simulates realistic low-prevalence logistics data at approximately 1% defect prevalence and requires learning abnormality from almost defect-free data.Initial baselines span supervised classifiers and unsupervised anomaly detection methods.Source: §6 · p. 8

A. Efficiency Metric

The efficiency metric combines detection quality with deployment costs, reflecting that throughput, memory, and power can matter as much as detection quality in practice.

A. Efficiency Metric

  • The metric combines detection performance with throughput, memory usage, and power consumption in one scalar efficiency score.Source: A. Efficiency Metric · p. 12
  • Its design goals require a bounded, interpretable score that responds to every component and penalizes severe weakness in any dimension.Source: A. Efficiency Metric · p. 12
  • Throughput uses diminishing returns beyond a service-level target, unlike memory and power, whose theoretical bounds differ.Source: A. Efficiency Metric · p. 12

A.1. Metric Definition

The metric normalizes performance and resource measurements, then combines them with a geometric mean to reward balanced efficiency across dimensions.

A.1. Metric Definition

  • Each raw measurement is mapped to [0, 1]; challenge performance scores such as SegF1,avg and APany already lie in that range.Source: §A.1 · p. 12
  • Throughput is soft-capped against a service-level target, so doubling an already-fast model contributes less than doubling a slow one.The mapping is monotonically increasing and bounded.Source: §A.1 · p. 12
  • The score uses the complement of normalized memory usage because lower memory consumption is preferable; exceeding the device budget leads to out-of-memory errors.Source: §A.1 · p. 12
  • The score likewise uses the complement of normalized power consumption because lower power draw is preferable.Source: §A.1 · p. 12
  • The four normalized components—performance, throughput, memory, and power—are combined using a geometric mean.Source: §A.1 · p. 12
  • The geometric mean collapses to zero if any component is zero and favors balanced values, while keeping the score bounded in [0, 1].For example, (0.9, 0.9, 0.9, 0.9) scores approximately 0.9 versus approximately 0.88 for (1.0, 1.0, 1.0, 0.6), despite equal arithmetic means.Source: §A.1 · pp. 12–13

A.2. Reference Operating Point

The experiments use reference operating-point parameters described as representative of a mid-range edge deployment scenario.

A.2. Reference Operating Point

A.3. Experimental Analysis

Controlled sweeps examine the efficiency score as individual components and combinations vary, while later analyses consider deployment-specific configuration, extensibility, and metric limitations.

A.3.1. Single-Component Sensitivity

  • Fourth-root growth makes small low-end performance gains meaningful, while throughput saturates and memory and power penalties reach zero at full budget use.The score decreases monotonically as normalized memory and power usage increase.Source: §A.3.1 · p. 13, Figure 2 · p. 13

A.3.2. Composite Sweep

  • The composite score responds approximately linearly when all four components are swept together, distributing sensitivity evenly across them.F1 and throughput increase while memory and power decrease in the sweep.Source: §A.3.2 · p. 13, Figure 3 · p. 13

A.3.3. Joint F1–Throughput Surface

  • With memory and power fixed at 50%, a high composite score requires both good detection quality and reasonable throughput.Excelling on only one axis is insufficient.Source: §A.3.3 · pp. 13–14, Figure 4 · p. 14

A.4.1. Configurable Reference Points

  • The reference points tmax, mmax, and pmax can be configured for deployment targets without changing the metric’s functional form.Source: §A.4.1 · p. 14

A.4.2. Extensibility

  • The geometric-mean score can extend to n normalized factors, including model size or latency jitter.Source: §A.4.2 · p. 14

A.5. Limitations

  • Equal weighting across the four components may not suit practitioners who prioritize detection quality over efficiency.The passage suggests a weighted geometric mean as an alternative.Source: §A.5 · p. 14
  • Throughput normalization saturates after 0.5, so a logarithmic alternative may better suit applications where extreme speed is critical.Source: §A.5 · p. 14

Related papers

Loading 2610.04392v1…