PaperlayerView exact revision on arXiv

Source-linked AI summary

Frame-Level Evaluation in Weakly Supervised Video Anomaly Detection Mostly Measures Video-Level Ranking

Inpyo Song, Jangwon Lee

arXiv:2608.21854v1cs.CV

TL;DR

Weakly supervised video anomaly detection commonly evaluates temporal localization with pooled frame metrics that also compare frames across videos, leaving their localization evidence unclear. The paper decomposes Micro-AUROC by video identity and tests video-constant classifiers and score-mean interventions. Across benchmarks and outputs, high pooled performance persists despite removing within-video variation, showing that triage and localization should be evaluated separately.

  • Problem

    Pooled Micro-AUROC or AP can separate videos without ordering anomalous and normal moments within them, limiting what pooled scores establish about temporal localization.

  • Method

    The paper decomposes Micro-AUROC into Within-AUROC for same-video temporal ordering and Cross-AUROC for cross-video comparisons, then tests video-constant controls and temporal interventions.

  • Results

    Across three benchmarks, only 0.071%-0.388% of anomalous-normal comparisons are within-video, while video-constant outputs reach 81.40%-97.18% Micro-AUROC and mean replacement preserves a median 98.6% margin above chance.

  • Takeaways & Limitations

    High pooled scores can persist without identifying anomalous moments, so video-level triage and temporal localization should be named and evaluated separately.

  • Takeaways & Limitations

    The exact decomposition applies only to AUROC; for XD-Violence AP, the video-constant result is an achieved lower bound rather than a certified optimum.

Abstract

from arXiv · show

Weakly supervised video anomaly detectors are trained with video-level labels but are commonly evaluated as temporal localizers using Micro-AUROC or AP over pooled test frames. Because these metrics compare frames from different videos, a detector can score well by separating videos without accurately ordering moments within them. We exactly decompose Micro-AUROC by video identity into Within-AUROC for temporal ordering within videos and Cross-AUROC for comparisons across videos. Across ShanghaiTech, XD-Violence, and UCF-Crime, only 0.071-0.388% of comparisons between anomalous and normal frames occur within the same video. When both classes remain distributed across V videos, this share decreases as O(1/V), a benchmark property we call temporal dilution. We train anomaly video binary classifiers under the same video-level supervision and repeat each video score across all frames. These video-constant outputs reach 81.40-97.18 Micro-AUROC despite having no within-video variation. Across 72 controlled runs, replacing every frame score with its video mean preserves a median 98.6% of the Micro-AUROC margin above chance. The same empirical pattern holds for author-released outputs and for XD-Violence under its official AP evaluation. A detector can therefore achieve a high pooled score even when it assigns the same score to every moment within each video.

1 INTRODUCTION

The paper argues that pooled frame metrics can reward video-level separation without measuring temporal localization. It decomposes Micro-AUROC and tests video-constant scores, finding that high pooled performance can persist without within-video variation.

  • Evaluation ambiguity: Micro-AUROC can reward different overall video scores even when every frame within each video receives one constant value.Same-video comparisons test temporal ordering, whereas cross-video comparisons can be won through video-level separation.
  • Evaluation ambiguity: 41 of 42 surveyed peer-reviewed methods report pooled Micro-AUROC or AP, while temporal localization identifies where an event occurs.Video-level alerts identify which recording to inspect; localization identifies the anomalous interval.
  • Decomposition and dilution: 0.071%-0.388% of anomalous-normal frame comparisons are within the same video across ShanghaiTech, XD-Violence, and UCF-Crime.Under diffuse class distribution across V videos, this share decreases as O(1/V), termed temporal dilution.
  • Video-constant controls: 81.40%-97.18% Micro-AUROC is achieved by video-constant classifiers that repeat one video prediction across every frame.These classifiers use the same video-level supervision but have no within-video variation.
  • Video-constant controls: 98.6% of the Micro-AUROC margin above chance is preserved at the median when frame scores are replaced by video means across 72 controlled runs.The same behavior appears in author-released outputs, including XD-Violence under official AP evaluation.
  • Implication: A high pooled score does not establish when an anomaly occurs, so video-level triage and temporal localization require separate evaluation.The paper recommends evaluating these capabilities separately.

2 RELATED WORK

Prior work modifies pooled rankings or adds localization-oriented measures, while this paper attributes pooled performance through video-based decomposition and diagnostic controls.

  • Evaluation strategies: AUCA removes all-normal videos before pooling, whereas Macro-AUROC averages per-video performance and changes the estimand.Other alternatives include region-, track-, temporal-IoU-, event-level, and fixed-operating-point false-alarm measures.
  • Attribution and diagnostics: The paper adapts decomposition and diagnostic-substitution ideas by conditioning on video identity, then combines exact accounting with video-constant controls and interventions.The design includes learned video-constant probes and tests on controlled and author-released outputs.

3 HOW MUCH OF MICRO-AUROC DIRECTLY TESTS TEMPORAL ORDERING?

The paper decomposes Micro-AUROC into within-video and cross-video comparisons, showing that same-video pairs receive only a tiny share of the metric’s pair mass. This share declines as O(1/V) when class mass is distributed across videos, limiting the direct contribution of temporal ordering.

  • Exact decomposition: Micro-AUROC separates into Within-AUROC for same-video ordering and Cross-AUROC for cross-video comparisons.Within-AUROC uses pair-count weighting over mixed-label videos; the decomposition assumes both pair populations are nonempty.
  • Pair-mass allocation: Less than 0.4% of Micro-AUROC pair mass consists of same-video comparisons across the three benchmarks.Table 1 states that same-video pairs receive less than 0.4% of the pooled pair mass.
  • Pair-mass allocation: w is the probability that sampled positive and negative frames come from the same video.The same-video share is w = W/(AN), where W counts same-video positive–negative pairs.
  • Temporal dilution: When positive and negative frame mass spans Θ(V) videos, temporal dilution makes w = O(1/V).The result is conditional on diffuse class mass or exact replication of the test set.
  • Temporal dilution: 0.036–0.194 percentage points is the maximum Micro-AUROC contribution from improving Within-AUROC from chance to perfection on the three benchmarks.This bound holds with Cross-AUROC fixed and depends on benchmark composition rather than the detector.

4 HOW HIGH CAN A SCORER RANK WITHOUT LOCALIZING?

The paper tests how highly a scorer can rank frames without temporal localization by evaluating video-constant outputs. Such scorers can achieve high Micro-AUROC while their Within-AUROC remains at chance.

  • Video-constant scoring: Within-AUROC = 50 for scorers assigning one constant value to every frame of each video.These scorers have no temporal resolution and cannot indicate when an anomaly occurs.
  • Optimal ranking: Video-constant Micro-AUROC is maximized by ordering videos by their ground-truth anomaly fraction ai/(ai + ni).Ties between videos with equal anomaly fractions may be broken arbitrarily.
  • Metric capacity: 98.35 Micro-AUROC on ShanghaiTech, 95.46 on XD-Violence’s auxiliary Micro-AUROC, and 92.95 on UCF-Crime are attainable without within-video variation.These values establish the metric capacity of the video-constant scorer family, not ordinary training performance.
  • Learned video-level probes: 97.18 Micro-AUROC on ShanghaiTech, 90.78 on XD-Violence, and 81.40 on UCF-Crime are reached by video-constant learners, while every Within-AUROC equals 50.Table 2 reports three-seed means; the largest sample SD across 24 cells is 2.12 points.
  • Learned video-level probes: 31.40–47.18 points above AUROC chance is achieved by the strongest video-constant heads across the three benchmarks.Their Cross-AUROC exceeds Micro-AUROC by only 0.03–0.12 points, while Within-AUROC remains at chance by construction.

5 DOES DETECTOR MICRO-AUROC DEPEND ON WITHIN-VIDEO SCORE VARIATION?

The audit tests whether Micro-AUROC depends on within-video score variation by replacing frame scores with video means and decomposing performance into within- and cross-video terms. Across controlled and author-released outputs, pooled scores remain largely intact despite removing temporal variation, while Within-AUROC can differ substantially.

  • Video-mean replacement: 69 of 72 controlled runs retain at least 90% of their Micro-AUROC chance margin after video-mean replacement.All three seeds remain separate observations in benchmark medians.
  • Within-video ordering: 1.69 points of pooled-score difference coexist with a 37.60-point Within-AUROC gap between CLIP-TSA and RTFM on ShanghaiTech.The methods use the same CLIP representation in the cited comparison.
  • Within-video ordering: 2.93 points of Micro-AUROC range coexist with a 58.92-point range in temporal ordering across the controlled RTFM–CLIP ShanghaiTech seeds.The corresponding Micro-AUROC mean is 96.05, versus a Within-AUROC mean of 44.99.
  • AP and auxiliary controls: 81.9–95.1% of the official XD-Violence AP chance margin remains after video-mean replacement for seven author-released outputs.Anomaly-only pooling raises the same-video pair share only to 0.24–1.90%, and video-constant scorers reach 77.68–84.92 AUCA.
  • Decomposition: Cross-AUROC matches Micro-AUROC within 0.09 points across all 17 Table 4 rows, whereas Within-AUROC uses pair-count-weighted same-video comparisons.The comparison includes 14 author-released results and three ShanghaiTech reproductions.
  • Interpretation: Micro-AUROC survives temporal interventions, showing that its detector value depends only marginally on within-video score variation.Within-video permutation preserves each video’s score multiset while randomizing alignment with temporal labels.

6 IMPLICATIONS FOR EVALUATION AND BENCHMARK DESIGN

The findings separate pooled frame ranking from temporal localization and motivate reporting designs that preserve same-video evidence. Pooled scores can support video-level triage, but localization claims require within-video or event-level evaluation.

  • Detector developers: Similar pooled scores can conceal large differences in Within-AUROC, so pooled frame AUROC should not stand alone as localization evidence.The ShanghaiTech CLIP-TSA–RTFM comparison differs by 37.60 points in Within-AUROC despite a 1.69-point pooled gap.
  • Detector developers: Cross-AUROC numerically coincides with pooled frame AUROC in every audited output, supporting pooled evaluation for whole-recording triage.The recommended explicit video-level measure is AUROC computed from one score per video against video-level labels.
  • Metric choice: The UCF-Crime VADTree–S3R ordering reverses between Within-AUROC and Macro-AUROC, demonstrating that metric choice changes practitioner selection.VADTree leads by 11.20 points under Within-AUROC, while S3R leads by 7.70 points under Macro-AUROC.
  • Benchmark builders: O(1/V) dilution of direct temporal weight should be treated as a controlled benchmark-design quantity when videos are added.If localization is the target, benchmark construction should preserve same-video comparisons by design.
  • Practitioners and deployers: Within-AUROC or Macro-AUROC should be paired with boundary, delay, fragmentation, and thresholded false-alarm measures for moment localization.Because within-video summaries exclude all-normal videos, false alarms should also be reported per normal video or per hour at a prespecified operating point.

7 LIMITATIONS

The conclusions are bounded by metric scope, weighting concentration, method coverage, and annotation dependence. These limitations affect interpretation and generalization, while the structural decomposition remains supported for fixed labels.

  • Operating-point coverage: Boundary accuracy, detection delay, fragmentation, and thresholded false alarms lie outside pairwise ranking populations.The proposed reporting protocol therefore complements rather than replaces event-level and operating-point evaluation.
  • Average Precision: AP lacks additive pair identity, so the exact decomposition applies only to AUROC.On XD-Violence, the reported AP evidence is a video-constant lower bound plus temporal interventions, not an exact AP attainability result.
  • Weight concentration: Within-AUROC weights can be dominated by a few long or anomaly-dense videos, unlike Macro-AUROC’s equal weighting of mixed-label videos.Stability requires video-level resampling or clustered-U-statistic variance analysis.
  • Coverage: Aligned frame scores limit coverage, with only one eligible author-released output on ShanghaiTech.The three benchmarks also do not span every domain or labeling protocol.
  • Annotation dependence: Annotation disagreement conditions the interpretation of Within-AUROC levels and gaps, especially for event duration and endpoints.The decomposition identity and dilution bound hold exactly for any fixed labeling, but sensitivity of Within-AUROC levels remains unquantified.

8 CONCLUSION

The paper reframes pooled frame evaluation as separate same-video temporal ordering and cross-video ranking estimands. Its audited benchmarks show that high pooled scores can persist without moment identification, so triage and localization must be evaluated separately.

  • Conclusion: High pooled scores can be obtained and preserved without identifying anomalous moments.Video-constant controls and temporal interventions support this conclusion on the audited benchmarks.
  • Conclusion: Triage and localization must be named and evaluated separately.The conclusion does not make video-level triage useless; it separates that use from temporal localization.
  • Scope: The work re-analyzes public surveillance-video benchmarks and introduces no new video collection or detector.Its analyses use existing benchmark data and detector outputs.

A PROOFS FOR AUROC ATTRIBUTION

The proofs decompose pooled AUROC into within-video and cross-video contributions, then characterize how video-constant scores and evaluation controls behave under this decomposition.

  • AUROC attribution: Within-AUROC averages same-video positive–negative ordering with pair-count weights a_i n_i/W, while Macro-AUROC gives each mixed-label video equal weight.Videos without both labels contribute no same-video pairs.
  • AUROC attribution: If same-video pair mass is W, Micro-AUROC is a weighted combination of Within-AUROC and Cross-AUROC with weight w = W/(AN).At W = 0 it reduces to Cross-AUROC; when AN − W = 0 it reduces to Within-AUROC.
  • Temporal dilution: w = O(1/V) when positive and negative frame mass is spread across Θ(V) videos, and exact replication reduces w by k.The result is conditional on diffuse class mass or exact replication.
  • AUROC attribution: Changing Within-AUROC from 0.5 to 1 changes Micro-AUROC by w/2 when Cross-AUROC is fixed.This follows directly from the decomposition’s weighting.

E DOES THE RESULT SURVIVE WITHIN-VIDEO PERMUTATION?

Within-video permutation preserves each video’s score histogram while randomizing its alignment with temporal labels, yet most pooled performance remains.

  • Permutation intervention: The permutation intervention makes expected Within-AUROC exactly 0.5 while preserving every video’s score histogram.Expected Micro-AUROC is computed analytically rather than by Monte Carlo.
  • Permutation intervention: 89.5% is the median exact expected Micro-AUROC chance-margin retention after uniformly permuting each video’s score multiset.The range across 14 official outputs is 81.7–96.0%.

F PAIR-MASS CONCENTRATION AND COMPLETE MICRO-AUROC RECORDS

The paper quantifies concentration in the within-video term, separates alternative summaries, and extends the video-constant analysis to AP and annotation perturbations.

  • Pair-mass concentration: Neff = 18.7–43.7 videos is the equivalent equal-weight count for the concentrated pair mass underlying Within-AUROC.Neff is not a sample size or uncertainty estimate.
  • Alternative summaries: Within-AUROC and Macro-AUROC differ by 3.34 points on average and up to 10.88 points across the controlled 72-run grid.Within-AUROC uses pair-count weighting; Macro-AUROC gives each mixed-label video one vote.
  • Annotation perturbation: All 14 official frame-score outputs remain below the video-constant Micro-AUROC optimum under paired annotation perturbation.The tightest interval is DSANet on XD-Violence: mean gap 0.108 points with interval [0.044, 0.175].
  • Average precision: 86.75% AP is an achieved lower bound for the video-constant family on XD-Violence, against an AP chance level of 23.08%.Because the ordering is feasible rather than globally optimal, it is not an exact ceiling.
  • Controlled design: The 72-run grid spans four objectives, two representations, three benchmarks, and three seeds per cell without rerolls or test-performance-based seed selection.This defines the controlled comparison design.

J METRIC REPORTING IN WSVAD METHODS

The survey finds that pooled frame metrics dominate WSVAD reporting, while Macro-AUROC is absent from the surveyed methods and newer alternatives appear only selectively.

  • Metric taxonomy: The metric mapping treats frame-level AUC as Micro-AUROC, anomaly-subset variants as AUCA, and per-video averaging as Macro-AUROC.Table 14 counts reporting by metric family.
  • Reporting prevalence: 41 of 42 surveyed WSVAD methods report pooled Micro-AUROC or AP, while none reports Macro-AUROC.The survey covers peer-reviewed methods published between 2018 and 2026.
  • Reporting variation: Some papers replace or omit pooled metrics: Zhang et al. report only event-level mAP, while Acharya et al. use AP and fixed-FPR recall.Other reporting differences include XD-Violence Micro-AUROC instead of official AP and declarations without numeric results.
  • Reporting trends: Event-level mAP emerges in 2022 and expands from 2024 to 2026, while two of seven CVPR 2026 entries abandon or demote pooled scoring.FAR appears in main comparisons only from 2018 to 2023.
Loading 2608.21854v1…