PaperlayerView exact revision on arXiv

Source-linked AI summary

A VLM Answer Is Not an Anomaly Score: Rank Compression in Training-Free Video Anomaly Detection

Inpyo Song, Jangwon Lee

arXiv:2608.21244v1cs.CV

TL;DR

VLM-based VAD needs a defined answer interface because benchmarks rank scalar scores, not raw generated answers. This paper formalizes scale and readout choices, finding that probability readout consistently outperforms generated readout by preserving ordering lost to decoding ties.

  • Problem

    VAD benchmarks rank scalar scores, but VLM generation returns answers from a distribution, so the answer-to-score interface is an under-specified evaluation choice.

  • Method

    The paper separately evaluates answer scales and readout rules across frozen VLMs, using GEN’s most likely answer and PROB’s full answer distribution.

  • Results

    PROB wins every model–scale–dataset–metric comparison, with average gains of 7.66 AUROC and 12.75 AP points on UCF-Crime and 5.09 AUROC and 10.97 AP points on XD-Violence.

  • Takeaways & Limitations

    The answer interface is a consequential component of VLM-based VAD and should be explicitly specified and evaluated.

  • Takeaways & Limitations

    The four evaluated VLMs are a fixed set, so the reported intervals are conditional on these models and do not establish an effect for VLMs generally.

Abstract

from arXiv · show

Vision-language models enable training-free video anomaly detection by answering questions about video segments. VAD benchmarks, however, require a scalar anomaly score for each segment and evaluate the resulting ranking using the AUROC or AP. A VLM-based detector should therefore define an answer interface: the answer scale specifies the admissible answers, and the readout rule maps the model's output distribution to a score. Because this interface can change the evaluated ranking, it is part of the detector rather than a formatting detail. The generated readout uses only the most likely answer, whereas the probability readout uses the full distribution over admissible answers. Across four 7-8B VLMs, the probability readout outperforms the generated readout for every tested combination of answer scale, benchmark, and metric, with average gains ranging from 5 to 13 points across the four benchmark-metric pairs. The gap arises because the generated readout keeps only one answer value per segment, so segment with different answer distributions can receive the same score and lose their relative order. We call this loss of relative order generated-answer rank compression. Even when the answer scale allows 91 answers, the generated readout produces only 4-18 distinct scores, whereas the probability readout retains substantially finer score resolution. The advantage persists under every decoding strategy, prompt wording, and joint scoring-explanation prompt we test. The answer interface is therefore a consequential component of VLM-based VAD and should be explicitly specified and evaluated.

1 INTRODUCTION

VLM-based VAD must convert generated answers into scalar scores that benchmarks can rank. This answer interface affects resolution and ordering: probability readout preserves distinctions that generated answers compress.

  • VLM-based VAD divides videos into temporal segments, asks a frozen model to judge each segment, and converts responses into anomaly scores.
  • Decoded answers can tie segments with different answer distributions, while reading answer probability preserves their relative order.For example, “Yes” probabilities of 0.56 and 0.94 decode to the same answer but remain ordered under probability readout.
  • The answer interface independently specifies admissible answers and the readout rule converting model outputs into scores.GEN keeps the most likely answer; PROB uses the full answer distribution and is called probability readout.
  • Generated-answer rank compression is the loss of relative order when different answer distributions share the same most likely answer.
  • PROB wins every model–scale–dataset–metric comparison across four 7–8B VLMs and seven answer scales.Average gains are 7.66 AUROC and 12.75 AP points on UCF-Crime, and 5.09 AUROC and 10.97 AP points on XD-Violence.

2 RELATED WORK

Prior work uses probabilistic outputs to improve ranking, but VAD introduces temporal and metric-specific consequences when generative answers become ranked anomaly scores.

  • Classical VAD learns scoring functions under semi-supervised or weakly supervised training, whereas generative VAD converts frozen-model judgments into ranked scores.
  • VLM-based VAD methods differ in where scoring occurs and what the model is asked to emit.Some methods score outside generation using aligned visual features, while generative systems move scoring into language output.
  • Probabilistic judging and ranking repair decoded-score compression by using likelihoods over admissible labels.Related work includes probability-weighted sums, normalized relevance-label likelihoods, and correctness probability from a “Yes” token.
  • VAD differs because finer scales do not systematically benefit PROB, temporal ties concentrate near the top, and smoothing post-processes scores.

3 THE ANSWER INTERFACE

The answer interface makes score extraction an explicit choice of answer scale and readout rule. GEN is a many-to-one reduction that creates ties, whereas PROB preserves within-answer ordering.

  • The answer interface: A forward pass produces an answer distribution, and a readout rule reduces it to the scalar sequence ranked by the benchmark.
  • Answer scales: Each answer scale defines a finite ordinal set normalized to evenly spaced values in [0, 1].The study uses verbal binary, numeric binary, ordinal 0–5 and 0–9, and floating-point scales with 11, 51, or 91 values.
  • Generated readout: GEN retains only the most likely admissible answer, so its score sequence contains at most |A_k| distinct values.The greedy decoder is idealized with a fully compliant parser, isolating reduction effects from formatting failures.
  • Probability readout: PROB normalizes answer likelihoods and scores each unit by the expected normalized answer value.For Yes/No, this is the normalized probability mass on “Yes”; both readouts use one forward pass.
  • Generated-answer rank compression: GEN removes relative ordering among distributions sharing a top answer, while PROB can order frames receiving the same answer.AUROC gives tied pairs half credit, whereas AP penalizes ties more when they occur near the ranking top.
  • Empirical comparison: PROB outperforms GEN in all 28 scale–metric cells, while finer scales mainly benefit GEN and PROB remains comparatively stable.Figure 2 reports four-model means and model-wise ranges rather than confidence intervals.

4 EXPERIMENTAL PROTOCOL

The evaluation uses four frozen VLMs on UCF-Crime and XD-Violence with a fixed temporal-unit protocol. Scores are reported using frame-level AUROC and AP.

  • Models and benchmarks: The study evaluates four frozen VLMs on 290 UCF-Crime test videos and 800 XD-Violence test videos.
  • Protocol and metrics: Each video is divided into 16-frame units, with each unit judged once from eight frames sampled over a centered 300-frame window.
  • Protocol and metrics: Frame-level AUROC and AP are reported, with AUROC primary on UCF-Crime and AP primary on XD-Violence.Unit scores are broadcast to the unit’s frames.

5 RESULTS

PROB consistently outperforms GEN because it preserves within-answer ordering that decoding discards. The advantage remains across metrics, scales, models, benchmarks, smoothing, decoding, and prompt variants.

  • 5.1 PROBABILITY READOUT IMPROVES EVERY COMPARISON: +7.66 AUROC points on UCF-Crime and +10.97 AP points on XD-Violence are the mean primary-metric gains of PROB over GEN.PROB wins every model, scale, dataset, and metric comparison, with paired bootstrap intervals excluding zero.
  • 5.2 TIE-BREAKING EXPLAINS THE GAP AND ITS METRIC ASYMMETRY: The gap is wider under AP than AUROC in 53 of 56 model–scale–dataset cells, at about 1.7× and 2.2× the AUROC gain.The asymmetry is attributed to where ties occur in the ranking.
  • 5.2 TIE-BREAKING EXPLAINS THE GAP AND ITS METRIC ASYMMETRY: 93–95% of the mean PROB–GEN gap is recovered by breaking GEN’s ties, showing that discarded within-answer ordering drives most of the difference.Random tie-breaking recovers under a fifth of the AP gap, while probability-based ordering recovers the informative order.
  • 5.2 TIE-BREAKING EXPLAINS THE GAP AND ITS METRIC ASYMMETRY: At a false-alarm rate of 0.01, PROB roughly doubles detected anomalous frames on both benchmarks, including 16.7% versus 8.6% on UCF-Crime.Ordering only the tied frames covers most of this distance.
  • 5.2 TIE-BREAKING EXPLAINS THE GAP AND ITS METRIC ASYMMETRY: GEN leaves AP underdetermined by more than 42 points on average, whereas PROB leaves an interval of about one point.Reported AP can depend on incidental tie ordering; grouped-threshold AP removes that dependence.
  • 5.3 A FINER SCALE DOES NOT SUBSTITUTE FOR THE READOUT: Fine scales help GEN more than PROB, but mean-scale conclusions are not per-model prescriptions because several binary-versus-finer intervals cover zero.Yes/No or 0–1 is best for six of eight PROB model–dataset pairs, while finer scales are best for all eight GEN pairs.
  • 5.3 A FINER SCALE DOES NOT SUBSTITUTE FOR THE READOUT: Floating-point GEN realizes at most 18 values despite admitting up to 91, with one score covering 72.4% and 55.1% of units on average.Under the same prompts, PROB assigns distinct scores to more than 99.8% of units.
  • 5.3 A FINER SCALE DOES NOT SUBSTITUTE FOR THE READOUT: Within a single generated answer, PROB still finds anomaly structure, with anomaly rates reaching 3.8× and 4.6× the group averages.This ordering is observed among frames InternVL answers “No” under Yes/No on UCF-Crime and XD-Violence.

6 LIMITATIONS

The evaluation is limited to a fixed set of models, ranking quality, and selected benchmarks and settings. PROB additionally requires answer likelihoods that many proprietary interfaces do not expose.

  • The four evaluated VLMs are a fixed set rather than a random sample, so reported intervals are conditional on those models.Generalizing the effect to VLMs broadly would require sampling checkpoints as a population.
  • The measured gap is an unsmoothed, ranking-only quantity, although it survives matched smoothing in aggregate.The pooled interval excludes zero at σ = 4 and σ = 8, while individual configuration results vary.
  • A deployment-relevant smoothing estimate would fix σ and the answer scale on a held-out split rather than the evaluation split.The oracle-σ residual has no interval because both σ values are chosen on the test set.
  • PROB requires exposed answer likelihoods, so proprietary interfaces may leave generated readout as the only option.The reported results quantify the cost of decoded answers and partial recovery from finer scales and matched smoothing, but do not provide a remedy there.
  • The paper identifies additional limitations involving the confounded scale axis, unmeasured mechanism, model and benchmark coverage, and unconstrained generation.

7 CONCLUSION

The conclusion frames answer-to-score conversion as a consequential part of VLM-based anomaly detection. Decoding creates ties that probabilities avoid, and the resulting choices should be reported with the detector.

  • Decoding creates ties that finer prompts do not reliably prevent, while reading answer probabilities avoids them.
  • 93–95% of the readout gap is accounted for by decoding ties.
  • The probability advantage survives matched smoothing and score-first explanation prompts.
  • Scale, readout, output order, tie policy, and smoothing jointly define a generative detector and should be reported with it.
  • Choosing these interface components deliberately turns the same frozen model into a stronger detector at no additional cost.

AI USE STATEMENT

The authors used generative AI for isolated inference-code utility functions and for proofreading-related writing checks, with author verification of the assisted work.

  • Generative AI tools were used to implement isolated utility functions in model inference code, which the authors verified and tested.
  • The authors also used generative AI for proofreading, grammar checking, and typographical checks during writing.
  • The authors state that every reported number was produced by the deterministic evaluation pipeline.

ETHICS STATEMENT

The study evaluates scoring rules on two public video-anomaly benchmarks using standard test annotations without redistributing video data. Generated outputs are not used to identify, profile, or disparage individuals.

  • The evaluation uses UCF-Crime and XD-Violence with their standard test annotations and redistributes no video data.
  • The videos depict real events involving people and are used only to evaluate frame-level anomaly rankings.
  • Generated scores and captions are not used to identify, profile, or disparage individuals, and generated captions will not be released.

REPRODUCIBILITY STATEMENT

The study uses frozen public VLM checkpoints, fixed temporal inputs, paired readouts, and controlled prompt, polarity, scale, and decoding evaluations. Across these controls, PROB consistently outperforms GEN, while score resolution and residual ties depend on the interface and inference arithmetic.

  • Experimental setup: Four frozen public 7–8B VLMs evaluate identical non-overlapping 16-frame units using eight frames sampled over a centered 300-frame window.The models run in bfloat16, with frame expansion back to the annotation grid before evaluation.
  • Experimental setup: GEN and PROB share one forward pass, visual input, model parameters, and answer likelihoods, differing only in final reduction.This makes their paired comparisons exact rather than matched.
  • Robustness controls: PROB leads GEN in all eight prompt-wording cells on both metrics, with gaps remaining positive under every tested wording.The readout gap spans 8.55–11.09 AUROC and 8.80–10.62 AUROC points across the reported wording comparisons.
  • Robustness controls: PROB leads GEN under both anomaly and normality question polarities on both metrics, so the gap is not an artifact of answer polarity.Reversed-question gaps are 7.29 and 9.17 AUROC points, versus 8.98 and 9.96 under the original question.
  • Statistical evaluation: All 56 model–scale–dataset cells have positive PROB–GEN gains, concentrated on binary scales.Every per-scale mean interval excludes zero, while five of 28 UCF-Crime AUROC pairs are exceptions at the individual model–scale level.
  • Score resolution: GEN score occupancy remains compressed: the modal score covers 72.4% of UCF-Crime units and 55.1% of XD-Violence units on average.PROB produces distinct scores for more than 99.8% of XD-Violence units in the reported configurations; residual ties arise from boundary overlap or identical probabilities.

J DECOMPOSITION OF THE READOUT GAP

The decomposition attributes most of the PROB–GEN gap to ties created when generated answers collapse distinct distributions, while threshold placement and answer reordering contribute more limited or setting-dependent effects. Sampling answers approaches PROB but remains inferior, and the gap persists across models and evaluation choices.

  • Gap decomposition: Tie-breaking dominates every PROB–GEN decomposition cell, while answer reordering grows only toward the finest floating-point scales.The reordering share increases where argmax and expected answer values disagree more often.
  • Threshold versus compression: 7.03–7.60 AUROC points remain as compression after oracle thresholding on UCF-Crime Yes/No, across all four models.Compression is the model-invariant part of the binary deficit; no binary readout recovers it.
  • Tie-breaking headroom: 15.66 and 9.65 AUROC points are the perfect-tie-breaking headroom on UCF-Crime and XD-Violence, while observed gap ratios are 1.66× and 2.16×.The headroom ratios overstate realized asymmetry by about 40% on both benchmarks.
  • Tie-breaking headroom: PROB recovers 7.31 and 4.81 AUROC points but only 11.84 and 10.44 AP points, reflecting AP’s sensitivity to normal frames above anomalies.The top tied blocks contain 34.9% and 58.9% anomalous frames on the two benchmarks.
  • Sampled-answer averaging: Sixteen sampled answers recover 72.2% and 59.3% of the GEN–PROB gap, while 256 recover 95.3% and 82.4% without reaching PROB.The sampling curve approaches PROB from below, so one forward-pass probability readout remains better at every tested budget.
  • Model scaling: The gap survives model scaling: PROB gains 1.58 and 1.54 AUROC points from larger models, while GEN changes by −0.06 and +0.63.This control covers one benchmark, one scale, and one size step per model family, so it tests persistence rather than scaling trends.
  • AP estimation: Under heavily tied GEN scores, trapezoidal AP violates the attainable-precision bound in 30 of 56 streams, whereas the non-interpolated estimator violates it in none.For InternVL3.5-8B on XD-Violence Yes/No, no threshold exceeds 59.8% precision, but trapezoidal AP reports 75.25.

P PER-CLASS RESULTS

Across anomaly classes and tested decoding settings, PROB consistently improves ranking over GEN by retaining information that generated answers compress into ties. The gains persist across benchmarks, metrics, smoothing strengths, and joint scoring–explanation prompts.

  • Per-class performance: Every UCF-Crime and XD-Violence anomaly class improves on both AUROC and AP under PROB.AP gains are largest for visually salient events, while subtle classes remain difficult under either readout.
  • Per-class performance: AP rises from 10.7 to 49.3 for the XD-Violence Abuse class under PROB, whereas UCF-Crime Abuse remains near zero at 0.02% anomaly prevalence.The contrast shows that recovered ranking evidence can be expressed when prevalence leaves AP room to distinguish performance.
  • Smoothing and robustness: Best-σ selection improves GEN more than PROB, by 6.81 UCF-Crime AUROC points and 10.42 XD-Violence AP points versus 1.24 and 2.67 for PROB.On binary interfaces, unsmoothed PROB beats GEN at GEN’s best smoothing strength in 14 of 16 pairs.
  • Smoothing and robustness: PROB exceeds GEN in all 288 matched comparisons across 16 model–scale cells, two metrics, and nine smoothing settings.The mean PROB–GEN gap shrinks monotonically as smoothing increases but remains positive throughout the tested range.
  • Joint scoring and explanation: Explanation-first decoding reduces PROB by 3.39 points on average, compared with 0.44 for GEN, while compressing the answer distribution by 42%.Score-first prompting changes PROB by only −0.90 points on average, and explanation-first decoding lowers entropy in all eight cells.
  • Numerical resolution: Float32 rescoring raises distinct PROB values from 137 to 65,885 and narrows the AP tie interval from 2.06 to 0.26 points.The prompt, temporal protocol, model readouts, and ranking otherwise remain unchanged.

U ADDITIONAL LIMITATIONS

The evidence is bounded by a narrow evaluation grid and several unresolved design confounds. The paper also limits its claims about unconstrained generation, cross-system comparisons, and the cause of retained within-answer order.

  • Evaluation scope: The study covers four open 7–8B VLMs, two surveillance benchmarks, and one temporal protocol, so other models, domains, and temporal units may differ.Several auxiliary controls cover only subsets of the evaluation grid, and inference-precision controls cover a single cell.
  • Generation scope: Unconstrained generation is out of scope because admissible-mass controls cover only two models on one scale.Systems answering outside the admissible set could introduce parsing failures not represented by the tested interfaces.
  • Design confounds: The scale axis is confounded because changing the number of levels also changes label wording and tokenization.The study cannot isolate granularity while holding the verbalizer and its tokenization fixed.
  • Mechanistic limitation: The study does not measure why ranking order survives within a fixed answer.Testing the proposed continuous-likelihood explanation would require controlled visual perturbations while tracking answer likelihoods.
  • External comparison: The fixed-interface comparison with published VAD systems does not establish a new state of the art.Compared pipelines differ in visual context, temporal design, post-processing, and AP estimators.
Loading 2608.21244v1…