PaperlayerView exact revision on arXiv

Source-linked AI summary

VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness

Maksim Plinskiy, Aleksandr Gushchin, Sergey Lavrushkin, Dmitriy S. Vatolin, Anastasia Antsiferova

arXiv:2610.08936v1cs.CV

TL;DR

Video classification lacks a comparable standardized infrastructure for adversarial-robustness evaluation, while temporal and protocol choices complicate comparisons. VCR-Bench brings models, attacks, defenses, and metrics into a common benchmark framework. Its evaluation finds adaptive targeted attacks remain effective against many models, and clean accuracy does not reliably indicate robustness.

  • Problem

    Video classification lacks image-like standardized robustness infrastructure, while temporal sampling, perturbation budgets, and aggregation complicate evaluation and reproducibility.

    Source: §1 · p. 1

  • Method

    VCR-Bench standardizes video loading, classifier wrappers, attacks, defenses, preprocessing, temporal sampling, perceptual metrics, configurations, and result logging under a common protocol.

    Source: §1 · pp. 1–2

  • Results

    Adaptive targeted attacks produce ASR above 60% for many models despite strong clean accuracy, and the clean-accuracy leader UniFormerV2 remains highly vulnerable.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    The benchmark supports joint, auditable comparisons of clean and attacked performance, perceptual distortion, runtime, and memory using reproducible presets.

    Source: §5 · p. 4, §3.2 · p. 4

    The evaluation is limited to a Kinetics-400 subset, and stronger budgets, longer attacks, or adaptive variants may change method rankings.

    Source: §4 · p. 4

Abstract

from arXiv · show

Robustness of image classification has several benchmarks, but their video counterparts are absent. In video classification temporal dimension introduces additional degrees of freedom for adversarial attacks, defenses, and preprocessing. Temporal sampling, perturbation budgets, and metric aggregation also interact in ways with no direct analogue in the image setting. Therefore, robustness for video classifiers is studied across scattered, incompatible implementations, making reported numbers hard to reproduce and analyze. We introduce VCR-Bench, a modular open-source benchmark framework that standardizes video loading, wrappers for classifiers, adversarial attacks and defenses, perceptual metrics, configuration presets, and result logging. VCR-Bench currently integrates 30 video classification models, 14 adversarial attacks, and 10 defense wrappers under a common evaluation protocol. We evaluate representative video classifiers, attacks, and defenses on Kinetics-400 subset, reporting clean accuracy, attack success rate, perceptual quality, runtime, and memory usage. VCR-Bench is released with documented installation, reproducible run presets, component-extension interfaces, and scripts for reproducing the reported results at https://github.com/msu-video-group/vcr-bench.

1 Introduction

Video classification lacks the standardized adversarial-robustness infrastructure available for image classification, despite temporal and protocol choices that complicate evaluation. VCR-Bench addresses this gap with an open-source framework that jointly supports clean evaluation, attacks, and defenses under a common protocol.

1 Introduction

  • Video robustness results depend on clip sampling, temporal stride, perturbation budgets, aggregation, and attack adaptivity, making comparisons sensitive to protocol and implementation choices.Defenses can also make non-adaptive attacks overestimate robustness.Source: §1 · p. 1
  • VCR-Bench standardizes video loading, classifier wrappers, attacks, defenses, preprocessing, temporal sampling, perceptual metrics, resource logging, and reproducible configuration presets.It jointly supports clean evaluation, adversarial attacks, and defenses under a common protocol.Source: §1 · pp. 1–2
  • The framework integrates datasets, classifiers, attacks, defenses, metrics, presets, and result logging, with stable interfaces and convention-based discovery for adding components.Its configurations cover clip sampling, preprocessing, perturbation budgets, attack success, accuracy, perceptual distortion, runtime, and memory usage.Source: §1 · p. 2
  • VCR-Bench includes 30 video classifiers, 14 adversarial attacks, and 10 defense methods, alongside documented run presets and extension examples.The paper presents it as the first open-source benchmark to jointly support clean evaluation, attacks, and defenses under a common protocol, to the authors’ knowledge.Source: §1 · p. 2

2 VCR-Bench Framework

VCR-Bench provides a common evaluation pipeline and modular components for video classification robustness. It records task, perceptual, and computational measures, and supports configurable, reproducible experiments and component extensions.

2.1 Evaluation Pipeline

  • The pipeline decodes and deterministically samples video into clips, optionally applies a defense, preprocesses clips for the classifier, and aggregates clip logits into a per-video label.Adversarial evaluation iterates attack queries until a stopping condition or budget limit; non-adaptive queries omit the defense, which is applied in final evaluation.Source: §2.1 · p. 2

2.2 Models, Attacks, and Defenses

  • The registries cover 30 video classifiers, 14 attacks, and 10 purification defenses across convolutional, temporal, transformer, masked-pretraining, and multimodal models, and varied attack and defense approaches.Examples include gradient-based and query-based attacks, universal perturbations, temporal shuffling, and diffusion-based purification.Source: §2.2 · p. 2

2.3 Metrics, Outputs, and Reproducibility

  • VCR-Bench reports clean and robust accuracy, defended accuracy, attack success rate, perceptual distortion, runtime, and memory, and records run settings and outputs in CSV and JSON.It also exports clean, adversarial, and defended videos for side-by-side inspection.Source: §2.3 · p. 2

2.4 Configuration, Extensibility, and Release

  • Experiments combine shared defaults, local paths, component presets, and run presets; stable component interfaces and convention-based discovery support extensions without rewriting the evaluation loop.The release includes installation instructions, documentation, run presets, and optional Slurm helpers.Source: §2.4 · pp. 2–3

3 Evaluation and Results

Experiments use a fixed Kinetics-400 test subset and evaluate models and defenses under specified attack settings. Results show that adaptive targeted attacks remain effective against many classifiers, while robustness and clean accuracy do not align consistently.

3.1 Experimental Setup

  • The main experiments use a randomly selected fixed subset of 100 Kinetics-400 test videos, deterministic temporal sampling with seed 42, and model-specific preprocessing.Experiments run on an NVIDIA A100 80GB.Source: §3.1 · p. 3
  • White-box attacks use an L∞ perturbation budget of 8/255, while iterative white- and black-box attacks use 20 and 200 iterations, respectively.When defenses are differentiable or approximable, adaptive evaluation is reported; otherwise the setting is marked non-adaptive.Source: §3.1 · p. 3

3.2 Results

  • Adaptive targeted attacks yield ASR above 60% for many models despite strong clean accuracy, and TAdaFormer has the lowest ASR and highest robust accuracy but substantially larger perceptual distortion.UniFormerV2 has the highest clean accuracy yet remains highly vulnerable; results suggest temporal evidence aggregation is more relevant to resistance than broad architecture categories.Source: §3.2 · p. 3, Table 2 · p. 4
  • Defense comparisons report defended robust accuracy and ASR aggregated across models by attack, and should be read as a search for the best accuracy–robustness trade-off.The benchmark also tracks perceptual distortion, runtime, memory, and query budget as diagnostics alongside attack success.Source: Table 3 · p. 4, §3.2 · p. 4

4 Limitations

VCR-Bench is a standardized evaluation framework, not evidence that any classifier is secure. Its current evaluation has scope and operating-point boundaries, and some component combinations may be incompatible.

4 Limitations

  • The benchmark is not proof that any particular video classifier is secure.Its current evaluation is limited to a Kinetics-400 subset and may not transfer across video domains, data distributions, or deployment settings.Source: §4 · p. 4
  • Stronger perturbation budgets, longer attacks, or adaptive attack variants may change method rankings.The reported perturbation budget and attack settings represent a specific operating point.Source: §4 · p. 4
  • Some combinations of models, attacks, and defenses may be incompatible.This is an engineering constraint associated with the framework's breadth.Source: §4 · p. 4

5 Conclusion

VCR-Bench provides a modular framework and common experimental contract for video-classification robustness evaluation. Its presets and execution metadata support auditable comparisons and future component integration.

5 Conclusion

  • VCR-Bench defines a common experimental contract for datasets, classifiers, attacks, defenses, perceptual metrics, configuration presets, and structured result logs.The framework is modular and intended for evaluating adversarial robustness in video classification.Source: §5 · p. 4
  • The representative evaluation jointly reports clean accuracy, attacked and defended performance, perceptual distortion, runtime, and memory usage.Reproducible presets and recorded execution metadata make comparisons auditable rather than tied to one-off scripts.Source: §5 · p. 4
  • Released code, documentation, run presets, output schemas, extension interfaces, and a leaderboard provide a reusable baseline for comparing methods and integrating future components.The stated scope includes future models, attacks, defenses, and datasets under the same protocol.Source: §5 · p. 4

Related papers

Loading 2610.08936v1…