Source-linked AI summary
RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos
Alexandru-Bogdan Dura, Sebastian Balmus, Radu Tudor Ionescu
TL;DR
Context-dependent, hard-to-enumerate anomalies make robotic manipulation VAD difficult to train for unforeseen failures. RoboVAD benchmarks detection across unseen tasks and anomaly categories; all evaluated methods remain below 70% micro-averaged frame-level AUC when both are unseen.
Problem
Robotic manipulation failures depend on task context and are difficult to enumerate, while detectors need to generalize to failure categories absent from training.
Source: §I · p. 1Method
RoboVAD is a large-scale benchmark with protocols for unseen tasks, unseen anomaly categories, and their combination, evaluated alongside existing methods and two FlowJigsaw models.
Source: §I · p. 2Results
All evaluated approaches remain below 70% micro-averaged frame-level AUC when both manipulation tasks and anomaly types are unseen.
Source: §I · p. 2Takeaways & Limitations
RoboVAD exposes substantial challenges beyond conventional in-domain evaluation and provides a benchmark for comparing methods under cross-domain conditions.
Source: §I · p. 2
Abstract
from arXiv · showhide
Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic safety. The task is also relevant for robotic arm interactions, where it has several downstream applications, including learning better interaction and manipulation abilities, triggering recovery procedures when anomalies occur, etc. Despite its relevance, the exploration of anomaly detection in robotic arm manipulation videos is limited by the low number of available resources. To this end, we introduce RoboVAD, a large-scale benchmark for video anomaly detection that comprises challenging cross-domain evaluation scenarios, where certain actions (tasks executed by a robotic arm) and anomaly types (mistakes that occur while performing certain tasks) remain unseen during training. RoboVAD is designed to benchmark VAD methods in realistic scenarios, where robotic arms can perform unforeseen tasks, and thereby encounter new anomaly types. We train and evaluate several state-of-the-art VAD methods, including a novel method specifically adapted for robotic arm manipulation. While the proposed method outperforms many state-of-the-art competitors, all methods remain below a micro-averaged frame-level AUC threshold of 70% in the most challenging evaluation setup, confirming the difficulty of the proposed benchmark. We publicly release our dataset and code at https://zenodo.org/records/22754659.
I. INTRODUCTION
Robotic manipulation anomaly detection must contend with rare, context-dependent failures that cannot be exhaustively anticipated during training. RoboVAD benchmarks generalization to unseen tasks and anomaly categories, where evaluated methods remain below 70% micro-averaged frame-level AUC when both are unseen.
I. INTRODUCTION
- Robotic failures are context-dependent, making representative training examples impractical to collect for every failure a robot may encounter.A manipulation outcome can be normal in one task but abnormal in another; most VAD methods therefore learn normality from anomaly-free videos, while alternatives use video-level abnormal labels.Source: §I · p. 1
- Existing robotic failure resources are limited in task coverage, failure categories, or sensory configuration, and often do not provide frame-level temporal localization.Evaluation restricted to familiar tasks and anomalies can give an overly optimistic estimate of practical performance.Source: §I · pp. 1–2
- Its cross-domain protocols separately test generalization to unseen manipulation tasks, unseen failure categories, or both together.The combined setting simulates monitoring new robot behaviors and mistakes not anticipated or exhaustively collected during training.Source: §I · p. 2
- The paper benchmarks methods from several VAD families and introduces FlowJigsaw-SSL and FlowJigsaw-Supervised, motion-oriented models for normal-only and supervised training.The evaluation includes a broad empirical comparison under consistent frame-level protocols.Source: §I · p. 2
- RoboVAD comprises 620,979 frames, 1,570 temporally localized anomalies, two synchronized camera views, and five manipulation tasks and anomaly categories.The benchmark separates task and anomaly-category domain variation, including a combined setting where both are unseen.Source: §I · p. 2
- All evaluated approaches remain below 70% micro-averaged frame-level AUC when both manipulation tasks and anomaly types are unseen.This cross-task+anomaly result highlights challenges not captured by conventional in-domain evaluation.Source: §I · p. 2
A. Video Anomaly Detection
Video anomaly detection methods commonly learn normality or use labeled abnormal examples, with newer approaches adding self-supervised tasks and vision-language models. Open-set evaluation tests generalization beyond training anomaly categories, while robotic manipulation presents a distinct setting with subtle, domain-specific anomalies.
A. Video Anomaly Detection
- Many VAD methods learn normality from anomaly-free data, detecting significant deviations at test time,.Approaches include reconstruction, future-frame prediction, object-centric learning, appearance-motion modeling, and optical-flow methods, –.Source: A. Video Anomaly Detection · p. 2
- Self-supervised VAD uses proxy tasks such as temporal ordering, event completion, and spatial or temporal jigsaw solving, –.Later frameworks scale these tasks with modern backbones and add region filtering, inpainting, motion-weighted learning, or self-distillation,.Source: A. Video Anomaly Detection · pp. 2–3
- Weakly supervised methods use video-level labels, while vision-language and foundation-model approaches support alignment or training-free and zero-shot detection,, –, –.Examples include discriminative temporal features, normal and abnormal prototypes, and caption-based reasoning.Source: A. Video Anomaly Detection · p. 3
- Open-set evaluation tests generalization beyond training anomaly categories; RoboVAD applies normal-only and supervised open-set formulations to robotic manipulation,.Its focus is subtle, domain-specific anomalies in robot manipulation videos rather than typical surveillance anomalies,.Source: A. Video Anomaly Detection · p. 3
B. Robotic Manipulation Failure Detection and Datasets
Robotic manipulation failures include physical execution errors, which are not equivalent to visual anomalies: unusual observations can be benign, and failures can be visually subtle. Existing datasets vary in modalities and objectives; RoboVAD focuses on frame-level localization of physical failures and supports cross-domain evaluation.
B. Robotic Manipulation Failure Detection and Datasets
- Visual anomalies and manipulation failures are distinct: unusual observations may be benign, while some failures are visually subtle.RoboVAD annotates physical manipulation failures and uses visual anomaly detection as the learning formulation for localizing them.Source: B. Robotic Manipulation Failure… · p. 3
- Prior failure-detection methods combine modalities or model nominal motion and action-conditioned outcomes,,,,.Examples include RGB, depth, and audio fusion, optical-flow dynamics, and detecting violations of expected action effects.Source: B. Robotic Manipulation Failure… · p. 3
- Existing datasets differ in modality and annotation granularity, spanning frame-level labels, multimodal failure classification, action phases, and execution outcomes –,,.Their target settings include placement, tabletop actions, human-robot handover, pouring, and contact-rich assembly.Source: B. Robotic Manipulation Failure… · p. 3
- RoboVAD targets frame-level localization of physical failures in low-cost, multi-view tabletop manipulation and supports normal-only, supervised open-set, cross-task, and camera-view evaluation.The paper reports that it is the only field dataset with challenging cross-domain setups, including cross-task and cross-anomaly, and contains 3× more anomaly instances than existing benchmarks.Source: B. Robotic Manipulation Failure… · p. 3
III. ROBOVAD BENCHMARK
RoboVAD benchmarks frame-level anomaly detection in robotic manipulation across five tasks and five anomaly categories, including protocols with unseen tasks, anomaly types, or both. It combines temporally localized annotations with multiple camera views and episode-level dataset splits.
Overview and motivation
- RoboVAD is the only cross-domain robotic-arm evaluation benchmark and contains 3× more anomaly instances than other datasets.The comparison also notes that larger existing datasets typically lack frame-level annotations.Source: Table I · p. 4
Setup and dataset splits
- The dataset provides both camera views, robot state and action data, frame-level labels, and episode-level train, validation, and test assignments.The two held-out tasks are rice pouring and ring insertion; source tasks are assigned across partitions according to anomaly categories.Source: §III · p. 4
Tasks and anomaly types
- The benchmark covers five manipulation tasks and five anomaly categories, with events defined by deviations from intended robot-object-environment interaction.An episode may contain multiple anomalous events and still achieve its task goal.Source: §III · p. 4
Episodes and preprocessing
- Episodes continue after an anomaly while further progress remains possible, so a single execution can contain multiple events.Annotations mark frame-level event intervals, severity, and visibility across the two camera views.Source: §III · p. 4
Preprocessing and annotation
- Two authors independently annotated 50 episodes, achieving Cohen’s κ = 1.00 for episode-level detection, κ = 0.79 for frame-level labels, and κ = 0.71 for anomaly categories.Disagreements were cross-reviewed and resolved through discussion.Source: §III · p. 4
Overview and motivation
- RoboVAD evaluates in-domain, cross-task, cross-anomaly, and combined cross-task+anomaly generalization, with unseen tasks and anomaly categories held out during training.Its 1,078 episodes are recorded from two camera viewpoints, and anomalous events are temporally localized.Source: §III · pp. 3, 5
IV. METHODS
The methods compare robotic-failure, weakly supervised, language-guided, and self-supervised detectors, alongside optical-flow models tailored to manipulation. FlowJigsaw-SSL learns motion representations through three proxy tasks, while FlowJigsaw-Supervised trains a binary classifier using normal executions and known failures.
Compared methods
- Comparisons include robotic failure detection, weakly supervised VAD, language-guided detection, and self-supervised Jigsaw-VAD.LaGoVAD accepts natural-language anomaly definitions at inference; weakly supervised baselines use video-level labels without temporal failure annotations.Source: §IV · p. 5
Multi-task self-supervised model
- FlowJigsaw-SSL uses optical-flow clips to learn manipulation motion through arrow-of-time prediction, motion irregularity detection, and inpainting.The three tasks target deviations such as object slips, drops, and collisions.Source: §IV · p. 6
Model architecture
- Flow fields from two camera views are concatenated into six-channel clips and processed by a shared 3D convolutional backbone with three task-specific heads.The flow channels encode horizontal displacement, vertical displacement, and motion magnitude.Source: §IV · p. 6
Inpainting and training
- For inpainting, each frame is divided into a 4×4 grid, 25% of frame area is masked, and a third head reconstructs missing regions from remaining spatial and temporal context.The total loss is Ltotal = LAoT + LMI + 0.1 LIP.Source: §IV · p. 6
Inference and supervised model
- At inference, normalized arrow-of-time, motion-irregularity, and inpainting scores are averaged into a clip-level anomaly score, then overlapping clip scores are averaged per frame.FlowJigsaw-Supervised replaces the proxy-task heads with a binary normal-or-anomalous classifier trained on normal executions and available anomaly categories.Source: §IV · p. 6
A. Implementation Details
The evaluation retains the chosen methods’ architectures while adapting data loading and evaluation to RoboVAD. FlowJigsaw variants use optical-flow inputs and distinct training and inference configurations.
A. Implementation Details
- Official architectures are retained; inputs are resized to 128 × 128, with 307 normal episodes for normal-only methods and all 367 training episodes for supervised and weakly supervised methods.Loading and evaluation protocols are dataset-specific.Source: A. Implementation Details · p. 6
- DSANet, VadCLIP, and LaGOVAD use frozen CLIP features, UR-DMU uses frozen RGB I3D features, and Jigsaw-VAD, FrameShield, and FIDeL use their stated method-specific training or memory setups.The passage specifies epochs, batch sizes, optimizers, and learning rates for the trained components; FIDeL builds normal reference memories with a fixed DINOv2 encoder.Source: A. Implementation Details · p. 6
- Both FlowJigsaw variants train from random initialization for 15 epochs on six optical-flow frames, with FlowJigsaw-Supervised using stride four and class-weighted focal loss.The supervised variant uses batch size 64, while the SSL variant uses batch size 32 and no weight decay.Source: A. Implementation Details · pp. 6–7
- At inference, FlowJigsaw-SSL sums scores from its three proxy tasks and averages reconstruction error over eight random masks.Source: A. Implementation Details · p. 7
B. Anomaly Detection Results
Across four evaluation settings, FlowJigsaw variants lead several comparisons, but performance falls when tasks and anomaly categories are both unseen. Qualitative examples also show a lower score for a placement failure in an unseen task and anomaly category.
B. Anomaly Detection Results
- All methods remain below 70% micro AUC when tasks and anomalies are unseen; FlowJigsaw-SSL leads micro AUC at 67.4%, while UR-DMU leads macro AUC at 71.9%.Source: B. Anomaly Detection Results · p. 7
- 79.6% micro and 82.5% macro AUC: FlowJigsaw-SSL leads cross-anomaly evaluation, outperforming its supervised counterpart by 10.6% on both measures.Source: B. Anomaly Detection Results · p. 7
- 94.6% micro AUC: FlowJigsaw-Supervised leads in-domain, exceeding UR-DMU by 6.1%, while both methods attain 94.4% macro AUC.Source: B. Anomaly Detection Results · p. 7
- 72% micro AUC: FlowJigsaw-Supervised leads on unseen tasks, while UR-DMU leads macro AUC at 77.5%.Source: B. Anomaly Detection Results · p. 7
- For ring insertion, three anomalous events produce clear score peaks, but the placement failure receives a lower score as an unseen task-and-anomaly case.Figures 3 and 4 illustrate cross-anomaly detection in cube stacking and cross-task or combined cross-task-and-anomaly detection in ring insertion.Source: B. Anomaly Detection Results · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
VI. CONCLUSION
RoboVAD evaluates frame-level anomaly detection across unseen robotic manipulation tasks and anomaly categories. Results remain challenging when both are unseen, and the current dataset is limited to a single robotic platform and tabletop environment.
VI. CONCLUSION
- All evaluated methods remain below 70% micro-averaged frame-level AUC when both tasks and anomaly categories are unseen.Source: §VI · p. 7
- Optical flow consistently improves performance over RGB inputs for both FlowJigsaw variants.Source: §VI · p. 7
- The dataset covers one robotic platform and a tabletop environment; future work could expand task and environment diversity and investigate online anomaly detection for robot recovery.Source: §VI · pp. 7–8