Source-linked AI summary
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision
Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, Zhiwei Yang
TL;DR
The paper addresses limited scale, scene diversity, and modality coverage in violence detection by releasing XD-Violence and proposing a multimodal relation-modeling network for weakly supervised detection. The method outperforms existing baselines on XD-Violence and UCF-Crime, while experiments support audiovisual input and explicit relation modeling.
Problem
Weakly supervised violence detection lacks large, multi-scene audiovisual datasets and methods that effectively model relations among video snippets.
Method
The paper releases XD-Violence and uses HL-Net to model long-range, local, and score-based relations, with an HLC approximator for online detection.
Results
The method shows clear advantages over existing baselines on XD-Violence and UCF-Crime, and multimodal input improves performance over unimodal input.
Takeaways & Limitations
The experiments support multimodality and explicit modeling of relations among video snippets for weakly supervised violence detection.
Takeaways & Limitations
Violent features can be smoothed by non-violent features when violent segments occupy only a small part of the video.
Abstract
from arXiv · showhide
Violence detection has been studied in computer vision for years. However, previous work are either superficial, e.g., classification of short-clips, and the single scenario, or undersupplied, e.g., the single modality, and hand-crafted features based multimodality. To address this problem, in this work we first release a large-scale and multi-scene dataset named XD-Violence with a total duration of 217 hours, containing 4754 untrimmed videos with audio signals and weak labels. Then we propose a neural network containing three parallel branches to capture different relations among video snippets and integrate features, where holistic branch captures long-range dependencies using similarity prior, localized branch captures local positional relation using proximity prior, and score branch dynamically captures the closeness of predicted score. Besides, our method also includes an approximator to meet the needs of online detection. Our method outperforms other state-of-the-art methods on our released dataset and other existing benchmark. Moreover, extensive experimental results also show the positive effect of multimodal (audio-visual) input and modeling relationships. The code and dataset will be released in https://roc-ng.github.io/XD-Violence/.
1 Introduction
The paper targets weakly supervised violence detection in untrimmed, multimodal videos, addressing limitations of short-clip, single-scenario, and hand-crafted multimodal approaches. It introduces XD-Violence and a relation-aware network with online detection support, outperforming baselines on two benchmarks.
- Research gap: Weakly supervised violence detection uses video-level labels to reduce the cost of frame-level annotation for large-scale untrimmed-video datasets.The paper frames each video as a bag of snippets and learns instance-level annotations from bag-level labels.
- Method: HL-Net models snippet relations through parallel holistic and localized branches based on similarity and proximity priors.The holistic branch captures long-range dependencies, while the localized branch models short-range interactions.
- Method: An HLC approximator uses previous snippets to support online detection when HL-Net cannot access the whole video.The approximator is designed to generate predictions guided by HL-Net.
- Contributions: XD-Violence contains 4754 untrimmed audio-visual videos from movies and in-the-wild scenarios, supporting multimodal weakly supervised detection.The dataset covers six common violence types and totals 217 hours.
- Results: The method shows clear advantages over existing baselines on XD-Violence and UCF-Crime, while multimodal input outperforms unimodal input.The paper attributes complementary value to visual and audio cues when visual signals are ambiguous.
3 XD-Violence Dataset
XD-Violence is a large-scale, multi-scenario audiovisual dataset designed to support weakly supervised violence detection. It contains untrimmed videos, multiple violence labels, and varied sources that address limitations of earlier datasets.
- Dataset scale and scope: The dataset combines movies and YouTube in-the-wild scenes, alongside sources such as sports, games, news, live scenes, and CCTV recordings.This multi-scenario collection is intended to broaden scene variation beyond datasets drawn from only one source type.
- Annotations and video structure: Violent videos may receive one to three labels because multiple violent events can co-occur, with label order reflecting event importance.The dataset is multi-label rather than restricted to a single violence category per video.
- Annotations and video structure: The videos are untrimmed, and the dataset reports distributions of video length and the percentage of violence in each test video.These statistics characterize how violence occupies only part of some videos.
- Comparison with prior datasets: Compared with prior datasets, XD-Violence is larger, more varied in scenarios, and includes audiovisual signals for multimodal violence detection.The paper describes it as more than 300 times the total of small datasets, four times the total of medium datasets, and almost twice UCF-Crime.
4 Methodology
The method extracts visual and audio snippet features, fuses them, and models relationships among snippets through holistic, localized, and score branches. An HLC approximator supports online inference, while multiple-instance-learning objectives train violence predictions from video-level labels.
- Multimodal Fusion: Visual and audio feature extractors process untrimmed videos with a sliding-window mechanism, producing snippet-level feature matrices for each modality.Each feature corresponds to a video snippet.
- Multimodal Fusion: The method concatenates visual and audio features across channels and transforms them through two fully connected layers with ReLU and dropout to obtain fusion features.The fully connected layers have 512 and 128 nodes, respectively.
- Holistic and Localized Networks: The holistic branch models long-range dependencies using a feature-similarity relation matrix, with normalization and thresholding used to filter weak relations.The similarity-based relation matrix supports interactions between arbitrary feature positions.
- Holistic and Localized Networks: The localized branch preserves temporal proximity by defining relations from the positions of snippet features, with hyperparameters controlling the distance influence.This branch complements the holistic branch’s position-independent interactions.
- Online Detection: The score branch updates relations from predicted-score closeness, while the HLC approximator uses previous snippets to generate online predictions guided by HL-Net.The approximator contains stacked fully connected layers, ReLU, and a causal one-dimensional convolution; online inference uses only the approximator.
- Training Objective: Multiple-instance-learning losses use K-max activations for video-level classification, and knowledge distillation encourages the HLC approximator to approximate HL-Net outputs.K-max instances in positive bags are treated as likely violent instances, while those in negative bags are treated as hard instances.
5 Experiments
Experiments evaluate modality choices, branch contributions, online versus offline detection, and comparisons with existing methods using AP. Audio-visual fusion improves over visual-only inputs, all three HL-Net branches contribute, and the proposed method outperforms the listed baselines on XD-Violence.
- AP is evaluated with precision-recall curves because AP focuses on positive violence samples under class imbalance.
- The effect of modality: Audio plus Flow improves AP by 3.5% over Flow-only input, while audio plus RGB improves AP by 3.2% over RGB-only input.These results support complementary information from audio and visual signals.
- The effect of holistic, localized and score branches: Removing any holistic, localized, or score branch harms performance, while HL-Net performs best with all three branches together.The ablation compares versions formed by deleting one or two of the three parallel branches.
- Online detection vs Offline detection: Offline detection outperforms online detection by 5% on AP, with the difference attributed to HL-Net’s three feature-aggregation types.The offline model computes relations using the whole video, whereas the online setting uses the HLC approximator.
- Qualitative Results: Qualitative examples show successful detection of violent and non-violent videos, while multimodal input more precisely localizes violence with fewer false positives and false negatives in videos with audio.The examples also show online detection as slightly worse than offline detection, with higher false alarms because of limited contextual information.
- Comparisons with State-of-the-Arts: 78.64 AP is reported for Ours on XD-Violence, compared with 73.67 for Ours (Online), 67.19 for Ours (C3D), and 73.20 for Sultani et al.The authors state that their method outperforms current state-of-the-art methods; C3D is inferior to I3D by a large margin in this task.
6 Conclusions
The paper studies large-scale weakly supervised violence detection with audio-visual modalities. It releases a dataset, models relations among video snippets, and reports benefits from multimodality and explicit relation modeling while identifying stronger online detection as future work.
- The paper releases a large-scale violence dataset and proposes explicit relation modeling among video snippets for weakly supervised audio-visual detection.
- Extensive experiments show that the dataset is applicable, multimodality significantly improves performance, and explicitly exploiting relations is highly effective.
- Future work includes adding audio-dominated violence classes, extending to multi-class detection, and exploring more powerful online detection.
Supplementary Materials Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak
The supplementary materials section is represented here only by the authors’ affiliation information. No supplementary technical or experimental content is supplied in the selected passages.
- The listed authors are affiliated with the School of Artificial Intelligence, Xidian University, Xi’an, China.
1 Video Collection
The dataset collection process uses movie and YouTube sources, expert annotation, and repeated random splitting to obtain suitable training and test proportions.
- Movie collection spans multiple genres, while annotators randomly cut sections containing clear violent or non-violent events and assign video-level labels.
- The dataset is randomly split into training and test sets multiple times, retaining the split with a suitable proportion.
2 Dataset Comparisons
The paper compares the violence types represented in common datasets, including XD-Violence and UCF-Crime.
- Common violence datasets cover categories such as fighting, abuse, assault, explosions, shooting, and robbery.
3 Similarity Computation Functions
The section introduces alternative similarity-computation functions and reports that their performance is comparable, with Version 2 slightly ahead.
- The section presents two additional versions of the similarity function.
- Three similarity-function versions achieve similar performance, while Version 2 leads by a narrow margin.The passage attributes Version 2's advantage to its learnable weights, which can learn better similarity.
4 The Effect of Length of Sampling
The method samples long videos to balance whole-video processing, GPU-memory constraints, accuracy, and computation cost.
- Long untrimmed videos cannot always be processed directly because GPU memory is limited.The sampling strategy addresses the tension between using entire videos under video-level labels and limiting memory use.
- 200 is selected as the sampling length because it provides a tradeoff between accuracy and computational cost.Videos shorter than the threshold are processed entirely; longer videos use a uniformly extracted segment.
- Increasing the sampling threshold raises training-epoch runtime, while performance first improves and then fluctuates slightly.
5 Investigating Perclass Performance with Different Multimodal Cues
Experiments compare multimodal cues, benchmark performance, precision–recall behavior, and qualitative failure cases across offline, online, RGB-only, and Audio+RGB settings.
- Per-class multimodal performance: Audio+RGB improves per-class performance over single-signal input except for abuse, with clear gains for shooting, riot, and car accident classes.The passage suggests the abuse exception may reflect its small number of samples.
- State-of-the-art comparison: Our method outperforms current state-of-the-art methods on the UCF-Crime dataset.
- Precision–recall comparison: The method's precision–recall curve encloses competitors across thresholds, while online detection and RGB-only lack the maximum area under the curve.The passage attributes these deficits to missing contextual information for online detection and missing audio information for RGB-only input.
- Qualitative failure cases: RGB-only input produces false alarms in changing scenes, football, and an airplane plummeting, while a ceiling mirror can also confuse the method.
- Qualitative failure cases: GCN over-smoothing can miss violence when violent segments occupy only a small part of the video.Violent features are smoothed by non-violent features in this setting.