Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

1 Introduction

• Weakly supervised labels reduce annotation effort compared with frame-level labels, addressing a barrier to large-scale untrimmed-video datasets and practical detection.Earlier approaches often classified trimmed clips, limiting their ability to locate violence in untrimmed videos.Source: §1 · p. 2
• Visual cues usually discriminate and locate events, while audio can distinguish visually ambiguous events; the paper combines both to exploit complementary information.Source: §1 · p. 2
• HL-Net models snippet relations through a holistic branch using similarity priors for long-range dependencies and a localized branch using proximity priors for local interactions.A dynamic score branch adds a holistic relation, with weights determined by predicted scores.Source: §1 · p. 3
• Because HL-Net needs the whole video to compute snippet relations, the HLC approximator supports online detection by processing a local neighborhood and learning predictions guided by HL-Net.Source: §1 · p. 3
• XD-Violence contains 4,754 untrimmed audio-visual videos, spans 217 hours, covers six violence types, and includes movies and in-the-wild scenarios.The paper casts weakly supervised detection as multiple instance learning, treating videos as bags of snippets and learning instance-level annotations from bag-level labels.Source: §1 · pp. 2–3
• The method shows clear advantages over existing baselines on XD-Violence and UCF-Crime, while experiments also find multimodal input superior to unimodal input.Source: §1 · p. 3

2 Related Work

• Earlier violence-detection work often relied on hand-crafted features such as SIFT, STIP, HOG, and optical-flow histograms, commonly using small-scale datasets.Later deep-learning approaches included convolutional LSTMs, bidirectional convolutional LSTMs, and other neural methods.Source: §2 · pp. 3–4
• Prior multimodal violence methods largely used hand-crafted audio features, whereas this work extracts high-level audio features with a CNN; related work also models relations with GCNs, temporal relation networks, and self-attention.Source: §2 · p. 4

3.1 Selecting Violence Categories

• The dataset defines six physically violent categories: Abuse, Car Accident, Explosion, Fighting, Riot, and Shooting.The categories were selected for their clear definitions, frequent occurrence, widespread use, and adverse safety impact.Source: §3.1 · p. 4

3.2 Collection and Annotation

• The dataset spans movies and YouTube in-the-wild scenes, with 4,754 videos split into 3,954 training and 800 test videos.The test set includes 500 violent and 300 non-violent videos; violent test videos have temporal annotations based on averaged annotations from multiple annotators.Source: §3.2 · pp. 4–5

3.3 Dataset Statistics

• The dataset includes varied scenarios and assigns each violent video one to three labels to represent co-occurring violent events.It consists of untrimmed videos, with distributions reported for video length, category labels, and the percentage of violence in test videos.Source: §3.3 · pp. 5–6, Figure 2 · p. 6, Figure 3 · p. 6

3.4 Dataset Comparisons

• XD-Violence is more than 300 times the total of small datasets, 4 times the total of medium datasets, and almost 2 times UCF-Crime in size.The paper identifies large scale, scenario diversity, and audio-visual signals as the dataset’s distinguishing traits.Source: Table 1 · p. 7, §3.4 · p. 7
• Compared with prior datasets, XD-Violence covers a wider variety of scenarios and includes audio-visual signals, while UCF-Crime lacks audio.The paper notes that quite a few Violent-Flows and CCTV-Fights videos are silent or contain only background music.Source: §3.4 · p. 7

4.1 Multimodal Fusion

• Visual and audio snippet features are concatenated and passed through two fully connected layers to produce fusion features.The features are extracted using a sliding-window mechanism.Source: §4.1 · pp. 7–8

4.2 Holistic and Localized Networks

• The holistic branch uses thresholded feature-similarity relations for global dependencies, while the localized branch uses temporal proximity to retain positional relations.The holistic branch normalizes relations with softmax; the localized branch uses hyperparameters to control the influence of temporal distance.Source: §4.2 · pp. 8–9

4.3 Online Detection

• The HLC approximator uses previous snippets and HL-Net guidance to predict online, while its causal-convolution classifier adds a dynamic score branch based on predicted-score closeness.Its architecture includes two fully connected layers followed by ReLU and a 1D causal convolution with kernel size 5 and stride 1.Source: §4.3 · p. 10

4.4 Training Based on MIL

• MIL training averages temporal K-max activations for CP and CS, uses binary cross-entropy for classification, and distills HL-Net outputs to the HLC approximator.Positive-bag K-max instances are treated as likely true violence instances, while negative-bag K-max instances are hard instances expected to be far from them.Source: §4.4 · pp. 10–11

4.5 Inference

• Inference can be offline or online; online inference uses only the HLC approximator, with HL-Net removed, and sigmoid outputs provide violence confidence scores in [0, 1].Source: §4.5 · p. 11

5.1 Evaluation Metric

• The evaluation uses frame-level precision-recall curves and AP rather than ROC curves and AUC because AUC can be optimistic on class-imbalanced data.PRC and AP focus on positive violence samples.Source: §5.1 · p. 11

5.2 Implementation Details

• The default input fuses I3D RGB features with VGGish audio features, and the generalized HL-Net uses three two-layer branches with residual connections.Videos are sampled at 24 FPS with 16-frame windows; audio is split into overlapping 960-ms segments aligned to video snippets. Training uses Adam for 50 epochs with batch size 128.Source: §5.2 · pp. 11–12

5.3 Ablation Studies

• Audio+Flow improves AP by 3.5% over Flow-only, while Audio+RGB improves AP by 3.2% over RGB-only.Visual-only input is significantly superior to audio-only input; the authors attribute the fusion gains to audio providing complementary information to visual signals.Source: §5.3 · pp. 12–13, Figure 5 · p. 12
• Removing any holistic, localized, or score branch harms performance, and HL-Net performs best with all three branches together.The three separate branches achieve similar performance when evaluated individually.Source: §5.3 · p. 13, Table 2 · p. 14
• Offline detection exceeds online detection by 5% on AP, except during the initial training stage, when the lightweight HLC approximator performs better.The authors attribute the early-stage result to the approximator being easier to train, while HL-Net needs more time to train; offline detection otherwise benefits from its three types of feature aggregation.Source: Figure 6 · p. 12, §5.3 · p. 13, Table 3 · p. 14

5.4 Comparisons with State-of-the-Arts

• Our method outperforms current state-of-the-art methods on XD-Violence, and C3D-plus-audio features perform substantially worse than I3D-plus-audio features.All baselines in the comparison use I3D RGB and VGGish audio features.Source: §5.4 · p. 13, Table 3 · p. 14

5.5 Qualitative Results

• Qualitative examples show multimodal input localizing violence more precisely, while online detection has more false alarms due to missing context.In one example, Audio+RGB detects an explosion using sound as well as fire; another video retains a violence score for an overturned car after the accident ends, although that part is not labeled violent.Source: §5.5 · pp. 13–14, Figure 7 · p. 14

6 Conclusions

• Experiments show that the dataset is applicable, multimodality significantly improves performance, and explicitly exploiting relationships is highly effective.The authors identify adding audio-dominated violence classes, extending to multi-class detection, and improving online detection as future work.Source: §6 · p. 14
• The work releases a large-scale violence dataset and proposes a method to model relationships among video snippets and learn powerful representations.The study addresses violence detection with audio-visual modalities under weak supervision.Source: §6 · p. 14
• Future work will add audio-dominated violence classes, extend detection to multiple classes, and further explore more powerful online detection.The authors note that XD-Violence is a multi-label dataset, enabling the extension to multi-class violence detection.Source: §6 · p. 14

1 Video Collection

• Movies were sampled across genres, cut into variable-length clips, labeled at video level by eight annotators, and checked to correct errors and remove unsuitable videos.The dataset was repeatedly split into training and test sets, retaining the split with a suitable proportion.Source: §1 · p. 19
• YouTube candidates were gathered through text searches, with non-violent videos selected to match violent-video backgrounds and reduce reliance on scene context.Videos were removed if soundless, containing only background sounds, ambiguous, blurry, or containing very little violence.Source: §1 · p. 19

• Table 1 lists the violence types represented in common datasets.Source: §2 · p. 19

3 Similarity Computation Functions

• The three versions perform similarly, but Version 2 narrowly outperforms the other two with learnable weights that can better learn similarity.Source: §3 · p. 20

4 The Effect of Length of Sampling

• For videos shorter than Γ, the method processes the entire feature sequence; for longer videos, it uniformly extracts a Γ-length segment to represent the video.This sampling strategy addresses video-level supervision while avoiding direct processing of very long videos.Source: §4 · p. 20
• As Γ increases, training-epoch runtime rises while performance first improves and then fluctuates slightly; the paper selects Γ=200 as a tradeoff between accuracy and computational cost.The authors describe Γ=200 as a good tradeoff between accuracy and computation burden.Source: §4 · pp. 20–21, Table 3 · p. 21

5 Investigating Perclass Performance with Different Multimodal Cues

• Audio+RGB improves per-class performance over single signals except for abuse, with clear boosts for Shoot, Riot, and Car Accident.The passage suggests abuse’s small sample count may explain the exception.Source: §5 · p. 21

6 Comparisons with State-of-the-Arts

• On UCF-Crime, the method outperforms current state-of-the-art methods in AUC comparisons.The supplied text reports the comparison but gives no AUC values.Source: §6 · p. 21, Table 5 · p. 22
• On XD-Violence, the method’s PRC curve completely encloses competitors, while online detection and RGB-only input lack the maximum area under the curve.The passage attributes the shortfall to missing contextual information for online detection and missing audio information for RGB-only input.Source: §6 · p. 21, Figure 1 · p. 22

7 More Qualitative Results

• RGB-only input produces false alarms when scenes change in live video and when football resembles a fight or an airplane plummets through the sky.These are the qualitative examples labeled (a), (b), and (c).Source: §7 · p. 22
• A ceiling mirror is identified as a possible cause of a false alarm in one test video.This is the example labeled (d).Source: §7 · p. 22
• Offline detection misses a violent region in one example, which the authors attribute to over-smoothing.The passage notes that over-smoothing usually occurs in GCN.Source: §7 · p. 22
• The missed alarm is attributed to violent features being smoothed by non-violent features because the violent segment occupies a small part of the video.Source: §7 · p. 23