PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Isolated Entities: Relation-Aware Multi-Entity Modeling for Unsupervised Video Anomaly Detection

Zhongpeng Pan, Xina Cheng, Kailun Yang

arXiv:2609.31753v1eess.IVcs.CVcs.RO

TL;DR

Unsupervised video anomaly detection must identify abnormal interactions among familiar entities despite changing configurations and normal-only training data. INTERACT models relational history and verifies predicted interactions against motion and semantic consistency. It reports state-of-the-art ShanghaiTech performance, while broader generalization to mobile first-person views remains to be established.

  • Problem

    In normal-only anomaly detection, models must learn temporal regularities among entities and recognize deviations despite changing entity configurations.

    Source: §I · pp. 1–2

  • Method

    INTERACT uses POAMI to represent entity states and context, TGRIP to predict interactions from relational history and target geometry, and MIRA to check motion reconstruction and semantic consistency.

    Source: §I · p. 2

  • Results

    84.5% frame-level AUC on ShanghaiTech is state-of-the-art; disabling cross-entity information exchange reduces AUC by 5.32 percentage points.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The ablation supports the contribution of relational modeling, while two Go2 sequences suggest potential transfer to mobile first-person views without retraining.

    Source: §V · p. 8

    Broader evaluation is needed to establish generalization to mobile first-person views.

    Source: §V · p. 8

Abstract

from arXiv · show

Video anomaly detection for patrol robots and surveillance systems must recognize abnormal interactions among familiar entities. Existing pixel-reconstruction and isolated-entity methods may fail when individual entities appear normal but their spatial or motion relations are abnormal. This work presents Interaction-Centric Network for Temporal Entity-Relation Analysis and Consistency Testing (INTERACT), a framework for unsupervised video anomaly detection. Person-Object Appearance-Motion Interaction (POAMI) moves beyond isolated-entity encoding by jointly modeling the appearance, motion, and spatial configurations of persons and objects in each frame. The resulting representations capture both individual entity states and their cross-entity context. Based on these representations, Target Geometry-Guided Relational Interaction Prediction (TGRIP) predicts target entity interaction states from historical relational memory and target-frame geometry without explicit identity tracking. Motion-Interaction Reconstruction and Alignment (MIRA) then evaluates these predictions through conditional flow reconstruction and semantic consistency checking, providing complementary anomaly evidence beyond prediction error alone. INTERACT achieves state-of-the-art performance, obtaining a frame-level AUC of 84.5% on the ShanghaiTech benchmark. Ablation studies show that removing cross-entity attention causes the largest performance drop, demonstrating the necessity of relational modeling. INTERACT is particularly effective for anomalies caused by changes in relations among people and objects, while maintaining competitive performance in general scenarios. The source code will be made publicly available at https://github.com/ppworkhard/INTERACT.

I. INTRODUCTION

INTERACT frames unsupervised anomaly detection as testing whether multi-entity interactions are predictable from relational history and consistent with observed motion. Its representation, prediction, and verification components model relations among familiar entities, with state-of-the-art frame-level AUC reported on ShanghaiTech.

I. INTRODUCTION

  • The normal-only setting must handle open-ended anomalies and changing entity configurations, including entity counts, positions, and participants.Source: §I · pp. 1–2
  • Anomalies can arise from relations among individually normal entities, but existing cues may fail to capture dependencies across appearance, motion, position, and time.Source: Figure 1 · p. 1, §I · p. 2
  • INTERACT defines normality through predictable, motion-consistent multi-entity interactions rather than isolated appearances or motions.Source: §I · p. 2
  • TGRIP predicts entity interaction representations from relational history and target-frame geometry without explicit cross-frame identity tracking.Source: §I · p. 2
  • MIRA conditions motion reconstruction on predicted representations and checks semantic consistency, complementing interaction-state deviations with reconstruction and representation-consistency errors.Source: §I · p. 2
  • POAMI jointly models appearance, motion, and spatial position through cross-modal attention, producing entity tokens that encode entity state and inter-entity context.Source: §I · p. 2
  • 84.5% frame-level AUC on ShanghaiTech is state-of-the-art, with competitive performance on Avenue and NWPU Campus_sub.Source: §I · p. 2

A. Robot Anomaly and Safety Monitoring

Prior work covers robot-state and geometric safety monitoring, residual-based video anomaly detection, and entity-centric or supervised interaction modeling. INTERACT instead evaluates whether relations among entities are predictable from history and consistent with observed motion.

A. Robot Anomaly and Safety Monitoring

  • Robot fault diagnosis monitors internal actuation signals, while human-robot safety methods use geometric and kinematic constraints; neither assesses whether scene configurations match normal behavior.Source: A. Robot Anomaly and Safety Monitoring · pp. 2–3

B. Video Anomaly Detection

  • Video anomaly detectors commonly score frame- or region-level residuals, which may not reveal divergence between individually plausible entity trajectories.Source: B. Video Anomaly Detection · p. 3

C. Entity-Centric and Interaction Modeling

  • Entity-centric methods largely reconstruct or cluster instances, while supervised interaction detectors use closed vocabularies; INTERACT models full-set relations and scores unseen relational deviations.Source: C. Entity-Centric and Interaction… · pp. 3–4

III. METHODOLOGY

INTERACT predicts target interaction states from normal-video history and scores deviations, without explicit cross-frame identity tracking. POAMI represents both entity-level appearance and motion and the context among persons and objects.

A. Problem Formulation and INTERACT Overview

  • INTERACT predicts a target state from the preceding frames and uses its deviation as anomaly evidence, without explicit cross-frame identity tracking.It is optimized in two stages: Stage 1 predicts global and entity interaction states, and Stage 2 verifies entity predictions with Stage 1 frozen.Source: A. Problem Formulation and INTERACT… · p. 4

B. POAMI: Relation-Aware Multi-Entity Appearance-Motion Representation

  • POAMI jointly models appearance, motion, and spatial structure so each entity representation captures its own state and relations across the entity set.Appearance and motion features are extracted independently; bounding-box centers are discretized on a 30×54 grid for spatial encoding.Source: B. POAMI: Relation-Aware Multi-Entity… · p. 4
  • Person and object queries attend over the full entity set through intramodal and cross-modal streams, which are gated and fused into entity tokens.A pooled global relation token is also formed; POAMI encodes multi-entity context without explicit graph construction or identity association.Source: B. POAMI: Relation-Aware Multi-Entity… · p. 4
  • Fig. 3 depicts cross-entity attention in scenes with different numbers of people and uses heatmaps to show Stage 1 score changes after local occlusion.The displayed person-person and person-object connections illustrate how entities aggregate context from the surrounding set.Source: Figure 3 · p. 4, B. POAMI: Relation-Aware Multi-Entity… · p. 4

C. Dual-Branch Relational Interaction Prediction

The prediction stage models normal interaction regularity at both global and entity levels. It uses historical relational information to predict target states despite changing entity configurations.

C. Dual-Branch Relational Interaction Prediction

  • Separate global and entity branches model whether the target interaction configuration is predictable from relational history.The global branch processes historical global tokens with a bidirectional GRU to predict the target global token, providing a scene-level relation signal.Source: C. Dual-Branch Relational Interaction… · p. 4
  • TGRIP uses historical entity tokens as relational memory and target-frame geometry queries to predict each target entity token without explicit ID matching.The geometry query includes normalized center, box extent, and person-or-object class; query self-attention captures the target configuration, while cross-attention retrieves historical relations.Source: C. Dual-Branch Relational Interaction… · p. 4
  • An EMA teacher supplies stable target entity tokens during Stage 1 training.The teacher's target-frame encoding replaces the target token in the entity prediction loss.Source: C. Dual-Branch Relational Interaction… · p. 5
  • The Stage 1 objective combines global and entity prediction losses with variance and covariance regularizers for the global relation sequence.The regularizers prevent representation collapse.Source: C. Dual-Branch Relational Interaction… · p. 5

D. MIRA: Motion-Interaction Reconstruction and Alignment

MIRA checks predicted entity interactions against observed motion and POAMI semantics. Its motion verification reconstructs target flow under predicted entity conditions, while semantic verification compares POAMI representations.

D. MIRA: Motion-Interaction Reconstruction and Alignment

  • Motion-space verification masks target-frame flow within entity regions and reconstructs the full flow conditioned on predicted entity tokens.During training, shuffled predicted tokens provide incorrect conditions for a margin constraint; this branch does not produce an additional inference score.Source: D. MIRA: Motion-Interaction… · p. 5
  • Semantic verification compares POAMI tokens from target RGB and reconstructed-flow patches with tokens encoded from observed target RGB and flow.Semantic consistency is defined by POAMI representations rather than manual interaction labels.Source: D. MIRA: Motion-Interaction… · p. 5
  • The Stage 2 objective combines ROI and full-image flow reconstruction, condition-sensitivity margin, and semantic consistency losses.The listed weights are λroi = 1.0, λimg = 0.1, λmargin = 0.2, and λcons = 0.05; Stage 1 is frozen while the conditional autoencoder is updated.Source: D. MIRA: Motion-Interaction… · p. 5

E. Anomaly Score Fusion and Inference

Inference combines entity and global prediction errors with motion-reconstruction and entity-reconstruction signals into a frame-level anomaly score. Entity-level signals are max-aggregated, and the component weights are fixed manually.

E. Anomaly Score Fusion and Inference

  • At test time, dimension-normalized errors are computed for each valid target entity before signal aggregation.The final frame-level scores use Gaussian temporal smoothing with sigma=8.Source: E. Anomaly Score Fusion and Inference · p. 5
  • Entity-level signals are aggregated by their maximum, while the global signal uses global-token prediction error.The four signals are combined with fixed weights: we = 0.30, wg = 0.10, wr = 0.10, and wc = 0.50; overlapping clip scores are averaged by coverage count, then temporally smoothed.Source: E. Anomaly Score Fusion and Inference · p. 5
  • Four residual signals from interaction prediction and MIRA verification are combined into a frame-level score for deviations from learned interaction patterns.The signals include entity and global prediction errors, ROI motion reconstruction, and entity semantic reconstruction.Source: E. Anomaly Score Fusion and Inference · p. 5

IV. EXPERIMENTS

Experiments evaluate INTERACT on ShanghaiTech, Avenue, and NWPU Campus_sub using frame-level anomaly detection. The benchmarks cover fixed-camera and pedestrian-walkway videos, while NWPU Campus_sub is curated to emphasize scene-independent behavioral and interaction anomalies.

IV. EXPERIMENTS

  • ShanghaiTech uses 330 normal-event training videos and 107 test videos across 13 fixed camera views, with 130 annotated anomalous events in the test set.Evaluation follows the official split and reports frame-level anomaly-detection performance.Source: A. Datasets · p. 5
  • Avenue comprises 16 training and 21 test clips from a pedestrian walkway, with training on normal frames and frame-level AUC reported on the full test set.Test anomalies include throwing objects, running, and wrong-direction motion.Source: A. Datasets · p. 5
  • INTERACT achieves 84.5% frame-level ROC-AUC on ShanghaiTech, matching the best reported result, and competitive results on Avenue and NWPU Campus_sub.The NWPU Campus_sub result is not directly comparable with methods evaluated on the original NWPU Campus test set.Source: Table I · p. 6, A. Datasets · p. 6
  • NWPU Campus_sub retains 118 normal and 64 anomalous videos, selected to reduce location-specific bias and emphasize scene-independent entity behaviors or interactions.It is a manually curated subset of the NWPU Campus test set; its results are not directly comparable to results on the original test set.Source: A. Datasets · pp. 5–6

B. Evaluation Protocol

Evaluation predicts the sixteenth frame from the preceding fifteen, with different training and test strides. Implementation uses precomputed detections and optical flow, followed by separate relational-prediction and conditional-flow stages.

B. Evaluation Protocol

  • Each 16-frame clip uses the first 15 frames as historical context and frame 16 as target, with strides of 8 for training and 1 for testing.Frame-level ROC-AUC pools test-frame scores within each evaluation set after score fusion and smoothing.Source: B. Evaluation Protocol · p. 6

C. Implementation Details

  • The implementation uses precomputed YOLOv8 detections and RAFT flow, filters boxes at confidence 0.5, and labels class 0 as person and all other classes as objects.RGB and flow crops are resized to 96×96; spatial positions use a 16 × 16-patch grid.Source: C. Implementation Details · p. 6
  • Independent appearance and motion CNNs feed 512-dimensional relation features and 1024-dimensional entity tokens, followed by a three-layer relation stack and two-layer entity predictor.Stage 2 uses a conditional flow autoencoder at 256×448 resolution; Stage 1 and Stage 2 are trained separately with Adam, and Stage 1 is frozen during Stage 2.Source: C. Implementation Details · p. 6

D. Comparison with Existing Methods

INTERACT matches the best reported ShanghaiTech result and remains competitive on Avenue and NWPU Campus_sub. Ablations show the largest drop when cross-entity exchange is disabled, while verification adds runtime and parameters alongside an AUC improvement.

D. Comparison with Existing Methods

  • INTERACT achieves 84.5% AUC on ShanghaiTech, 90.6% on Avenue, and 72.4% on NWPU Campus_sub; ShanghaiTech matches the best reported result.The authors describe Avenue performance as competitive but not the highest, and the curated NWPU Campus_sub focuses on scene-independent interaction anomalies.Source: D. Comparison with Existing Methods · p. 6
  • Figure 4 shows sample frames and anomaly-score curves for Fall and Jump on Unitree Go2 first-person video, with ground-truth anomalous intervals shaded.The curves compare Prediction Only, Verification Only, and full INTERACT.Source: Figure 4 · p. 7

E. Ablation Studies

  • Disabling cross-entity exchange yields 79.18% AUC, a 5.32-point drop—the largest ablation degradation; prediction-only and verification-only variants score 80.93% and 82.93%.Removing semantic consistency yields 83.43% AUC, while the full model outperforms both single-branch configurations.Source: E. Ablation Studies · p. 7
  • Occluding predominantly blue regions of anomalous object O1 lowers the Stage 1 anomaly score in both qualitative scenes.This suggests those regions’ appearance and motion cues contribute to the original anomaly response.Source: E. Ablation Studies · p. 7

F. Computational Efficiency

  • Adding MIRA increases latency by 17.09 ms (10.9%) and adds 4.96M parameters; the corresponding ablation raises ShanghaiTech AUC from 80.93% to 84.50%.Stage 1 accounts for approximately 90% of full-model latency in the three-entity setting.Source: F. Computational Efficiency · p. 7

G. Preliminary Evaluation on a Quadruped Robot

INTERACT reaches 84.5% frame-level AUC on ShanghaiTech, and disabling cross-entity information exchange reduces AUC by 5.32 percentage points. A preliminary evaluation on two Unitree Go2 videos suggests transfer to mobile first-person views without retraining, while broader evaluation remains needed.

G. Preliminary Evaluation on a Quadruped Robot

  • 74.0% AUC on Fall and 84.9% on Jump in a preliminary evaluation using a ShanghaiTech-trained model applied without retraining or finetuning.Both videos were captured with a Unitree Go2 built-in camera; the results suggest learned representations remain informative when moving from fixed surveillance views to a mobile first-person viewpoint.Source: G. Preliminary Evaluation on a… · pp. 7–8

V. CONCLUSION

  • INTERACT assesses normality through interaction predictability, conditional motion reconstruction, and semantic consistency while jointly modeling entity-level appearance and motion with cross-entity dependencies.The work presents this framework for unsupervised video anomaly detection in surveillance and robotic monitoring.Source: §V · p. 8
  • 84.5% frame-level AUC on ShanghaiTech; disabling cross-entity information exchange reduces AUC by 5.32 percentage points.The conclusion also notes that broader evaluation is needed to establish generalization to mobile first-person views.Source: §V · p. 8
  • The conclusion identifies longer temporal context, more diverse mobile-platform scenarios, and scene-aware extensions as future work.These directions are intended for robotic patrol applications.Source: §V · p. 8
Loading 2609.31753v1…