PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Temporal Regularity in Video Sequences

Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K. Roy-Chowdhury, Larry S. Davis

arXiv:1604.04574v1cs.CV

TL;DR

The paper studies how to identify meaningful or anomalous moments in long videos when meaningfulness is ambiguous and supervision is limited. It learns regular temporal motion patterns with fully connected and fully convolutional autoencoders, including models that learn features end to end. Across multiple datasets, the models capture temporal regularities, support several video-analysis applications, and perform comparably to state-of-the-art anomaly-detection methods.

  • Problem

    The paper addresses learning meaningful temporal characteristics in long, cluttered videos despite ambiguous definitions and very limited supervision.

  • Method

    The paper trains fully connected and fully convolutional autoencoders to learn regular motion patterns, with the convolutional model jointly learning local features and regular patterns.

  • Results

    Across CUHK Avenue, Subway, and UCSD Pedestrian datasets, the models capture temporal regularities and perform comparably to state-of-the-art anomaly-detection methods.

  • Takeaways & Limitations

    The learned regularity supports regular-frame synthesis, irregular-object delineation, past and future motion prediction, and abnormal-event detection.

  • Takeaways & Limitations

    Training assumes that all events in the training videos are regular, and evaluation can label a model-flagged usual motion as a false alarm when ground truth calls it normal.

Abstract

from arXiv · show

Perceiving meaningful activities in a long video sequence is a challenging problem due to ambiguous definition of 'meaningfulness' as well as clutters in the scene. We approach this problem by learning a generative model for regular motion patterns, termed as regularity, using multiple sources with very limited supervision. Specifically, we propose two methods that are built upon the autoencoders for their ability to work with little to no supervision. We first leverage the conventional handcrafted spatio-temporal local features and learn a fully connected autoencoder on them. Second, we build a fully convolutional feed-forward autoencoder to learn both the local features and the classifiers as an end-to-end learning framework. Our model can capture the regularities from multiple datasets. We evaluate our methods in both qualitative and quantitative ways - showing the learned regularity of videos in various aspects and demonstrating competitive performance on anomaly detection datasets as an application.

1. Introduction

The paper addresses unsupervised temporal regularity learning for identifying irregular moments in long, cluttered videos, where meaningful activity is difficult to define. It uses autoencoders to learn regular motion patterns from multiple datasets with limited supervision and applies them to anomaly detection and related video-analysis tasks.

  • Problem: Ambiguous definitions of meaningful moments make unsupervised or minimally supervised temporal segmentation of long videos a fundamental computer-vision problem.The paper motivates applications including annotation, summarization, indexing, temporal segmentation, anomaly detection, and activity recognition.
  • Problem: The approach models dense regular temporal dynamics instead of sparsely supervised irregular or meaningful moments.The training assumption is that all events in the training videos belong to regular patterns.
  • Method: Autoencoders provide a computationally efficient alternative to sparse coding while preserving spatio-temporal information during dynamic encoding.The learned model reconstructs regular motion with low error and irregular motion with higher error; regularity is computed from reconstruction error.
  • Method: The paper develops two autoencoder variants: a fully connected model using handcrafted motion features and a fully convolutional model that learns features and regular patterns jointly.The convolutional variant is motivated by the possibility that handcrafted motion features are suboptimal for temporal regularity.
  • Results: Models trained across CUHK Avenue, Subway, and UCSD Pedestrian datasets learn regular patterns across datasets without compensating for dataset bias.The authors report applications including regular-frame synthesis, irregular-object delineation, past and future motion prediction, and abnormal-event detection.
  • Results: The method performs comparably to state-of-the-art methods on anomaly detection across multiple datasets.The paper presents anomaly detection as a quantitative application of the learned temporal regularity model.

3. Approach

The approach learns video regularity with autoencoders that reconstruct regular motion accurately and assign higher reconstruction error to irregular motion. It combines handcrafted motion features with a fully connected autoencoder, and learns features end-to-end with a fully convolutional autoencoder using temporal video cuboids.

  • Autoencoders model regular video dynamics by reconstructing regular motion with low error and irregular motion with higher error.The reconstruction error serves as the basis for measuring regularity.
  • The first method feeds handcrafted HOG, HOF, and improved trajectory features into a fully connected autoencoder.The concatenated motion descriptor has 204 dimensions.
  • The second method learns local motion features and regular patterns jointly with a fully convolutional autoencoder trained end-to-end from video.The design preserves spatial information needed to reconstruct input frames.
  • The handcrafted-feature autoencoder encodes 204-dimensional inputs through four encoder layers and three decoder layers, with a 30-neuron bottleneck.The small middle layer captures compact semantics and reduces noisy information.
  • The fully convolutional model uses convolutional and pooling layers in its encoder, followed by reverse-order deconvolutional and unpooling layers in its decoder.The final deconvolutional layer reconstructs the input cuboid.
  • The convolutional model stacks T consecutive frames into a temporal cuboid, and larger T produces more discriminative regularity scores after training.Increasing T also makes training loss take more iterations to converge.

4. Experiments

The experiments evaluate qualitative regularity visualization, temporal prediction, cross-dataset generalization, and abnormal-event detection. The models capture regular patterns across datasets while identifying irregular motion, though deviations from regularity can produce extra false alarms.

  • Experimental Setup: The models are evaluated qualitatively and quantitatively on multiple video datasets, including Avenue, UCSD Pedestrian, and Subway.The experiments use models trained across datasets and compare abnormal-event detection performance with state-of-the-art methods.
  • Generalization: Models trained on all datasets are not degraded by additional datasets and transfer effectively to unseen datasets.The comparisons indicate limited overfitting to any one dataset and a balance between overfitting and underfitting.
  • Visualizing Temporal Regularity: The convolutional and improved-trajectory autoencoders synthesize regular frames and visualize irregularity at pixel or patch precision.The convolutional autoencoder selects minimum-reconstruction-cost pixels over time, while the improved-trajectory model localizes irregularity less precisely.
  • Temporal Prediction: The convolutional autoencoder predicts a few past and future regular frames from a single center frame by processing a temporal clip.A temporal cube padded with zero-valued frames allows the model to extrapolate regular motion around the center frame.
  • Abnormal Event Detection: Regularity scores support weakly supervised abnormal-event detection by locating meaningful local minima in their framewise time series.Persistence-based processing expands and groups minima into final abnormal temporal regions.
  • Abnormal Event Detection: The model performs comparably to or better than state-of-the-art methods but produces more false alarms when unusual motion is not labeled abnormal.Running is detected as irregular in one example even though the ground truth treats it as normal.
  • Filter Responses: Learned convolutional filters can respond oppositely to the same irregular object, representing regularity and irregularity responses.The filter-response visualization uses an irregular bag as an example of contrasting learned responses.

5. Conclusion

The paper learns temporal regularity with autoencoders under limited supervision, using both handcrafted features and a fully convolutional model. It visualizes regularity and reports competitive anomaly-detection performance.

  • The method learns regular patterns with limited supervision using a fully connected autoencoder and a fully convolutional autoencoder.The latter learns local features and classifiers in a single framework.
  • The convolutional autoencoder analyzes learned regularity by visualizing regular frames, frame and pixel regularity, and predicting regular video from a single image.
  • The method performs competitively with state-of-the-art anomaly-detection methods in quantitative evaluation.

6. Dataset Details

The evaluation uses Avenue, UCSD Pedestrian, and Subway surveillance datasets curated for anomaly or abnormal-event detection. They vary in scene, duration, resolution, and event annotations.

  • The study evaluates three datasets: Avenue, UCSD Pedestrian, and Subway.UCSD Pedestrian contains Ped1 and Ped2 scenes, while Subway contains entrance and exit videos.
  • Avenue: Avenue contains 16 training and 21 testing sequences, with approximately 1–2 minutes per sequence and 640 × 360 resolution.The training and testing sets contain 15,328 and 15,324 frames, respectively.
  • UCSD Pedestrian: UCSD-Ped1 has 34 training and 36 testing clips, while UCSD-Ped2 has 16 training and 12 testing clips.Ped1 clips contain 200 frames at 238 × 158 resolution; Ped2 clips contain 150–200 frames at 360 × 240 resolution.
  • Subway: Subway videos come from entrance and exit surveillance cameras at 512 × 384 resolution, with roughly 10 people per frame.
  • Subway: Subway-Entrance contains 66 unusual events across five types, whereas Subway-Exit contains 19 events across three types.The listed categories include wrong-direction walking and loitering, with additional miscellaneous events.

7. Learned Temporal Regularity

Temporal regularity is visualized by synthesizing regular frames and accumulating regularity scores as heatmaps. The convolutional autoencoder provides finer-grained regularity scores than the trajectory-based autoencoder in the presented examples.

  • The convolutional autoencoder synthesizes a regular frame and produces an accumulated regularity-score heatmap for each video.The synthesized frame uses the pixel value with the lowest reconstruction score across all video frames.
  • The improved trajectory-based autoencoder produces patch-level scores and cannot capture regularity as well as the convolutional autoencoder.
  • The heatmaps reveal trajectories of moving objects, including an SUV and two people, in example videos.
  • Long videos can yield relatively high regularity scores because irregular motion is averaged over long durations.The passage notes that clock ticking is not part of the regular motions in the example.
  • The IT-autoencoder shows a very high accumulated irregularity score in stair regions for one Subway example.

8. Object Detection in Irregular Motion

Regularity scores localize objects involved in irregular motion across the evaluated videos. The examples identify moving cars, fast-moving people, prominent leg motion, and flying papers.

  • Regularity scores locate objects involved in irregular motion, which are usually the objects of interest.Responses are high at object edges where motion changes most significantly.
  • In one example, the bottoms of legs receive high scores, while flying papers are captured in two other videos.
  • Moving cars are localized in several videos, and fast-moving people are localized in the corresponding examples.
  • Moving people are easily localized in the additional example videos, including Subway-like footage.

9. Predicting Past and Future Regular Frames

The model predicts temporally regular frames before and after a given image, producing plausible regular-motion sequences across multiple videos.

  • Prediction setup: The visualization uses a single image to predict regular frames from 0.1 seconds before and after the input moment.The sequence also includes a reconstructed regular frame at the input moment.
  • Qualitative predictions: In video 1, the crowd and foreground person gradually appear and disappear in the predicted regular sequence.
  • Qualitative predictions: In video 13, the predicted future frame places the paper closer to the ground than the past frame.
  • Qualitative predictions: In video 20, the car moves slightly upward in the predicted future frame.
  • Qualitative predictions: In video 4, the predicted car appears clearer and moves slightly farther south than in the past-frame predictions.

10. Anomalous Event Detection and Generalization Analysis on Multiple Datasets

The regularity score identifies anomalous temporal regions and supports evaluation of models trained on target, combined, or held-out datasets. Generalized and transfer models remain effective across datasets, while score interpretation depends on how anomaly definitions relate to temporal regularity.

  • Detection and Generalization: Low regularity scores identify video segments containing anomalous events within local temporal windows.The visualization compares conventional, generalized, and transfer models across training-set configurations.
  • Detection and Generalization: The generalized model performs very well, matching the conventional target-trained model, while the transfer model also performs decently on CUHK Avenue.Here, conventional uses Avenue only, generalized uses all datasets, and transfer excludes Avenue during training.
  • Detection and Generalization: Comparisons indicate that training on additional datasets does not harm performance and that the model generalizes to unseen videos without excessive dataset overfitting.The authors characterize the network as balanced between overfitting and underfitting.
  • Detection and Generalization: Similar performance patterns are reported on another dataset, with the generalized model matching the conventional model and the transfer model performing decently.
  • Detection and Generalization: Regular motions may receive high regularity scores inside annotated anomalous events when the dataset’s anomaly definition differs from temporal ordinariness.This distinction explains high scores in some anomalous regions and frames.
  • Detection and Generalization: Anomalous events are well captured when the dataset defines anomalies similarly to the model’s regularity concept of avoiding abrupt motions.Some anomalous regions can retain higher regularity scores when the anomaly is spatially small or visually blended with regular activity.

11. Filter Response Visualization

The convolutional filter responses visualize how the learned network represents regularity across several video datasets. The responses are examined in CHUK Avenue, UCSD-Ped1, UCSD-Ped2, Subway Enter, and Subway Exit videos.

  • Layerwise Responses: Early convolutional layers capture low-level structural patches, while deeper layers represent higher-level structure at larger scales.Different filters respond differently to the same patch, providing complementary information; deconvolutional layers hierarchically unpack encoded information.
  • Dataset Visualizations: Learned convolutional filter responses are visualized on videos from CHUK Avenue, UCSD-Ped1, UCSD-Ped2, Subway Enter, and Subway Exit.
  • Dataset Visualizations: On UCSD-Ped1 and UCSD-Ped2, different filters produce varied colored responses on the same region, indicating multiple captured aspects of regularity.For UCSD-Ped2, first-layer background outputs also appear in various colors.

12. Filter Weights Visualization

The learned temporal-regularity filters occupy small spatial regions while spanning up to ten frames. Their visualized patterns differ from object-recognition filters because they capture spatial and temporal structure together.

  • Filter Structure: The learned filters use small spatial regions and temporal dimensions spanning up to 10 frames.Because ten-frame cubes are difficult to visualize, the visualization shows only the first three frames.
  • Filter Structure: Temporal-regularity filters lack obvious spatial structure compared with object-recognition filters because they jointly capture spatial and temporal patterns.
Loading 1604.04574v1…