Source-linked AI summary
Privacy-Preserving Human Activity Recognition from Extreme Low Resolution
Michael S. Ryoo, Brandon Rothrock, Charles Fleming, Hyun Jong Yang
TL;DR
The paper addresses how to recognize human activities while preventing cameras from obtaining privacy-sensitive high-resolution video. It introduces inverse super resolution, which learns transformations to generate multiple low-resolution training videos from high-resolution data. Experiments on three public datasets confirm benefits for extreme low-resolution activity recognition, especially in first-person video.
Problem
Privacy-preserving activity recognition requires useful understanding from anonymized video without acquiring detailed high-resolution visual data, while conventional low-resolution training can produce unstable decision boundaries.
Method
Inverse super resolution learns recognition-optimized transformations that generate multiple low-resolution labeled training videos from publicly available high-resolution videos.
Results
Experiments on three public datasets confirm ISR effectiveness, with particularly successful recognition on first-person video datasets where ego-motion is important.
Takeaways & Limitations
ISR enables activity recognition from extreme low-resolution videos while using high-resolution videos only during training.
Takeaways & Limitations
Transformation-set learning can require many MCMC iterations and a separate validation set, reducing the training data available for learning transformations.
Abstract
from arXiv · showhide
Privacy protection from surreptitious video recordings is an important societal challenge. We desire a computer vision system (e.g., a robot) that can recognize human activities and assist our daily life, yet ensure that it is not recording video that may invade our privacy. This paper presents a fundamental approach to address such contradicting objectives: human activity recognition while only using extreme low-resolution (e.g., 16x12) anonymized videos. We introduce the paradigm of inverse super resolution (ISR), the concept of learning the optimal set of image transformations to generate multiple low-resolution (LR) training videos from a single video. Our ISR learns different types of sub-pixel transformations optimized for the activity classification, allowing the classifier to best take advantage of existing high-resolution videos (e.g., YouTube videos) by creating multiple LR training videos tailored for the problem. We experimentally confirm that the paradigm of inverse super resolution is able to benefit activity recognition from extreme low-resolution videos.
Introduction
The paper addresses the tension between privacy-preserving video capture and useful human activity recognition. It proposes anonymized extreme low-resolution videos and inverse super resolution to learn from high-resolution training videos while avoiding detailed visual data.
- Motivation: Ubiquitous surveillance, wearable, and robotic cameras create a societal need to recognize activities without recording privacy-sensitive visual details.The desired system must prevent access to detailed data such as faces while still capturing information about objects and events.
- Motivation: Prior software-level approaches may expose original high-resolution videos, and feature visualizations can recover substantial visual information.These limitations motivate privacy protection at acquisition rather than after high-resolution recording.
- Anonymized video: Anonymized videos use extreme low resolution, such as 16x12, or operations including Gaussian blurring and superpixel clustering.The approach limits the system to obtaining anonymized videos instead of first acquiring high-resolution footage.
- Limitation: Conventional resizing of training videos is unstable because sub-pixel viewpoint changes produce very different low-resolution features for the same object or person.Figure 1 illustrates that slight projection changes can substantially alter pixel values and visual structures.
- Proposed direction: Inverse super resolution learns optimal image transformations that generate multiple informative low-resolution training videos from each high-resolution video.The generated videos are used to learn low-resolution decision boundaries while testing remains based on anonymized video.
Related works
Prior work addressed privacy or extreme-low-resolution recognition separately, but the paper situates inverse super resolution as a framework connecting privacy-preserving video capture with activity recognition.
- Activity recognition: Human activity recognition has been studied in YouTube, surveillance, wearable-camera, and robotic videos, generally without privacy considerations.
- Privacy protection: Privacy-focused methods include detecting sensitive locations and turning cameras off, but they do not eliminate the risk associated with acquiring original high-resolution video.
- Extreme low-resolution recognition: Dai et al. demonstrated activity recognition from extreme low-resolution third-person videos captured by static cameras.Their results showed that computer-vision features and classifiers can operate on very low-resolution videos.
- Conventional framework: The conventional framework resizes original training videos to the target resolution, a strategy associated with unstable decision-boundary learning under sub-pixel viewpoint changes.
Inverse super resolution
Inverse super resolution generates multiple low-resolution training samples from each high-resolution image or video by applying learned transformations before downsampling. The transformations are optimized for recognition, helping classifiers learn in the low-resolution feature space.
- Concept: Inverse super resolution learns image transformations that generate a set of low-resolution training images from one high-resolution image.Candidate transformations include sub-pixel translation, scaling, rotation, and other affine motions.
- Privacy-preserving setting: ISR targets settings where high-resolution videos are available for training but prohibited during testing for privacy protection.It does not enhance the resolution of the testing video; instead, it enriches training through multiple sub-pixel transformations.
- Formulation: The ISR formulation reverses super resolution: instead of reconstructing one high-resolution image from multiple low-resolution images, it generates multiple low-resolution images from one high-resolution image.
- Formulation: The forward model represents each low-resolution image through motion transformation, blurring, downsampling, and noise applied to a high-resolution image.In ISR, noise and lens blur can be omitted to avoid contaminating generated samples and confusing classifiers.
- Transformation learning: The transformation set is learned in feature space because it must maximize recognition performance for the selected features and classifier.The paper describes MCMC-based decision-boundary matching and notes that the approach can work with different classifiers, including nonlinear-kernel SVMs.
- Training-data generation: The method applies learned motion transforms and downsampling to each high-resolution training image, producing multiple labeled low-resolution samples for classifier training.For videos, the same operators are applied frame by frame and the outputs are concatenated.
Transformation learning
The paper learns a compact set of image transforms for generating informative low-resolution training samples, using decision-boundary matching or maximum-entropy selection. These transforms are selected from high-resolution training videos to improve the activity classifier’s low-resolution training data.
- Method 2 - maximum entropy: Inverse super resolution transform learning searches for an optimal set S of filters that generate low-resolution training videos from high-resolution data.The target set contains a specified number of filters or samples.
- Method 1 - decision boundary matching: Decision-boundary matching uses MCMC to add or remove motion transforms while seeking transforms whose classifier boundary matches an ideal boundary.The ideal boundary is approximated using a large transform pool, and candidate sets are evaluated through classification-function similarity.
- Method 1 - decision boundary matching: The MCMC procedure samples transform sets from S0 = {} through Sm and selects the sampled set with maximum π(S), subject to |S| ≤ n.The proposal is accepted according to a Metropolis-Hastings transition probability.
- Method 1 - decision boundary matching: The decision-boundary approach requires many MCMC iterations and a separate validation set, reducing the training data available for transformation learning.This motivates the alternative entropy-based methodology.
- Method 2 - maximum entropy: Maximum-entropy selection iteratively adds the transform whose generated low-resolution samples have the greatest uncertainty under the current classifier.The process continues until the nth round and can measure entropy with or without ground-truth labels.
- Method 2 - maximum entropy: The entropy-based method supports unsupervised transform learning because its entropy criterion does not require ground-truth labels.It greedily updates the transform set by adding one transform at each iteration.
Experiments
Experiments evaluate inverse super resolution on three datasets resized to extreme low resolutions, using multiple features, kernels, and training-sample generation settings. ISR consistently improves low-resolution activity recognition, including comparisons with prior methods.
- Experimental setup: The primary setting is 16x12 video created by averaging source pixels within low-resolution boundaries, with cropping for non-4:3 videos.The experiments also test 32x24 videos.
- Experimental setup: Features include HOG, HOF, CNN, and their combinations; the CNN uses a three-layer architecture designed for low-resolution frames.The CNN extracts 256-D features from very low resolution images.
- DogCentric: DogCentric experiments compare data augmentation, rotation augmentation, uniform sampling, and two ISR methods across four feature types, three kernels, and six values of n.The evaluation uses multiple random half-half training/test splits and reports highly consistent behavior in Table 1.
- DogCentric: ISR consistently outperforms conventional approaches, including data augmentation and uniform transform selection, across features and kernels.Method2 reaches 62.0 versus 60.9 for data augmentation with a linear kernel, while method1 is more reliable overall.
- DogCentric: 67.4% accuracy on 16x12 DogCentric videos improves the prior PoT result of 64.6% using the same features, representation, and classifier.The prior 73% result used 320x240 videos.
- HMDB: On HMDB, ISR improves low-resolution recognition in all cases, with gains appearing even when only n = 2 additional ISR samples are generated.Classification accuracies at both 16x12 and 32x24 are reported; 32x24 generally performs better.
- Cross-dataset results: At 16x12, the proposed methods outperform HOF/HOG at 20.0% and ActionBank at 26.9%, despite severe visual-information loss.Trajectory-based features are unavailable in this setting.
Conclusion
The paper presents inverse super resolution for improving classification on extreme low-resolution video and evaluates it across three public datasets. Recognition is particularly successful for first-person videos, while the method remains computationally efficient compared with traditional data augmentation.
- The method improves classification performance on extreme low-resolution video across three different public datasets.
- Recognition is particularly successful on first-person video datasets, where capturing ego-motion is most important.
- n=16 versus n=175 examples illustrates that the method can achieve similar performance with an order of magnitude fewer examples than traditional data augmentation.
Discussions
The discussion examines whether extreme low-resolution videos sufficiently hinder face recognition and whether resolution enhancement can recover facial details. At 16x12 resolution, faces are often only a few pixels, and enhancement sharpens edges without recovering actual details.
- Reliable face recognition is difficult because faces in 16x12 videos are at most 5x7 pixels and are often only 2x2 or 1x1 pixels.
- 50∼60% accuracy was reported for low-resolution face recognition using 16x16 face images, with performance expected to decrease in real-world settings containing more subjects.
- Higher-scale enhancement performed worse, while x4 deep learning-based enhancement sharpened edges but failed to recover actual facial details in 16x12 videos.