Source-linked AI summary
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition
Jinwoo Choi, Chen Gao, Joseph C. E. Messou, Jia-Bin Huang
TL;DR
Existing video datasets encode scene bias because activities often occur in characteristic contexts, potentially weakening representation generalization. The paper adds scene-adversarial and human-mask confusion losses to action-classification pre-training, then transfers the model across three tasks and finds consistent improvement over a non-debiased baseline.
Problem
Video models may exploit scene context instead of human-action evidence, and scene bias can limit generalization to unseen action classes and novel tasks.
Method
The method augments action-classification cross-entropy with a scene-adversarial loss and a human-mask confusion entropy loss during video-representation pre-training.
Results
Transfer learning shows consistent improvement over a baseline without debiasing across action classification, temporal action localization, and spatio-temporal action detection.
Takeaways & Limitations
The two losses encourage representations that are invariant to scene types and do not predict actions when humans are masked out, supporting transfer across action-understanding tasks.
Takeaways & Limitations
The work addresses only scene bias, and its final performance remains behind other state-of-the-art models because it builds on relatively weak baselines.
Abstract
from arXiv · showhide
Human activities often occur in specific scene contexts, e.g., playing basketball on a basketball court. Training a model using existing video datasets thus inevitably captures and leverages such bias (instead of using the actual discriminative cues). The learned representation may not generalize well to new action classes or different tasks. In this paper, we propose to mitigate scene bias for video representation learning. Specifically, we augment the standard cross-entropy loss for action classification with 1) an adversarial loss for scene types and 2) a human mask confusion loss for videos where the human actors are masked out. These two losses encourage learning representations that are unable to predict the scene types and the correct actions when there is no evidence. We validate the effectiveness of our method by transferring our pre-trained model to three different tasks, including action classification, temporal localization, and spatio-temporal action detection. Our results show consistent improvement over the baseline model without debiasing.
1 Introduction
Scene context can bias action-recognition models toward shortcuts rather than human-action evidence, limiting generalization. The paper proposes debiasing losses and evaluates transfer across action-understanding tasks.
- CNNs may recognize scene context or objects instead of the actual human activity, producing correct predictions for wrong reasons.
- UCF-101 has scene representation bias of log(59.7/1.0) = 4.09, compared with 0 for a completely unbiased dataset.The measure compares action accuracy from Places365-pre-trained features with random-chance accuracy.
- Scene bias arises because human activities often occur in specific contexts, allowing masked actors’ actions to be inferred from the scene.
- The proposed method pre-trains CNNs with standard action cross-entropy plus scene-adversarial and human-mask confusion losses.
- Scene-adversarial learning targets scene-invariant representations, while human-mask confusion discourages correct action predictions without visible human evidence.
- Transfer experiments cover action classification, temporal action localization, and spatio-temporal action detection, with consistent improvement over a baseline without debiasing.
2 Related Work
Related work uses context for recognition and occlusion for robustness, while this paper learns scene-invariant representations from biased datasets. The approach connects to adversarial training and broad video-representation transfer.
- Prior bias-mitigation work resamples datasets to generate less-biased action-recognition data, whereas this method learns scene-invariant representations without resampling.
- Scene context supports several vision tasks, but this paper instead targets scene-bias invariance and reports better generalization to new datasets and tasks.
- Unlike architectures focused on two-stream inputs, 3D CNNs, or longer-range temporal dependencies, the proposed debiasing approach is model-agnostic.
- Video models commonly pre-train on large-scale datasets before fine-tuning for action classification, temporal localization, or spatio-temporal detection.
- Adversarial training has been applied to domain adaptation and bias mitigation by predicting protected variables or measuring statistical dependency.
- This work adapts artificial occlusion by masking detected humans, but focuses on debiasing action representations rather than the prior applications listed.
3 Method
The method pre-trains a CNN with action classification and two debiasing losses, then transfers its feature extractor to downstream action-understanding tasks.
- Pre-training: The feature extractor is pre-trained on video-action pairs with a standard cross-entropy action-classification loss and can use either 3D or 2D CNNs.The approach is model-agnostic at the feature-extraction stage.
- Transfer learning: After debiased pre-training, the feature-extractor weights initialize models for action classification, temporal localization, and spatio-temporal action detection, followed by task-specific fine-tuning.The scene-prediction and human-masked action-prediction heads are removed before downstream training.
- Scene adversarial loss: The scene adversarial loss encourages action-useful representations that are invariant to scene types by adversarially training a scene classifier against the feature extractor.The feature extractor maximizes the scene-classification loss while the scene classifier minimizes it.
- Scene adversarial loss: Pseudo scene labels are obtained by applying a Places365-pre-trained ResNet-50 to the Kinetics dataset because common action datasets lack scene annotations.These labels supply scene categories for the adversarial objective.
- Human mask confusion loss: The human mask confusion loss applies entropy to action predictions on videos with detected humans masked out, discouraging correct action inference without visible human evidence.Both the feature extractor and action classifier maximize this loss on masked videos.
- Human mask confusion loss: Human regions are masked offline using cached detector outputs, replacing each detected bounding-box region with the frame’s average pixel value.The detector is run on the Kinetics dataset before training.
4 Experimental Results
The experiments test whether debiasing reduces scene-dependent representations and improves transfer across action classification, temporal localization, and spatio-temporal detection. Results show reduced scene predictability, consistent transfer gains, and complementary benefits from the two debiasing losses.
- Experimental design: The experiments evaluate scene-bias mitigation, cross-dataset action classification, temporal localization, spatio-temporal detection, and ablations of pseudo-label and loss choices.The study pre-trains on Mini-Kinetics-200 and transfers to UCF-101, HMDB-51, Diving48, THUMOS-14, and JHMDB.
- Transfer learning for action classification: Debiasing consistently outperforms the non-debiased baseline on UCF-101, HMDB-51, and Diving48 action classification.The reported relative improvement on Diving48 is 14.1%; Diving48 has a scene representation bias of 1.26.
- Transfer learning for action classification: ρ = −0.896 with p-value 0.006 indicates a strong negative correlation between relative improvement and scene representation bias across HMDB-51, UCF-101, and Diving48 splits.The reported pattern associates larger gains with datasets having less scene bias.
- Transfer learning for other activity understanding tasks: The debiased representation consistently outperforms the baseline at all reported IoU thresholds for THUMOS-14 temporal action localization.The results are reported using video mean average precision at various IoU threshold values.
- Transfer learning for other activity understanding tasks: Debiasing outperforms the non-debiased baseline on JHMDB spatio-temporal action detection, while both proposed losses improve performance individually and jointly perform best.The ablation attributes the combined gain to complementary regularization by scene adversarial and human mask confusion losses.
5 Conclusions
The paper mitigates scene bias in video representation learning with two auxiliary losses and transfers the resulting model to three target tasks. Its performance remains behind other state-of-the-art models, and the method addresses only scene bias.
- The method augments action-classification cross-entropy with an adversarial scene-class loss and an entropy loss for videos with masked humans.These losses encourage the network to focus on the actual action.
- The pre-trained model is transferred to three target action-understanding tasks to evaluate the debiasing method.
- The model’s final performance still falls behind other state-of-the-art models because it is built upon relatively weak baseline models.
- The proposed debiasing method addresses scene bias but does not address other biases such as object and person bias.Extending the method to those biases is identified as future work.