PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidential Deep Learning for Open Set Action Recognition

Wentao Bao, Qi Yu, Yu Kong

arXiv:2107.10161v2cs.CV

TL;DR

Real-world action recognition requires recognizing known actions while rejecting unknown ones, with video-specific temporal dynamics and static bias creating additional open-set challenges. DEAR uses evidential deep learning with uncertainty calibration and contrastive debiasing, and consistently improves multiple action recognition models and benchmarks. The approach provides a principled, flexible way to quantify uncertainty for unknown-action identification while retaining high closed-set performance.

  • Problem

    Open set action recognition must recognize known actions and reject unknown ones despite uncertain temporal dynamics and static bias in video actions.

  • Method

    DEAR formulates action recognition through evidential deep learning, calibrates evidential training, and debiases representations with a contrastive learning module.

  • Results

    DEAR consistently improves existing action recognition models on challenging benchmarks with small- and large-scale unknown videos while maintaining high closed-set performance.

  • Takeaways & Limitations

    DEAR offers a principled and efficient uncertainty evaluation approach for identifying unknown actions and can flexibly extend existing action recognition models.

Abstract

from arXiv · show

In a real-world scenario, human actions are typically out of the distribution from training data, which requires a model to both recognize the known actions and reject the unknown. Different from image data, video actions are more challenging to be recognized in an open-set setting due to the uncertain temporal dynamics and static bias of human actions. In this paper, we propose a Deep Evidential Action Recognition (DEAR) method to recognize actions in an open testing set. Specifically, we formulate the action recognition problem from the evidential deep learning (EDL) perspective and propose a novel model calibration method to regularize the EDL training. Besides, to mitigate the static bias of video representation, we propose a plug-and-play module to debias the learned representation through contrastive learning. Experimental results show that our DEAR method achieves consistent performance gain on multiple mainstream action recognition models and benchmarks. Code and pre-trained models are available at {\small{\url{https://www.rit.edu/actionlab/dear}}}.

1. Introduction

Open set action recognition must recognize known actions while rejecting unknown ones, but video-specific temporal diversity and static bias make this harder than closed-set recognition. DEAR addresses these challenges with evidential uncertainty, calibration, and contrastive debiasing, and improves existing action recognition models.

  • Motivation: Open set action recognition requires simultaneous recognition of known actions and identification of unknown actions.This setting is important for real-world applications but is more challenging than closed-set recognition.
  • Challenges: Video OSAR must model diverse temporal action patterns while remaining aware of uncertainty on unknown actions.The temporal nature of videos creates action-pattern diversity that is not central to image-based open-set recognition.
  • Challenges: Static scene, object, and human cues can cause models to falsely recognize out-of-context videos in open-set settings.For example, background cues associated with an action may fail when the same action appears in a different scene.
  • Method: DEAR formulates OSAR as evidential uncertainty estimation, regularizes EDL with model calibration, and debiases representations through contrastive learning.Its contributions include the DEAR framework, Evidential Uncertainty Calibration, and Contrastive Evidential Debiasing.
  • Results: DEAR consistently boosts state-of-the-art action recognition models on challenging benchmarks and maintains high closed-set recognition performance.The reported open-set evaluations include both small- and large-scale unknown videos.

2. Related Work

Related work spans classical and deep open-set recognition, uncertainty-based methods, and closed-set video action recognition. DEAR introduces evidential deep learning to video OSR and is designed to extend existing action recognition models to open-set testing.

  • Open Set Recognition: Classical open-set recognition methods use SVM-based rejection and confidence calibration, while deep methods include OpenMax, generative unknown synthesis, and reconstruction errors.These approaches were developed primarily for image datasets and general OSR settings.
  • Open Set Action Recognition: Open set action recognition has received limited attention despite greater difficulty from temporal dynamics and video-specific uncertainty.Existing OSAR work includes incremental class learning, Bayesian uncertainty, and open-set domain adaptation.
  • Deep Learning Uncertainty: Predictive uncertainty is used as an out-of-distribution scoring function, but Bayesian neural networks face posterior-inference, prior-selection, and sampling challenges.EDL is presented as an alternative uncertainty-learning framework for classification and regression.
  • Video Action Recognition: DEAR can be applied to representative 3D convolution, 2D convolution, two-stream, and neck-structure action recognition models.The selected backbones include I3D, TSM, SlowFast, and TPN.

3. Approach

DEAR formulates open-set action recognition as uncertainty-aware evidential learning, then calibrates uncertainty and debiases video representations to address over-confidence and static cues. Its design combines an EDL-based recognition head with EUC calibration and CED representation debiasing for flexible integration into action-recognition models.

  • 3.1. Deep Evidential Action Recognition: DEAR uses an Evidential Neural Network head to predict class-wise evidence, form a Dirichlet distribution, and classify low-uncertainty videos while rejecting high-uncertainty videos as unknown.The head sits on an action-recognition backbone and jointly determines categorical probabilities and predictive uncertainty.
  • 3.1. Deep Evidential Action Recognition: EDL replaces softmax point estimates with evidence-based uncertainty modeling, linking class evidence to Dirichlet parameters and deterministically computing predictive uncertainty.For class k, α_k = e_k + 1, predicted probability is α_k/S, and uncertainty is u = K/S.
  • 3.2. Evidential Uncertainty Calibration: Evidential Uncertainty Calibration (EUC) regularizes training so accurate predictions become certain and inaccurate predictions become uncertain, improving the Accuracy versus Uncertainty utility.The loss uses confidence and uncertainty terms with an annealing weight that increases from λ0 to 1 across training.
  • 3.2. Evidential Uncertainty Calibration: The annealing factor balances calibration losses because inaccurate predictions dominate early training, whereas accurate predictions dominate later training.This weighting emphasizes the inaccurate-and-certain loss early and the accurate-and-uncertain loss later.
  • 3. Approach: DEAR’s EUC and CED components are designed for flexible open-set deployment: EUC avoids distribution-shifted validation data, while CED is plug-and-play across existing action-recognition models.The authors report stronger calibration effects for open-set than closed-set recognition and compatibility with multiple state-of-the-art backbones.
  • 3.3. Contrastive Evidence Debiasing: Contrastive Evidential Debiasing (CED) learns unbiased features with a Conv3D branch and contrasts them against biased evidence from temporally shuffled inputs and Conv2D features.HSIC-based minmax optimization encourages the unbiased feature to be independent of both biased feature sources.

4. Experiments

Experiments evaluate DEAR on open-set action recognition using UCF-101 as known data and HMDB-51 or MiT-v2 as unknown data. DEAR consistently improves open-set metrics across recognition models while largely preserving closed-set accuracy.

  • Experimental Setup: Models are trained on UCF-101 and tested with HMDB-51 or MiT-v2 as sources of unknown actions.The evaluation includes closed-set accuracy, open-set AUC, and Open maF1 across randomized unknown-class selections.
  • Evaluation Protocol: Open maF1 summarizes (K+1)-class performance by weighting macro-F1 across increasing openness.Openness increases as randomly selected HMDB-51 or MiT-v2 classes are introduced as unknown.
  • Comparison with State-of-the-art: DEAR consistently outperforms baselines on Open maF1 and Open Set AUC while reducing Closed Set Accuracy by less than 1%.With SlowFast, DEAR improves over MC Dropout by almost 8% in open-set AUC and 15% in Open maF1.
  • Comparison with State-of-the-art: DEAR achieves the best Open maF1 across varying openness, although methods converge as MiT-v2 unknown classes increasingly dominate macro-F1.Macro-F1 becomes sensitive to class imbalance when many of MiT-v2’s 305 categories are treated as unknown, but DEAR remains better than the baselines.
  • Out-of-distribution Detection: DEAR’s EDL uncertainty better detects out-of-distribution samples than the vanilla DEAR variant using only LEDL training.The comparison uses MiT-v2 as unknown and I3D as the recognition model.
  • Ablation Study: All proposed components contribute to OSAR gains, with the CED module’s h2D(x) contributing the most.Joint CED training performs slightly better than alternative training in the TPN and HMDB-51 ablation setting.
  • Ablation Study: LEUC reduces Expected Calibration Error in both open- and closed-set settings, with a larger calibration effect in the open-set setting.The result supports the role of the EUC module in calibrating OSAR models.
  • Representation Debiasing: CED significantly improves accuracy on the unbiased Mimetics dataset while causing only minor changes on the biased Kinetics dataset.The experiment evaluates TSM models trained and tested across biased and unbiased action data.

5. Conclusion

The conclusion presents DEAR as an evidential approach to open-set action recognition. It combines uncertainty estimation, calibration, and representation debiasing, with experiments showing effectiveness across existing action recognition models.

  • Conclusion: DEAR addresses OSAR by using EDL to quantify predictive uncertainty for distinguishing known and unknown samples.The conclusion identifies uncertain temporal dynamics and static background bias as the central OSAR challenges.
  • Conclusion: EUC and CED extend EDL to address uncertainty calibration and static bias in open-set action recognition.The conclusion names these as the proposed extensions of the evidential framework.
  • Conclusion: Extensive experiments show that DEAR works with most existing action recognition models in open-set settings.

Supplementary Material

The supplementary material provides methodological background, implementation details, and additional experiments supporting the main paper.

  • Supplementary Material: Section A explains evidential deep learning and model calibration to support understanding of the main methodology.
  • Supplementary Material: Section B supplies implementation details intended to support reproduction of the proposed method.
  • Supplementary Material: Sections C and D provide additional experimental results complementing the main paper.

A. Detailed Methodology

The methodology replaces softmax-based confidence with evidential predictions that represent class probabilities and uncertainty through a Dirichlet model. Evidence-based training and calibration support recognition of known actions and identification of unknowns.

  • Motivation: Softmax can produce over-confident predictions for unknown data, making it poorly suited to open-set recognition.
  • Evidential Deep Learning: EDL treats classification as a K-dimensional subjective opinion and models class probabilities with a Dirichlet distribution.The opinion includes belief masses, uncertainty, and a base rate.
  • Evidential Deep Learning: The base rate is typically set to ak = 1/K, and the subjective opinion is constrained so its probability interpretation is maintained.
  • Evidential Deep Learning: For K = 2, the Dirichlet distribution reduces to a Beta distribution and produces a binomial subjective opinion.
  • Evidential Deep Learning: EDL represents nonnegative class evidence through Dirichlet strength, with expected class probability pk = αk/S.Predictive uncertainty is determined after the Dirichlet strengths are obtained.
  • Evidential Deep Learning: A neural network predicts evidence from input data using a nonnegative evidence function such as exp, softplus, or ReLU.The evidence is trained with a negative log-likelihood objective for K-class classification.

A.2. EDL for Open Set Action Recognition

DEAR adapts evidential deep learning to video action recognition, combining uncertainty calibration with contrastive debiasing for efficient open-set inference.

  • EDL for Open Set Action Recognition: DEAR replaces the KL regularizer with Evidential Uncertainty Calibration because the digamma function is numerically unstable for large-scale video data.The calibration term compensates for EDL’s over-fitting risk.
  • EDL for Open Set Action Recognition: The complete DEAR objective combines EDL, EUC, and Contrastive Evidence Debiasing with weights w1=1.0 and w2=0.1.The contrastive term is defined from the paper’s CED losses.
  • EDL for Open Set Action Recognition: DEAR predicts class probabilities as p̂_k = α_k/S and predictive uncertainty as u = K/S during inference.High uncertainty indicates insufficient evidence for closed-set classification and suggests an unknown action.
  • EDL for Open Set Action Recognition: EDL deterministically learns predictive uncertainty without posterior approximation or computationally expensive sampling, supporting efficient training on large-scale video data.This contrasts with Bayesian neural networks and deep Gaussian processes.
  • EDL for Open Set Action Recognition: HSIC measures dependence between two high-dimensional variables, with the unbiased estimator computed from kernelized matrices and equal to zero exactly under independence.The matrices use RBF kernels, zeroed diagonals, and an all-one vector in the estimator.

A.4. Evaluation of Model Calibration

The paper evaluates EUC calibration by comparing confidence with empirical accuracy across confidence bins, using ECE as the aggregate miscalibration measure.

  • Evaluation of Model Calibration: Model calibration requires predicted confidence to represent the true probability that the predicted label is correct.The paper uses expected calibration error to evaluate EUC.
  • Evaluation of Model Calibration: Predicted confidences are grouped into M equal-width bins over [0,1] because perfect calibration is infeasible with finite samples.Each bin has width 1/M.
  • Evaluation of Model Calibration: For each bin, accuracy is estimated from predictions whose confidence falls within that bin, using the predicted and ground-truth labels.The bin contains the corresponding prediction indices.
  • Evaluation of Model Calibration: ECE measures the expected gap between accuracy and confidence across M bins and N samples.It aggregates calibration discrepancies across the confidence range.
  • Evaluation of Model Calibration: ECE=0 denotes perfect calibration, while higher ECE indicates poorer calibration.

B. Implementation Details

DEAR is implemented across four video recognition architectures, with dataset-specific training choices and plug-in integration of EUC and CED.

  • Implementation Details: DEAR and its baselines use I3D, TSM, SlowFast, and TPN with ResNet-50 backbones initialized from Kinetics-400.For Kinetics and Mimetics validation experiments, models are trained from scratch instead.
  • Implementation Details: For I3D, DEAR replaces cross-entropy with EUC-regularized EDL loss and inserts CED before the recognition head.
  • Implementation Details: Training uses exp as the evidence function, λ0=0.01 in EUC, λ=1.0 in CED losses, and loss weights 1.0 and 0.1.The reported performance is robust to these hyperparameters.
  • Implementation Details: Experiments use split1 for UCF-101 and HMDB-51, while MiT-v2 contributes only its testing set for evaluation.The CED validation uses 10 action categories shared by Kinetics and Mimetics.

C. Quantitative Results

Quantitative evaluations cover open-set action recognition and out-of-distribution detection across multiple architectures and unknown-data settings.

  • Quantitative Results: Open-set curves are reported for I3D, TSM, SlowFast, and TPN in addition to the main-paper I3D results.
  • Quantitative Results: When HMDB-51 testing data are unknown, DEAR significantly outperforms other baselines with large margins.
  • Quantitative Results: When MiT-v2 testing data are unknown, DEAR achieves the best performance at relatively low openness.
  • Quantitative Results: OOD detection results compare DEAR with all listed baselines on HMDB-51 and MiT-v2 using I3D, TSM, SlowFast, and TPN.Because OpenMax, SoftMax, and RPL do not predict uncertainty, their maximum categorical probability is used instead.

D. Qualitative Results

The qualitative results examine open-set recognition through macro-F1 trends, out-of-distribution detection visualizations, confusion matrices, and examples of representation debiasing. They compare unknown datasets, backbone models, and DEAR variants with and without CED.

  • Open Set Confusion Matrix: When HMDB-51 is unknown, TSM and SlowFast show less known-to-unknown misclassification than I3D and TPN.This comparison concerns the bottom-left confusion-matrix region.
  • Representation Debiasing Examples: Examples from Kinetics and Mimetics compare DEAR variants with and without CED across playing piano, writing, and golf driving.The examples indicate that CED helps DEAR recognize actions in both biased and out-of-context datasets.
  • Open macro-F1: Open macro-F1 is evaluated against varying Openness with HMDB-51 and MiT-v2 alternately serving as the unknown testing set.Figures 9 and 10 report these two unknown-dataset settings.
  • Open Set Confusion Matrix: Confusion matrices compare ground-truth and predicted labels, separating known UCF-101 classes from unknown HMDB-51 or MiT-v2 classes.Values are uniformly scaled into [0,1], with lighter colors indicating higher values.
Loading 2107.10161v2…