Source-linked AI summary
Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
Yu-Gang Jiang, Zuxuan Wu, Jun Wang, Xiangyang Xue, Shih-Fu Chang
TL;DR
Video categorization requires bridging complex high-level semantics while combining multiple features and exploiting relationships among semantic classes. The paper introduces rDNN, which jointly learns both relationship types through regularized DNN training. Across benchmarks, rDNN consistently outperforms alternative approaches and is presented as efficient for large-scale applications, alongside the FCVID dataset release.
Problem
Video categorization must represent complex high-level semantics, while existing methods often use simple feature fusion and neglect inter-class semantic relationships.
Method
rDNN jointly learns feature and class relationships by imposing trace-norm regularizations on tailored fusion and output layers within one DNN.
Results
rDNN consistently outperforms several alternative approaches in extensive action- and event-recognition experiments on popular benchmarks.
Takeaways & Limitations
The framework supports efficient large-scale video categorization and the released FCVID benchmark contains 91,223 videos across 239 manually annotated categories.
Takeaways & Limitations
The framework uses pre-computed features, including CNN features from off-the-shelf models, rather than jointly learning feature representations and classification.
Abstract
from arXiv · showhide
In this paper, we study the challenging problem of categorizing videos according to high-level semantics such as the existence of a particular human action or a complex event. Although extensive efforts have been devoted in recent years, most existing works combined multiple video features using simple fusion strategies and neglected the utilization of inter-class semantic relationships. This paper proposes a novel unified framework that jointly exploits the feature relationships and the class relationships for improved categorization performance. Specifically, these two types of relationships are estimated and utilized by rigorously imposing regularizations in the learning process of a deep neural network (DNN). Such a regularized DNN (rDNN) can be efficiently realized using a GPU-based implementation with an affordable training cost. Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed rDNN is more suitable for modeling video semantics. With extensive experimental evaluations, we show that rDNN produces superior performance over several state-of-the-art approaches. On the well-known Hollywood2 and Columbia Consumer Video benchmarks, we obtain very competitive results: 66.9\% and 73.5\% respectively in terms of mean average precision. In addition, to substantially evaluate our rDNN and stimulate future research on large scale video categorization, we collect and release a new benchmark dataset, called FCVID, which contains 91,223 Internet videos and 239 manually annotated categories.
1 INTRODUCTION
Video categorization must bridge a semantic gap while combining multimodal features and exploiting relationships among semantic classes. The paper proposes rDNN, a unified DNN framework that jointly learns these relationships through regularization.
- 1 INTRODUCTION: High-level video categorization is challenging because low-level features must represent complex semantics in diverse, intrinsically multimodal videos.Applications include Internet video search.
- 1 INTRODUCTION: Existing systems typically use large feature sets, but often overlook cross-feature correlations and complementary information during fusion.The paper highlights relationships between visual, motion, and acoustic descriptors.
- 1 INTRODUCTION: Inter-class relationships can provide clues for recognizing related categories, yet prior methods generally addressed feature fusion and class relationships separately.For example, evidence for running or diving can alter confidence in soccer.
- 1 INTRODUCTION: The proposed rDNN jointly learns feature and class relationships within one DNN and performs video categorization using the learned relationships.The framework is motivated by limitations of existing techniques and the growing use of DNNs.
- 1 INTRODUCTION: Trace-norm regularization in the fusion and output layers models feature correlation, feature diversity, and semantic-class commonalities.The framework is designed for GPU execution with affordable training cost.
- 1 INTRODUCTION: FCVID contains 91,223 YouTube videos and 239 manually annotated categories spanning events, scenes, and objects.The dataset was introduced to support large-scale video categorization research.
2 RELATED WORK
Prior video-categorization research developed diverse features, fusion methods, class-context models, and multi-task approaches. The paper positions rDNN as a unified neural framework that explicitly regularizes relationships across features and classes.
- 2 RELATED WORK: Video-categorization systems commonly extract varied descriptors before classification, including trajectory, audio, STIP, and CNN-based features.SVMs were historically dominant, while neural networks became increasingly common with deep learning.
- 2.1 Exploiting Feature Relationships: Early fusion assumes feature complementarity, whereas late fusion combines separately trained prediction scores and cannot fully explore feature relationships.Both strategies require fusion weights, which may be fixed or learned.
- 2.1 Exploiting Feature Relationships: Advanced fusion methods studied low-rank denoising and audio-visual correlations, including shared low-rank matrices and joint codebooks.These approaches targeted noise removal, correlation discovery, or faster codebook training.
- 2.2 Exploiting Class Relationships: Class-relationship methods use context, co-occurrence, semantic diffusion, domain adaptation, or hierarchy and exclusion graphs to improve recognition.Some video-classification methods use co-occurrence statistics as semantic feature representations.
- 2.2 Exploiting Class Relationships: Co-occurrence-based approaches cannot address classes sharing commonalities without explicitly co-occurring in the same video.rDNN injects a class relationship matrix into a unified learning objective to learn and use such commonalities.
- 2.3 Multi-Task Learning: Multi-task learning shares commonalities across related tasks, but earlier approaches did not inject these regularizations into DNNs.The paper interprets output units as class-specific tasks and hidden neurons as shared features.
3 REGULARIZED DNN
The regularized DNN extends standard deep networks with a fusion layer and relationship-aware regularization for both multiple features and semantic classes. It jointly learns these relationships and network weights to support video classification.
- DNN formulation: The DNN maps feature inputs through nonlinear layers to produce semantic-class predictions for video samples.Standard training minimizes empirical loss between network outputs and ground-truth labels while adding regularization to prevent overfitting.
- Feature fusion: A fusion layer transforms multiple feature representations into a shared representation for video categorization.The layer applies feature-specific transformations followed by nonlinear mapping to generate the fused representation.
- Feature fusion: Trace-norm regularization models inter-feature correlations so correlated feature dimensions receive similar weights while feature diversity is retained.The feature relationship matrix is symmetric and positive semidefinite, and its entries are learned together with the network weights.
- Class relationships: Class-relationship regularization trains multiple classifiers jointly and enables knowledge sharing among related semantic classes.The formulation learns the class relationships alongside the prediction model rather than assuming them as given prior knowledge.
- Unified objective: The unified rDNN combines feature fusion and class knowledge sharing within one DNN formulation to bridge low-level features and high-level video semantics.Its training alternates between estimating relationship matrices from network weights and refining the weights using those relationships.
4 EXPERIMENTS
Experiments evaluate rDNN on three video benchmarks using CNN, motion, and audio features, comparing relationship regularization with alternative fusion and classification methods. Results show gains from feature and class relationship regularization, especially with limited data and on FCVID.
- Experimental setup: The evaluation uses Hollywood2, CCV, and FCVID, reporting mean average precision under standard training and testing splits.AP is computed per category and mAP summarizes all categories.
- Video features: The feature inputs combine static CNN representations, dense trajectory motion descriptors, and audio features including MFCCs and spectrogram SIFT.Motion descriptors are quantized into 4,000-dimensional bag-of-words representations.
- Feature relationships: rDNN-F achieves the best performance among the compared feature-fusion methods, with consistent gains across the evaluated datasets.The comparison includes individual features, alternative fusion methods, and a neural-network baseline without explicitly enforced feature relationships.
- Class relationships: 1.6, 0.8, and 3.0 percentage-point drops occur on Hollywood2, CCV, and FCVID when the class relationship matrix is fixed to the identity.This setting treats classes independently, supporting the contribution of learned class relationships.
- Combined relationships: The full rDNN obtains higher performance than using either relationship type alone, while deeper six- and eight-layer DNNs perform significantly worse because additional parameters encourage over-fitting with limited samples.The full framework also outperforms dropout-based DNN structures on all datasets.
- Limited samples: rDNN’s performance gains are more significant with fewer training samples, and it requires less training data to reach results comparable to the non-regularized version.The exception is FCVID with 10 training samples, which are too few to distinguish its 239 categories.
- Comparison with state of the art: 66.9 percent mAP is achieved on Hollywood2 and 73.5 percent mAP on CCV, while class-relationship gains are more significant on FCVID.The authors attribute the stronger FCVID effect to its larger number of classes sharing useful commonalities.
5 CONCLUSION
The paper presents rDNN as a unified approach for exploiting feature and class relationships in video categorization, with experiments showing consistent gains over alternatives. It also introduces FCVID, while identifying joint feature-extraction and classification learning as future work.
- rDNN exploits feature and class relationships jointly through trace-norm regularization in its fusion and output layers.The framework learns a fused representation from multiple inputs and uses commonalities among semantic classes.
- The approach consistently outperforms several alternative methods on action and event recognition benchmarks.
- FCVID is introduced as a new dataset for large-scale video categorization and related research.
- The current framework uses pre-computed features, leaving joint learning of feature representations and classification models for future work.The adopted CNN features are computed from off-the-shelf models and are not tuned simultaneously with the regularized classification network.