PaperlayerView exact revision on arXiv

Source-linked AI summary

An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

Sicheng Zhao, Yunsheng Ma, Yang Gu, Jufeng Yang, Tengfei Xing, Pengfei Xu, Runbo Hu, Hua Chai, Kurt Keutzer

arXiv:2003.00832v1cs.CVcs.HCcs.MM

TL;DR

Emotion recognition in user-generated videos remains challenging because of varied video structure, diverse emotional content, and sparse keyframes, while existing approaches have used shallow pipelines. VAANet combines attention-based visual and audio CNNs with a polarity-consistent loss, and outperforms state-of-the-art approaches on two benchmarks.

  • Problem

    User-generated-video emotion recognition remains challenging because of large intra-class variation, low structural consistency, and sparse keyframes, while prior approaches also neglect temporal and emotion relationships.

    Source: Introduction · p. 1

  • Method

    VAANet uses visual and audio CNN streams with spatial, channel-wise, and temporal attention, guided by polarity-consistent cross-entropy loss.

    Source: Introduction · p. 2

  • Results

    VAANet improves performance by 2.0% and 0.9% over the best state-of-the-art video emotion recognition approach on VideoEmotion-8 and Ekman-6, respectively.

    Source: Conclusion · p. 8

  • Takeaways & Limitations

    VAANet can focus on discriminative key segments and regions in user-generated videos while using less auxiliary data.

    Source: Conclusion · p. 7

    Attribute-based methods require auxiliary data to train classifiers, which may be unavailable in real applications.

    Source: Comparison with the State-of-the-art · p. 6

Abstract

from arXiv · show

Emotion recognition in user-generated videos plays an important role in human-centered computing. Existing methods mainly employ traditional two-stage shallow pipeline, i.e. extracting visual and/or audio features and training classifiers. In this paper, we propose to recognize video emotions in an end-to-end manner based on convolutional neural networks (CNNs). Specifically, we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture that integrates spatial, channel-wise, and temporal attentions into a visual 3D CNN and temporal attentions into an audio 2D CNN. Further, we design a special classification loss, i.e. polarity-consistent cross-entropy loss, based on the polarity-emotion hierarchy constraint to guide the attention generation. Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6 datasets demonstrate that the proposed VAANet outperforms the state-of-the-art approaches for video emotion recognition. Our source code is released at: https://github.com/maysonma/VAANet.

Introduction

Emotion recognition in user-generated videos matters for understanding creators’ psychological states and viewers’ responses, but remains challenging because of varied content and sparse emotional cues. The paper proposes an end-to-end visual-audio attention approach to address limitations in prior methods.

Introduction

  • The paper identifies emotion recognition in user-generated videos as important for human-centered computing, including understanding video creators’ psychological states and viewers’ emotional responses.The introduction also notes potential relevance to advertising, recommendation, customer analysis, and Internet management.Source: Introduction · p. 1
  • UGV emotion recognition remains unsolved amid large intra-class variation, low structural consistency, and sparse keyframes that convey emotion.The rest of a video may provide background and context rather than directly determining its emotion.Source: Introduction · p. 1
  • Existing approaches largely use shallow two-stage pipelines and neglect adjacent-frame correlations, keyframes across discrete segments, and relationships among emotions.Some methods also depend on auxiliary data that may be unavailable in real applications.Source: Introduction · p. 2
  • VAANet processes segmented video and audio end-to-end, using spatial, channel-wise, and temporal visual attention, audio features, and polarity-consistent cross-entropy loss.The loss is designed around the polarity-emotion hierarchy constraint to guide attention generation.Source: Introduction · p. 2
  • The authors present end-to-end user-generated-video emotion recognition as the paper’s first contribution.They contrast this contribution with existing approaches described as two-stage shallow pipelines.Source: Introduction · p. 2

2. We develop a novel network architecture, i.e. VAANet,

VAANet combines attention mechanisms across visual and audio CNN streams and uses polarity-consistent cross-entropy to guide attention. Experiments on two benchmarks show superiority over state-of-the-art approaches.

2. We develop a novel network architecture, i.e. VAANet,

  • VAANet integrates spatial, channel-wise, and temporal attention into a visual 3D CNN and temporal attention into an audio 2D CNN.Its polarity-consistent cross-entropy loss is proposed to generate polarity-preserving attention maps.Source: §2 · p. 2
  • Experiments on VideoEmotion-8 and Ekman-6 show VAANet’s superiority over state-of-the-art approaches.The passage reports the benchmark-level comparison without specifying metric values.Source: §2 · p. 2

Related Work

Prior video-emotion work spans categorical and dimensional representations, movie and GIF methods, and approaches for user-generated videos. The paper situates VAANet as an end-to-end approach combining attention mechanisms that had not previously been studied for emotion recognition in user-generated videos.

Related Work

  • Video emotion research can represent affect as categorical emotion states or as dimensions in a valence-arousal-dominance space.The paper adopts categorical states because they are easy for users to understand and label.Source: Related Work · p. 2
  • Earlier video-emotion studies focused on structured movies and animated GIFs, using low-level features, multimodal classifiers, 3D ConvNets, or human-related information and visual attention.The cited work covers movie affective-event detection and several approaches to GIF emotion recognition.Source: Related Work · p. 2
  • User-generated-video methods used low-level features, attributes, detected semantics, deep features, or joint representations, while remaining distinct from the proposed end-to-end approach.The passage characterizes earlier UGV methods as two-stage shallow pipelines.Source: Related Work · p. 2
  • VAANet’s framework divides soundtrack MFCCs and visual information into segments, applies attention within the visual and audio streams, then concatenates attended semantic vectors.The visual stream uses spatial, channel-wise, and temporal attention; the audio stream uses temporal attention.Source: Figure 2 · p. 3
  • Attention methods include spatial, semantic, channel-wise, and temporal forms, previously applied across vision tasks such as image classification, captioning, and action recognition.The cited overview describes attention as dynamic feature extraction combining contextual fixations over time.Source: Related Work · p. 3
  • Prior attention-based emotion recognition addressed images and speech, but the paper states that attention had not been studied for UGV emotion recognition.VAANet investigates spatial context, channel interdependency, and the importance of different video segments.Source: Related Work · p. 3

Visual-Audio Attention Network

VAANet combines visual and audio streams with attention mechanisms, then trains the network end to end using a polarity-consistent loss. Its attention modules weight informative regions, channels, and video segments.

Visual-Audio Attention Network

  • VAANet uses visual and audio streams, with spatial, channel-wise, and temporal attention in the visual stream and temporal attention in the audio stream.The framework is trained end to end with a polarity-consistent cross-entropy loss.Source: Visual-Audio Attention Network · p. 3

Visual Representation Extraction

  • The visual stream sparsely samples successive-frame snippets from equal-duration video segments and processes them with a 3D ResNet-101 backbone.The backbone produces super-frame feature maps for subsequent attention modules.Source: Visual Representation Extraction · p. 3

Visual Spatial Attention Estimation

  • Spatial attention assigns weights to regions within each super-frame to emphasize regions contributing to emotion prediction.The module uses a two-layer network and softmax to generate spatial attention distributions.Source: Visual Spatial Attention Estimation · p. 4

Visual Channel-Wise Attention Estimation

  • Channel-wise attention weights feature-map channels, treated as semantic attributes, to produce weighted feature maps for each super-frame.The channel weights are generated for each super-frame.Source: Visual Channel-Wise Attention Estimation · p. 4

Visual Temporal Attention Estimation

  • Visual temporal attention weights super-frames and combines them into a visual embedding, focusing on segments containing discriminative keyframes.The segment features are formed by spatially average-pooling the channel-weighted maps.Source: Visual Temporal Attention Estimation · p. 4

Audio Representation Extraction

  • The audio stream uses MFCC descriptors divided into segments and processes them independently with a 2D ResNet-18 backbone.Spatial average pooling produces a feature vector for each segment.Source: Audio Representation Extraction · p. 4

Audio Temporal Attention Estimation

Experiments

Across VideoEmotion-8 and Ekman-6, the experiments compare VAANet with prior methods and test its attention and polarity-consistent loss components. The reported results favor VAANet and show gains from the attention and loss ablations.

Experimental Settings

Comparison with the State-of-the-art

Visualization

  • Figure 3 examples from the VE-8 test set show attention on salient regions and changing across event-bearing frames, including caterpillar exposure and sudden dog movement.Source: Figure 3 · p. 7, Visualization · p. 7

Ablation Study

Conclusion

The conclusion presents VAANet as an attention-based method that focuses on discriminative video segments and regions, with polarity-consistent cross-entropy guiding attention. It reports improvements over the best state-of-the-art approach and outlines future directions in classification, regression, and key-frame attention.

  • VAANet combines visual and audio attention with a polarity-consistent cross-entropy loss to focus on discriminative key segments and regions while using less auxiliary data.The loss guides attention generation.Source: Conclusion · pp. 7–8
  • VAANet achieves 2.0% and 0.9% performance improvements over the best state-of-the-art video-emotion recognition approach on VideoEmotion-8 and Ekman-6.Source: Conclusion · p. 8
  • Future work will extend VAANet to fine-tuned emotion classification and regression and investigate attention that better concentrates on key frames within each video segment.Source: Conclusion · p. 8

Related papers

Loading 2003.00832v1…