Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization

1 INTRODUCTION

• Emotion in user-generated videos is difficult to analyze because complex spatio-temporal interactions and sparse expressive frames complicate understanding.The paper contrasts this with static-image analysis and notes that emotion may appear only in limited keyframes or clips.Source: §1 · p. 1
• Auxiliary emotional images provide an Image Transfer Encoding that aggregates frame-level features into a video-level emotion-sensitive representation.The encoding supports supervised video emotion recognition and the related attribution and summarization tasks.Source: §1 · p. 1
• Text-derived semantic relationships support recognition of emotion classes unseen during training, a setting the authors describe as their first exploration of zero-shot emotion recognition.The motivation is that real-world systems trained on a small label set will encounter other emotion types.Source: §1 · pp. 1–2
• Video emotion attribution identifies each frame's contribution to the video's overall emotion, while emotion-oriented summaries balance emotion, information content, and length.The paper motivates attribution partly by noting that emotionally strong videos can contain many neutral frames.Source: §1 · p. 2
• The framework transfers knowledge from heterogeneous image and text sources to video emotion understanding, and the paper introduces two emotion-centric video datasets.The datasets are VideoStory-P14 and YF-E6.Source: §1 · pp. 1–2
• A user study finds that the proposed summaries capture emotional content consistent with the original video's overall emotion, and the authors report better summaries than methods that do not consider emotion.The paper also reports experiments on multiple datasets demonstrating the framework's effectiveness.Source: §1 · p. 2

2.1 Psychological Theories of Emotion

• Emotion theories range beyond basic categories to varied, dynamic experiences, motivating recognition of emotion classes absent from training, despite the practical cost of labeling diverse descriptions.The paper frames zero-shot recognition as testing whether heterogeneous-source knowledge can adapt to unseen emotional labels.Source: §2.1 · p. 2

2.2 Automatic Emotion Analysis

• Prior work covers image sentiment, video emotion, and related methods, but the paper identifies unseen-class recognition and emotion-oriented attribution and summarization as unaddressed needs.It also transfers emotion information from SentiBank images for video analysis.Source: §2.2 · pp. 2–3

2.3 Zero-shot Learning

• The paper uses a text-corpus semantic vector space and image-feature regressors to connect visual and textual emotion representations for zero-shot recognition.Vector operations support reasoning over emotion categories and recognition of emotions unknown to the system.Source: §2.3 · p. 3

2.4 Video Emotion Attribution

• Video emotion attribution assigns a video's emotion to constituents such as frames or clips, extending beyond the binary positive-versus-negative sentiment attribution described for text.The paper notes the difference in media as well as the consideration of multiple emotions.Source: §2.4 · p. 3

2.5 Video Summarization

• Existing summarization methods use varied visual and semantic cues, but the paper identifies general video emotion as an overlooked cue for selecting meaningful highlights.It argues that omitting emotion risks losing moments such as those in a birthday party or graduation ceremony.Source: §2.5 · p. 3

2.6 Multi-Instance Learning

• The work treats videos as multi-instance bags of frames and encodes them into single-instance representations using auxiliary image data, for multiclass rather than typical binary emotion recognition.Figure 1 summarizes how image-derived information supports video encoding and emotion tasks, while text-corpus information supports zero-shot recognition.Source: §2.6 · p. 4, Figure 1 · p. 4

3 APPROACH

• The framework covers problem setup, auxiliary image transfer encoding, zero-shot emotion recognition, video emotion attribution, and emotion-oriented summarization.Figure 1 shows an overview of the framework.Source: §3 · p. 4

3.1 Problem Setup

• Training videos pair frame features with a video emotion label and a video-level feature vector, while test labels come from classes absent in training.Auxiliary image data and text are used to transfer knowledge; image features are extracted with AlexNet’s fc7 layer.Source: §3.1 · p. 4

3.2.1 The Encoding Scheme

• The encoding scheme clusters auxiliary emotional-image features and represents each video by accumulating frame assignments to its nearest cluster centers.Each frame is assigned to its K nearest centers, producing a D-dimensional video-level Bag-of-Words vector.Source: §3.2.1 · p. 5

3.2.2 Rationale for ITE

• ITE is designed to give emotionally informative frames more influence: such frames yield sparse encodings, whereas non-emotional frames tend to yield denser, higher-entropy encodings.Unlike soft-weighting BoW, ITE permits one frame to contribute equally to multiple bins, reflecting that a frame may evoke multiple emotions.Source: §3.2.2 · p. 5

3.3 Zero-Shot Emotion Recognition

• For zero-shot recognition, text-derived word embeddings represent emotion labels, video features are regressed to label embeddings, and one-step self-training adjusts unseen-class prototypes before cosine-based classification.The training and test emotion-label sets are disjoint; self-training is performed once to prevent semantic drift.Source: §3.3 · pp. 5–6

3.4 Video Emotion Attribution

• Emotion attribution scores each frame by its cosine similarity to the video-level emotion vector, and the procedure also extends to pre-partitioned clips.Frame features use the auxiliary image dataset to evaluate emotion.Source: §3.4 · p. 6

3.5 Emotion-Oriented Video Summarization

• Emotion-oriented summarization selects key frames to balance emotional content with information coverage and reduced redundancy.The method uses ITE similarity for emotion and within-video feature similarity for representativeness, with λ empirically set to 1.Source: §3.5 · p. 6

4.1 Datasets and Settings

• Evaluation covered YouTube-8 and its 24-class reannotation, YF-E6, and VideoStory-P14, with auxiliary data comprising 110K Flickr images and a 7-billion-word text corpus.YouTube-24 annotation agreement was Cohen’s kappa 0.62, and YF-E6 agreement was 0.73; the latter retained 1,637 videos across six emotion classes.Source: §4.1 · p. 7

4.2 Supervised Emotion Recognition

• ITE improved over AvgP by 1.2, 3.2, 4.6, and 0.8 absolute percentage points on VideoStory-P14, YF-E6, YouTube-8, and YouTube-24, respectively.ITE significantly outperformed the four alternatives across the supervised benchmarks. The authors attribute its advantage over CCE and Mi-FV to knowledge transferred from auxiliary emotional images.Source: §4.2 · p. 8, Figure 2 · p. 9

4.2.1 Hyper-Parameters and Deep Network Configurations

• ITE outperformed AvgP over most auxiliary-cluster settings; fully connected features significantly outperformed convolutional features, for which conv5 and conv4 results were 22.5 ± 2%.ITE performance improved as auxiliary image clusters increased from 100 to 2,000, then saturated with a slight drop. ITE with MFCC outperformed concatenating all features.Source: Table 2 · p. 8, §4.2.1 · pp. 9–10

4.3 Zero-Shot Emotion Recognition

• ITE+T1S achieved the best zero-shot accuracy, exceeding the second-best baseline by 3.6, 4.8, and 1.2 absolute percentage points, respectively.T1S helped most when training and unseen classes were similar, while ITE mattered more when they differed; YouTube-24 showed the largest margin over baselines.Source: Figure 4 · p. 10, §4.3 · p. 11

4.4 Video Emotion Attribution

• ITE-selected clips achieved the best emotion-attribution accuracy, outperforming face-presence sampling by 16–26 absolute percentage points.The method identified clips that conveyed emotions recognizably similar to the full video, including when emotion was expressed through body language or color rather than faces; results remained best in both annotator groups.Source: Figure 8 · p. 11, §4.4 · pp. 11–12
• The attribution example assigns higher scores to frames in clips with stronger emotional content, supporting the method’s effectiveness.The video is sampled uniformly every 10 frames, with key frames shown above the score bars.Source: §4.4 · p. 12

4.5 Emotion-Oriented Video Summarization

• EmotionVS outperforms the baselines on accuracy and emotion; its emotion score exceeds the best baseline by 0.87, while coverage is -0.13 and quality is -0.04 versus their best baselines.The user-study results are shown in Figure 10.Source: Figure 10 · p. 12, §4.5 · p. 12
• Ten participants rated summaries from EmotionVS and four baselines on accuracy, coverage, quality, and emotion, using fixed summary lengths based on the original video duration.Summaries omitted audio; videos longer than one minute used six-second summaries, while shorter videos used summaries equal to 10% of the original duration.Source: §4.5 · p. 12
• Qualitative examples show the method capturing the therapist’s sadness and a father’s reveal, his daughter’s surprise, and their subsequent hug.Other methods focused on the therapy procedure or baseball game rather than these emotional elements.Source: §4.5 · p. 12
• Top retrieval results using video-summary cosine similarity share the same category and common visual characteristics, indicating the method can find emotional clips.The example uses all available videos and cosine similarity between videos and frames in Eq (10).Source: §4.5 · pp. 12–13

• The framework applies heterogeneous-source knowledge transfer to supervised and zero-shot emotion recognition, emotion attribution, and emotion-oriented summarization.Source: §5 · p. 13
• It learns encoding schemes from a large-scale emotional image dataset and a 7-billion-word text corpus to create representations for video emotion tasks.Source: §5 · p. 13
• Experiments on three challenging datasets demonstrate benefits from external knowledge, and a user study finds summaries capture emotion consistent with videos’ overall emotion.Source: §5 · p. 13
• Future work would combine emotion-oriented with story-driven summarization and encode motion; the framework lacks a large-scale auxiliary emotional-motion dataset deemed essential for knowledge transfer.Source: §5 · p. 13