EmoVid: A Multimodal Emotion Video Dataset for Emotion-Centric Video Understanding and Generation

• EmoVid covers animation, movie clips, and animated stickers with eight emotion labels, visual attributes, and VLM-generated captions.The authors explore emotion distributions, color-emotion correlations, temporal transitions, and semantic links.Source: Introduction · p. 1
• The benchmark evaluates visual quality and emotion accuracy in text-to-video and image-to-video tasks.The work also describes a pipeline for generating animated stickers of any character with any emotion.Source: Introduction · p. 1
• Fine-tuning Wan2.1 on EmoVid significantly improves emotional expression in video generation.Source: Introduction · p. 2
• The paper examines spatial and temporal emotional patterns and their relationships with captions and visual attributes.Source: Introduction · pp. 1–2

Emotion Analysis and Affective Computing

• Video affect research builds on static-image work, but creative domains with semantically rich emotional content remain underexplored.Static images lack temporal dynamics needed to model transitions and nuances in affect.Source: Emotion Analysis and Affective Computing · p. 2

Video Generation and Editing

• Video generators emphasize coherence, motion, realism, or aesthetic control, while affective conditioning rarely addresses stylized, non-realistic creative domains.Prior affective video work often centers on human or conversational tasks; meme and sticker emotion is typically implicit.Source: Video Generation and Editing · pp. 2–3

Emotion-related Datasets

• Existing emotion-video datasets have limitations in modality, domain, or size, and many focus on constrained facial expression rather than varied narrative content.The passage describes EmoVid as a 22,758-clip multimodal dataset spanning animation, film scenes, and stickers, where emotion is semantically important.Source: Emotion-related Datasets · p. 3

The EmoVid Dataset

• Existing emotion-video datasets lack scale or modalities and focus on real-world scenarios, limiting their coverage of stylized content.The paper identifies this as a challenge for multimodal emotion analysis and transfer of emotional priors to general video generation.Source: The EmoVid Dataset · p. 3

Data Collection

• 2,807 animation face clips, 13,255 movie clips, and 6,696 animated stickers make up EmoVid’s stylized-video collection.Each clip also has emotion annotations, brightness, colorfulness, hue, and a textual caption.Source: Data Collection · p. 3
• Animation clips come from MagicAnime, movie clips from Condensed Movies, and animated stickers from Tenor searches for emotion labels and synonyms.Movie segments are retained at 4–30 seconds, and each sticker is manually checked for the intended emotion.Source: Data Collection · pp. 3–4

Data Labeling

• EmoVid uses Mikels’ eight-emotion scheme and arranges its categories by valence and arousal.The categories are amusement, awe, contentment, excitement, anger, disgust, fear, and sadness.Source: Data Labeling · p. 4, Figure 2 · p. 4
• A human-machine process labels clips: three annotators assess 20% of animation and movie data, while a fine-tuned NVILA-Lite-2B labels the remaining 80%.A clip is retained when at least two annotators agree; validation found human–VLM kappa differences below 4%.Source: Data Labeling · p. 4
• Colorfulness, brightness, and hue are computed from HSV samples taken every 20 frames, and captions are generated with NVILA-8B-Video.Colorfulness and brightness are normalized to 0–1 and rounded to one decimal place.Source: Data Labeling · p. 4
• The annotation set also includes audio tracks, quantitative visual features, and free-form captions describing perceived content and sentiment.These annotations support audio-visual emotion fusion and emotion-attribution analysis.Source: Data Labeling · p. 4

Properties of EmoVid

• In the feature visualization, animation and movie samples separate, while sticker samples overlap both clusters.The paper describes the sticker overlap as reflecting their hybrid content characteristics.Source: Figure 4 · p. 5, Properties of EmoVid · p. 5
• 22,758 videos span 140,580 seconds, including 10,049 clips with human-annotated emotion labels.The remaining clips receive labels from a fine-tuned VLM, whose quality is verified in the reported experiments.Source: Properties of EmoVid · p. 5
• Animation and movie clips show imbalanced emotion distributions, with anger and sadness more frequent than amusement and awe.The paper attributes this imbalance to the real-world emotional landscape of those domains.Source: Figure 3 · p. 4, Properties of EmoVid · p. 5
• EmoVid is presented as a foundation for emotional storytelling, T2V and I2V generation, and emotionally grounded video editing.The benchmark evaluates T2V and I2V and assesses emotion accuracy in binary and eight-category forms.Source: Properties of EmoVid · p. 5, Table 3 · p. 6

Emotional Structure and Dynamics

• Positive-to-total emotion ratios rise with colorfulness and brightness; positive-valence categories are brighter and slightly more colorful, while high-arousal ones are darker but more colorful.ANOVA reports statistically significant but small effects (p < 0.01; η2 < 1%).Source: Emotional Structure and Dynamics · p. 5, Figure 4 · p. 5
• Across consecutive movie clips, all eight emotions show strong self-persistence—especially fear (0.53), anger (0.46), and amusement (0.46)—with within-valence transitions typically 0.08–0.18 versus under 0.08 across valences.Negative sequences show sadness → fear/anger and fear → anger.Source: Emotional Structure and Dynamics · p. 5
• Caption phrases provide semantic cues for emotion, including “funny reaction” for amusement and “approaching danger” for fear.The paper reports extracting frequent 2–4 word phrases for each emotion after filtering redundant or noisy expressions.Source: Emotional Structure and Dynamics · pp. 5–6

Evaluation of EmoVid

• The benchmark samples 240 videos across three video types and eight emotions, appending an explicit emotion label to each video’s caption; both Wan2.1 tasks are fine-tuned on EmoVid.The fine-tuning excludes benchmark data and uses LoRA.Source: Evaluation of EmoVid · p. 6

Quantitative Results

• 92.08% (2-class) and 72.92% (8-class) are the fine-tuned I2V model’s highest emotion classification accuracies; T2V emotion-alignment metrics also gain while FVD and CLIP remain comparable.The I2V model surpasses all competitors on the reported emotion-classification accuracies.Source: Quantitative Results · p. 6

Qualitative Results

• Compared with original Wan2.1-I2V, fine-tuned outputs more precisely express emotion through facial expressions, contextual cues, and mood-consistent motion.The original model often produces neutral or mismatched expressions.Source: Qualitative Results · p. 6, Figure 5 · p. 7
• The fine-tuned I2V model generates emotion-conditioned stickers for different characters, while multi-LoRA T2V combines emotion, identity, and visual-style priors.The authors identify social media as a potential setting for the sticker outputs.Source: Qualitative Results · pp. 6–7

Discussion

• For creative uses such as films, operas, miniseries, and expressive memes, the authors prioritize emotional clarity over realism or temporal fidelity and identify avatar generation and controllable editing as potential applications.They report that improvements often occur where baseline models fail to express the intended emotion.Source: Discussion · p. 7

Conclusion

• EmoVid combines emotion labels, visual attributes, and textual captions for animations, movie clips, and animated stickers, and fine-tuning generative models improves emotional expressiveness in both T2V and I2V.The paper presents the dataset as tailored to creative contexts.Source: Conclusion · p. 7
• The dataset assumes each clip conveys one specific emotion, although real-world expressions can be composite; the authors also identify better use of audio as future work.They describe audio integration as a step toward a unified video-audio-text model.Source: Conclusion · p. 7

Ethics Statement

• Access is limited to non-commercial research by academic institutions; redistribution and commercial use are prohibited.Use must comply with intellectual property laws and ethical research standards.Source: Ethics Statement · p. 8

Detailed Statistics

• The section adds statistical insights and detailed analyses of EmoVid to complement the main manuscript.Source: APPENDIX Detailed Statistics · p. 10

Duration Distribution across Styles

• EmoVid totals 140,580 seconds of video, with clips averaging 6.18 seconds and ranging from 0.18 to 29.98 seconds.Most clips fall between approximately 2 and 8 seconds, covering brief emotional moments and longer arcs.Source: Duration Distribution across Styles · p. 10
• Movies make up 58.24% of the dataset and average 8.75 seconds per clip, versus 29.42% and 2.91 seconds for stickers and 12.33% and 5.12 seconds for animation.The style mix covers emotional expression across different content styles.Source: Duration Distribution across Styles · p. 10

Color Attributes

• Positive-valence emotions are generally brighter and more colorful, while high-arousal emotions tend to be darker but more colorful; effects are significant but small (η2 < 1%).The color-space layout aligns with valence and arousal, but color cues are insufficient for standalone classification.Source: Color Attributes · p. 10

Caption Analysis

• Positive-emotion videos have a higher proportion of positively valenced captions, while frequent phrases reflect category-specific affective semantics.The analysis used caption polarity and extracted frequent 2–4-word phrases for each emotion category.Source: Caption Analysis · p. 10, Table 6 · p. 12

Construction Details

• Construction combines manual annotation with classifier-assisted verification and retrieves stickers using eight emotion categories plus root-merged synonyms.For example, amusement, amusing, and amused are normalized to amuse.Source: Construction Details · p. 10, Manual Annotation Protocol · p. 11

Manual Annotation Protocol

• Twelve annotators supported fine-grained labeling: two validated emotion stickers, while three annotators labeled each sampled movie or animation clip.A clip was retained when at least two annotators agreed; sticker filtering preserved 6,696 of 9,633 clips.Source: Manual Annotation Protocol · p. 11

Classifier Comparison

• NVILA-Lite-2B reached 87.5% accuracy on EmoSet-118k and showed human-comparable annotation agreement on EmoVid.The EmoVid validation yielded average inter-human kappa of 0.311 and average human–VLM kappa of 0.301.Source: Classifier Comparison · p. 11

Metrics and Implementation Details

• The benchmark reports visual fidelity, text-video alignment, first-frame consistency, temporal flicker, and binary or eight-class emotion accuracy.CLIP uses up to eight sampled frames, while SD and Flicker use all frames.Source: Metrics and Implementation Details · pp. 11–13

More Experiment Results

• Fine-tuned Wan2.1-I2V generates character expressions for specified emotions, while Multiple LoRA with Wan2.1-T2V combines visual styles with targeted emotions.Examples include animated sticker generation and Studio Ghibli-style videos.Source: More Experiment Results · p. 13, Figure 10 · p. 13, Figure 11 · p. 14

User Study

• Wan-Finetuned was ranked first for Emotion Expression in 66.2% of comparisons and achieved a 57.9% Top-1 rate for Aesthetic Quality, with advantages across all eight emotion categories.All pairwise preferences were significant (p ≪0.001); mean Kendall’s W was 0.371 for Emotion and 0.333 for Aesthetics.Source: User Study · p. 13, Figure 12 · p. 15