Source-linked AI summary
Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
Sidong Zhang, Trang Nguyen, Shiv Shankar, Gauri Jagatap, Deepak Chandran, Andrea Fanelli, Madalina Fiterau
TL;DR
Audience-reaction prediction lacks large-scale, standardized data because induced emotions are variable and costly to collect. Video2Reaction addresses this gap with social-media-derived clip-level emotion distributions and evaluates LoRA-finetuned VLMs for in-domain prediction and transfer to VCE. The models improve reaction prediction and transfer effectively, with LLaVA-NeXT-Video-7B reaching 0.682 top-3 accuracy using 500 VCE samples.
Problem
Large-scale data and standardized evaluation for induced emotion are limited, while controlled audience-reaction collection is costly, slow, and difficult to scale.
Method
Video2Reaction aggregates social-media comments into clip-level categorical emotion distributions and uses them to finetune VLMs for audience-reaction prediction and transfer to VCE.
Results
LLaVA-NeXT-Video-7B + V2R achieves 0.682 top-3 accuracy after finetuning on only 500 VCE samples, matching reported SOTA of 0.689 and LLaVA trained on 5,000 samples at 0.701.
Takeaways & Limitations
Video2Reaction provides learnable supervision for VLM audience-reaction prediction and supports transfer to an out-of-domain induced-emotion dataset under limited supervision.
Abstract
from arXiv · showhide
We introduce Video2Reaction, a multimodal dataset that maps short movie segments to the induced emotional reactions of viewers in the wild, as expressed through social media comments. Video2Reaction captures the natural diversity of emotional responses by aggregating reactions from online comments at scale, modeling labels as distributions over categorical emotions to better reflect the subjective and ambiguous nature of emotional perception. We benchmark two vision-language models (VLMs) finetuned with LoRA, showing that VLMs learn effectively from Video2Reaction and outperform specialized baselines on dominant reaction prediction. We further demonstrate that VLMs pre-finetuned on Video2Reaction transfer effectively to VCE, another induced emotion dataset with a different taxonomy and video domain. Notably, LLaVA-NeXT-Video-7B pre-finetuned on Video2Reaction and adapted on only 1% of VCE training data achieves a top-3 accuracy of 0.682, on par with the best reported VCE performance trained on the full dataset. The dataset is available at https://huggingface.co/datasets/infofusionlab/Video2Reaction
1. Introduction
Video2Reaction addresses the underexplored problem of predicting audience reactions to video by providing scalable data for induced-emotion modeling. It targets gaps in large-scale data, standardized evaluation, and updateable real-world reaction datasets.
- Induced-emotion prediction remains underexplored in affective computing despite its potential use in pre-screening media and refining content before release.
- Existing datasets largely measure perceived emotions expressed by characters or scenes, while relatively few target viewers’ induced emotional responses.
- Induced emotions vary with individual, cultural, and temporal factors, making audience responses more complex and variable.
- Controlled participant-based approaches capture reactions at fixed times but are costly, slow, and difficult to scale.
- Video2Reaction aggregates nearly one million social-media comments across approximately 10,000 videos into clip-level emotion distributions for scalable audience-reaction prediction.
- Finetuning Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Video-7B on Video2Reaction substantially improves audience reaction prediction and supports promising out-of-distribution generalization under limited supervision.
2. Video2Reaction Dataset
Video2Reaction constructs a fine-grained induced-emotion dataset from licensed movie clips and social-media comments using a scalable multi-agent LLM annotation pipeline. The design supports frequent updates while addressing implicit reactions and annotation consistency.
- The dataset uses licensed CondensedMovies clips, retaining videos with at least 10,000 views and 10 comments to ensure meaningful audience engagement.
- Viewer comments extend through 2025, giving each clip at least six years of audience commentary.
- The final reaction taxonomy contains 21 fine-grained emotions after removing seven severely under-represented GoEmotions categories.
- Each comment passes through rephrasing and reaction-label extraction stages, with the first stage interpreting implicit or off-topic remarks.
- Three multilingual instruction-tuned LLMs use majority voting, enabling parallel inference rather than sequential annotator–critic interaction.
- Annotation quality is evaluated through human–LLM alignment and dual-blind verification.
OUTPUT
The annotation pipeline rephrases comments, filters non-reactions, and extracts labels through three-agent majority voting. Its outputs show human-comparable alignment, high verified label accuracy, and substantial variation across clips and reaction categories.
- Stage 1 rephrases comments to make reactions explicit and removes comments without discernible reactions; Stage 2 extracts labels using three-agent majority voting.
- The LLM pipeline achieves a mean inter-rater correlation of 0.402, compared with 0.428 for human annotators.
- 86% of LLM-assigned labels are judged correct, while 7.8% are incorrect and 6.2% indeterminate in dual-blind verification.
- Clip-level reaction distributions vary within movies, with a median Chebyshev distance of 0.48, supporting clip-level rather than movie-level labeling.
- The 21 reaction categories have an imbalance factor of 28.36, while the dominant emotion has a median probability of only 0.4.
3. Methods
Video2Reaction frames audience reaction prediction as label distribution learning and evaluates VLM finetuning for in-domain prediction and transfer to VCE under shifted videos and taxonomies.
- 3.1. Problem Setup: Audience reactions are represented as label distributions over emotion classes, capturing the proportion of viewers associating each label with a video clip.The model predicts a distribution approximating the target distribution by minimizing KL divergence.
- 3. Methods: The study evaluates Video2Reaction through in-domain VLM finetuning and out-of-domain transfer to the VCE induced-emotion dataset.VCE introduces shifts in video content and label space while requiring costly human annotation.
- 3.2. Finetuning VLMs to Predict Audience Reaction: Two VLMs are finetuned with LoRA on attention layers, using next-token probabilities over candidate emotion labels as predicted distributions.The setup uses rank-8 LoRA with approximately 6% trainable parameters and requires distinct first tokens for candidate emotions.
- 3.2. Finetuning VLMs to Predict Audience Reaction: Transfer is tested by comparing Video2Reaction-pre-finetuned VLMs adapted on small VCE subsets against models trained from scratch on the same subsets.This tests whether pre-finetuning can reduce target-domain supervision while maintaining competitive performance.
- 3.5. Baseline Methods: In-domain evaluation compares finetuned VLMs with LDSVR, SA-BFGS, and zero-shot Gemini 2.5 Flash.The evaluation uses classic LDL baselines and a proprietary VLM reference.
4. Results
Video2Reaction-trained VLMs outperform specialized LDL baselines for dominant reaction prediction and transfer effectively to VCE with limited target-domain supervision.
- 4.1. In-Domain Results: MRR, TPE, and class-weighted Top-k F1 evaluate dominant reaction ranking, probability estimation, and classification quality.These metrics assess complementary aspects of dominant reaction prediction.
- 4.1. In-Domain Results: LoRA-finetuned VLMs substantially outperform LDSVR and SA-BFGS across most dominant reaction metrics, while zero-shot VLMs perform poorly.Word-option consistently beats Label-option, indicating that emotion words better exploit pretrained semantic knowledge.
- 4.2. Out-of-Domain Transferable Results: Video2Reaction pre-finetuning consistently outperforms training from scratch across VCE subset sizes.The comparison evaluates both VLMs with and without Video2Reaction pre-finetuning.
5. Next Steps
Future work will test whether Video2Reaction knowledge can support automatic VCE labeling and whether transfer benefits vary with VLM scale.
- Transfer annotation knowledge for automatic labeling: Pre-finetuned VLMs will be used to automatically label VCE, followed by VideoMAE training on 50,000 VLM-labeled samples.The resulting quality gap against human annotation will assess whether induced-emotion dataset curation can scale at lower cost.
- Understand the effect of model scale on knowledge transfer: Future experiments will evaluate compact and larger LLaVA and Qwen models beyond the current 7B-scale VLMs.The goal is to determine whether transfer benefits scale proportionally or show diminishing returns.
A.1. Implementation Details
The annotation implementation uses multilingual LLM agents and two prompt stages to rephrase, filter, and extract structured audience reactions from clip descriptions and comments.
- Annotation models: Three multilingual instruction-tuned LLMs form an ensemble for processing the large volume of raw comments.The ensemble includes LLaMA-3.1-8B-Instruct, Qwen2.5-14B-Instruct, and DeepSeek-R1-Distill-Qwen-7B.
- Pipeline inputs: The pipeline requires a clip description and a user-written YouTube comment as inputs.Descriptions come from Movieclips or can be generated by a video understanding model when unavailable.
- Stage 1: Rephrase and Filter Comment Prompt: Stage 1 rephrases comments from a third-person perspective and filters remarks that do not provide generalizable audience reactions.The prompt asks for concise reasoning about what the audience member is likely feeling because of the clip.
- Stage 2: Extract Reaction Labels Prompt: Stage 2 extracts one or more high-level and fine-grained reaction labels from the clip description and rephrased comment.The prompt returns structured JSON fields for reaction categories.
A.2. Human Evaluation & Additional Error Analysis
The paper evaluates annotation quality through human–LLM agreement, dual-blind human verification, and analysis of annotation errors. These evaluations are reported across multiple tables.
- Table 4 compares inter-rater correlation with LLM–human correlation across 21 emotion classes.
- Table 5 summarizes dual-blind human evaluation on the test set.
- Tables 6 and 7 analyze errors from the annotation pipeline across movie genres and emotion categories.
B. Data Statistics
Video2Reaction is summarized at the movie, clip, and comment levels, with substantial imbalance across reaction categories and considerable variation in reaction probabilities across clips.
- 28.36 is the dataset's imbalance factor across 21 reaction categories.
- Approximately 0.4 is the median top-1 reaction probability across clips.
- Table 8 summarizes Video2Reaction at the movie, clip, and comment levels.