PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion Understanding in Streaming Video with Trajectory-Aware Reliability

Qingsong Wang, Qigong Lei, Zitong Wang, Bohan Yu, Zhiang Dong, Jian liu, Weiqiang Wang, Chang Yao, Jingyuan Chen

arXiv:2608.26786v1cs.MM

TL;DR

Streaming emotion understanding must make reliable decisions from incomplete, evolving evidence rather than complete offline video. TRACE uses belief trajectories to route stable cases through low-latency prediction and ambiguous cases through contextual reinterpretation, improving the accuracy-cost trade-off across evaluated datasets.

  • Problem

    Streaming emotion understanding lacks reliability-aware decisions over evolving beliefs, since offline methods assume complete video and single-prefix confidence can reflect unstable evidence.

  • Method

    TRACE forms evolving emotion beliefs from audio prefixes, estimates trajectory-based reliability, and selectively invokes visual, textual, and neighboring-utterance contextual reinterpretation.

  • Results

    TRACE improves the accuracy-cost trade-off across StreamMER, MER2024, and MELD; on StreamMER, Accuracy rises from 59.87% to 69.47% with reinterpretation for 55.58% of samples.

  • Takeaways & Limitations

    Belief stability is an important signal for reliable real-time emotion understanding and helps determine when to commit early or request stronger contextual reasoning.

  • Takeaways & Limitations

    The reliability estimator may be less reliable under substantial domain shifts because it relies on trajectory patterns from calibration data.

Abstract

from arXiv · show

Video emotion understanding is commonly studied as an offline classification problem, where the complete video segment is available before prediction. Real-time interaction, however, requires emotion decisions from incomplete and evolving evidence. This paper studies streaming video emotion understanding as a reliability-aware decision process over evolving emotion beliefs. In this setting, a single confident prefix prediction can still be unreliable when the underlying belief trajectory is unstable or repeatedly switches across emotion classes. We propose TRACE, a trajectory-aware reliability framework that forms low-latency emotion beliefs from streaming audio prefixes, estimates reliability from confidence, entropy, stability, and class-switching patterns, and selectively invokes contextual belief reinterpretation with visual, textual, and neighboring-utterance evidence. TRACE keeps stable cases in the low-latency online pathway while allocating stronger multimodal reasoning to uncertain cases that remain ambiguous. Experiments on StreamMER, MELD, and MER2024 show that TRACE improves the accuracy-cost trade-off, retaining most full-context gains while reducing unnecessary contextual reasoning.

1 Introduction

The paper formulates streaming video emotion understanding as continuously forming, evaluating, and revising emotion beliefs from partial evidence rather than classifying independent truncated prefixes. TRACE uses low-latency audio trajectories to decide when immediate commitment is reliable and when contextual multimodal reinterpretation is necessary.

  • Motivation: Streaming emotion understanding must respond as interactions unfold because offline methods assume the complete video segment is available before prediction.This requirement arises in applications including embodied intelligence, online mental health support, and remote meeting analysis.
  • Motivation: Prediction trajectories reveal how emotional evidence accumulates, beliefs stabilize, and uncertainty persists, unlike independent static predictions at separate time steps.The same utterance can change dynamically as new information arrives during streaming inference.
  • Motivation and framework: TRACE reframes emotion understanding from offline full-observation prediction to online stream decision-making over evolving emotion beliefs.The framework continuously estimates emotion beliefs and uses their trajectories to guide decisions during streaming inference.
  • Framework: TRACE forms low-latency beliefs from audio prefixes and adaptively invokes costly cross-modal reasoning using visual, textual, neighboring-segment, and trajectory evidence.Stable trajectories support immediate online commitment, whereas unstable trajectories trigger contextual reinterpretation.
  • Contributions: The paper defines streaming video emotion understanding and introduces StreamMER to evaluate emotion recognition under partial, progressively revealed observations.Its contribution also emphasizes asymmetric evidence trajectories and adaptive allocation of cross-modal reasoning.

2 Related Work

Prior work advances efficient streaming video understanding and multimodal video emotion understanding, but largely assumes full-context prediction. TRACE addresses real-time emotion judgments by using streaming audio prefixes and selectively adding richer contextual reasoning when belief trajectories are unreliable.

  • Streaming video understanding: Streaming video methods target real-time and long-form understanding through frame selection, retrieval-based representations, and efficient processing of continuous video streams.Representative approaches include BOLT, VLog, Flash-VStream, and StreamingVLM.
  • Video emotion understanding: Multimodal emotion models improve video emotion understanding through instruction tuning, emotion-specific datasets, facial and acoustic cues, and multimodal reasoning.Representative systems include Emotion-LLaMA (Cheng et al., 2024), AffectGPT (Lian et al., 2025), EmoLLM (Yang et al., 2024), Emotion-Qwen (Huang et al., 2025), and Omni-Emotion (Yang et al., 2025).
  • Video emotion understanding: Unlike prior offline methods with complete video, TRACE forms low-latency emotion beliefs from streaming audio prefixes and invokes richer visual, textual, and contextual reasoning only when trajectories are unreliable.This makes TRACE complementary to full-context multimodal emotion models while targeting judgments before an utterance is fully observed.

3 Method

TRACE models streaming emotion understanding as evolving belief estimation under incomplete utterance evidence, then selectively adds multimodal context when belief trajectories are unreliable. Its reliability-aware policy preserves low-latency processing for stable cases while allocating reinterpretation to ambiguous or unstable cases.

  • Streaming setting: Streaming prediction observes only the current utterance prefix and historical utterances, not future content or the complete utterance, transcript, and multimodal context.Voice activity detection segments video into ordered utterance units, and the current multimodal prefix evolves over time.
  • Online Prefix Belief Formation: Online Prefix Belief Formation uses continuously arriving audio cues to update emotion distributions, supervises all prefixes, and weights later prefixes more heavily because early evidence may be weak.Predictions form a belief trajectory without feeding earlier predicted labels autoregressively into later inputs.
  • Trajectory-Calibrated Reliability Estimation: Trajectory-Calibrated Reliability Estimation uses confidence, entropy, class switching, distributional stability, and observation ratio to estimate prediction risk and contextual correction gain.A shared encoder with two sigmoid heads predicts whether the current belief is wrong and whether additional context can correct it, informing commit, wait, or reinterpret actions.
  • Overall method behavior: TRACE achieves performance close to full contextual reinterpretation while invoking reinterpretation for 55.58%, 54.63%, and 58.21% of samples on StreamMER, MELD, and MER2024, respectively.These overall results are reported in Table 1, with percentages denoting reinterpretation invocation rates across the three datasets.
  • Contextual Belief Reinterpretation: Contextual Belief Reinterpretation supplements unstable audio beliefs with visual, textual, preceding-utterance, memory, and trajectory evidence rather than applying cross-modal reasoning by default.The reinterpretation model re-judges high-risk utterances and uses the belief trajectory to diagnose instability or competition among classes.

4 Experiments

TRACE preserves most of full contextual reinterpretation’s accuracy gains across StreamMER, MELD, and MER2024 while selectively invoking costly multimodal reasoning. Experiments show that trajectory-aware reliability improves the accuracy-cost trade-off by identifying unstable or ambiguous streaming beliefs.

  • Overall results: TRACE raises StreamMER Accuracy from 59.87% to 69.47% over online belief formation while requesting contextual reinterpretation for only 55.58% of samples, remaining close to full reinterpretation on MER2024 and MELD.Reinterpretation rates are 58.21% on MER2024 and 54.63% on MELD, with Accuracy and F1 remaining close to full contextual reinterpretation.
  • Latency and cost: TRACE achieves an RTF of 0.091, between online prefix formation at 0.055 and full contextual reinterpretation at 0.127.Applying multimodal contextual reasoning to every sample is therefore more costly than TRACE’s selective strategy.
  • Overall results: Full contextual reinterpretation achieves the best overall performance on StreamMER, MELD, and MER2024, while online prefix formation alone reaches 59.87% Accuracy on StreamMER.The online result outperforms Affect and ViDEmo and approaches Emotion-Qwen, indicating that short audio prefixes contain useful emotion cues.
  • Selective decision strategies: On StreamMER, Early Commit reaches 57.98 Accuracy, Full Commit 58.98, and Adaptive Commit 59.65, showing that additional audio observation alone rarely resolves ambiguity.TRACE instead reaches 69.47 Accuracy and 68.10 W-F1 at 55.58 RR, compared with 70.18 Accuracy and 100.00 RR for Always Reinterp.
  • Case studies: Representative cases show TRACE triggering reinterpretation when beliefs shift or fluctuate, then using visual, textual, and dialogue context to revise ambiguous predictions.A joy-to-surprise shift is revised to anger using confrontational dialogue, direct gaze, and accusatory tone; another trajectory fluctuates among surprise, anxiety, and anger before resolution.
  • Ablation: Trajectory features improve Accuracy from 66.87 to 68.83 and reduce reinterpretation from 69.33 to 60.12 under a linear rule, while reliability modeling lowers reinterpretation further to 55.71.Under the reliability model, trajectory features improve Accuracy from 67.98 to 68.83; overall, trajectory features capture reliability beyond single-prefix confidence.

5 Conclusion

TRACE frames streaming video emotion understanding as belief evolution under partial observation, using trajectory-aware reliability to decide between early commitment and contextual reinterpretation. Across StreamMER and two public benchmarks, it improves the accuracy-cost trade-off and identifies belief stability as a key reliability signal.

  • 5 Conclusion: TRACE models emotion prediction as belief evolution under partial observation, using belief trajectories to decide when to commit early or invoke contextual reinterpretation.This trajectory-aware decision process supports reliable real-time emotion understanding.
  • 5 Conclusion: TRACE improves the accuracy-cost trade-off across StreamMER and two public benchmarks, highlighting belief stability as an important signal for reliable real-time emotion understanding.

Limitations

TRACE improves the streaming accuracy–cost trade-off but remains limited by domain-shift sensitivity in reliability estimation and added latency and computation when contextual reinterpretation is invoked.

  • TRACE’s trajectory-based reliability estimator may become less reliable under substantial domain shifts because it depends on patterns learned from calibration data.
  • Invoking contextual belief reinterpretation adds latency and computation, making TRACE less suitable for extremely strict real-time applications.

Ethics Statement

StreamMER is designed for academic research using publicly accessible media while avoiding redistribution of copyrighted content.

  • Ethics Statement: StreamMER releases only derived annotations and metadata, requiring benchmark users to obtain the original media resources themselves.Released materials include emotion labels, timestamps, speaker identifiers, context summaries, rationales, and non-reconstructive features when applicable.

A Dataset Construction and Statistics

StreamMER is constructed from the first two seasons of Friends by segmenting episodes into locally complete interaction scenes and annotating utterances with multimodal contextual evidence. Its streaming evaluation uses 815 utterances with prefix-video assets and selectively requests contextual reinterpretation when belief trajectories appear unreliable.

  • Data source and segmentation: StreamMER segments the first two seasons of Friends into coherent narrative units preserving preceding dialogue, speaker relations, and local emotional development.Original videos, scripts, and subtitles support segmentation into locally complete interaction scenes.
  • Annotation protocol: Each utterance is annotated with its speaker, dialogue, timestamp, emotion label, context summary, rationale, and audio/visual evidence.Annotations use the video segment, aligned dialogue, and local script context.
  • Human verification: Initial annotations are generated with Gemini-3.1-Pro, manually checked by three experienced graduate annotators, and finalized by majority vote.Verification covers labels, timestamps, speakers, rationales, and audio/visual evidence against the available multimodal context.
  • Data release and copyright: The released artifact contains derived annotations and metadata rather than copyrighted videos, subtitles, scripts, screenshots, or other original media.The metadata includes episode and timestamp references, utterance-level labels, speaker identifiers, context summaries, rationales, and provenance-related evidence.
  • Streaming evaluation split: 815 utterances with prefix-video assets form the streaming test split, where partial audio prefixes are used online and contextual reinterpretation is requested selectively for unreliable belief trajectories.The evaluation separates Online Prefix Belief Formation from selectively invoked Contextual Belief Reinterpretation.

B Additional Analysis of Belief Trajectories

Belief trajectories provide dynamic reliability signals beyond single-prefix confidence. As observation grows, confidence rises and entropy falls overall, yet accuracy can decline at intermediate prefixes and vary sharply with label switching.

  • Trajectory dynamics: Figure 7 shows confidence generally increasing and entropy decreasing with longer prefixes, while accuracy is not strictly monotonic.At some intermediate observation lengths, confidence increases even as accuracy drops, so a single confident prefix can remain unreliable.
  • Trajectory dynamics: 72.6% accuracy without label switches falls to 38.3% with one switch and 24.1% with three switches, showing switching predicts trajectory instability.Online Prefix Belief Formation is substantially less accurate as predicted labels switch more often.
  • Trajectory dynamics: Contextual Belief Reinterpretation provides larger correction benefits for unstable trajectories, including 74 helpful corrections for samples with one switch.This supports using trajectory stability to determine when contextual reinterpretation is needed.
  • Trajectory dynamics: Trajectory information reveals whether beliefs form stably, competing labels remain unresolved, and contextual reinterpretation is needed.Thus, trajectories are useful reliability signals rather than merely intermediate prediction records.

C Effect of Context Scope in Contextual Belief Reinterpretation

Context scope strongly affects contextual belief reinterpretation on StreamMER: Previous + Current substantially outperforms Current-only and remains preferable to adding future context. It also preserves real-time processing despite higher latency.

  • Recognition performance: Previous + Current + Next reaches 68.83 Accuracy, underperforming Previous + Current because future context can add distracting or irrelevant evidence.Future information is also incompatible with the online streaming protocol, motivating Previous + Current as TRACE’s default scope.
  • Inference efficiency: Previous + Current + Next costs nearly the same as Previous + Current—0.8999s latency and 0.2058 RTF—yet performs worse, indicating a relevance rather than computational limitation.Combining recognition and efficiency results, Previous + Current offers the best overall trade-off by improving accuracy while preserving real-time inference.
  • Recognition performance: Previous + Current achieves 70.18 Accuracy, 70.00 W-F1, and 58.29 UAR, improving substantially over Current-only’s 62.53 Accuracy.Table 9 shows that preceding dialogue provides important emotional cues when the target clip alone is insufficient.
  • Inference efficiency: Previous + Current increases average latency from 0.3636s to 0.8959s but remains real-time at 4.8809 video seconds per second and 0.2049 RTF.Its RTF remains below 1 despite the added contextual processing cost.

D Ablation of Rationale and Memory Summary

TRACE’s contextual reinterpretation benefits from both explicit classification rationales and compact memory summaries. Removing either component degrades performance, with rationale generation causing the larger drop in overall recognition metrics.

  • Ablation of Rationale and Memory Summary: Removing rationale generation reduces Accuracy to 63.31%, W-F1 to 61.84%, and UAR to 55.31%.This is the larger degradation among the two ablations and indicates that explicit explanations help ground emotion labels in multimodal and contextual evidence.
  • Ablation of Rationale and Memory Summary: Removing memory summary reduces Accuracy from 70.18% to 66.01% and W-F1 from 70.00% to 65.69%, while UAR remains 61.81%.The reduction indicates that preceding conversational state helps resolve current-utterance ambiguity, although some minority-class sensitivity is preserved while overall recognition reliability declines.
  • Ablation of Rationale and Memory Summary: Jointly predicting the emotion label, generating a rationale, and maintaining contextual memory yields the strongest reinterpretation performance.The rationale grounds decisions in audio-visual cues, dialogue semantics, and contextual evidence, while memory preserves preceding conversational state.

E Implementation Details … H Complete Prefix Spectrogram Visualization

TRACE combines a low-latency prefix belief tracker, trajectory-calibrated reliability trigger, and selective multimodal reinterpretation, with implementation controls for fair evaluation and leakage prevention. Its feature design and prompts operationalize streaming evidence, while spectrograms support early audio prefixes as the primary signal.

  • E Implementation Details: TRACE uses Qwen2.5-Omni-3B for online prefix beliefs and Qwen2.5-Omni-7B for contextual reinterpretation, with LoRA fine-tuning for 10 epochs.LoRA uses rank 16, α = 32, and dropout 0.05, applied to attention and feed-forward projections.
  • E Implementation Details: Audio prefixes begin at 1.5 seconds and extend with a 1.0-second stride, while every prefix is supervised using the current utterance emotion label.Audio is resampled to 16 kHz, and complete utterances are retained as full-prefix training samples.
  • F Feature Details: The reliability trigger combines confidence, entropy, margins, temporal changes, JS divergence, label switches, persistence, entropy AUC, observation ratio, and class-probability features.These features distinguish confident stable beliefs from confident unstable ones when deciding whether to request contextual reinterpretation.
  • E Implementation Details: Reliability training uses out-of-fold training predictions and excludes test examples from fitting, threshold selection, policy calibration, and hyperparameter choice.Experiments use predefined or official dataset splits, supporting leakage-controlled evaluation.
  • F Feature Details: Trajectory features characterize current uncertainty, temporal fluctuation, distributional stability, label persistence, accumulated entropy, and progress from partial to nearly complete observation.High entropy AUC indicates sustained uncertainty, while larger JS divergence indicates substantial belief changes after new evidence arrives.
  • G Prompt Templates: The prompts restrict online prediction to observed prefixes and prioritize vocal cues, while contextual reinterpretation integrates current audio, video, dialogue, trajectory, and auxiliary preceding context.When modalities disagree, the contextual prompt favors current target audio for affective state unless it is corrupted, silent, off-screen, or from another speaker.
  • H Complete Prefix Spectrogram Visualization: Figure 8 shows category-dependent energy distribution, spectral structure, and temporal rhythm in short audio prefixes, supporting streaming audio as the primary low-latency signal.The visualization complements Figure 3 by providing a more complete view of early acoustic evidence across emotion categories.
Loading 2608.26786v1…