Source-linked AI summary
TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection
Bo-Yuan Cheng, Kuan-Yu Chen, Po-Han Huang, Jeng-Lin Li, Jian-Jiun Ding
TL;DR
Multimodal highlight detectors assume continuously available streams, although real videos can have localized frame loss or complete-stream outages, and MSE does not directly optimize ranking. TSMD combines structured dropout with correlation- and peak-oriented losses; across evaluated conditions, its specialized and mixed variants perform robustly, with TSMD-Mix ranking best or second-best.
Problem
Multimodal highlight detectors generally assume complete, uninterrupted streams, despite localized frame corruption and full-stream outages; pointwise MSE also does not explicitly optimize relative ordering or top-segment retrieval.
Source: §1 · p. 1Method
TSMD simulates temporal, complete-stream, or mixed missingness and combines pointwise MSE with per-video Pearson correlation and peak-oriented RankNet losses.
Source: §1 · p. 2Results
Takeaways & Limitations
Robustness is strongest when training corruption matches the test failure, while TSMD-Mix provides a robust compromise when the failure type is unknown at deployment.
Source: §3.3.3 · p. 4The reported missingness scheme uses zero-masking; future work will extend to feature noise, transcript errors, and cross-stream misalignment.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Existing multimodal video highlight detectors typically assume that visual, audio, and textual streams are continuously available. In practice, however, inputs may suffer from localized frame missingness or complete-stream outage. We formulate this robustness challenge along two dimensions: temporal missingness, where frames are missing independently in each modality, and stream-level missingness, where one modality is unavailable throughout a video. Moreover, we find that the mean squared error (MSE) loss is misaligned with both the evaluation metrics and the peak-driven nature of highlights. Therefore, we propose Temporal-Stream Modality Dropout (TSMD), which combines structured missingness simulation with a joint objective comprising pointwise MSE, per-video Pearson correlation, and peak-oriented RankNet loss terms. TSMD has three variants: temporal, stream-level, and mixed dropout. On the MoSu and Mr. HiSum datasets, TSMD-Temporal improves mAP@15 by 7.06 and 3.41 points over TripleSumm under 50% independent temporal removal, whereas TSMD-Stream performs the best under complete-stream removal. TSMD-Mix retains most of these complementary benefits and ranks the best or the second-best across the evaluated temporal and stream-level conditions.
1. INTRODUCTION
The introduction frames robust video highlight detection around localized, asynchronous modality corruption and complete-stream outages, alongside a mismatch between pointwise MSE training and ranking-based evaluation. TSMD combines structured missingness simulation with an objective aligned to salience profiles and highlight ordering, and evaluates temporal, stream-level, and mixed policies.
1. INTRODUCTION
- Real inputs can lose frames asynchronously or an entire modality, and TripleSumm loses 6.67 and 5.92 mAP@15 on MoSu under temporal and stream removal, respectively.Examples include packet loss, degraded visual features, incorrect transcripts, or an unavailable audio track or transcript.Source: §1 · p. 1
- Prior missing-modality methods do not target dense video highlight detection or examine how temporal and complete-stream dropout policies interact during training.The introduction contrasts those settings with prior gesture, classification, emotion, action, and speech work.Source: §1 · p. 1
- Pointwise MSE does not explicitly optimize relative ordering or top-segment retrieval, while prior ranking-loss studies use presegmented clips rather than dense frame-level salience on a 1-Hz grid.The introduction motivates balancing correlation-based and peak-oriented metrics.Source: §1 · pp. 1–2
- TSMD combines temporal, stream-level, or mixed missingness simulation with MSE, per-video Pearson correlation, and peak-oriented RankNet loss, without adding parameters or inference computation.The mixed variant allocates corrupted training examples equally between temporal and stream-level regimes.Source: §1 · p. 2
- TSMD-Temporal gains up to 7.06 mAP@15 over TripleSumm under 50% temporal removal, while TSMD-Stream excels under stream removal and TSMD-Mix ranks best or second-best across conditions.The reported evaluations cover MoSu and Mr. HiSum.Source: §1 · p. 2
2. METHOD
The method treats multimodal highlight detection as sequence-to-sequence regression and addresses ranking-oriented supervision and missing inputs through a metric-aligned objective and structured modality dropout.
2. METHOD
- A detector maps aligned visual, audio, and text feature sequences to frame-level salience scores, trained against per-timestep ground-truth scores.The feature sequences and target scores are defined on the same N-timestep video grid.Source: §2 · p. 2
2.1. Metric-Aligned Training Objective
- The objective combines pointwise MSE, per-video Pearson correlation, and RankNet supervision to align score magnitudes, salience profiles, and highlight-frame ordering.RankNet samples candidate frame pairs per video, including frames from the top 15% by ground-truth score, and retains pairs with sufficient score separation.Source: §2.1 · p. 2
2.2. Temporal and Stream-Level Modality Removal
- A unified availability mask zeroes selected modality features while preserving sequence length, timestamps, targets, and the padding mask.Temporal removal masks exactly round(rN) timesteps independently across modalities; stream removal zeros one modality across all timesteps.Source: §2.2 · p. 2
2.3. Temporal-Stream Modality Dropout (TSMD)
- TSMD-Temporal samples temporal masking ratios, TSMD-Stream selects clean or single-stream-outage instances, and TSMD-Mix allocates training examples across clean, temporal, and stream-dropout states.All variants retain 25% clean-data exposure; they share the joint objective and add no auxiliary head, architectural modification, or inference computation.Source: §2.1 · p. 2, §2.3 · p. 3
3. EXPERIMENTS
Experiments benchmark temporal and stream-level missingness on MoSu and Mr. HiSum, comparing the joint objective and TSMD dropout variants. Robustness is strongest when training corruption matches the test-time failure, while TSMD-Mix performs well across both regimes.
3.1. Datasets and Base Model
- MoSu provides 52,678 trimodal videos; Mr. HiSum’s trimodal version retains 30,452 accessible videos, and TripleSumm is used as the unchanged backbone.Models use released pre-extracted features and are trained across three random seeds; checkpoints are selected by validation τ + ρ on uncorrupted videos.Source: §3.1 · p. 3
3.2. Evaluation Metrics and Missingness Benchmark
- The evaluation uses Kendall’s τ and Spearman’s ρ for sequence alignment, and mAP@50 and mAP@15 for segment retrieval, with temporal removal swept from r = 0 to 0.5.Stream-removal results average across −V/−A/−T; temporal-removal evaluation uses three fixed mask seeds and averages over three training seeds.Source: §3.2 · pp. 3–4
3.3.3. Stream Removal and Cross-Regime Complementarity
- TSMD-Stream raises the worst missing-stream mAP@15 cases to 42.07 on MoSu and 38.62 on Mr. HiSum, with only a marginal decrease on Mr. HiSum −T.The clean-trained joint objective is most affected by visual absence on MoSu (36.37 mAP@15) and audio absence on Mr. HiSum (36.77 mAP@15).Source: Table 3 · p. 4, §3.3.3 · p. 4
3.3.1. Clean-Input Performance and Objective Selection
- Pearson loss improves τ and ρ by +0.008/+0.008 on MoSu and +0.008/+0.009 on Mr. HiSum; adding RankNet yields 45.82 and 41.81 mAP@15 peak retrieval.The MSE-only retraining gives the lowest mAP@15 among evaluated objectives on both datasets.Source: §3.3.1 · p. 4
- On clean inputs, the joint objective is second-highest in MoSu correlation and achieves the top Mr. HiSum peak-retrieval score, 42.31 mAP@15; TSMD variants match or exceed its MoSu mAP@15 and stay within 0.25 points on Mr. HiSum.Table 2 reports three-seed mean results for objective components under clean inputs.Source: §3.3.1 · p. 4, Table 2 · p. 4
3.3.2. Robustness Under Temporal Removal
- +7.06 MoSu and +3.41 Mr. HiSum mAP@15 over TripleSumm† under 50% independent temporal removal, with τ within 0.01 of clean values.Across the removal sweep, temporal-dropout training limits the mAP@15 reduction from r = 0 to r = 0.5 to less than 0.5 points under both objectives; the joint objective retains a 0.83–0.91-point advantage over MSE-only training.Source: §3.3.2 · p. 4
3.3.3. Stream Removal and Cross-Regime Complementarity
- Under complete-stream removal, TSMD-Stream ranks first on all eight metrics, beating TripleSumm† by 4.41 MoSu and 2.59 Mr. HiSum mAP@15.TSMD-Mix ranks second on all metrics under stream removal and remains within 0.24 mAP@15 of TSMD-Stream; under temporal removal, it matches TSMD-Temporal within 0.13 mAP@15.Source: §3.3.3 · p. 4
4. CONCLUSION
The evaluated TSMD variants address temporal and stream-level missingness, with each specialized variant strongest for its matched failure scale. TSMD-Mix remains among the best across both conditions without changing the architecture or inference procedure.
4. CONCLUSION
- Robustness is strongest when training corruption matches test-time failure: TSMD-Temporal for localized frame missingness and TSMD-Stream for complete-stream outages.Across evaluated conditions, TSMD-Mix is best or second-best without changing model architecture or inference procedure.Source: §4 · p. 4
- The evaluated conclusion covers MoSu and Mr. HiSum and zero-masking; future work targets feature noise, transcript errors, and cross-stream misalignment.Source: §4 · p. 4
- The study formulates missingness along temporal and stream-level dimensions.Source: §4 · p. 4