PaperlayerView exact revision on arXiv

Source-linked AI summary

UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection

Ye Liu, Siyuan Li, Yang Wu, Chang Wen Chen, Ying Shan, Xiaohu Qie

arXiv:2203.12745v2cs.CV

TL;DR

Video understanding requires retrieving query-relevant moments and detecting salient highlights, yet these tasks have rarely been jointly optimized. UMT unifies visual-audio-text modeling for joint or individual-task operation, and experiments across four datasets report effectiveness, superiority, and flexibility under varied settings.

  • Problem

    Retrieving relevant moments and detecting highlights are important video needs, but their joint optimization remains an emerging research problem.

  • Method

    UMT is a unified, flexible visual-audio-text framework that supports joint or individual-task optimization and models moment retrieval as keypoint detection.

  • Results

    Experiments on QVHighlights, Charades-STA, YouTube Highlights, and TVSum show UMT outperforms existing approaches across joint moment retrieval, moment retrieval, and highlight detection settings.

  • Takeaways & Limitations

    UMT provides one framework that remains effective across modality-quality variations, text-query conditions, and both joint and individual component tasks.

Abstract

from arXiv · show

Finding relevant moments and highlights in videos according to natural language queries is a natural and highly valuable common need in the current video content explosion era. Nevertheless, jointly conducting moment retrieval and highlight detection is an emerging research topic, even though its component problems and some related tasks have already been studied for a while. In this paper, we present the first unified framework, named Unified Multi-modal Transformers (UMT), capable of realizing such joint optimization while can also be easily degenerated for solving individual problems. As far as we are aware, this is the first scheme to integrate multi-modal (visual-audio) learning for either joint optimization or the individual moment retrieval task, and tackles moment retrieval as a keypoint detection problem using a novel query generator and query decoder. Extensive comparisons with existing methods and ablation studies on QVHighlights, Charades-STA, YouTube Highlights, and TVSum datasets demonstrate the effectiveness, superiority, and flexibility of the proposed method under various settings. Source code and pre-trained models are available at https://github.com/TencentARC/UMT.

1. Introduction

Video’s information richness creates costly demands for locating relevant moments and quickly scanning highlights. UMT addresses these related needs with a unified, flexible multimodal framework evaluated across joint and individual-task settings.

  • Video’s visual, audio, and linguistic richness makes finding relevant moments and scanning highlights costly in the content-explosion era.
  • Moment retrieval and highlight detection are closely related tasks, but joint study only recently emerged with QVHighlights and Moment-DETR.
  • UMT integrates visual, audio, and text learning in one flexible framework that handles missing, unreliable, or noisy modalities.
  • The framework supports joint optimization and degeneration to individual tasks, including highlight detection when text is unavailable.
  • Experiments cover QVHighlights, Charades-STA, TVSum, and YouTube Highlights, with comparisons against state-of-the-art methods and component ablations.

2. Related Works

Related work spans moment retrieval, highlight detection, text-guided video understanding, and visual-audio fusion. UMT differs by supporting unreliable or absent text and by introducing multimodal learning for joint moment retrieval and highlight detection.

  • Moment retrieval locates video moments from natural-language queries, while highlight detection identifies salient segments across domains such as sports, social media, and first-person video.
  • QVHighlights extends prior moment retrieval by annotating multiple query-relevant moments and supports text-guided highlight detection.
  • Text queries are essential for moment retrieval but uncommon in highlight detection, where UMT supports both absent and unreliable queries.
  • Figure 2 depicts a flexible architecture that deactivates unavailable modality encoders and substitutes learnable moment queries when text is absent.
  • Prior multimodal highlight detectors fuse visual and audio features, but UMT is presented as the first multimodal framework for joint moment retrieval and highlight detection.

3. Method

UMT uses a transformer encoder-decoder with uni-modal and cross-modal processing, adaptive moment queries, and prediction heads for joint moment retrieval and highlight detection.

  • Overview: UMT processes video and text with separate pretrained feature extractors, representing visual, audio, and textual inputs as feature collections.Visual and audio features are contextualized by separate uni-modal encoders before cross-modal fusion.
  • Query Decoder and Prediction Heads: The query decoder preserves encoder sequence length and feeds two prediction heads that produce moment retrieval and clip-level highlight detection outputs.The decoder enables saliency prediction and formulates each moment through its temporal center and duration; additional heads predict center offsets.
  • Uni-modal Encoder: Uni-modal encoders add global temporal context, addressing the local correlations produced by sliding-window video and audio feature extractors.Self-attention aggregates global context into each clip, followed by a two-layer feed-forward projection.
  • Cross-modal Encoder: The bottleneck transformer compresses visual-audio information into fewer bottleneck tokens, then expands it back into each modality with reduced computation.It separates cross-modal feature compression and expansion; the number of bottleneck tokens is much smaller than the number of video clips.
  • Query Generator: A query generator creates temporally aligned moment queries from joint visual-audio representations and textual features, or from joint representations plus positional encodings without text.Joint representations act as queries while textual features provide keys and values when text is available.

4. Experiments

Experiments evaluate UMT across joint and individual retrieval or highlight-detection settings, showing strong performance, benefits from multimodal and multitask learning, and flexibility across query conditions.

  • Datasets and Experimental Settings: QVHighlights provides joint annotations for moment retrieval and highlight detection, while Charades-STA, YouTube Highlights, and TVSum evaluate individual tasks.QVHighlights contains 10,310 queries and 18,367 annotated moments; the other datasets support task-specific evaluation.
  • Joint Retrieval and Highlight Detection: UMT outperforms existing approaches, including Moment-DETR, on both moment retrieval and highlight detection on QVHighlights.The comparison holds with and without pre-training using automatic speech-recognition captions.
  • Individual Tasks: UMT performs better than previous methods on Charades-STA and surpasses video-only and existing multimodal methods on YouTube Highlights and TVSum.On Charades-STA, replacing audio with optical flow produces similar performance.
  • Ablation Studies: Multimodal learning generally improves performance across datasets and tasks by combining visual and audio information.Compared with Moment-DETR+, UMT is reported to better exploit complementary modality information while suppressing noise during transfer.
  • Ablation Studies: Joint co-optimization produces both tasks simultaneously and significantly improves moment retrieval compared with individual-task training.The paper attributes UMT’s retrieval advantage to keypoint-based moment modeling and clip-aligned text-guided decoder queries.
  • Ablation Studies: Relevant text queries improve highlight detection, with larger gains when query relevance is higher.The comparison includes weakly relevant TVSum queries and highly relevant QVHighlights queries.

5. Conclusion

The conclusion presents UMT as a unified framework for joint moment retrieval and highlight detection and for their individual component tasks. It emphasizes multimodal integration, effectiveness across datasets, robustness to modality quality, and flexibility under text-query conditions.

  • Conclusion: UMT unifies joint moment retrieval and highlight detection with solutions for their individual component problems.The paper describes it as the first framework for this unified setting.
  • Conclusion: The framework integrates multimodal learning and is evaluated against relevant methods on diverse public datasets and settings.The conclusion reports demonstrated effectiveness and superiority without specifying numerical results.
  • Conclusion: UMT is reported to handle modality-quality variations and different text-query conditions flexibly.This includes settings where text queries may vary in availability or relevance.

Appendix

The appendix supplements the main paper with architectural and implementation descriptions, additional ablations, and QVHighlights visualizations.

  • Appendix: The appendix adds model architecture and implementation details beyond the main paper.It also incorporates additional ablation studies and QVHighlights visualization.

A. Model Architecture

The architecture uses positional encodings and removes unnecessary cross-modal components in single-modality or no-text settings. Without text, visual-audio representations become moment queries.

  • Positional Encodings: Learnable positional encodings with 0.1 dropout are applied differently across uni-modal encoders, cross-modal encoders, and the query decoder.They are added to specified query and key matrices according to each module’s feature-compression or expansion operation.
  • Modality Configurations: Cross-modal encoders are removed for video-only or audio-only schemes, while their normalization is moved to the corresponding uni-modal encoders.This adapts the architecture to settings without cross-modal inputs.
  • Query Generation: When text queries are unavailable, joint visual-audio representations plus learnable positional encodings construct the moment queries.The query generator outputs the joint representations for this no-text setting.

B. Implementation Details

The implementation uses Gaussian heatmap supervision for moment retrieval, adaptive window parameters, and ASR-caption pre-training, alongside reported fusion-comparison and token-ablation setups.

  • Moment retrieval supervision: Each ground-truth moment establishes a 1D Gaussian kernel on the temporal heatmap using its quantized center and window.The kernel radius and standard deviation are window-adaptive parameters.
  • Moment retrieval supervision: The moment-retrieval hyperparameters are fixed at 0.2, and testing treats all clips as centers to increase recall.The reported retrieval performance is insensitive to these hyperparameters.
  • Pre-training: ASR-caption pre-training disables saliency loss, uses batch size 2048, and runs for 100 epochs to reduce overfitting.The paper reports faster convergence than Moment-DETR under its novel moment-retrieval formulation.
  • Evaluation setup: The cross-modal fusion comparison evaluates moment retrieval and highlight detection across YouTube Highlights, TVSum, and QVHighlights validation data.MR denotes moment retrieval and HD denotes highlight detection.
  • Evaluation setup: The bottleneck-token ablation evaluates token-count effects on QVHighlights validation using mAP for both moment retrieval and highlight detection.The table defines MR and HD as moment retrieval and highlight detection, respectively.

C. Ablation Studies

The ablations support bottleneck-transformer fusion over simple fusion operations and report little sensitivity to the number of bottleneck tokens.

  • Cross-modal fusion: Bottleneck-transformer fusion improves both moment retrieval and highlight detection over simple cross-modal fusion operations across multiple datasets.Table A compares bottleneck tokens with baseline fusion methods.
  • Bottleneck-token count: Performance is insensitive to the number of bottleneck tokens on QVHighlights validation.The paper attributes this to feature compression and expansion eliminating undesirable noise.

D. Visualization

Qualitative results show complementary visual and audio contributions, while failure cases expose weaknesses in temporal, action-level, and abstract-language understanding.

  • Complementary modalities: Combining visual and audio information improves both moment retrieval and highlight detection relative to using either modality alone.Video-only and audio-only models fail on different aspects of the queried moment, including refinement and boundary discrimination.
  • Failure cases: The model fails to interpret the temporal clause “after a tiring trip” and instead retrieves moments matching the subjects and setting.The reported failure highlights incomplete understanding of compositional text queries.
  • Failure cases: Similar visual appearances provide few clues for separating “shot” from other actions, while abstract words remain difficult to comprehend.The failure analysis reports better understanding of nouns than abstract words and suggests a stronger language model may help.
  • Visualization: Figure A visualizes UMT predictions on the QVHighlights validation split for queries involving cats, family conversation, and listening to audio.The figure includes multiple query-specific examples of retrieval and highlight outcomes.
  • Visualization: Figure B presents QVHighlights validation failures involving a mother’s trip, a cat getting a shot, and abstract or visually ambiguous queries.These cases are used to illustrate limitations in query comprehension and visual disambiguation.
Loading 2203.12745v2…