Source-linked AI summary
Query-Dependent Video Representation for Moment Retrieval and Highlight Detection
WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, Jae-Pil Heo
TL;DR
MR/HD must localize query-relevant moments and estimate clip-wise saliency, yet transformer methods may not fully exploit query information. QD-DETR builds query-dependent video representations with cross-attention, negative-pair learning, and an input-adaptive saliency predictor. It outperforms state-of-the-art methods across QVHighlights, TVSum, and Charades-STA, while relying on meaningful query context.
Problem
Transformer-based MR/HD methods may underuse text-query information when predicting moments and saliency, despite the tasks' dependence on query-video accordance.
Method
QD-DETR uses early cross-attention, low-saliency training on irrelevant video-query pairs, and a saliency-token-based input-adaptive predictor to build query-dependent video representations.
Results
QD-DETR outperforms state-of-the-art methods on QVHighlights, TVSum, and Charades-STA, including a 4.2% average improvement over the previous SOTA model for highlight detection.
Takeaways & Limitations
The studies verify the importance of building query-dependent representations for MR/HD and emphasizing descriptive contexts in text queries.
Takeaways & Limitations
The approach expects meaningful query context, and training may be ineffective with noisy, mismatched, or irrelevant ground-truth texts.
Abstract
from arXiv · showhide
Recently, video moment retrieval and highlight detection (MR/HD) are being spotlighted as the demand for video understanding is drastically increased. The key objective of MR/HD is to localize the moment and estimate clip-wise accordance level, i.e., saliency score, to the given text query. Although the recent transformer-based models brought some advances, we found that these methods do not fully exploit the information of a given query. For example, the relevance between text query and video contents is sometimes neglected when predicting the moment and its saliency. To tackle this issue, we introduce Query-Dependent DETR (QD-DETR), a detection transformer tailored for MR/HD. As we observe the insignificant role of a given query in transformer architectures, our encoding module starts with cross-attention layers to explicitly inject the context of text query into video representation. Then, to enhance the model's capability of exploiting the query information, we manipulate the video-query pairs to produce irrelevant pairs. Such negative (irrelevant) video-query pairs are trained to yield low saliency scores, which in turn, encourages the model to estimate precise accordance between query-video pairs. Lastly, we present an input-adaptive saliency predictor which adaptively defines the criterion of saliency scores for the given video-query pairs. Our extensive studies verify the importance of building the query-dependent representation for MR/HD. Specifically, QD-DETR outperforms state-of-the-art methods on QVHighlights, TVSum, and Charades-STA datasets. Codes are available at github.com/wjun0830/QD-DETR.
1. Introduction
Video moment retrieval and highlight detection seek query-relevant moments and clip-level saliency, but existing transformer methods may underuse the text query. QD-DETR builds query-dependent video representations through cross-attention, negative-pair learning, and adaptive saliency prediction.
- Motivation: Video moment retrieval and highlight detection address the need to locate requested or highlighted moments in increasingly prevalent but time-consuming video data.QVHighlights jointly supports moment retrieval and highlight detection by pairing human-written queries with moments and clip-level saliency labels.
- Research gap: Recent transformer approaches fuse text and video effectively but may overlook the text query's role in predicting query-relevant moments and saliency.The paper's investigation reports that existing methods may assign the query an insignificant role, producing equivalent low saliency for positive and negative queries in ground-truth moments.
- QD-DETR: QD-DETR injects query context into video representations by placing cross-attention layers at the beginning of its transformer encoder.Video features act as queries, while text features provide keys and values, enforcing text-query engagement during video representation extraction.
- QD-DETR: Negative video-query pairs are created by mixing original pairs and trained with low saliency scores to strengthen prediction dependence on query-video relevance.The same video is therefore encouraged to receive different predictions for relevant and irrelevant queries.
- QD-DETR: An input-adaptive saliency predictor dynamically defines the criterion for assigning saliency scores to each video-query instance.The saliency token is introduced to preserve query-dependent variation in the saliency prediction process.
2. Related Work
MR localizes moments matching text descriptions, while HD estimates clip importance; QVHighlights brings both tasks together with query-linked saliency labels. QD-DETR extends this detection-transformer setting with query-dependent encoding and adaptive saliency processing.
- Moment Retrieval and Highlight Detection: Moment retrieval localizes the moment relevant to a given text description through cross-modal interaction or temporal video-context modeling.Some approaches also use subtitles or target faster inference.
- Moment Retrieval and Highlight Detection: Highlight detection measures clip-wise importance and includes supervised, weakly supervised, and unsupervised settings with different annotation requirements.Supervised highlight detection uses fine-grained scores described as expensive to collect and annotate.
- Joint MR/HD: QVHighlights combines moment retrieval and highlight detection using human-written text queries, corresponding moments, and clip-level saliency labels.Moment-DETR introduced the dataset and a modified DETR for jointly localizing query-relevant moments and saliency scores; UMT adds video and audio features.
- Detection Transformers: DETR-like architectures simplify prediction by eliminating anchor generation and non-maximum suppression, despite DETR's slow convergence.These architectures have become popular in image and video tasks.
- Detection Transformers: QD-DETR follows the Moment-DETR design while transforming video features into query-dependent representations with cross-attention and irrelevant-pair training.Its architecture also incorporates an input-adaptive saliency mechanism through a saliency token.
3. Query-Dependent DETR
QD-DETR builds video representations conditioned on the text query, then trains and scores them so retrieved moments and saliency depend on video-query relevance. Its design combines cross-attentive encoding, negative-pair learning, an input-adaptive saliency predictor, and transformer-based moment decoding.
- Query-dependent representation: QD-DETR produces query-dependent video representations so each clip can be conditionally assessed against the text query for moment retrieval and highlight detection.The model targets clip-wise saliency and moment localization using video and text representations.
- Cross-attentive transformer encoder: Cross-attention is placed at the encoder’s first layers, using projected video clips as queries and projected text features as keys and values.The resulting attention integrates text context into the original video representations.
- Learning from negative relationship: Negative pair learning mixes videos and queries from different pairs and suppresses saliency scores for the resulting irrelevant pairs.This forces the same video to receive different predictions for positive and negative queries rather than relying only on inter-clip relationships.
- Input-adaptive saliency predictor: The input-adaptive saliency predictor appends a learnable saliency token to query-dependent video tokens and derives scores from their projected scaled dot product.Transformer encoding reorganizes the saliency token using input-dependent context, producing pair-specific prediction criteria.
- Decoder and objectives: The decoder represents video moments with center coordinate mc and duration mσ, while highlight detection combines margin ranking and rank-aware contrastive losses.The margin loss ranks high-saliency clips above low-rank clips inside and outside the ground-truth moment; contrastive learning refines segmented saliency levels.
4. Evaluation
QD-DETR is evaluated on moment retrieval and highlight detection across QVHighlights, TVSum, and Charades, with ablations examining query-dependent components and saliency behavior.
- QVHighlights: QD-DETR outperforms state-of-the-art approaches on all evaluation metrics for jointly learned moment retrieval and highlight detection in QVHighlights.The evaluation uses recall@1 and mAP for moment retrieval, and mAP and HIT@1 for highlight detection.
- QVHighlights: 36% gains in R1@0.7 and mAP@0.75 are reported over previous state of the art among video-only methods.With video and audio sources, QD-DETR improves moment-retrieval metrics by 11.84% on average over the multimodal state-of-the-art method.
- Charades: QD-DETR surpasses existing state-of-the-art methods for every evaluated feature type on the Charades dataset.Comparisons are conducted separately within VGG, C3D, and SF+C feature backbones using prior data-preparation settings.
- TVSum: 4.2% average improvement over the previous state of the art establishes new highlight-detection results across video-only and multimodal settings.QD-DETR outperforms baselines in 9 of 10 categories with video only and 8 of 10 with video and audio.
- Ablation study: CATE, negative-pair learning, and the input-adaptive saliency predictor benefit both moment retrieval and highlight detection, whereas DAM primarily improves moment retrieval.The ablation identifies CATE as enforcing text-query contributions, negative-pair learning as exploiting query context, and adaptive prediction as replacing a fixed MLP criterion.
- Ablation study: CATE improves query-dependent representation beyond merely adding encoder layers, while its combination with negative-pair learning separates positive and negative saliency distributions.CATE is more beneficial than an equal-depth self-attention encoder; adaptive prediction further makes positive-query scores nearly optimal.
5. Limitation and Conclusion
QD-DETR addresses the underused role of text queries in MR/HD through query-dependent representations, negative-pair training, and input-adaptive saliency prediction. The authors report superior performance in extensive experiments but caution that noisy or irrelevant text descriptions can reduce training effectiveness.
- Limitation: Noisy, mismatched, or irrelevant ground-truth text descriptions may make QD-DETR training less effective.The proposed components expect the provided query to maintain meaningful context.
- Conclusion: QD-DETR uses a cross-attentive encoder, negative-pair training, and an input-adaptive saliency predictor to make predictions depend on text queries.The encoder injects query context into video representations, irrelevant pairs receive low saliency targets, and the saliency criterion adapts to each video-query pair.
- Conclusion: Extensive experiments validated QD-DETR with superior performances.
6. Training Details
The experiments use a standardized transformer configuration while varying feature sources and training settings across QVHighlights and Charades-STA. The encoder has four layers, with cross-attention followed by self-attention, and the decoder has two layers.
- Model configuration: The encoder uses 4 transformer layers—2 cross-attention and 2 self-attention—while the decoder uses 2 layers.For the TVSum highlight-detection dataset, only encoding layers are used.
- Optimization: The transformer hidden dimension is 256, and Adam optimization uses weight decay 1e-4.
- QVHighlights: QVHighlights uses pretrained SlowFast and CLIP video features with CLIP text embeddings, trained for 200 epochs with batch size 32 and learning rate 1e-4.
- Charades-STA: Charades-STA uses official VGG video features and GloVe text embeddings, with additional tests using pretrained C3D, SlowFast, and CLIP features.
7. Further study on model performance on varying lengths of the query.
The study examines QD-DETR across short-, medium-, and long-length queries on QVHighlights. It reports robustness to meaningless words, while irrelevant or wrong descriptions can degrade effectiveness.
- Query-length evaluation: QD-DETR is evaluated separately on short, medium, and long queries for both moment retrieval and highlight detection.The validation set is divided into three query-length groups, with results reported in Table 6.
- Findings: Irrelevant or wrong text descriptions for video contexts can degrade QD-DETR’s effectiveness.
- Findings: QD-DETR is robust against meaningless words commonly present in text queries.