Source-linked AI summary
TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection
Hao Sun, Mingyao Zhou, Wenjing Chen, Wei Xie
TL;DR
Joint moment retrieval and highlight detection methods often use separate task heads and underuse the tasks’ reciprocity. TR-DETR combines multimodal alignment and visual refinement with task cooperation that passes information between retrieval and highlight prediction. It outperforms prior methods on the reported datasets, though its use of audio remains inefficient.
Problem
Joint MR&HD methods use separate task heads after shared feature extraction and interaction, leaving the reciprocity between the tasks underused.
Source: Introduction · p. 1Method
TR-DETR aligns multimodal features, refines visual features to remove query-irrelevant information, and uses task cooperation to exchange information between MR and HD.
Source: Introduction · p. 2Results
TR-DETR outperforms the current best approach on all metrics, with R1@0.7 and mAP@0.75 improvements of 3.98% and 3.75%, respectively.
Source: Comparison with Other Methods · p. 6Takeaways & Limitations
The experiments on QVHighlights, Charades-STA, and TVSum demonstrate the effectiveness of exploiting reciprocity between MR and HD.
Source: Introduction · p. 2, Conclusion · p. 8TR-DETR cannot efficiently utilize audio-modality data.
Source: Conclusion · p. 8
Abstract
from arXiv · showhide
Video moment retrieval (MR) and highlight detection (HD) based on natural language queries are two highly related tasks, which aim to obtain relevant moments within videos and highlight scores of each video clip. Recently, several methods have been devoted to building DETR-based networks to solve both MR and HD jointly. These methods simply add two separate task heads after multi-modal feature extraction and feature interaction, achieving good performance. Nevertheless, these approaches underutilize the reciprocal relationship between two tasks. In this paper, we propose a task-reciprocal transformer based on DETR (TR-DETR) that focuses on exploring the inherent reciprocity between MR and HD. Specifically, a local-global multi-modal alignment module is first built to align features from diverse modalities into a shared latent space. Subsequently, a visual feature refinement is designed to eliminate query-irrelevant information from visual features for modal interaction. Finally, a task cooperation module is constructed to refine the retrieval pipeline and the highlight score prediction process by utilizing the reciprocity between MR and HD. Comprehensive experiments on QVHighlights, Charades-STA and TVSum datasets demonstrate that TR-DETR outperforms existing state-of-the-art methods. Codes are available at \url{https://github.com/mingyao1120/TR-DETR}.
Introduction
Natural-language moment retrieval and highlight detection address related video-search needs, but joint methods largely leave their task reciprocity unused. TR-DETR is proposed to exploit that reciprocity through aligned features and task cooperation.
Introduction
- The growth in video volume and variety motivates methods that help users find desired moments and browse videos using natural-language queries.The paper identifies video complexity and diverse user needs as challenges for query-based MR and HD.Source: Introduction · p. 1
- MR searches videos for query-related moments, while query-based HD assigns each clip a highlight score according to user needs.MR commonly extracts video and text features before cross-modal interaction and moment scoring.Source: Introduction · p. 1
- Joint MR&HD methods share multimodal extraction and interaction but use isolated task heads, leaving the reciprocity between the tasks underused.MR highlight scores can help eliminate query-irrelevant clips, while retrieved moments can improve video and user-need understanding for HD.Source: Introduction · pp. 1–2
- TR-DETR combines local-global multimodal alignment, visual feature refinement, and task cooperation to exploit reciprocity between MR and HD.Its cooperation module injects highlight scores into retrieval and uses retrieved moments to refine highlight scores.Source: Introduction · p. 2
Related Works
Prior work addresses moment retrieval and highlight detection separately or jointly, and multimodal alignment methods model cross-modal correspondences. This paper is distinguished by its focus on exploiting reciprocity between the two tasks.
MR and HD
- Proposal-based MR ranks generated candidate moments but adds processing and depends on candidate quality, whereas proposal-free MR directly predicts moment boundaries.Proposal-free methods avoid ranking many candidates and improve training efficiency.Source: MR and HD · p. 2
- Joint MR&HD work includes DETR-based QVHighlights methods, audio for missing-query scenarios, and query-guided joint representations, while this paper focuses on task reciprocity.Earlier query-based HD approaches include text ranking without direct text-highlight alignment and graph-based clip-text interaction in thumbnail generation.Source: MR and HD · p. 2
Multi-Modal Alignment
- Multimodal alignment work uses contrastive or cycle-consistency losses to model cross-modal correspondences, but does not explicitly align modality semantics before interaction.Examples include clip-to-word cycle consistency and multi-level contrastive alignment for moment retrieval.Source: Multi-Modal Alignment · p. 2
Method
TR-DETR combines feature extraction, local-global multimodal alignment, query-guided visual refinement, and reciprocal cooperation between moment retrieval and highlight detection. Its pipeline uses task-specific signals to refine retrieval and highlight prediction.
Method
- TR-DETR has four core modules: feature extraction, local-global multimodal alignment, visual feature refinement, and task cooperation.The overview also notes that two multi-head self-attention components share weights.Source: Method · pp. 2–3, Figure 1 · p. 3
Feature Extraction
- Video clips are encoded with pre-trained visual networks and audio features, while a pre-trained CLIP text encoder represents the natural-language query.The video is divided into non-overlapping clips before visual feature extraction.Source: Visual Features · p. 3, Textual Features · p. 3
Local-Global Multi-Modal Alignment
- Local and global alignment regulators address visual-textual mismatch: local alignment distinguishes query-irrelevant clips, while global contrastive alignment brings paired video and query representations together.The global representations are formed by averaging clip and word-level features.Source: Local-Global Multi-Modal Alignment · pp. 3–4
Visual Feature Refinement for Modal Interaction
- Query-guided refinement suppresses irrelevant visual content while retaining temporal cues, then cross-attention incorporates query features into joint representations.The refinement uses clip-word similarities and combines clip-level visual and textual features.Source: Visual Feature Refinement for Modal… · p. 4
Task Cooperation
- Task cooperation uses highlight scores to weight joint features for retrieval, while retrieved moments and the query inform highlight-score refinement.The retrieval path re-encodes the highlight-weighted features before moment prediction.Source: Task Cooperation · p. 4, HD2MR · pp. 4–5, MR2HD · p. 5
Objective Losses
- The objective combines moment-retrieval and highlight-detection losses with local and global regulator losses.A coefficient weights the local-global regulator losses; the moment and highlight losses follow QD-DETR.Source: Objective Losses · p. 5
Experiment
Evaluation spans QVHighlights, Charades-STA, and TVSum, with joint-task and task-specific comparisons. Reported results include gains on QVHighlights and video-only TVSum, alongside comparable Charades-STA performance on VGG and SF+C features.
Datasets
- The evaluation uses QVHighlights’ 10,148 videos, Charades-STA’s 9,848 videos and 16,128 queries, and TVSum’s 10 categories of five videos each.QVHighlights test annotations are inaccessible, so predictions are assessed through the dataset’s CodaLab platform.Source: Datasets · p. 5
Metrics and Experimental Settings
- Metrics include QVHighlights Recall@1 and mAP at specified IoU thresholds plus HD mAP and HIT@1, Charades-STA Recall@1, and TVSum top-5 mAP.The experiments use dataset-specific feature extractors and training schedules; λlg is set to 0.3.Source: Metrics and Experimental Settings · p. 5
Comparison with Other Methods
- TR-DETR outperforms the current best approach on all QVHighlights metrics, improving R1@0.7 by 3.98% and mAP@0.75 by 3.75%.The comparison reports visual features only; adding audio decreases performance on a few indicators, which the authors attribute to directly spliced, misaligned features.Source: Table 1 · p. 6, Comparison with Other Methods · p. 6
- On Charades-STA, TR-DETR performs comparably with VGG and SF+C features, while some metrics decline when audio is added.The authors suggest that aligning VGG visual, PANN audio, and GLoVe text features is challenging because their semantics differ across feature extractors.Source: Comparison with Other Methods · pp. 6–7
- Video-only TR-DETR exceeds the previous method by approximately 3.1% on the average top-5 mAP across TVSum’s 10 categories.The evaluation also reports top-5 mAP for each video category.Source: Comparison with Other Methods · p. 7, Table 3 · p. 8
Visualization
- Qualitative examples show more accurate retrieval and highlight-score distributions than QD-DETR on QVHighlights, while TVSum includes both a well-fit score distribution and an unreasonable puppy-focused result.The authors attribute the reported improvements to the combination of proposed modules.Source: Figure 2 · p. 6, Visualization · p. 7, Figure 3 · p. 7
Ablation
- The ablations use module combinations to test task reciprocity; setting (i) shows a large improvement from combining local-global alignment loss with visual feature refinement.Table 4 compares combinations against a Moment-DETR baseline with cross-attention and a DAB-DETR decoder.Source: Ablation · p. 8, Table 4 · p. 8
- Adding the local-global regularization term improves performance by about 5%; λlg=0.3 is the peak before performance declines slowly, with about 7% total improvement over λlg=0.The coefficient ablation is conducted on QVHighlights while introducing audio features.Source: Figure 4 · p. 7, Ablation · p. 8
Conclusion
TR-DETR combines local-global alignment, visual feature refinement, and reciprocal task cooperation for joint highlight detection and moment retrieval. Experiments on several datasets demonstrate its effectiveness, though the model cannot efficiently use audio data.
Conclusion
- Extensive experiments on several datasets demonstrate the effectiveness of TR-DETR.Source: Conclusion · p. 8
- TR-DETR aligns visual and textual features, refines joint features, and uses highlight scores and retrieved moments to improve the two task pipelines.Source: Conclusion · p. 8
- TR-DETR cannot efficiently utilize audio data, motivating future work on multimodal feature interaction networks.Source: Conclusion · p. 8