EviDETR: Preserving Query-Relevant Temporal Evidence for Moment Retrieval and Highlight Detection

1. INTRODUCTION

• Existing DETR-style joint MR/HD pipelines lack explicit mechanisms to preserve query-relevant temporal evidence through encoding, query refinement, and cross-task prediction.The cited gaps are irrelevant clips mixed during encoding, a shared FFN applied across temporal queries, and indirect transfer of span-level MR evidence to clip-level HD.Source: §1 · p. 1
• SFR estimates query–clip relevance and enhances relevant representations before decoding through semantic reweighting and cross-modal interaction.This targets query-conditioned emphasis during encoding.Source: §1 · pp. 1–2
• TTop2MoE routes each temporal query to a sparse subset of experts, with Top-2 aggregation combining two expert transformations for conditional refinement.This introduces query-specific adaptation where a standard decoder would apply the same feed-forward transformation across temporal queries.Source: §1 · pp. 1–2
• MR2HD transfers span-level localization evidence from MR through multi-scale temporal aggregation so retrieved moment cues directly guide clip-level saliency estimation.The framework thus explicitly connects retrieval evidence to highlight detection.Source: §1 · pp. 1–2
• On QVHighlights, EviDETR achieves 69.29 R1@0.5, 54.77 R1@0.7, and 48.41 Avg. mAP for MR, plus 41.83 HD-mAP and 68.33 HIT@1 for HD.The paper also reports leading or second-best results across QVHighlights metrics and strong cross-dataset performance on TACoS and Charades-STA.Source: §1 · p. 2

2. PROPOSED METHOD

• EviDETR first enhances query-relevant temporal representations, then applies query-adaptive sparse expert refinement and transfers moment-level localization evidence to clip-level highlight prediction.SFR builds query-aware representations before decoding; TTop2MoE uses sparse expert routing, and MR2HD carries retrieval evidence into highlight prediction.Source: §2 · p. 2

2.1. Semantic-aware Feature Reweighting

• SFR estimates query–clip relevance before deeper temporal modeling, reweights relevant clips, and uses tri-linear similarity and bidirectional video–text attention to form query-aware memory.The memory is used for subsequent decoding.Source: §2.1 · p. 2

2.2. Temporal Top-2 Mixture-of-Experts Decoder

• Top-2 routing combines two expert transformations for each temporal query, enabling complementary refinement while retaining sparse conditional computation across three sequential decoder stages.The design replaces a shared FFN that does not explicitly adapt to individual temporal queries.Source: §2.2 · pp. 2–3

2.3. MR-to-HD Evidence Fusion

• MR2HD uses predicted moments and foreground confidence to guide highlight detection, aggregating confidence-weighted span evidence at temporal scales 1, 2, and 4 before transferring it to clips.A GRU encodes the highest-confidence span, and a 1D ROIAlign adaptation converts predicted spans into fixed-length representations.Source: §2.3 · p. 3

2.4. Training Objective

• Hungarian matching and foreground classification, L1 span regression, and GIoU losses train moment retrieval; deep supervision applies the same MR loss to two intermediate decoder stages.Deep supervision provides direct optimization signals to earlier decoder stages.Source: §2.4 · p. 3

3.1. Experimental Setup

• QVHighlights jointly evaluates MR and HD, whereas TACoS and Charades-STA mainly evaluate temporal moment localization.All three datasets use CLIP+SlowFast features; QVHighlights results use validation, and the cross-dataset results use test splits for EviDETR.Source: §3.1 · p. 3

3.2. Main Results

• 69.29 R1@0.5, 54.77 R1@0.7, 48.41 Avg. mAP, 41.83 HD-mAP, and 68.33 HIT@1; EviDETR ranks first on both recall and both HD metrics, and second on the three mAP metrics.At R1@0.7, it exceeds second-best TD-DETR by 1.10 percentage points.Source: §3.2 · p. 3, Table 1 · p. 4
• 44.09 R1@0.5 and 26.94 R1@0.7 on TACoS, and 60.28 R1@0.5 and 40.13 R1@0.7 on Charades-STA, exceed the listed baselines on both datasets.EviDETR is evaluated on each dataset's test split, while baselines follow their reported splits.Source: Table 2 · p. 4, §3.2 · p. 4

3.3. Ablation Studies

• SFR raises R1@0.7 from 29.61 to 35.61 over the baseline, while adding MR2HD raises it to 54.77 and HIT@1 from 65.80 to 68.33.The baseline omits SFR and MR2HD and uses a decoder without multi-stage refinement, auxiliary supervision, or MoE routing.Source: §3.3 · p. 4
• The full TTop2MoE decoder design raises R1@0.7 to 52.45, but its gain includes three-stage refinement and auxiliary supervision as well as sparse routing.The component-ablation TTop2MoE label denotes the full three-stage decoder design.Source: §3.3 · p. 4
• Doubling shared FFN width from 1024 to 2048 raises R1@0.7 from 46.31 to 53.68, explaining a substantial part of the decoder gain; at approximately matched active capacity, Top-2 reaches 54.77 versus 53.68.The comparison supports an additional benefit beyond simply widening the dense FFN.Source: §3.3 · p. 4
• Top-1 routing reduces R1@0.7 to 51.68 and HIT@1 to 65.68, whereas MR2HD further improves HIT@1 from 65.80 to 68.33.TTop1MoE and TTop2MoE use the same eight-expert pool.Source: §3.3 · p. 4

• EviDETR combines SFR, TTop2MoE, and MR2HD to enhance query-relevant representations, refine queries conditionally, and transfer retrieval evidence to highlight detection.Ablations further validate the proposed components and decoder design.Source: §4 · p. 4
• Experiments demonstrate strong joint moment retrieval and highlight detection performance on QVHighlights.Source: §4 · p. 4
• Results on TACoS and Charades-STA show consistent effectiveness across different temporal grounding benchmarks.Source: §4 · p. 4