PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing Where to Focus: Event-aware Transformer for Video Grounding

Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, Kwanghoon Sohn

arXiv:2308.06947v1cs.CVcs.LG

TL;DR

DETR-based video grounding avoids hand-crafted proposals, but its input-agnostic moment queries overlook video-specific temporal structure. EaTR builds dynamic queries from event units and fuses them with sentence information; it reports state-of-the-art performance on three video grounding benchmarks.

  • Problem

    DETR-based video grounding uses input-agnostic moment queries that overlook a video's intrinsic temporal structure and provide limited positional information.

    Source: §1 · p. 2

  • Method

    EaTR identifies video event units with slot attention, uses them as dynamic moment queries, and applies gated fusion to combine sentence information before timestamp prediction.

    Source: §5 · p. 9

  • Results

    EaTR outperforms state-of-the-art approaches on QVHighlights, Charades-STA, and ActivityNet Captions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Event-derived queries provide video-specific referential search areas, while gated fusion enhances sentence-relevant queries and filters irrelevant ones.

    Source: §5 · p. 9

    The study explores dynamic moment queries based on visual information, while consideration of sentence information remains underexplored.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent DETR-based video grounding models have made the model directly predict moment timestamps without any hand-crafted components, such as a pre-defined proposal or non-maximum suppression, by learning moment queries. However, their input-agnostic moment queries inevitably overlook an intrinsic temporal structure of a video, providing limited positional information. In this paper, we formulate an event-aware dynamic moment query to enable the model to take the input-specific content and positional information of the video into account. To this end, we present two levels of reasoning: 1) Event reasoning that captures distinctive event units constituting a given video using a slot attention mechanism; and 2) moment reasoning that fuses the moment queries with a given sentence through a gated fusion transformer layer and learns interactions between the moment queries and video-sentence representations to predict moment timestamps. Extensive experiments demonstrate the effectiveness and efficiency of the event-aware dynamic moment queries, outperforming state-of-the-art approaches on several video grounding benchmarks.

1. Introduction

Video grounding localizes a sentence-relevant moment, but earlier approaches depend on hand-crafted proposals or use input-agnostic moment queries that provide fixed search areas. EaTR uses video-specific event units as dynamic queries and reports state-of-the-art results on three benchmarks.

1. Introduction

  • Video grounding localizes the timestamps of a moment corresponding to a natural-language sentence, while proposal-based approaches rely on hand-crafted components such as proposals and non-maximum suppression.The task requires aligning video-language information and reasoning about temporal areas.Source: §1 · p. 1
  • DETR-based methods replace heuristic proposals with learned moment queries, but their input-agnostic queries provide fixed search areas despite videos containing events of varying lengths.The queries also contribute equally during aggregation, which can miss salient information and slow convergence.Source: Figure 1 · p. 1, §1 · p. 2
  • EaTR treats video-specific event units as dynamic moment queries, identifying events with slot attention and using gated fusion to emphasize sentence-relevant queries before timestamp prediction.The fused queries interact with video-sentence representations in the transformer decoder.Source: §1 · p. 2
  • EaTR outperforms state-of-the-art approaches on QVHighlights, Charades-STA, and ActivityNet Captions.The paper presents extensive experiments on all three video grounding benchmarks.Source: §1 · p. 2

2. Related Work

Related work spans proposal-driven and proposal-free video grounding, DETR-based methods, event-boundary detection, and slot attention. EaTR draws on event grouping while using the video itself as positional guidance for its moment queries.

2. Related Work

  • Video grounding methods either generate and rank candidate proposals or directly predict moments through video-sentence interactions; both paradigms have used hand-crafted components or redundant proposals.Proposal-driven methods include sliding windows and temporal anchors, while proposal-free approaches use attention, dense predictions, or complementary visual features.Source: §2 · p. 2
  • Prior DETR-based video grounding methods use learnable moment queries, while EaTR uses the video itself as interpretable positional guidance rather than relying on extra modalities such as audio or optical flow.The paper frames ineffective use of moment queries as a limitation on temporal reasoning.Source: §2 · p. 2
  • DETR streamlines detection by removing hand-crafted components, but its object queries can converge slowly; spatial priors have been used to improve query design and cross-attention.Prior approaches encode coordinates or constrain cross-attention with sparse sampling, Gaussian priors, or conditional weights.Source: §2 · pp. 2–3
  • EaTR’s overview proceeds from feature extraction to feature interaction and event reasoning, then to moment reasoning that predicts timestamps.Slot-attention event outputs initialize moment queries, which are fused with sentence features before interacting with video-sentence representations.Source: Figure 2 · p. 3
  • Generic Event Boundary Detection identifies instants perceived as event boundaries, and EaTR adopts a contrastive kernel from UBoCo to generate pseudo event information while designing its own reasoning network.UBoCo uses boundary-sensitive temporal self-similarity features and recursive parsing.Source: §2 · p. 3
  • Slot attention groups input features into object-centric representations, and EaTR uses its grouping property to aggregate visually similar frames into events.The event reasoning network is trained with an event localization loss.Source: §2 · p. 3

3. Proposed Method

EaTR addresses ambiguous search areas from input-agnostic moment queries by deriving dynamic queries from video-specific event units and combining them with sentence information. Its method encodes video and sentence features, reasons over events, then refines and predicts moment spans through moment reasoning.

3.1. Background and motivation

  • Input-agnostic moment queries can make referential search areas ambiguous; EaTR uses video-specific event units to initialize dynamic queries and fuses them with sentence information.The queries are intended to provide a more precise referential search area.Source: §3.1 · p. 3

3.2. Feature extraction and interaction

  • Pretrained backbones encode video and sentence features, which interact through transformer encoder layers to form video-sentence representations and frame-level saliency scores.A saliency loss enlarges the score gap between sampled frames inside and outside the ground-truth interval.Source: §3.2 · p. 4

3.3. Event reasoning

  • N learnable event slots iteratively group visually similar frames; their outputs initialize content queries, while projected centers and widths initialize positional queries.Pseudo event timestamps are obtained by thresholding and sampling boundary scores from a temporal self-similarity matrix, and event localization uses Hungarian assignment; slot attention is reported to outperform cross-attention with higher efficiency.Source: §3.3 · pp. 4–5

3.4. Moment reasoning

  • Moment reasoning uses self-attention to enhance queries, gates them with sentence information to suppress irrelevant queries, then attends to video-sentence representations to predict spans and confidence scores.The decoder’s remaining layers update positional queries using offsets predicted from corresponding content queries.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate EaTR on QVHighlights, Charades-STA, and ActivityNet Captions, comparing models with the same feature representations for fairness. The analyses cover benchmark results, component and hyperparameter ablations, training convergence, and attention.

4.3. Comparison with state-of-the-art

  • On QVHighlights, EaTR establishes state-of-the-art results on several video-grounding metrics; for highlight detection, it exceeds Moment-DETR by 1.46% mAP and 3.05% HIT@1 but trails other approaches.Table 1 reports QVHighlights validation results, averaged over five seeds with standard deviations.Source: Table 1 · p. 6, §4.3 · p. 7
  • 2.88% and 3.69% R1@0.5 performance improvements are reported on Charades-STA and ActivityNet Captions, respectively, using fewer than 10 moment queries without hand-crafted components.Table 2 reports results on Charades-STA test with I3D features and ActivityNet Captions val 2 with C3D features.Source: Table 2 · p. 6, §4.3 · p. 7

4.1. Datasets and evaluation protocols

  • QVHighlights supports grounding and highlight detection, while evaluation uses Recall@1 at IoU 0.5 and 0.7 and, for QVHighlights, mAP and HIT@1.The official test splits lack QVHighlights ground truth, so evaluation uses its validation set; Charades-STA uses its test split and ActivityNet Captions uses val 2.Source: §4.1 · p. 6

4.2. Implementation details

  • Features come from SlowFast and CLIP for QVHighlights, I3D for Charades-STA, and C3D for ActivityNet Captions; models train for 200 epochs with batch size 32 and AdamW.The QVHighlights video features are sampled every two seconds; sentence features use token-level CLIP text features.Source: §4.2 · pp. 6–7

4.4. Ablation study and discussion

  • EaTR converges faster and reaches higher performance than Moment-DETR; its video-specific search areas also balance content and positional attention, unlike Moment-DETR’s fixed input-agnostic search area.Figure 4 compares five-seed mean performance and range using mAP and Lmoment; the attention visualization is in Figure 5.Source: Figure 4 · p. 7, §4.4 · pp. 8–9, Figure 5 · p. 8
  • Sequentially adding event reasoning and event localization loss improves R1@0.5 by 2.61% and 4.0%, while all components with gated fusion improve it by 6.26%.The component ablation is on the QVHighlights validation set.Source: Table 3 · p. 7, §4.4 · p. 8
  • Performance improves as moment-query count N rises, but excessive N hinders capturing long events; N=10 performs best, while λ_event values from 1 to 3 yield monotonically increasing performance.Figure 3 covers the query-count and event-loss-weight analyses on QVHighlights validation.Source: Figure 3 · p. 7, §4.4 · p. 8
  • Addition and concatenation perform poorly, and multi-head cross-attention only slightly improves on them; all remain below gated fusion, which better emphasizes informative moment queries.The fusion ablation is reported on QVHighlights validation.Source: Table 4 · p. 7, §4.4 · p. 8

5. Conclusion and Future Work

EaTR uses event and moment reasoning to ground video moments, with dynamic queries derived from video events. The authors identify sentence information in dynamic queries as an underexplored direction.

5. Conclusion and Future Work

  • Slot attention identifies a video's event units, which become initial dynamic moment queries and provide video-specific referential search areas.These queries incorporate content and positional information from the video.Source: §5 · p. 9
  • A gated fusion transformer layer enhances sentence-relevant moment queries and filters irrelevant ones before they interact with video-sentence representations.The decoder-layer interactions produce timestamps for video grounding.Source: §5 · p. 9
  • The authors leave consideration of sentence information in dynamic moment queries as an underexplored direction.Their explored dynamic moment queries are based on visual information.Source: §5 · p. 9

Appendix

The appendix provides further implementation details and experiments for the paper, including ablations and qualitative results.

Appendix

  • The supplementary material provides more concrete implementation details for generating pseudo event timestamps.These details are covered in Section A.Source: Appendix · p. 12
  • Additional experimental results include ablation studies.The studies are included in Section B.Source: Appendix · p. 12
  • The supplementary material also includes qualitative video-grounding results.These results are included in Section B.Source: Appendix · p. 12

A. Pseudo event timestamps generation

Pseudo event timestamps provide supervision for event reasoning and are generated without learnable parameters using a procedure inspired by temporal self-similarity.

A. Pseudo event timestamps generation

  • Pseudo event timestamps supervise event reasoning and are detected without learnable parameters, following inspiration from prior work.The procedure uses a temporal self-similarity matrix (TSM).Source: A. Pseudo event timestamps generation · p. 12
  • The procedure computes a temporal self-similarity matrix from cosine similarities between video representations and defines a contrastive kernel with size 5.The kernel is designed to imitate the boundary pattern in the TSM.Source: A. Pseudo event timestamps generation · p. 12
  • Boundary scores are computed by convolving the kernel over the TSM diagonal, then thresholded at the average score and filtered with a size-3 sliding max filter.Remaining indices are treated as event boundaries; pseudo event timestamps use the center coordinate and duration between boundaries.Source: A. Pseudo event timestamps generation · p. 12

B. Additional Experiments

The additional experiments section covers component analysis on QVHighlights, ablations on two datasets, and qualitative video-grounding results.

B. Additional Experiments

B.1. Additional component analysis

The additional analysis examines event-reasoning attention, slot-attention iterations, and transformer depth. It favors slot attention and selects three iterations and three layers as performance-efficiency trade-offs.

B.1. Additional component analysis

  • Slot attention outperforms cross-attention while using fewer GFLOPs and parameters through parameter reuse across iterations.Its event slots compete to cover distinctive video semantics, unlike cross-attention's independent slot weights.Source: §B.1 · p. 12
  • Three slot-attention iterations balance performance and training efficiency; more than three degrade performance.Performance improves from K = 1 to 2 to 3, with slightly lower computational efficiency as K increases.Source: §B.1 · p. 12
  • Three transformer layers balance performance and efficiency, since fewer than three underlearn video-sentence interaction and more increase complexity.Higher performance is attainable with more layers, but computational complexity also rises.Source: §B.1 · p. 12

B.2. Ablation study

The ablations examine EaTR components, moment-query counts, and event-loss weighting. Component additions consistently improve results, while the preferred query count and loss weight vary by dataset.

B.2. Ablation study

  • 2.66% and 5.9% R1@0.7 gains result from full component usage on Charades-STA and ActivityNet Captions, respectively.Each component introduces consistent improvement on both datasets.Source: §B.2 · p. 13
  • N = 6 is optimal for Charades-STA, while N = 10 is optimal for ActivityNet Captions.The paper relates the smaller count to Charades-STA's fine-grained videos with fewer events, versus the coarser, more numerous events in ActivityNet Captions.Source: §B.2 · p. 13, Figure A1 · p. 13, Figure A2 · p. 13
  • Event-loss weights of 2–3 improve Charades-STA results, while weights of 1–2.5 improve ActivityNet Captions results.Weights below 1 or above 3.5 degrade performance across all three datasets.Source: §B.2 · p. 13, Figure A1 · p. 13, Figure A2 · p. 13

B.3. Qualitative results

Qualitative examples show EaTR localizing target moments and activating different moment queries for different sentences. The reported failure case is a fine-grained video with visually similar frames.

B.3. Qualitative results

  • EaTR correctly localizes QVHighlights target moments regardless of their length.The examples show cross-attention weights between video-sentence representations and the highest-confidence moment queries.Source: §B.3 · p. 13, Figure A3 · p. 14
  • For one Charades-STA video, different sentences activate different moment queries and yield correct final predictions.The examples use the sentences “A person is in an entryway eating a sandwich” and “The person puts the cup down.”Source: §B.3 · pp. 13, 15, Figure A4 · p. 15
  • EaTR fails to localize sentences in fine-grained videos composed of visually similar frames.The paper attributes this failure to the difficulty of providing informative referential search areas from visual content alone.Source: §B.3 · pp. 13–14, Figure A5 · p. 15

Related papers

Loading 2308.06947v1…