Source-linked AI summary
Dense Regression Network for Video Grounding
Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, Chuang Gan
TL;DR
Video grounding must identify query-relevant temporal segments despite sparse positive frame annotations and difficult video-language localization. The paper proposes DRN, which regresses boundary distances densely and adds IoU-aware scoring; it reports state-of-the-art performance across three benchmarks, including 42.49% versus 36.90% on ActivityNet-Captions.
Problem
Video grounding is difficult because videos contain many frames but may provide only a few annotated starting and ending frames for training.
Method
DRN densely regresses each frame’s distances to the query segment’s starting and ending boundaries and uses an IoU regression head to estimate localization quality.
Results
DRN outperforms state-of-the-art methods on Charades-STA, ActivityNet-Captions, and TACoS; on ActivityNet-Captions it achieves 42.49% versus 36.90%.
Takeaways & Limitations
Using distances to ground-truth boundaries converts frames within the segment into positive training samples, while IoU scoring explicitly accounts for localization quality.
Abstract
from arXiv · showhide
We address the problem of video grounding from natural language queries. The key challenge in this task is that one training video might only contain a few annotated starting/ending frames that can be used as positive examples for model training. Most conventional approaches directly train a binary classifier using such imbalance data, thus achieving inferior results. The key idea of this paper is to use the distances between the frame within the ground truth and the starting (ending) frame as dense supervisions to improve the video grounding accuracy. Specifically, we design a novel dense regression network (DRN) to regress the distances from each frame to the starting (ending) frame of the video segment described by the query. We also propose a simple but effective IoU regression head module to explicitly consider the localization quality of the grounding results (i.e., the IoU between the predicted location and the ground truth). Experimental results show that our approach significantly outperforms state-of-the-arts on three datasets (i.e., Charades-STA, ActivityNet-Captions, and TACoS).
1. Introduction
Video grounding must localize query-relevant temporal boundaries despite complex video-language relationships and very sparse frame annotations. DRN addresses this imbalance with dense boundary regression and an IoU-aware ranking signal, achieving higher benchmark accuracy.
- Challenge: Sparse annotations make training difficult because videos may contain thousands of frames but only a few positive starting and ending frames.The task also requires connecting queries to complex video contents and precisely localizing actions against complex backgrounds.
- Dense supervision: Distance regression turns every frame inside the ground-truth segment into a positive training sample.Each frame predicts its distances to the segment’s starting and ending boundaries, increasing usable supervision.
- Proposed method: DRN combines video-query interaction, location regression, semantic matching, and IoU regression in a one-stage grounding framework.The IoU regression head explicitly considers prediction localization quality.
- Prediction ranking: Semantic matching scores indicate query relevance, while IoU scores explicitly estimate localization quality for selecting among dense predictions.The model produces a temporal bounding box and ranking signals for each frame location.
- Results: 42.49% accuracy on ActivityNet-Captions surpasses He et al.’s state-of-the-art 36.90%.The paper reports evaluation on three video grounding datasets.
2. Related work
Related work spans two-stage proposal-and-ranking systems, direct one-stage predictors, and anchor-free detection ideas adapted to video grounding. DRN differs by using many frames inside the ground truth as positive samples.
- Two-stage methods: Two-stage methods generate proposals and rank them by proposal-query similarity, but thousands of proposals increase computation and performance depends on proposal quality.These methods use a propose-and-rank pipeline.
- One-stage methods: One-stage methods directly regress temporal coordinates or predict starting and ending probabilities at each frame.These approaches avoid the conventional proposal-generation pipeline.
- DRN distinction: Unlike methods selecting only boundary frames as positives, DRN leverages many positive frames within the ground-truth segment.The paper reports this distinction as a source of improved grounding performance.
- Anchor-free detection: Anchor-free detection predicts distances from each pixel inside an object to its boundaries without predefined anchor boxes.The paper uses this distance-regression idea as motivation for video grounding.
3. Proposed method
The proposed dense regression network grounds query-described video segments by using frame-level boundary distances as dense supervision, then combines semantic matching with predicted localization quality. Its multi-level video-query interaction captures temporal and language information before producing and selecting dense temporal boxes.
- 3.1. General scheme: The grounding module contains location regression, semantic matching, and IoU regression heads, producing a temporal box and associated scores at each frame or feature-map location.The semantic score measures query-content matching, while the IoU score estimates localization quality against the ground truth.
- 3.1. General scheme: At inference, DRN selects the predicted temporal box with the highest product of semantic matching score and IoU score.This combines query-semantic correspondence with predicted temporal-boundary quality.
- 3.2. Multi-level video-query interaction module: The video-query interaction module uses hierarchical feature maps, multi-level query fusion, temporal location embedding, and FPN to represent varied temporal scales and ordering cues.On Charades-STA, ground-truth durations range from 2.4s to 180.8s.
- 3.3. Location regression head: DRN predicts the distances from each frame inside the ground truth to the segment’s starting and ending boundaries, making every such frame a positive training sample.Locations outside the ground truth are excluded from location-regression training.
- 3.5. IoU regression head: The IoU regression head estimates the IoU between each predicted box and its ground truth, explicitly incorporating localization quality during training and testing.Its input concatenates features from the semantic matching and location regression heads, and its target is the corresponding box-ground-truth IoU.
4. Experiments
Experiments evaluate DRN on Charades-STA, ActivityNet-Captions, and TACoS using standard grounding metrics and feature settings. DRN achieves the strongest reported results across these benchmarks, including substantial gains over prior methods.
- Datasets and evaluation: Charades-STA contains 6,672 videos and 16,128 video-query pairs, with 12,408 pairs for training and 3,720 for testing.Videos average 29.76 seconds, with 2.4 annotated moments lasting 8.2 seconds on average.
- Datasets and evaluation: The evaluation metric R@n, IoU=m measures the percentage of test samples with at least one prediction whose IoU exceeds m among the top n predictions.C3D features are used for fair comparisons with prior methods.
- Benchmark comparisons: 8.7% absolute improvement over R-W-M is achieved on Charades-STA for R@1, IoU=0.5 using the same C3D features.DRN reaches the highest scores across all IoU thresholds and also outperforms MAN and ExCL under their reported feature settings.
- Benchmark comparisons: 23.17% versus 20.01% is reported on TACoS, where DRN outperforms the previous best result despite the dataset's multiple-query difficulty.DRN achieves the highest scores for both R@1 and R@5 when IoU=0.5.
5. Ablation studies
Ablations examine dense positive sampling, IoU-based ranking, multi-level fusion, and temporal location embeddings. The reported results support each component's contribution to grounding performance, with IoU regression and multi-level representations improving selection or accuracy.
- Positive training samples: DRN variants test how selecting all, half, random, or center frames within the ground truth as positives affects training.DRN-All uses every frame within the ground truth, while DRN-Half, DRN-Random, and DRN-Center use subsets or extreme location choices.
- IoU regression: The IoU regression head consistently improves R@1, IoU=0.5 on both Charades-STA and ActivityNet-Captions.Removing the head leaves matching scores to rank predictions, while replacing it with centerness slightly decreases accuracy.
- Multi-level fusion: 43.79% versus 43.04% on Charades-STA and 40.61% versus 39.78% on ActivityNet-Captions show gains from multi-level fusion.The comparisons remove MLF and measure the resulting performance differences.
- Multi-level fusion: 44.76% versus 45.40% on Charades-STA indicates that using different query features at different fusion levels outperforms using the same query feature.The comparison is between MLF-Same and DRN.
- Temporal location embedding: Location embeddings consistently improve performance on temporal queries, with a 1.7% gain when training and testing on the temporal subset.The subset contains queries using temporal words such as before, after, while, and then.
6. Conclusions
The paper proposes a dense regression network that increases positive training samples by predicting distances to segment boundaries and adds IoU-based localization quality modeling. DRN outperforms state-of-the-art methods on Charades-STA, ActivityNet-Captions, and TACoS, while extension to temporal action localization and dense video captioning remains future work.
- DRN predicts distances from each frame to the starting and ending frames of the query-described segment, significantly increasing positive training samples.
- The IoU regression head explicitly models the localization quality of grounding results.
- DRN outperforms state-of-the-art methods on Charades-STA, ActivityNet-Captions, and TACoS.
- Extending DRN to temporal action localization and dense video captioning is left for future work.
A. More details about our DRN
The DRN combines video-query interaction, dense temporal prediction, semantic matching, and IoU regression. Training uses a staged procedure, while implementation reduces computation by predicting boxes relative to segment centers and extracts video and query features with I3D and a bi-directional LSTM.
- Training first optimizes location and matching objectives, then trains the IoU head with fixed DRN parameters, and finally fine-tunes all modules jointly.
- The video-query interaction module preprocesses query sentences, video frames, and temporal coordinates, then fuses video and query features into a feature pyramid.
- DRN predicts one temporal bounding box relative to each segment’s central frame after dividing the video into K evenly sized segments.
- The model uses K=32 for Charades-STA and ActivityNet Captions, and K=128 for TACoS.
- I3D features are extracted from 64-frame snippets and pooled within segments.
- Query words use 300-dimensional GloVe embeddings, followed by a one-layer bi-directional LSTM with 512 units.
B.3. Location embedding
Each segment’s temporal coordinates are encoded as a location embedding and concatenated with video features before further video-query processing.
- The k-th segment is represented by a 3D temporal-coordinate vector.
- A linear layer maps the temporal coordinates to a 256D location embedding.
- The location embedding is concatenated with video features along the channel dimension.
B.4. Vision-Language Fusion Module
The vision-language fusion module applies textual attention and element-wise multiplication to combine query and video representations, while the grounding module predicts semantic matching, temporal location, and IoU quality.
- B.4. Vision-Language Fusion Module: Textual attention is applied to the input query feature to obtain an attended query representation.
- B.4. Vision-Language Fusion Module: The module processes a query such as “A person hits a wood panel into the ground” within the video-query interaction pipeline.
- Grounding Module: Figure C describes the grounding module’s input as the i-th feature-pyramid level with temporal dimension K 2i−1.
- B.4. Vision-Language Fusion Module: Attended query features are fused with lower-level pyramid features using element-wise multiplication.
- Grounding Module: The grounding module contains semantic matching, location regression, and IoU regression heads.
- Grounding Module: The semantic matching and location regression heads use two 1D convolution layers, while the IoU regression head uses three.
D. More visualization examples
Figure D provides additional qualitative results showing how the IoU regression head improves prediction selection.
- The IoU regression head selects predictions with larger IoU against the ground truth.
E.1. Details of centerness baseline
The centerness baseline replaces IoU regression with a centerness score trained from distances to the ground-truth boundaries.
- The experiment compares the IoU regression head with the centerness loss used in FCOS.
- The model predicts a centerness score for each location as the replacement training target.
- The centerness loss uses binary crossentropy, following FCOS.
E.2. Results of the centerness assumption
The study tests whether locations nearer the ground-truth center produce better grounding boxes by examining the best prediction locations across two datasets.
- The centerness assumption predicts that locations closer to an object’s center yield boxes with larger IoU against the ground truth.
- For each video-query pair, the experiment selects the predicted box with the largest IoU as the best grounding result.
- The best-location distribution is reported across three evenly divided relative-location portions within the ground truth.The portions are [0, 1/3), [1/3, 2/3), and [2/3, 1].
- The location statistics cover Charades-STA and ActivityNet-Captions, focusing on locations inside the ground truth because few fall outside it.