Source-linked AI summary
Boundary Proposal Network for Two-Stage Natural Language Video Localization
Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian Shao, Lu Ye, Jun Xiao
TL;DR
NLVL requires localizing a language-described segment in a long, untrimmed video, while existing anchor-based and anchor-free one-stage methods have complementary limitations. BPNet uses anchor-free proposals followed by visual-language matching, and it achieves state-of-the-art performance across three benchmarks. Its universal design permits replacing either stage with other effective models.
Problem
NLVL must localize a language-described segment in an untrimmed video, while anchor-based methods depend on heuristic proposals and anchor-free methods do not model segment-level interaction.
Method
BPNet generates high-quality segment proposals with an anchor-free backbone, then fuses each proposal with the language query and predicts a matching score.
Results
BPNet achieves new state-of-the-art performance on Charades-STA, TACoS, and ActivityNet Captions across all reported evaluation metrics.
Takeaways & Limitations
BPNet provides a universal NLVL framework whose proposal-generation and visual-language matching modules can be replaced by other effective methods.
Takeaways & Limitations
The reported implementation uses VSLNet as its anchor-free backbone for simplicity and efficiency, although more effective anchor-free backbones could further improve performance.
Abstract
from arXiv · showhide
We aim to address the problem of Natural Language Video Localization (NLVL)-localizing the video segment corresponding to a natural language description in a long and untrimmed video. State-of-the-art NLVL methods are almost in one-stage fashion, which can be typically grouped into two categories: 1) anchor-based approach: it first pre-defines a series of video segment candidates (e.g., by sliding window), and then does classification for each candidate; 2) anchor-free approach: it directly predicts the probabilities for each video frame as a boundary or intermediate frame inside the positive segment. However, both kinds of one-stage approaches have inherent drawbacks: the anchor-based approach is susceptible to the heuristic rules, further limiting the capability of handling videos with variant length. While the anchor-free approach fails to exploit the segment-level interaction thus achieving inferior results. In this paper, we propose a novel Boundary Proposal Network (BPNet), a universal two-stage framework that gets rid of the issues mentioned above. Specifically, in the first stage, BPNet utilizes an anchor-free model to generate a group of high-quality candidate video segments with their boundaries. In the second stage, a visual-language fusion layer is proposed to jointly model the multi-modal interaction between the candidate and the language query, followed by a matching score rating layer that outputs the alignment score for each candidate. We evaluate our BPNet on three challenging NLVL benchmarks (i.e., Charades-STA, TACoS and ActivityNet-Captions). Extensive experiments and ablative studies on these datasets demonstrate that the BPNet outperforms the state-of-the-art methods.
Introduction
Natural Language Video Localization (NLVL) identifies the start and end of a query-relevant segment in an untrimmed video. BPNet combines anchor-free proposal generation with segment-level visual-language matching to address limitations of one-stage methods and achieves state-of-the-art results across three benchmarks.
- NLVL localizes the video segment corresponding to a natural-language query by predicting its start and end points.
- Anchor-based methods match language queries against predefined temporal proposals, but require many candidates and heuristic settings for varied segment lengths.These methods model segment-level information but incur inefficient matching when proposal counts are large.
- Anchor-free methods directly predict segment boundaries or positive frames, avoiding fixed proposals but overlooking segment-level interaction.
- BPNet first generates high-quality proposals with an anchor-free backbone, then fuses each proposal with the query and predicts a matching score.The highest-scoring proposal is selected as the localized segment.
- The framework is universal: its proposal-generation and visual-language matching stages can be replaced with stronger corresponding models.
- BPNet achieves new state-of-the-art performance across Charades-STA, TACoS, and ActivityNet Captions on all reported evaluation metrics.
Boundary Proposal Generation
BPNet’s first stage uses an anchor-free backbone to generate high-quality video segment proposals from query-guided visual features and predicted boundaries. It samples the highest-scoring start–end positions as candidates.
- Boundary Proposal Generation: BPNet uses an anchor-free proposal extraction process that can incorporate other anchor-free approaches.The paper follows VSLNet’s span-based question-answering formulation while presenting BPNet as a universal framework.
- Boundary Proposal Generation: Visual and query features are projected into a shared dimension and processed by an embedding encoder with convolutions, self-attention, normalization, residual connections, and feed-forward layers.The encoder integrates contextual information within each modality.
- Boundary Proposal Generation: A visual-language attention layer computes frame–word similarities and row- and column-normalized attention weights to model cross-modal interaction.The resulting attention combines video and query representations before proposal generation.
- Boundary Proposal Generation: Two stacked LSTMs and feed-forward layers predict start and end boundary scores from query-guided visual features.The hidden states produce logits for the start and end boundaries, whose probabilities are combined into a two-dimensional score map.
- Boundary Proposal Generation: BPNet samples the N highest-scoring positions on the proposal score map and treats their corresponding segments as candidates.Each position represents a segment defined by a start boundary and an end boundary.
Visual-Language Matching
The second stage fuses each variable-length segment candidate with the language query and predicts a matching score. The highest-scoring candidate is selected as the final localized segment.
- Visual-Language Matching: Candidate visual features are converted to a common temporal length using temporal weighted pooling, while word features are pooled into a sentence-level query representation.This accommodates candidates with different temporal lengths before fusion.
- Visual-Language Matching: The candidate and sentence representations are concatenated and passed through a feed-forward layer to encode segment-level multimodal information.The fused feature is used by the subsequent matching process.
- Visual-Language Matching: A matching score rating layer predicts one score for each candidate using two feed-forward layers with ReLU and sigmoid activations.The most matched proposal is selected as the final result.
- Visual-Language Matching: The matching scores are supervised with candidate–ground-truth temporal IoU values, turning score prediction into an IoU regression problem.The paper states that predicted matching scores are positively associated with temporal IoU.
Training and Inference
BPNet trains boundary proposal generation and visual-language matching jointly with separate classification and regression losses. During inference, it ranks candidate scores and returns the highest-scoring segment.
- Training and Inference: Each training sample contains an untrimmed video, a language query, and the ground-truth video segment.The sample supports both frame-level boundary supervision and segment-level matching supervision.
- Training and Inference: Frame-level labels indicate whether each frame is a start or end boundary, while each candidate receives its temporal IoU with the ground-truth segment as a matching target.This provides supervision for both stages of BPNet.
- Training and Inference: Boundary proposal generation uses binary cross-entropy for start and end probabilities, and visual-language matching uses an L2 loss on predicted and ground-truth IoU scores.The two losses supervise the distinct prediction tasks.
- Training and Inference: The final objective combines boundary classification and matching regression losses with a hyper-parameter balancing their contributions.The loss is expressed as L = Lcls + λ × Lreg.
- Training and Inference: At inference, BPNet produces N candidates with matching scores, ranks them, and selects the candidate with the highest score.The selected candidate is the final localization result.
Datasets
BPNet is evaluated on three public NLVL benchmarks: Charades-STA, TACoS, and ActivityNet-Captions. The supplied dataset description details Charades-STA and introduces TACoS as a cooking-activity dataset.
- Datasets: Charades-STA contains 6,672 daily indoor-activity videos and 16,128 sentence–moment pairs.It has 12,408 training pairs and 3,720 testing pairs; average video and segment durations are 30.59s and 8.22s.
- Datasets: TACoS consists of 127 cooking-activity videos and includes a video-grounding task.The supplied passage truncates its sentence–moment-pair count.
Evaluation Metrics
The evaluation uses recall at specified IoU thresholds and mean IoU to measure localization accuracy.
- R@n, IoU=θ measures the percentage of test samples with at least one of the top-N results exceeding the IoU threshold θ.The experiments set n = 1 and θ ∈{0.3, 0.5, 0.7}.
- mIoU measures the average intersection-over-union with the ground truth across all test samples.
Implementation
BPNet uses down-sampled visual frames and fixed-dimensional language and visual representations, with specified encoder and proposal settings.
- Visual features are extracted with a Sports-1M-pretrained C3D network, then reduced to 500 dimensions using PCA.
- Each query word is initialized with a fixed 300-dimensional GloVe embedding.
- BPNet uses a 128-dimensional intermediate layer, four embedding-encoder convolution blocks, and kernel size 7.
- The model generates 128 boundary proposals for each video.
Comparisons with the State-of-the-Arts
BPNet achieves state-of-the-art performance across Charades-STA, TACoS, and ActivityNet Captions, including videos and segments with diverse lengths. Its two-stage design combines an anchor-free proposal generator with proposal-sentence matching, while the reported implementation retains VSLNet for simplicity and efficiency.
- BPNet achieves new state-of-the-art performance across all reported metrics and benchmarks.
- On Charades-STA, BPNet outperforms all baselines across every metric, including a 2.59 absolute improvement in IoU@0.7 over the second result.
- BPNet significantly outperforms other methods on TACoS and ActivityNet Captions using C3D features.
- The results demonstrate adaptability to videos and ground-truth segments with diverse lengths.
- The authors retain VSLNet as the backbone to prioritize simplicity and efficiency, although stronger anchor-free backbones could improve performance.
Ablation Study
The ablation study evaluates BPNet against anchor-based and anchor-free alternatives, including comparisons reported across benchmark settings.
- The ablation experiments compare BPNet with both anchor-based and anchor-free methods to assess the two-stage model.
- Table 4 compares an anchor-free model, an anchor-based model, and BPNet using the same backbone across three benchmarks.
GT BPNet
BPNet combines compact candidate generation with visual-language fusion to improve natural language video localization. Its qualitative evaluation and ablations support high-quality proposals and useful segment-level interaction modeling.
- Proposal quality: BPNet generates only 8 candidates while achieving higher performance than the anchor-based setting.This indicates that the proposal generator produces high-quality candidates with fewer proposals.
- Visual-language fusion: Adding the Visual-Language Fusion Layer improves performance by modeling segment-level video-query interaction.
- Framework: BPNet is a universal two-stage framework that separates proposal generation from visual-language matching and can incorporate alternative effective methods.