PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Video Localization: A Revisit in Span-based Question Answering Framework

Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

arXiv:2102.13558v3cs.CLcs.CV

TL;DR

NLVL requires locating a query-matched moment in an untrimmed video, but video and text differ in temporal continuity and how small changes affect meaning. The paper treats video as a passage and proposes span-based QA models, including VSLNet with query-guided highlighting and VSLNet-L for long videos. Experiments on three datasets demonstrate the effectiveness of the framework, while error analysis reports that VSLNet tends to predict shorter moments than ground truth.

  • Problem

    Video and text differ in temporal continuity and sensitivity to small changes, creating technical gaps for applying span-based QA directly to NLVL.

    Source: §1 · pp. 1–2

  • Method

    The paper treats video as a passage and target moment as an answer span, adding query-guided highlighting in VSLNet and multiscale splitting for VSLNet-L.

    Source: §1 · pp. 1–2

  • Results

    Experiments demonstrate the effectiveness of VSLNet and VSLBase on three datasets, and further analyses demonstrate VSLNet-L’s effectiveness on long videos.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    The authors conclude that span-based QA is a promising framework for NLVL and that query-guided highlighting directs search toward a predicted coarse region.

    Source: §5 · p. 13

    VSLNet tends to predict shorter moments than ground truth, while VSLBase tends to predict longer ones.

    Source: §4.5.5 · p. 12

Abstract

from arXiv · show

Natural Language Video Localization (NLVL) aims to locate a target moment from an untrimmed video that semantically corresponds to a text query. Existing approaches mainly solve the NLVL problem from the perspective of computer vision by formulating it as ranking, anchor, or regression tasks. These methods suffer from large performance degradation when localizing on long videos. In this work, we address the NLVL from a new perspective, i.e., span-based question answering (QA), by treating the input video as a text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework (named VSLBase), to address NLVL. VSLNet tackles the differences between NLVL and span-based QA through a simple yet effective query-guided highlighting (QGH) strategy. QGH guides VSLNet to search for the matching video span within a highlighted region. To address the performance degradation on long videos, we further extend VSLNet to VSLNet-L by applying a multi-scale split-and-concatenation strategy. VSLNet-L first splits the untrimmed video into short clip segments; then, it predicts which clip segment contains the target moment and suppresses the importance of other segments. Finally, the clip segments are concatenated, with different confidences, to locate the target moment accurately. Extensive experiments on three benchmark datasets show that the proposed VSLNet and VSLNet-L outperform the state-of-the-art methods; VSLNet-L addresses the issue of performance degradation on long videos. Our study suggests that the span-based QA framework is an effective strategy to solve the NLVL problem.

1 INTRODUCTION

The paper reframes natural language video localization (NLVL) as span-based question answering by treating video as a passage and the target moment as an answer span. It proposes query-guided highlighting for video–text differences and a multiscale extension for long videos.

1 INTRODUCTION

  • The paper casts NLVL as finding a query-matched video moment, with the video as a text passage and the moment as an answer span.This establishes the span-based QA perspective that underlies the proposed approach.Source: §1 · p. 1
  • Video and text differ because video events are temporally correlated and frame changes are often smooth, whereas words are discrete and small changes can alter sentence meaning.These differences are presented as technical gaps that prevent direct application of existing span-based QA methods to NLVL.Source: §1 · pp. 1–2
  • VSLNet adds query-guided highlighting (QGH) to span-based QA, treating the target moment as foreground and the rest of the video as background.The foreground extends beyond the target’s start and end boundaries to include surrounding video content.Source: §1 · p. 2
  • VSLNet-L applies a multiscale split-and-concatenate strategy to long videos, increasing the chance that a segment fully contains a target moment truncated at another scale.The paper frames long-video localization as multi-paragraph QA, with clip segments corresponding to paragraphs.Source: §1 · p. 2
  • The paper extends its conference publication with investigation of long-video performance degradation and further experiments on VSLNet-L.The authors state that the additional analyses demonstrate VSLNet-L’s effectiveness on long videos.Source: §1 · p. 2
  • The authors identify span-based QA as a new perspective for NLVL and propose VSLNet and VSLNet-L to address video–text differences and long-video degradation.The contribution list describes VSLNet-L as incorporating multi-paragraph QA.Source: §1 · p. 2

2 RELATED WORK

Prior NLVL work includes ranking, anchor-based, reinforcement-learning, regression, and QA approaches. This paper adopts a standard span-based QA framework and adds modules to address differences between video and text and to handle multiple clips.

2.1 Natural Language Video Localization

  • Ranking methods match query and video candidates but rely on dense sampling, which the authors identify as inefficient and inflexible.The related-work discussion also describes anchor-based methods as assigning multiscale temporal anchors to frames and selecting the highest-confidence anchor.Source: §2.1 · p. 2
  • Regression methods directly predict target-moment boundaries, while reinforcement-learning methods progressively observe or refine candidate moments.The section also describes methods that jointly model video–query interactions and regress temporal locations.Source: §2.1 · p. 2
  • Earlier QA-based NLVL methods predict start and end boundaries, but the authors say they do not explain the differences from traditional span-based QA or adopt its standard framework.The paper positions VSLBase as a simple standard span-based QA framework, with QGH added in VSLNet to address those differences.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Span-based Question Answering

  • VSLNet-L is conceptually related to multi-paragraph QA because both locate an answer span across multiple paragraphs or clips.The related-work discussion describes multi-paragraph methods using confidence modules, boundary and content predictions, global score normalization, ranking, or hierarchical spans.Source: §2.2 · p. 3

3 METHODOLOGY

The methodology section proceeds from adapting span-based QA to NLVL, through VSLBase and VSLNet, to VSLNet-L’s multiscale split-and-concatenate strategy.

3. METHODOLOGY

  • The section first describes how to address NLVL with a span-based QA framework.The model architectures are shown in Figure 2.Source: §3 · p. 3
  • The methodology then details VSLBase and VSLNet, with VSLBase covered in Sections 3.2–3.4 and VSLNet in Section 3.5.Source: §3 · p. 3
  • The section concludes with VSLNet-L and its multiscale split-and-concatenate strategy.Source: §3 · p. 3

3.1 Span-based QA for NLVL

The paper converts each NLVL example into a span-based QA instance, using visual features as the passage and the language query as the question. The target moment becomes the answer span, whose boundaries map to times in the video.

3.1 Span-based QA for NLVL

  • A pre-trained 3D ConvNet extracts a visual feature sequence that serves as the passage, while the language query supplies the question in a SQuAD-style triple.Each extracted visual feature vector plays the role of a passage token embedding.Source: §3.1 · p. 4
  • The target moment’s start and end times are mapped to rounded indices in the visual feature sequence, and predicted indices convert back to video times.The mapping uses the video duration and the number of extracted feature vectors.Source: §3.1 · p. 4
  • The resulting instance is a video-feature passage, a language-query question, and an answer span from the target’s start index to its end index.The notation identifies the untrimmed video and query as sequences of frames and words, respectively.Source: §3.1 · pp. 3–4

3.2 Feature Encoder

The feature encoder projects video and text representations into a common dimension, then processes them with a simplified QANet encoder whose parameters are shared across modalities.

3.2 Feature Encoder

  • Video and text feature vectors are projected into the same dimension before encoding.The paper obtains video and word representations, then applies separate linear layers to project them.Source: §3.2 · p. 4
  • A single simplified QANet encoder block uses four convolution layers, multihead attention, and a feed-forward layer.Layer normalization and residual connections are applied to each layer.Source: §3.2 · p. 4
  • The feature encoder shares its parameters between visual features and word embeddings.Source: §3.2 · p. 4

3.3 Context-Query Attention

Context-query attention captures interactions between visual and text features by computing their similarities and deriving attention weights.

3.3 Context-Query Attention

  • Context-query attention (CQA) captures cross-modal interactions between visual and text features.Source: §3.3 · p. 4
  • CQA computes similarity scores between each visual feature and query feature, then derives context-to-query and query-to-context attention weights.Source: §3.3 · p. 4
  • The attention weights use row- and column-wise SoftMax normalization, and the output includes a feed-forward layer and element-wise multiplication.Source: §3.3 · p. 4

3.4 Conditioned Span Predictor

The conditioned span predictor uses separate start and end boundary predictions, with the end-boundary LSTM conditioned on the start-boundary LSTM.

3.4 Conditioned Span Predictor

  • The predictor uses two unidirectional LSTMs and two feed-forward layers, conditioning the end-boundary LSTM on the start-boundary LSTM.The authors motivate this ordering by the temporal sequence of video events and the fact that the end boundary follows the start boundary.Source: §3.4 · p. 4
  • The authors report that unidirectional LSTMs perform similarly to bidirectional LSTMs with fewer parameters and higher efficiency.Source: §3.4 · p. 4
  • SoftMax converts start and end scores into probability distributions, and the training objective uses the start and end boundary labels.Source: §3.4 · p. 5
  • At inference, the predicted answer span maximizes the joint probability of its start and end boundaries.Source: §3.4 · p. 5

3.5 Query-Guided Highlighting

Query-Guided Highlighting (QGH) marks a query-aligned foreground region and predicts foreground confidence to guide span localization.

3.5 Query-Guided Highlighting

  • QGH extends the target moment's start and end boundaries by an α-controlled ratio, treating the extended region as foreground and the rest as background.The extension may include surrounding context and help the network focus on subtle frame differences.Source: §3.5 · p. 5
  • QGH assigns foreground label 1 and background label 0, then predicts whether visual features belong to the foreground or background.Source: §3.5 · p. 5
  • QGH encodes the query into a sentence representation and concatenates it with each query-conditioned visual feature before computing highlighting scores.Source: §3.5 · p. 5
  • The highlighting scores weight visual features used for span prediction, and VSLNet is trained end-to-end with the combined losses.Source: §3.5 · p. 5

3.6 Multi-scale Split-and-Concat

VSLNet-L extends VSLNet with multi-scale split-and-concatenation to address localization degradation on long videos. It scores clip segments, then combines their representations across scales to predict the target moment.

3.6 Multi-scale Split-and-Concat

  • VSLNet-L introduces multi-scale split-and-concatenation to address localization performance degradation on long videos.The method treats video segments at different scales and combines their representations for video-level localization.Source: §3.6 · p. 5, §3.6.3 · p. 6

3.6.1 Split: Clip Segment Level Detection

  • Each clip segment is processed into query-attended features, and the Nil Prediction Module estimates whether it contains or partially contains the target moment.The module highlights segments containing the target and suppresses others, narrowing the search scope; the video is split at the visual-feature level for computational efficiency.Source: §3.6.1 · pp. 5–6

3.6.2 Concat: Video Level Localization

  • All clip-segment representations are concatenated because one segment may not cover the full target moment and low-confidence segments may still provide useful context.The concatenated representations are used for video-level localization.Source: §3.6.2 · p. 6

3.6.3 Multi-scale Split-and-Concat

  • Multi-scale splitting uses different segment lengths to capture context and vary training samples, since separate processing may miss context across segments.At inference, VSLNet-L-Pm selects the candidate with the highest joint boundary probability, while VSLNet-L-U unions the two candidates with the largest overlap.Source: §3.6.3 · p. 6

4 EXPERIMENTS

Experiments on three benchmarks compare VSLBase, VSLNet, and VSLNet-L using Rank@1 at multiple IoU thresholds and mIoU. VSLNet improves results across datasets, while VSLNet-L shows its clearest advantage on longer videos.

4.1 Datasets

  • The evaluation uses Charades-STA, ActivityNet Captions, and two TACoS versions; metrics are Rank@1 at IoU thresholds 0.3, 0.5, and 0.7, plus mIoU.Video features use maximal lengths of 128 for Charades-STA, 300 for ActivityNet Captions, and 300 or 600 for TACoS; VSLNet-L uses multiple clip scales.Source: §4.1 · pp. 6–7, §4.2.1 · p. 7, §4.2.2 · p. 7

4.3 Comparison with State-of-the-Arts

  • On Charades-STA and both TACoS versions, VSLNet generally outperforms prior methods, and VSLNet-L further improves TACoS results, particularly with longer feature sequences.The reported Charades-STA gain over MAN is 7.47% at IoU = 0.7 versus 0.78% at IoU = 0.5; on TACoSorg, VSLNet-L(L) surpasses VSLNet-L(S).Source: Table 2 · p. 7, §4.3 · p. 8, Table 3
  • On ActivityNet Captions, VSLNet significantly boosts VSLBase, while VSLNet-L is comparable to state-of-the-art methods and yields less significant gains than on TACoS.The paper notes that 99% of ActivityNet Captions annotations are from videos shorter than four minutes.Source: §4.3 · pp. 8–9, Table 4 · p. 8

4.4 Performance on Videos with Different Length

  • VSLNet-L improves mIoU more on longer TACoS videos, with over 3% absolute gains above eight minutes versus under 2% below eight minutes on TACoSorg.ActivityNet Captions also shows larger improvements on longer videos, with VSLNet-L outperforming VSLNet across video lengths for both candidate-selection strategies.Source: §4.4 · p. 9

4.5 Ablation Studies

Ablations examine VSLBase modules, QGH's extension ratio, and VSLNet-L's split scales, alongside qualitative predictions and error patterns. The results support the contributions of CQA, QGH, and multi-scale splitting, while revealing a tendency for QGH-constrained predictions to be shorter.

4.5.1 Module Analysis

  • CQA and the CMF encoder each outperform their alternatives across metrics, with replacing CAT by CQA contributing more than replacing BiLSTM by CMF on Rank@1, IoU = 0.7.With CQA retained, replacing the encoder yields a 1.61% absolute gain.Source: §4.5.1 · pp. 10–11, Table 8 · p. 10, Table 9 · p. 10

4.5.2 The Impact of Extension Ratio in QGH

  • QGH improves performance across tested extension ratios, with its best results between α = 0.05 and 0.2; even α = ∞ remains better than VSLBase.At α = ∞, the video is not region-highlighted, and QGH acts as concatenation of the sentence representation with each visual feature.Source: §4.5.2 · p. 12

4.5.3 Single vs Multi-scale Split-and-Concat

  • Multi-scale split-and-concat improves over VSLNet across tested split lengths, and VSLNet-L-Pm (-U) further improves all metrics over single-scale VSLNet-L.The authors attribute the gains to reducing target-moment truncation and capturing video context.Source: Table 10 · p. 11, §4.5.3 · p. 12

4.5.4 Qualitative Analysis

  • Qualitative examples show VSLNet predictions closer to ground truth than VSLBase, while VSLNet-L predictions are more accurate than VSLNet on TACoS.The examples also show VSLNet-L constrained to positive clip segments; one case contrasts capturing only “washes” with correctly locating both the action and “cutting board” position.Source: §4.5.4 · p. 12

4.5.5 Error Analysis

  • VSLBase tends to predict longer moments, whereas QGH-constrained VSLNet tends to predict shorter ones.The length-error distributions include more VSLBase errors above 4 seconds on Charades-STA or 30 seconds on ActivityNet, and more VSLNet errors below −4 or −20 seconds, respectively.Source: §4.5.5 · p. 12, Figure 17 · p. 12
  • Two VSLNet failures illustrate incorrect QGH start placement and unequal confidence across multiple queried actions, causing only the “pushes” action to be retrieved.Both examples are selected from predictions with IoU below 0.2.Source: §4.5.5 · p. 12, Figure 18 · p. 13

5 CONCLUSION

The paper frames natural language video localization as span-based question answering, treating video as a text passage and the target moment as an answer span. It proposes VSLNet with query-guided highlighting and extends it to VSLNet-L for long videos; experiments report state-of-the-art progress and reduced long-video degradation.

5 CONCLUSION

  • VSLNet-L extends VSLNet with multi-scale video partitioning to capture context and address performance degradation on long videos.The approach treats a long video as a document with multiple paragraphs and partitions it using different clip lengths.Source: §5 · p. 13
  • VSLNet adds query-guided highlighting to VSLBase, directing span search to a predicted coarse region to address differences between video and text.Source: §5 · p. 13
  • Experiments on three benchmark datasets support span-based QA as a promising framework for NLVL and report that VSLNet-L prevents long-video performance degradation effectively.Source: §5 · p. 13
Loading 2102.13558v3…