PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Re-localization

Yang Feng, Lin Ma, Wei Liu, Tong Zhang, Jiebo Luo

arXiv:1808.01575v1cs.CV

TL;DR

Video re-localization asks how to find a reference-video segment that semantically corresponds to a query despite substantial appearance variation and limited paired data. The paper constructs an ActivityNet-based dataset and proposes cross gated bilinear matching with classification-based boundary prediction; it reports favorable performance over competing methods.

  • Problem

    Video re-localization seeks semantically corresponding segments despite variation in environment, subject, and viewpoint, unclear boundaries, and a lack of paired videos with localization information.

    Source: §1 · pp. 2–3

  • Method

    The authors reorganize ActivityNet into a dataset and use cross gated bilinear matching to compare reference features with attentively weighted query features, then classify time steps into localization classes.

    Source: §1 · pp. 3–4

  • Results

    The authors report favorable performance over competing methods on the new dataset.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The dataset’s class split reserves 20 action classes for testing, so test actions are unseen during training and can assess generalization to unseen actions.

    Source: §1 · p. 3

Abstract

from arXiv · show

Many methods have been developed to help people find the video contents they want efficiently. However, there are still some unsolved problems in this area. For example, given a query video and a reference video, how to accurately localize a segment in the reference video such that the segment semantically corresponds to the query video? We define a distinctively new task, namely \textbf{video re-localization}, to address this scenario. Video re-localization is an important emerging technology implicating many applications, such as fast seeking in videos, video copy detection, video surveillance, etc. Meanwhile, it is also a challenging research task because the visual appearance of a semantic concept in videos can have large variations. The first hurdle to clear for the video re-localization task is the lack of existing datasets. It is labor expensive to collect pairs of videos with semantic coherence or correspondence and label the corresponding segments. We first exploit and reorganize the videos in ActivityNet to form a new dataset for video re-localization research, which consists of about 10,000 videos of diverse visual appearances associated with localized boundary information. Subsequently, we propose an innovative cross gated bilinear matching model such that every time-step in the reference video is matched against the attentively weighted query video. Consequently, the prediction of the starting and ending time is formulated as a classification problem based on the matching results. Extensive experimental results show that the proposed method outperforms the competing methods. Our code is available at: https://github.com/fengyang0317/video_reloc.

1 Introduction

Video re-localization seeks semantically corresponding segments in a reference video when visual appearance varies, a case not handled by copy detection or predefined-action localization. The paper defines the task, reorganizes ActivityNet into a dataset, and proposes a cross gated bilinear model for matching and boundary prediction.

1 Introduction

  • Copy detection can fail when the same action appears in a different environment, while predefined-action localization cannot handle actions absent from its training classes.The paper proposes cropping the segment of interest as a query and locating semantically matched segments in full episodes.Source: §1 · pp. 2–3
  • Video re-localization detects the start and end of a reference-video segment that semantically corresponds to a query clip.The reference contains at least one corresponding segment, which may differ in appearance from the query.Source: §1 · p. 3
  • The task is challenging because corresponding videos can vary in environment, subject, and viewpoint, and segment boundaries may be unclear.The authors also identify a lack of paired query/reference videos with localization annotations.Source: §1 · p. 3
  • The ActivityNet dataset pairs an action segment with a reference video containing a same-class action and surrounding background, splitting classes so test actions are unseen in training.The proposed model matches each reference feature with an attentively weighted query representation, then classifies each time step into four localization classes.Source: §1 · pp. 3–4
  • The authors report that their model achieves favorable performance over competing methods on the new dataset.They formulate video re-localization as classification of starting, ending, inside, and outside time steps after matching query and reference videos.Source: §1 · p. 4

2 Related Work

Related work includes whole-video retrieval, segment localization, and language-based temporal retrieval. Video re-localization differs by matching a query video directly to a reference video and locating the corresponding segment.

2 Related Work

  • Content-based video retrieval returns entire videos, leaving users to seek manually for the relevant segment; video re-localization instead locates its exact start and end.Source: §2 · p. 4
  • Action localization finds segments from predefined classes, whereas video re-localization can target concepts beyond actions.One-shot recognition and unsupervised action discovery/localization are cited as attempts to move beyond predefined classes.Source: §2 · p. 4
  • Natural-language query methods localize temporal segments from text, while this method directly matches query and reference video segments in a single modality.Source: §2 · p. 5

3 Methodology

The model aggregates local video features, matches query and reference representations, and uses the matching results to localize a corresponding segment. Its matching design includes attention, cross gating, bilinear matching, and aggregation.

3 Methodology

  • The model comprises aggregation, matching, and localization components that exploit interactions between query and reference videos to predict segment boundaries.Source: §3 · p. 5

3.1 Video Feature Aggregation

  • Two LSTMs aggregate temporally ordered local features to incorporate longer-range context into the query and reference representations.The model excludes global video features because segment localization relies on local information.Source: §3.1 · p. 6

3.2 Cross Gated Bilinear Matching

  • Cross gated bilinear matching uses four modules: attention-weighted query generation, cross gating, bilinear matching, and matching aggregation.Source: §3.2 · p. 6

Attention Weighted Query.

At each reference-video time step, attention selects relevant query information for matching. Cross gating then filters irrelevant features and emphasizes relevant query-reference interactions.

Attention Weighted Query.

  • Each reference feature is matched against an attention-weighted query because only some query parts may correspond to that reference feature.The attention weights depend on the reference representation and the previous matching result, selecting relevant query information for a representation intended to match the current reference feature.Source: Attention Weighted Query · pp. 6–7
  • Cross gating makes each video’s gate depend on the other video, filtering both features when the reference is irrelevant and expected to enhance interaction when the features closely relate.Source: Attention Weighted Query · p. 7

Bilinear Matching.

Bilinear matching models interactions between query and reference representations using quadratic, linear, and bias terms. Factorization reduces the model’s parameter count while retaining these relationships.

Bilinear Matching.

  • An l-dimensional bilinear output normally uses l^3 + l parameters, compared with k × l × (l + 1) after factorization.The factorization is introduced to reduce the parameter count and make the model easier to learn.Source: Bilinear Matching · pp. 7–8
  • Factorized bilinear matching captures relationships between query and reference representations with fewer parameters.The factorized model introduces k × l × (l + 1) parameters, where k is much smaller than l.Source: Bilinear Matching · p. 8
  • Each bilinear matching dimension has quadratic, linear, and bias terms, with the quadratic term capturing complex interactions between the two video representations.Source: Bilinear Matching · p. 8

Matching Aggregation.

An LSTM aggregates the local matching results into contextual matching information. A reverse-direction LSTM also aggregates these results, with its hidden state concatenated with the forward state.

Matching Aggregation.

  • An LSTM aggregates local matching results, which capture interactions between the query and reference videos, into matching context.Source: Matching Aggregation · p. 8
  • A reverse-direction LSTM aggregates matching results in the opposite temporal direction, and its hidden state is concatenated with the forward hidden state.Source: Matching Aggregation · p. 8

3.3 Localization

Localization classifies each reference-video time step as a start, end, inside, or outside point. Training uses weighted cross-entropy with labels that distinguish boundaries and dynamically emphasizes the rare start and end positions.

3.3 Localization

  • At each reference-video time step, localization predicts one of four classes: starting point, ending point, inside point, or outside point.The matching-layer output indicates whether the content at that time step matches the query clip.Source: §3.3 · p. 8

3.4 Training

  • Training assigns outside points [0, 0, 0, 1], inside points [0, 0, 1, 0], and soft labels to start and end boundaries.When a segment occupies one time step, its label is [1/3, 1/3, 1/3, 0].Source: §3.4 · p. 9
  • Dynamic weighting gives greater attention to start and end losses because each reference video has only one position of each type.Source: §3.4 · p. 9

3.5 Inference

Inference selects the reference-video segment with the largest joint probability. The score uses a geometric mean of probabilities inside the segment so it is not affected by segment length.

3.5 Inference

  • Inference selects the segment with the largest joint probability, using a geometric mean for inside-segment probabilities so the score is not affected by segment length.Source: §3.5 · p. 9

4 The Video Re-localization Dataset

Existing datasets lack paired query and reference videos with annotated corresponding segments, so the authors reorganize ActivityNet to support re-localization, including evaluation on unseen action classes.

4 The Video Re-localization Dataset

  • Existing datasets do not directly provide paired query/reference videos with annotated corresponding boundaries, and manually collecting and localizing such pairs is labor-expensive.Source: §4 · p. 10
  • ActivityNet is reorganized and split by action class: 160 classes for training, 20 for validation, and 20 for testing, keeping validation and test classes unseen during training.Source: §4 · p. 10
  • After filtering videos with differently labeled overlapping segments, merging same-class overlaps, and removing segments longer than 512 frames, the dataset contains 9,530 segments.Source: §4 · p. 10
  • A segment can serve as a query paired with a reference containing an action of the same label; training pairs are random, while validation and test pairs are fixed.Source: §4 · p. 10

5 Experiments

The experiments compare the proposed model with three baselines, evaluate component contributions, and visualize attention. The proposed method outperforms the baselines, and the ablations report benefits from both cross gating and bilinear matching.

5.1 Baseline Models

5.2 Experimental Settings

  • Evaluation uses average top-1 mAP across tIoU thresholds 0.5–0.9 in 0.1 increments, with C3D features projected to 500 dimensions and downsampled by two.Source: §5.2 · p. 12

5.3 Performance Comparisons

  • The frame-level baseline is better than random guesses but significantly underperforms the proposed model; the video-level baseline is slightly better than the frame-level one.Source: §5.3 · p. 12
  • The proposed method is substantially better than all three baseline methods and accurately detects segment start and end positions.Source: §5.3 · p. 13
  • Testing examples show correct localization despite large query/reference visual differences and actions unseen during training.Source: §5.3 · p. 13, Figure 4 · p. 13

Contributions of Different Components.

Attention.

  • In the attention example, reference-video “hurling” parts receive larger weights when interacting with query-video “hurling” parts.Source: Figure 5 · p. 13, Attention · p. 14

6 Conclusions

The paper defines video re-localization, builds an ActivityNet-based dataset, and proposes a cross gated bilinear matching model with LSTM localization. The authors report that their model is effective and outperforms baseline methods.

6 Conclusions

  • Video re-localization locates a reference-video segment that semantically corresponds to a query video.The task detects the segment’s starting and ending points.Source: §6 · p. 14
  • The paper creates a new video re-localization dataset by reorganizing ActivityNet videos.Source: §6 · p. 14
  • The proposed cross gated bilinear matching method matches query and reference videos, and an LSTM uses the results to localize the query.Source: §6 · p. 14
  • The authors report that their model is effective and outperforms baseline methods.Source: §6 · p. 14

Related papers

Loading 1808.01575v1…