Source-linked AI summary
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, Kate Saenko
TL;DR
The paper addresses retrieving temporal video clips that match natural-language activity queries in untrimmed videos. It introduces multilevel early fusion through query-guided proposals, visual conditioning of word-level LSTM processing, and auxiliary captioning, achieving more accurate retrieval than previous methods on Charades-STA and ActivityNet Captions.
Problem
Text-to-clip retrieval requires matching natural-language activity descriptions to temporal segments in untrimmed videos, while prior independent vector embeddings lose fine-grained structure.
Method
The model uses query-conditioned segment proposals, a visual-conditioned word-level LSTM similarity model, and captioning as an auxiliary task.
Results
The approach performs more accurately than previous methods on two challenging benchmarks, Charades-STA and ActivityNet Captions.
Takeaways & Limitations
Earlier and tighter vision-language integration supports retrieval from many candidate clips in untrimmed videos.
Abstract
from arXiv · showhide
We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.
Introduction
Text-to-clip retrieval localizes video segments matching natural-language activity queries, requiring fine-grained understanding of objects, verbs, and temporal order. The paper proposes multilevel vision-language integration through query-guided proposals and word-level similarity modeling.
- Introduction: Text-to-clip retrieval finds temporal segments in untrimmed videos that match natural-language queries.The task requires understanding both language nuances and video contents.
- Introduction: Whole-sentence and whole-clip embeddings can lose word and frame ordering and cannot use one modality to modulate the other.This motivates tighter cross-modal integration.
- Introduction: Query-guided proposals re-weight video features by similarity to the input query, selecting more relevant clips for further processing.Weights can be pooled across queries to reduce computation on a shared video.
- Introduction: A recurrent LSTM processes query words conditioned on candidate-clip visual features and produces a nonlinear similarity score.Each word can associate with a potentially different part of the visual representation.
- Introduction: The model adds clip-to-text captioning as an auxiliary task to learn shared representations for retrieval.The paper reports that this multi-task training improves retrieval performance.
- Introduction: Compared with prior methods, the approach conditions segment proposals on queries and integrates visual features while processing sentences word-by-word.The paper contrasts this with query-independent proposals and sentence-level vector embeddings.
Related Work
Prior video-language localization methods use proposal-based or recurrent approaches, while cross-modal retrieval commonly relies on late fusion into a shared embedding space. This paper instead uses early, word-level fusion for text-to-clip localization and addresses inefficient or inflexible prior localization mechanisms.
- Temporal Activity Detection: Temporal activity detection methods evolved from sliding-window classification toward recurrent and proposal-based localization pipelines.Proposal-based approaches adapt ideas from object detection to video activity detection.
- Temporal Activity Detection: Language queries express richer semantic meaning than fixed activity-class vocabularies for video localization.The paper uses a proposal-based pipeline and adopts R-C3D proposal generation.
- Vision and Language: Late-fusion vision-language methods embed modalities into a common space, whereas early fusion combines representations before similarity measurement.These approaches apply across modalities including image, video, text, and sound.
- Vision and Language: For text-to-clip retrieval, prior models embed entire query sentences as vectors, which can lose fine-grained structure.The paper proposes combining video and query features at the word level instead.
- Vision and Language: Captioning and visual question answering are related vision-language tasks that can influence retrieval and localization research.The related-work discussion presents these tasks as interconnected rather than isolated.
- Localization-based Cross-modal Tasks: Video grounding extends language-based localization from images to retrieving video segments with query sentences.Related localization mechanisms include reconstructing the text query.
- Localization-based Cross-modal Tasks: Prior localization mechanisms are described as either inefficient because of sliding windows or inflexible because of hard-coded designs.This limitation motivates query-conditioned proposal generation.
Approach
The approach integrates language and vision at multiple stages: query-guided proposals narrow candidate segments, while an early-fusion LSTM learns fine-grained query–clip similarity and caption regeneration provides an auxiliary training objective.
- Approach: The model integrates language and vision before matching through query-specific proposals, early fusion, and a multi-task caption-regeneration formulation.These components target temporal activity localization and text-to-clip retrieval.
- Query-Guided Segment Proposals: A learned segment proposal network generates variable-length candidate segments instead of using handcrafted heuristics or multiscale sliding windows.The SPN encodes video frames with C3D and produces temporal segments with pooled C3D features for retrieval.
- Query-Guided Segment Proposals: Query features modulate the proposal network by re-weighting video features according to their similarity to the query representation.Attention weights are computed at temporal locations and applied across video-feature channels before proposal generation.
- Early Fusion Retrieval: The retrieval model feeds each query and candidate clip into a two-layer LSTM whose word-level processing interacts with visual features to predict a nonlinear similarity score.Unlike isolated embeddings with generic similarity metrics, the model permits different query words to associate with visual features through intertwined processing.
- Early Fusion Retrieval: Triplet-based retrieval training encourages a matching clip to score above a negative clip by margin η, with negatives drawn from the same or different videos.The model directly predicts σ(S, R) with the LSTM rather than using a generic measure such as cosine similarity.
- Multi-Task Learning: Captioning is added as an auxiliary loss so the second LSTM layer re-generates the query sentence from proposal visual features.The combined objective uses retrieval and captioning losses, with λ = 0.5 selected through cross-validation.
Experiments
Experiments on Charades-STA and ActivityNet Captions evaluate progressively integrated models, showing that early fusion, query-guided proposals, and captioning supervision improve retrieval. Qualitative and quantitative analyses also expose occasional localization errors and annotation issues.
- Setup: The study compares Random, VE, LSTM, LSTM+QSPN, and LSTM+QSPN+Cap on Charades-STA and ActivityNet Captions.VE separately embeds queries and clips; LSTM predicts query-clip similarity, while QSPN and Cap add query-guided proposals and captioning loss.
- Datasets: 12,408 training and 3,720 testing sentence-clip pairs comprise the Charades-STA retrieval annotations.The annotations were created by decomposing video descriptions, keyword-aligning sentences to segments, and manually verifying the alignments.
- Charades-STA results: The LSTM significantly outperforms VE on Charades-STA, distinguishing opposite actions and similar query subjects more effectively in qualitative similarity matrices.VE sometimes performs better on individual examples, so the improvement is not universal across every case.
- Charades-STA results: The query-guided proposal model improves all metrics over LSTM, while LSTM+QSPN+Cap achieves the highest results on most metrics except R@5 at tIoU 0.3.Captioning supervision further improves most metrics, and the captioning-loss weight is set to λ = 0.5 for retrieval.
- Qualitative analysis: A Charades-STA prediction for “Person takes out a towel” is correct despite being marked incorrect because of inaccurate ground truth.This example highlights an annotation problem affecting qualitative evaluation.
- ActivityNet Captions results: On ActivityNet Captions, LSTM outperforms VE, captioning improves LSTM, query-guided proposals improve all metrics, and the full model gets highest results in all metrics.The reported progression compares five model variants under the standard evaluation protocol.
- Qualitative analysis: Qualitative ActivityNet examples include precise hula-hoop and kite-event localization, but one kite-helping query is localized to an incorrect shot segment.Figure 4 marks predictions with temporal IoU above 0.5 as correct and uses green for correct versus red for incorrect predictions.
Conclusion
The paper addresses text-to-clip retrieval in untrimmed videos with a multilevel vision-language integration model. It reports more accurate retrieval than previous methods on two challenging datasets and identifies further visual-feature modulation as future work.
- Conclusion: The task is temporal localization of video events matching a natural-language query, requiring retrieval among many candidate clips in untrimmed videos.The paper frames this as text-to-clip retrieval rather than classification over a predefined action set.
- Conclusion: The model combines query-conditioned segment proposals, a two-layer LSTM for direct query-clip similarity prediction, and an auxiliary captioning loss.These components fuse language and vision earlier than independent vector-embedding approaches.
- Conclusion: The approach performs more accurately than previous methods on Charades-STA and ActivityNet Captions when retrieving clips from untrimmed videos.The conclusion states the cross-dataset outcome without reporting a specific metric value.
- Conclusion: Future work will further exploit language features to modulate visual-feature extraction.This direction is presented as an extension of the current vision-language integration strategy.