Source-linked AI summary
Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos
Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, Shilei Wen
TL;DR
Video grounding must localize open-ended language in videos without exhaustively enumerating candidate clips. The paper learns an RL agent that progressively adjusts temporal boundaries, enhanced by supervised multi-task learning, and reports state-of-the-art performance on two datasets while observing at most 10 clips per video.
Problem
Video-grounding methods typically slide over entire videos or exhaustively rank candidate clip-sentence pairs, limiting grounding of open-ended descriptions without exhaustive enumeration.
Method
An end-to-end reinforcement-learning agent reads the description, watches video content and current boundaries, and progressively moves temporal boundaries using a policy with supervised multi-task learning.
Results
The method achieves state-of-the-art performance on ActivityNet DenseCaption and Charades-STA, improving over prior methods while observing up to 10 clips per video.
Takeaways & Limitations
Sequential decision making with supervised boundary information provides an effective formulation for temporally grounding natural-language descriptions in videos.
Abstract
from arXiv · showhide
The task of video grounding, which temporally localizes a natural language description in a video, plays an important role in understanding videos. Existing studies have adopted strategies of sliding window over the entire video or exhaustively ranking all possible clip-sentence pairs in a pre-segmented video, which inevitably suffer from exhaustively enumerated candidates. To alleviate this problem, we formulate this task as a problem of sequential decision making by learning an agent which regulates the temporal grounding boundaries progressively based on its policy. Specifically, we propose a reinforcement learning based framework improved by multi-task learning and it shows steady performance gains by considering additional supervised boundary information during training. Our proposed framework achieves state-of-the-art performance on ActivityNet'18 DenseCaption dataset and Charades-STA dataset while observing only 10 or less clips per video.
1 Introduction
Video grounding seeks temporal boundaries for open-ended natural-language descriptions, but prior methods exhaustively enumerate candidates. The paper formulates grounding as sequential boundary adjustment with reinforcement learning and supervised multi-task learning.
- Motivation: Video grounding localizes an open-ended natural-language description within an untrimmed video by determining a matching clip’s start and end timestamps.Natural-language grounding also captures involved entities and their interactions, unlike approaches restricted to predefined activity labels.
- Motivation: Mapping descriptions to predefined action labels can lose temporal information, such as when a parent begins watching a child play drums.Action-localization methods are therefore not directly applicable to arbitrary natural-language descriptions.
- Motivation: Existing video-grounding methods slide over entire videos or rank all possible clip-sentence pairs, requiring exhaustive candidate enumeration.This limitation is especially problematic for long, untrimmed videos, where grounding should rely on several glimpses.
- Approach: The proposed agent repeatedly observes the description, video, current clip, and boundaries, then moves the temporal boundaries according to a learned policy.Each action changes the environment and receives a reward reflecting its performance.
- Approach: Supervised learning is combined with reinforcement learning in a multi-task framework to improve environmental information and state-space exploration.The framework includes supervised temporal IoU and boundary learning alongside policy learning.
- Results: Experiments on two well-known datasets report substantial improvements over state-of-the-art methods.The supplied introduction states that the results verify the effectiveness of the proposed method.
2 Related Work
Prior work grounds language in visual content through exhaustive regression or ranking, while related vision tasks address different outputs or restricted label spaces. This paper applies reinforcement learning to sequentially ground arbitrary descriptions in videos.
- Video grounding: Earlier video-grounding methods use sliding windows or exhaustive ranking of clip-sentence pairs, creating candidate-enumeration and false-positive challenges.Ranking methods may also require pre-segmented videos.
- Related visual tasks: Image grounding localizes phrases in images, whereas this work temporally grounds open-ended, arbitrary descriptions in videos.The paper distinguishes its focus from image grounding and other visual-content tasks.
- Related visual tasks: Video captioning generates text from visual data, while sentence-based video retrieval finds an entire video relevant to a query rather than a temporal segment.The paper presents visual grounding as the conjugate task of visual description generation.
- Reinforcement learning: The proposed architecture uses an actor-critic module to model sequential decisions for grounding descriptions.The supplied figure description specifies seven actions for adjusting temporal boundaries and a state vector combining textual, visual, and location features.
- Reinforcement learning: Reinforcement learning is suited to sequential decision-making problems and has been applied to task-specific policies in computer vision.The paper motivates using RL for iterative temporal-boundary adjustment.
3 Methodology
The method formulates video grounding as sequential decision making: an agent observes the description, video, current clip, and boundaries, then iteratively adjusts temporal boundaries. Reinforcement learning is combined with supervised boundary information in a multi-task objective.
- Environment and sequential decisions: The agent iteratively observes the description, video, current grounding clip, and boundaries, then selects actions to adjust the temporal grounding boundaries.It continues until a STOP action or the maximum number of steps is reached.
- Environment and action space: The action space contains seven predefined boundary-adjustment actions, including moving either endpoint, shifting both endpoints, and stopping.The shifting actions are included to enable faster temporal boundary adjustment.
- Observation and policy networks: The observation network fuses sentence, global-video, local-video, and normalized-boundary features into the state vector used for policy decisions.The features are encoded with fully connected layers before fusion.
- Observation and policy networks: An actor-critic module with a GRU outputs an action policy and estimates the value of the current state.The policy defines an action distribution, while the critic provides a state-value estimate.
- Reward design: The reward favors actions that increase temporal IoU, penalizes non-improving steps, and traces later rewards backward through a discount factor.The objective balances boundary accuracy against the number of actions.
- Supervised learning and RL: Multi-task learning combines reinforcement-learning losses with supervised tIoU and location regression losses to improve state representations and boundary prediction.The supervised loss predicts state-ground-truth matching and boundary locations, while the RL loss learns stepwise boundary adjustment.
4.1 Datasets and Evaluation Metric
The experiments use ActivityNet DenseCaption and Charades-STA, datasets containing descriptions paired with temporal boundaries. Models are evaluated with Acc@0.5, which measures whether predicted and ground-truth temporal segments have tIoU above 0.5.
- Datasets: ActivityNet DenseCaption provides 37,421 training and 17,505 test descriptions with corresponding ground-truth temporal boundaries.Charades-STA was processed from a video classification and captioning dataset for this grounding task.
- Evaluation metric: Acc@0.5 reports whether the temporal IoU between a model’s grounding result and the ground truth exceeds 0.5.This is the evaluation criterion used following prior work.
- Evaluation results: Table 1 presents the evaluation results of the different methods.The table is used to compare methods on the experimental benchmarks.
4.2 Implementation Details
Implementation uses Adam optimization, greedy action selection at test time, and dataset-specific visual modalities. The maximum decision horizon is 10 steps, with RGB-only input for DenseCaption and RGB plus optical flow for Charades-STA.
- Optimization and inference: Adam optimization uses an initial learning rate of 1e-3 during training.The test-time action is selected greedily at each time step.
- Initialization and modalities: The initial grounding location is the central half of the video, L(0) = [N/4, 3N/4].N denotes the total video length.
- Initialization and modalities: DenseCaption models use RGB frames only, whereas Charades-STA models use both optical flow and RGB modalities.The difference follows the availability of optical-flow frames in the datasets.
- Model configuration: Tmax is set to 10, and the description is embedded into a 2400-dimensional skip-thought vector.The observation network encodes description, global, local, and boundary features through fully connected layers.
4.3 Comparison with Baselines
The proposed method is compared with adapted action-localization and ranking baselines on DenseCaption and Charades-STA, achieving the best performance while avoiding exhaustive candidate enumeration.
- Baselines: The evaluation compares the proposed method with TALL, MCN, and RL-Loc baselines on ActivityNet DenseCaption and Charades-STA.TALL and MCN are adapted from action localization or ranking approaches, while RL-Loc adapts reinforcement learning for video grounding.
- Results: The proposed method achieves the best performance against all baselines on both datasets.Table 1 reports the evaluation results across the two benchmarks.
- Results: 10.0% and 10.9% accuracy improvements over TALL are reported on DenseCaption and Charades-STA, respectively.The reported improvements are 9.5% and 5.9% over MCN, and 2.2% and 4.2% over RL-Loc, on the same datasets.
- Efficiency: The proposed model and RL-Loc observe up to 10 clips per video, whereas MCN enumerates 21 clips and TALL examines 45.8 candidates on average.This comparison describes inference computation cost across the methods.
4.4 Ablation Study
The ablations examine reward settings, discounting, explicit location information, supervised regression components, and feature modalities, showing trade-offs between accuracy, computation, and state representation.
- Impact of Penalty Factors: Lower penalty factor φ increases both average steps and grounding accuracy, revealing a trade-off between computation cost and accuracy.With φ=1, the agent takes one step; with φ=0.001, it takes 9.99 steps and achieves the best accuracy, while φ=0.1 is presented as a balanced choice.
- Accumulated Reward: Intermediate discount factors outperform γ=0 and γ=1 because they model dependencies between successive actions without overextending reward accumulation.The experiment fixes φ=0.001 and varies γ from 0 to 1.
- Network Components: Removing all three components reduces Charades-STA Acc@0.5 from 36.7% to 13.0%, while leave-one-out tests also show substantial drops.The three components are explicit location features, tIoU regression, and location regression.
- Network Components: Removing explicit location features lowers Acc@0.5 by 3.4% on DenseCaption and 1.9% on Charades-STA.The results support using explicit location information to model the environment for the observation network.
- Supervised Learning: Disabling location regression lowers Acc@0.5 from 36.9% to 35.9% on DenseCaption and from 36.7% to 35.1% on Charades-STA.Disabling both tIoU regression and location regression lowers the values further to 34.5% and 34.9%, respectively.
- Multi-Modality Experiment: Combining RGB and optical-flow features raises Charades-STA accuracy to 36.7%, compared with 36.2% for RGB and 35.8% for flow alone.The paper notes that better feature fusion methods could improve performance further.
4.5 Qualitative Results
A qualitative action sequence shows the agent progressively adjusting temporal grounding boundaries until it correctly localizes the described content.
- Action Sequence: The agent correctly grounds a natural-language description after five boundary-adjustment steps from the initial location.The sequence visualizes how grounding boundaries are updated step by step.
5 Conclusion
The paper formulates temporal video grounding as sequential decision making and combines reinforcement learning with supervised multi-task learning in an end-to-end model. Experiments report new state-of-the-art performance on two datasets.
- Conclusion: Temporal grounding is modeled as an agent that reads the description, watches the video and current localization, and iteratively moves the grounding boundaries.The formulation treats the task as sequential decision making.
- Conclusion: Combining reinforcement learning with supervised learning in a multi-task framework enables end-to-end training of the model.The supervised components are used alongside the reinforcement-learning formulation.
- Conclusion: Experiments demonstrate new state-of-the-art performance on two well-known datasets.The conclusion summarizes the reported experimental outcome without specifying a dataset-specific metric.