PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos

Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu, Thomas Seidl, Gedas Bertasius

arXiv:2411.14901v1cs.CVcs.CL

TL;DR

VLMs struggle to locate queried events precisely in hour-long videos, where frame limits can discard temporal detail. ReVisionLLM recursively narrows from broad segments to event boundaries and progressively trains from short clips to long videos. It reports stronger results than prior state-of-the-art methods across datasets, including a +2.6% R1@.1 gain over RGNet on MAD.

  • Problem

    VLMs struggle to identify queried events’ start and end times in hour-long videos, and frame limitations can lose temporal detail.

    Source: §1 · p. 1

  • Method

    ReVisionLLM recursively narrows from broad video segments to precise event boundaries and trains progressively from short segments to hour-long videos.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    Across multiple datasets ReVisionLLM outperforms prior state-of-the-art methods, including by +2.6% R1@.1 over RGNet on MAD.

    Source: §1 · p. 2, §5 · pp. 6–7

  • Takeaways & Limitations

    ReVisionLLM establishes a recursive VLM approach for temporal event grounding in hour-long videos.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) excel at retrieving information from lengthy text, but their vision-language counterparts (VLMs) face difficulties with hour-long videos, especially for temporal grounding. Specifically, these VLMs are constrained by frame limitations, often losing essential temporal details needed for accurate event localization in extended video content. We propose ReVisionLLM, a recursive vision-language model designed to locate events in hour-long videos. Inspired by human search strategies, our model initially targets broad segments of interest, progressively revising its focus to pinpoint exact temporal boundaries. Our model can seamlessly handle videos of vastly different lengths, from minutes to hours. We also introduce a hierarchical training strategy that starts with short clips to capture distinct events and progressively extends to longer videos. To our knowledge, ReVisionLLM is the first VLM capable of temporal grounding in hour-long videos, outperforming previous state-of-the-art methods across multiple datasets by a significant margin (+2.6% R1@0.1 on MAD). The code is available at https://github.com/Tanveer81/ReVisionLLM.

1. Introduction

Long-video temporal grounding asks VLMs to locate queried events’ start and end times in hour-long videos, a task current models struggle with. ReVisionLLM recursively narrows attention from broad segments to precise boundaries and uses progressive training to support long-video processing.

1. Introduction

  • Long-video temporal grounding requires locating queried events’ start and end times, but current VLMs struggle with hour-long videos and lose temporal detail when frame-limited.Hour-long video training also poses memory and computational challenges, while poor confidence calibration can produce high-confidence false positives.Source: §1 · p. 1
  • Prior non-LLM methods lack flexible textual-instruction handling, while existing VLMs can take queries but underperform on long-video localization.The introduction also notes VLM-based approaches can underperform non-LLM approaches even on short-video localization.Source: §1 · p. 1
  • ReVisionLLM scans broadly for relevant segments, then recursively refines its focus to pinpoint event boundaries in hour-long videos.The illustrated hierarchy moves from segments such as five minutes toward boundaries such as 3.5 seconds.Source: §1 · p. 2, Figure 2 · p. 2
  • Training progresses from short segments to hour-long videos, using contrastive absent-event segments and temporal feature reduction to support calibration and efficient processing.Feature reduction compresses video segments into compact representations to reduce LLM input tokens.Source: §1 · p. 2
  • +2.6% R1@.1 over RGNet [15] on MAD; ReVisionLLM also processes 43% fewer frames on average than VTimeLLM.The authors report state-of-the-art results across multiple datasets.Source: §1 · p. 2

2. Related Work

Prior work includes VLMs for video tasks and specialized temporal-grounding models, but most VLMs remain limited to short videos. Long-video grounding methods generally lack instruction-following capability, motivating ReVisionLLM’s use of VLMs for long-video queries.

2. Related Work

  • VLMs support video tasks including summarization, captioning, question-answering, and temporal localization, but most remain limited to short videos and lack extensive temporal awareness.ReVisionLLM adds hierarchical perception and temporal feature reduction to adapter fine-tuning for hour-long video localization.Source: §2 · p. 2
  • Temporal-grounding VLMs use varied designs, but their training context length confines them to short clips and they generally underperform traditional non-LLM models.Examples include timestamp integration, time tokens, and dense-caption segment matching.Source: §2 · p. 2
  • Long-video grounding methods use proposal-free or proposal-based pipelines, but prior models lack instruction-following and instead match video and text inputs.The passage describes ReVisionLLM as integrating VLMs to process textual user queries in long-video grounding.Source: §2 · p. 3

3. Method

ReVisionLLM recursively narrows hour-long videos from broad candidate segments to precise event boundaries, using progressive training and confidence-based ranking to support temporal grounding across multiple time scales.

3.1. Problem Overview

  • Given a long, untrimmed video and a text query, the task is to predict the queried event’s precise start and end times.The video is represented as a sequence of RGB frames, and the query corresponds to the target event.Source: §3.1 · p. 3

3.2. The ReVisionLLM Model

  • The model uses sparse features to identify broad segments and dense features within them to refine event boundaries.The Hierarchical Adapter supplies downsampled sparse features at upper levels and dense features at the bottom; the LLM predicts boundaries at each level using prior predictions to refine its focus.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Training

  • Training progresses from short clips, where dense features teach boundary and event-absence prediction, to hour-long videos scanned with sparse features.Contrastive clips without the queried event help calibrate confidence; the long-video stage uses sparse features to identify broader regions of interest.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Inference with Calibrated Confidence

  • At inference, ReVisionLLM ranks boundary predictions using confidence derived from the LLM’s output-word entropy.It averages entropy across generated words, converts mean entropy to confidence, and uses the scores to select top-K predictions.Source: §3.4 · p. 5

4. Experimental Setup

The experiments compare ReVisionLLM with adapted video-language baselines and model variants, evaluating temporal grounding on MAD and VidChapters-7M with Recall@k at specified IoU thresholds.

4.1. ReVisionLLM Baselines

  • The baselines include an hour-long-video VTimeLLM and a short-segment VTimeLLM ranked with CONE’s fine-grained method.The latter aggregates CLIP frame features and ranks segments by similarity to the query.Source: §4.1 · p. 5

4.2. ReVisionLLM Variations

  • The default model scans from longer to shorter segments; unified and reverse-order variants trade parameter or frame-processing efficiency against accuracy.The unified model shares weights across hierarchies, while ReVisionLLM-I starts at the bottom hierarchy, processes all frames, and achieves higher accuracy with added computation.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • ReVisionLLM is reported to achieve state-of-the-art performance on both MAD and VidChapters-7M.Table 1 presents the main results for the two datasets.Source: Table 1 · p. 6

4.2.1 Datasets and Metrics

  • MAD contains about 1,200 hours of movies with 384,000 queries, while VidChapters-7M spans 817,000 videos and chapters up to 12 hours.Grounding is measured with Recall@k at IoU thresholds; text-to-video retrieval uses Recall at Rank k.Source: §4.2.1 · p. 6

5. Results

ReVisionLLM outperforms prior methods on MAD and VidChapters-7M, and its ablations examine modules, model variants, video length, and hierarchy depth. Qualitative examples and text-to-video retrieval results extend the evaluation beyond benchmark grounding.

5. Results

  • +2.6% in R1@.1 and +1.5% in R1@.3 over RGNet; ReVisionLLM sets a new state-of-the-art on MAD.ReVisionLLM-I exceeds RGNet by +4.9% in R1@.1 and +6.3% in R1@.3.Source: §5 · p. 6
  • +4.2% in R1@.7 and +8.8% in R1@.9 over M-DETR; ReVisionLLM sets a new state-of-the-art on VidChapters-7M.The dataset includes YouTube tutorial videos with queried steps, from short clips to videos up to 12 hours.Source: §5 · pp. 6–7

5.1. Ablation Studies

  • The cumulative module ablation reaches R1@.1 of 15.0% and R5@.1 of 25.1%, with the recursive process delivering the largest gains.Contrastive segments raise R1@.1 to 4.8%; confidence-based ranking raises it to 8.4% before recursion.Source: Table 2 · p. 7, §5.1 · p. 7
  • Default ReVisionLLM reaches R1@.1 of 15.0% and R5@.1 of 25.1% using 57% of input frames, while ReVisionLLM-I reaches 17.4% and 31.4%.The unified variant uses fewer trainable parameters but slightly reduces performance; the reverse-order variant uses more frames for higher accuracy.Source: Table 3 · p. 7, §5.1 · p. 7
  • Recursive processing maintains strong performance up to 10 hours, while the non-recursive method fails at that length.Videos were extended by repetition, with the ground-truth moment appearing only once; performance stabilizes beyond approximately five hours.Source: Figure 5 · p. 7, §5.1 · p. 7

5.2. Qualitative Results on MAD Dataset

  • Qualitative examples show accurate localization of queried events amid similar office scenes and of a 5.8-second description within a two-hour movie.The examples involve matching appearance and action details and distinguishing subtle visual differences.Source: Figure 6 · p. 8, §5.2 · p. 8

5.1. Ablation Studies

  • Performance improves with two and three hierarchies as irrelevant regions are progressively filtered and predictions revised recursively.Without hierarchy, the model cannot capture event boundaries; one hierarchy improves results but still struggles with false positives.Source: Table 4 · p. 8, §5.1 · p. 8

5.3. Generalization: Text-to-Video Retrieval

  • On MSRVTT, ReVisionLLM improves over prior state-of-the-art by +2.2% in R@5 and +0.6% in R@10 while remaining competitive in R@1.The method concatenates videos into one hour-long video and localizes the predicted video’s index.Source: Table 5 · p. 8, §5.3 · p. 8

6. Conclusion and Future Work

ReVisionLLM is presented as the first recursively structured VLM for temporal event grounding in hour-long videos. The conclusion reports state-of-the-art performance and identifies audio integration and multi-day videos as future directions.

6. Conclusion and Future Work

  • ReVisionLLM is presented as the first VLM designed with a recursive structure for temporal event grounding in hour-long videos.The paper says its recursive architecture locates events in extensive videos and establishes a new state-of-the-art, outperforming specialized models.Source: §6 · p. 8
  • Future work could integrate audio to improve event comprehension.Source: §6 · p. 8
  • Future work could expand the model to videos spanning multiple days.Source: §6 · p. 8

S1. Additional Implementation Details

The implementation uses CLIP-derived frame and query features, adapted into sparse and dense temporal representations for the language model. Training progresses from projector alignment and short-segment learning to long-video processing, while inference recursively refines candidate regions.

S1. Additional Implementation Details

  • The frozen CLIP L/14 encoder represents each frame with its CLS token and encodes the queried event with its text encoder.The Hierarchical Adapter produces sparse features through attention and feed-forward layers and dense features through linear projection.Source: S1. Additional Implementation Details · p. 12, Figure S1 · p. 12
  • Training first aligns the CLIP CLS token with the LLM embedding space, then fine-tunes the model on short-segment and long-video objectives.The short-video stage focuses on dense features at the lowest hierarchy; sparse features are introduced for upper hierarchies, and the long-video stage uses a separate LoRA module.Source: S1. Additional Implementation Details · pp. 12–13
  • Short-video training includes randomly sampled same-video segments that do not contain the queried event, teaching the model to distinguish similar scenes.Source: S1. Additional Implementation Details · p. 13
  • The unified variant alternates batches of short clips and long videos to train shared parameters across their differing feature and input requirements.The strategy is intended to mitigate catastrophic forgetting; inference samples 250 frames per sliding-window segment.Source: S1. Additional Implementation Details · p. 13
  • Default inference uses two hierarchies to identify regions of interest, then processes dense features from selected segments to pinpoint event boundaries.The inverse variant begins with all segments and recursively filters false positives through upper hierarchies, using their confidence scores to adjust initial predictions.Source: S1. Additional Implementation Details · p. 13

S2. Calibration Confidence

The section evaluates whether ReVisionLLM’s confidence estimates are calibrated, using Expected Calibration Error (ECE) across three IoU thresholds. It reports lower ECE than the VTimeLLM+CONE baseline at each threshold.

S2. Calibration Confidence

  • ECE weights each bin’s absolute confidence–accuracy gap by that bin’s share of the N video segments.Confidence is the average confidence in a bin, accuracy is the proportion of correct predictions, and correctness requires IoU above the specified threshold.Source: S2. Calibration Confidence · pp. 13–14
  • ReVisionLLM has lower ECE than VTimeLLM+CONE at IoU thresholds 0.1, 0.3, and 0.5.The table caption identifies the comparison and says lower values indicate better calibration; the accompanying text reports that ReVisionLLM outperforms the baseline across all thresholds.Source: Table S1 · p. 14, S2. Calibration Confidence · p. 14
  • The analysis uses 10 bins, and the authors report that ECE rises negligibly (+0.01%) as the IoU threshold increases.They interpret the small increase as further evidence of robustness.Source: S2. Calibration Confidence · p. 14

S3. Generalization: Text-to-Video Retrieval

The section adapts ReVisionLLM to text-to-video retrieval by treating multiple videos as segments in a combined sequence. It evaluates retrieval on MSR-VTT using the 9k-Train and 1k-Test splits.

S3. Generalization: Text-to-Video Retrieval

  • The adaptation combines videos into a sequence and represents each with 100 sampled frames encoded as 100 × 768 video features.These video features serve as the segments, while the query is encoded as textual features.Source: S3. Generalization: Text-to-Video… · p. 14
  • For retrieval, ReVisionLLM-I processes each video at the lowest hierarchy, then revises predictions at higher levels to identify the relevant video.The prompts first ask whether the event occurs in a video, then ask which video contains it.Source: S3. Generalization: Text-to-Video… · p. 14
  • MSR-VTT contains 10,000 videos with around 20 human-annotated captions each, ranging from 10 to 32 seconds.Experiments train on the 9k-Train split and evaluate on the 1k-Test set.Source: S3. Generalization: Text-to-Video… · p. 14

S4. Aditional Qualitative Results

Supplementary qualitative examples show ReVisionLLM localizing or retrieving events across MAD, MSRVTT, and VidChapters-7M. They include subtle moments in repetitive scenes, brief events, dynamic actions, and varied long-form video content.

S4. Aditional Qualitative Results

  • On MAD, ReVisionLLM localizes brief actions and subtle lighting changes in visually repetitive scenes from hour-long movies.Examples include a woman walking off on a dim street and an orange sunrise over Mumbai; Figure S2 contrasts these cases with a VLM baseline that fails to locate the events.Source: S4. Aditional Qualitative Results · pp. 14–15, Figure S2 · p. 15
  • On MSRVTT, ReVisionLLM retrieves examples with brief events and dynamic actions, including a dog crossing a road and a car moving along it.Figure S3 shows representative frames for four retrieved events.Source: S4. Aditional Qualitative Results · p. 15, Figure S3 · p. 15
  • On VidChapters-7M, examples span product reviews, Canva workflows, pipeline construction news, and a podcast cameraman’s reaction.Figure S4 presents long-video grounding examples across diverse YouTube content types.Source: S4. Aditional Qualitative Results · p. 15, Figure S4 · p. 16
Loading 2411.14901v1…