PaperlayerView exact revision on arXiv

Source-linked AI summary

Number it: Temporal Grounding Videos like Flipping Manga

Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, Xu Yang

arXiv:2411.10332v3cs.CV

TL;DR

Vid-LLMs can understand video content but struggle to localize event boundaries in time. NumPro adds unique numbers to video frames so models can associate events with temporal positions. The paper reports state-of-the-art performance in both training-free and fine-tuned settings.

  • Problem

    Vid-LLMs can recognize video events but struggle to describe when they begin and end, limiting their temporal grounding ability.

    Source: §1 · p. 2

  • Method

    NumPro overlays unique numerical identifiers on video frames, and NumPro-FT fine-tunes Vid-LLMs on a dataset augmented with those numbers.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    NumPro consistently achieves state-of-the-art performance in both training-free and fine-tuned settings.

    Source: §5 · p. 8

  • Takeaways & Limitations

    NumPro supports fine-grained temporal understanding while preserving general video comprehension.

    Source: §5 · p. 8

    The evaluated VTG datasets focus mainly on short videos, typically 30 seconds to 3 minutes, and frame numbers may obscure critical visual details, watermarks, or logos.

    Source: §14 · p. 15

Abstract

from arXiv · show

Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro.

1. Introduction

Vid-LLMs can recognize video events yet struggle to express when they begin and end. NumPro adds numbered frame identifiers so models can connect visual content with temporal positions, and NumPro-FT further improves VTG performance.

1. Introduction

  • Vid-LLMs can recognize events but may fail to describe their temporal boundaries, because visual-language alignment does not directly provide a mechanism for interpreting when events occur.The paper frames this as a gap between identifying what happens and describing when it happens.Source: §1 · p. 2
  • NumPro overlays unique numerical identifiers on video frames, using the sequence like numbered manga panels to link visual content to a timeline.Given an event query, the model associates relevant frame features with overlaid numbers and translates them into text.Source: §1 · p. 2
  • Unlike approaches that add temporal tokens or modify vocabulary, NumPro uses frame numbers as temporal cues and is described as transferable across tasks and datasets.The paper also presents fine-tuning on a NumPro-enhanced dataset, NumPro-FT, as a further performance improvement.Source: §1 · p. 2
  • The authors evaluate NumPro in training-free and fine-tuned settings across standard VTG benchmarks and report that it is effective across models and datasets.They also study number-prompt font size, color, and position to balance detectability against interference with video content.Source: §1 · p. 2

2. Related Work

Prior VTG approaches use temporal data, tokens, embeddings, or video structure, often requiring retraining or model adaptations. NumPro instead marks frames with numbers; the paper analyzes the recognition-boundary gap, presents training-free and fine-tuned variants, and selects a prompt design intended to balance number readability with video-content preservation.

2. Related Work

  • Prior VTG methods use timestamp-enriched fine-tuning, temporal tokens, embeddings, or video segmentation, often at the cost of retraining or specialized adaptations; NumPro uses visual number tags instead.The related-work section describes NumPro as the first use of simple numerical tags in video frames to improve temporal grounding.Source: §2 · pp. 2–3

3. Number-Prompt Approach

  • The approach has training-free and fine-tuned settings: NumPro marks frames directly, while NumPro-FT fine-tunes Vid-LLMs on a NumPro-augmented VTG dataset without architectural modifications.The paper describes the approach as a way to associate visual content with temporal information.Source: §3 · p. 3, §3.2 · p. 4

3.1. Attention Analysis

  • In a Qwen2-VL-7B analysis, query attention aligns with relevant frames, yet the model can still hallucinate temporal boundaries such as “from 200 to 599.”The authors use this observation to motivate mechanisms connecting spatial feature alignment with temporal reasoning.Source: §3.1 · pp. 3–4

3.2. NumPro and NumPro-FT

  • In the training-free setting, models use built-in OCR to read each frame number and link it to an event query; NumPro-FT fine-tunes on augmented data to align numbers with temporal spans.The fine-tuning description freezes the visual encoder and applies LoRA to adjust the LLM.Source: §3.2 · p. 4

3.3. Design of Numerical Prompt

  • The design study balances number recognition against caption fidelity, using CLIP-based Number Accuracy and Caption Accuracy measures on COCO image-caption pairs as proxies for video frames.It compares font size, color, and placement; the authors describe CLIP-based design choices as approximate rather than definitive guidance for Vid-LLMs.Source: §3.3 · pp. 4–5
  • Bottom-right placement, red color, and font size 40 are selected for NumPro; larger numbers improve number accuracy but can reduce caption accuracy, while center placement reduces caption accuracy.Red performs best for number accuracy, and black is least effective on that measure.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate NumPro and NumPro-FT on moment retrieval and highlight detection, then test design choices, frame sampling, and effects on general video-QA. NumPro-FT and training-free NumPro are compared with prior methods and across Vid-LLMs.

4.1. Implementation Details for NumPro-FT

  • NumPro-FT uses a 220k-sample instruction set built from 70k DiDeMo and ActivityNet Caption pairs plus VTimeLLM data, and trains LongVA-7B-DPO for three epochs.Training uses a total batch size of 128 and LoRA for the LLM component.Source: §4.1 · p. 5

4.2.1. Comparison with State-of-the-Art Methods

  • NumPro-FT surpasses prior SOTA by 11.8% on Charades-STA R@0.3 and +8.5% in highlight-detection mAP.It establishes new SOTA across the reported moment-retrieval metrics and also exceeds prior SOTA by +3.7% in HIT@1 for highlight detection.Source: Table 1 · p. 6, §4.2.1 · p. 6
  • Training-free NumPro increases GPT-4o’s ActivityNet mIoU by 9.9%, exceeding prior SOTA by 0.9%, while Qwen2-VL-7B averages a 24.7% mIoU increase across datasets.For highlight detection, NumPro averages +1.55% mAP and +1.6% HIT@1 across models.Source: §4.2.1 · pp. 5–6

4.2.2. Effectiveness of NumPro across Vid-LLMs

  • Across additional Vid-LLMs, NumPro yields average mIoU gains up to 18.1% on Charades and 14.0% on ActivityNet; NumPro-FT gains 9.8% over conventional fine-tuning on ActivityNet.The passage reports that NumPro-FT consistently outperforms conventional fine-tuning, particularly on longer videos.Source: §4.2.2 · p. 6

4.2.3. Qualitative Results

  • In two challenging ActivityNet cases, NumPro delineates event boundaries where TimeChat misses or mistimes events and VTimeLLM misses or overestimates them.The examples include minimal scene changes and, separately, short events with frequent scene changes.Source: §4.2.3 · pp. 6–7

4.3. Validation of NumPro Design

  • Bottom-right placement performs comparably to the best position, balanced font sizes avoid the poor results of very large or small text, and red performs best among tested colors.Directly overlaying timestamps such as “10.5s” performs worse than frame numbers.Source: §4.3 · p. 7

4.4. Investigation on the Sampling of NumPro

  • Performance rises with more labeled frames, uniform sampling generally retains higher accuracy, and labeling 20% of frames already provides a substantial boost.Uniformly labeling 80% of frames surpasses prior state-of-the-art.Source: §4.4 · p. 7

4.5. Influence on General Video-QA

  • On general video-QA, NumPro minimally affects comprehension metrics, with Distraction Overlap changing by -0.02 and Temporal Understanding by +0.1.The passage reports that models maintain robust general video-QA performance while excelling at precise VTG.Source: §4.5 · p. 7

5. Conclusion

NumPro overlays frame numbers to improve Vid-LLM temporal grounding, and NumPro-FT further advances performance. The paper reports state-of-the-art VTG results while maintaining robust video understanding on general video-QA.

5. Conclusion

  • NumPro is a visual prompt that overlays frame numbers and uses Vid-LLMs’ OCR and visual-language alignment to map events to temporal boundaries.The conclusion describes the approach as supporting fine-grained temporal understanding while preserving general video comprehension.Source: §5 · p. 8
  • NumPro achieves state-of-the-art performance in training-free and fine-tuned settings, with NumPro-FT establishing new SOTA across VTG tasks.The conclusion describes integration with both closed-source and open-source Vid-LLMs.Source: §5 · p. 8
  • The reported minimal impact on general video-QA indicates that NumPro augments VTG while maintaining robust video understanding.Table 4’s caption identifies correctness, detail orientation, contextual understanding, temporal understanding, and consistency as its video-QA metrics.Source: §5 · p. 8, Table 4 · p. 8

6. Experimental Details

Experimental frame extraction settings vary by task and training regime; all reported frames are resized to roughly 336 × 336 pixels.

6.1. Moment Retrieval

  • For moment retrieval, training-free NumPro extracts video frames at 1 FPS and resizes each frame to 336×336.The NumPro-FT setting instead extracts frames at 0.5 FPS during training and inference because of GPU memory constraints.Source: §6.1 · p. 13

6. Experimental Details

  • The reported frame resolution is 336 × 336 in both tasks, while extraction frequency differs between moment retrieval settings and highlight detection.Moment-retrieval NumPro-FT and both highlight-detection settings use 0.5 FPS.Source: §6.1 · p. 13, §6.2 · p. 13

6.2. Highlight Detection

  • For highlight detection, both NumPro and NumPro-FT extract frames at 0.5 FPS because saliency scores are labeled every two seconds.Each frame is resized to 336 × 336.Source: §6.2 · p. 13

7. Hallucination in Vid-LLMs for VTG

The paper reports hallucinated or repetitive temporal predictions from general and VTG-tuned Vid-LLMs. The examples include implausible frame ranges, short or repeated intervals, and coarse time increments.

7.1. General Vid-LLMs

  • Qwen2-VL-7B can predict frame 000–200 for a 19-frame video, while Qwen2-VL-72B can produce an incomplete interval; two LLaVA models often repeat short intervals.The reported short intervals account for over 50% of predictions for each LLaVA model.Source: §7.1 · p. 13

7.2. VTG-Tuned Vid-LLMs

  • VTimeLLM frequently predicts [34] and [0, 17], accounting for 49.34% and 16.34% of predictions, respectively.The passage characterizes this pattern as hallucination and overfitting to specific frame numbers.Source: §7.2 · p. 13
  • TimeChat often outputs intervals in multiples of five, with [0, 5] and [0, 10] appearing in over 42% of predictions.The passage characterizes the pattern as hallucination and overfitting.Source: §7.2 · p. 13

8. Additional Attention Analysis

Vid-LLMs can attend to relevant video segments yet struggle to express their temporal boundaries accurately. A simple attention-based boundary estimate improves mIoU over direct predictions.

8. Additional Attention Analysis

  • Attention maps highlight relevant video segments even when model outputs are incomplete, incorrect, or extend beyond the video duration.The analysis suggests models locate relevant segments but fail to translate them into precise textual boundaries.Source: §8 · p. 13
  • 11.9% higher mIoU than direct predictions is achieved by selecting the two highest-attention frames as segment boundaries.The experiment used Qwen2-VL-7B on Charades-STA.Source: §8 · p. 14
  • The analyses identify inaccurate textual boundary expression—not segment identification—as a primary Vid-LLM bottleneck for temporal grounding.Source: §8 · pp. 13–14

9. Additional Video Benchmark Results

NumPro improves temporal-understanding performance on additional video QA benchmarks and produces gains in highlight detection across general Vid-LLMs, including results that surpass prior state of the art.

9. Additional Video Benchmark Results

  • NumPro improves Vid-LLM generalization on MVBench and VideoMME, notably for Scene Transition and Temporal Perception.The evaluation used 1 FPS and red, size-40 numbers positioned at the bottom right.Source: Table 8 · p. 14, §9 · p. 14

10. Ablation Results on Highlight Detection

  • 10.8% higher mAP and 16.8% higher HIT@1 are achieved with NumPro-FT, surpassing state-of-the-art results.The results are on QVHighlights and generalize across various general Vid-LLMs.Source: §10 · p. 14

11. Ablation Results on NumPro-FT Designs

The design ablations favor unobtrusive frame numbers over larger ones, while comparisons and visualizations report effective grounding across timestamp formats, dialogue, moment retrieval, and highlight detection.

11. Ablation Results on NumPro-FT Designs

  • A font size of 60 causes a 7.8% drop in R@0.3 compared with size 40, alongside lower general performance, reduced Caption Accuracy, and more outliers.Although size 60 improves Number Accuracy, the authors attribute its trade-off to interference with video-content understanding.Source: §11 · p. 14

12. NumPro with Accurate Timestamps

  • Frame numbers perform comparably to minute-level timestamps on Charades-STA and outperform them on ActivityNet, which has longer videos and more annotations.Frame numbers are chosen as discrete units that map to timestamps through the sampling rate.Source: §12 · p. 14, Table 9 · p. 14

13.1. Dialogue

  • NumPro-equipped Qwen2-VL-7B supports multi-turn video dialogue, including tracking score changes, identifying celebrities, and extracting text via OCR.The passage contrasts this flexibility with VTG-specific models.Source: §13.1 · p. 14

13.2. Moment Retrieval

  • Additional visualizations show event-boundary identification across videos of varying lengths and content, with reported improvements over prior state-of-the-art methods.Source: §13.2 · p. 15

13.3. Highlight Detection

  • For highlight detection, NumPro predicts segment start and end times and assesses saliency scores on a 1-to-5 scale.The passage reports consistently high mAP values.Source: §13.3 · p. 15

14. Limitations

The authors identify dataset duration, possible visual obstruction, and the fixed sampling rate as limitations requiring further investigation.

14. Limitations

  • Current VTG datasets predominantly cover short videos of 30 seconds to 3 minutes, leaving hour-long video scalability and generalizability to be tested.Source: §14 · p. 15
  • Frame numbers may obscure critical visual details, watermarks, or logos in some scenarios.Suggested future options include adaptive positioning or opacity.Source: §14 · p. 15
  • NumPro was evaluated at a fixed 1 FPS, and the effect of other sampling rates remains underexplored.The authors propose investigating adaptive frame rates.Source: §14 · p. 15
Loading 2411.10332v3…