Source-linked AI summary
SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, Li Zhang
TL;DR
Vid-LLMs struggle to preserve both detailed frame semantics and broad temporal coverage, limiting fine-grained video understanding. SlowFocus grounds question-relevant segments, densely samples them, and combines local high-frequency features with global context, supported by specialized training and FineAction-CGR. The method significantly outperforms existing models on detailed temporal understanding while remaining competitive on coarse-grained benchmarks.
Problem
Vid-LLMs struggle to retain high-quality frame-level semantics and comprehensive video-level temporal information simultaneously.
Method
SlowFocus grounds question-relevant temporal segments, densely samples them, mixes local high-frequency details with global context, and trains Vid-LLMs for temporal localization and reasoning.
Results
SlowFocus significantly outperforms existing models on detailed temporal understanding and reasoning, achieving 66.68 mIoU for temporal grounding and 53.10% accuracy for reasoning.
Takeaways & Limitations
SlowFocus provides a framework for fine-grained video understanding and remains competitive with state-of-the-art models on coarse-grained video tasks.
Takeaways & Limitations
The method may still produce inaccurate predictions because spatial details remain ambiguous when high-resolution video research is limited.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.
1 Introduction
Vid-LLMs face a token-allocation trade-off: sparse sampling loses temporal details, while frame compression loses frame-level semantics. SlowFocus addresses this by grounding relevant segments, densely sampling them, and combining local high-frequency details with global context.
- Challenge: Vid-LLMs must trade off sampling frequency against frame-token capacity under fixed computational budgets.Sparse sampling omits temporal details, while excessive frame-feature compression degrades semantic and spatial context.
- SlowFocus: SlowFocus identifies question-relevant temporal segments and densely samples them to preserve high-frequency local features.The mechanism assumes relevant details are concentrated in one or several clips.
- SlowFocus: A specialized temporal encoder and multi-frequency mixing attention module combine local details with global contexts for temporal comprehension.The design models relationships between frames and captures inter-frame context.
- Training: Training and inference strategies improve temporal localization and fine-grained temporal reasoning for the SlowFocus mechanism.The training process uses dense video captioning and temporal grounding before adapting the model to high-frequency temporal reasoning.
- Evaluation: FineAction-CGR is introduced to evaluate fine-grained temporal understanding, especially reasoning based on temporal details.The benchmark addresses limitations in existing benchmarks for specific temporal-related tasks.
- Results: SlowFocus significantly outperforms existing models, particularly on tasks requiring detailed temporal understanding and reasoning.The supplied passage states the paper-level superiority claim without reporting a numerical comparison.
2 Related works
Related work extends LLM-based visual understanding from images to videos, while fine-grained video understanding focuses on locating and describing specific events. Prior work commonly frames this area through temporal grounding and dense video captioning.
- Vision large language models: Vision-language research has aligned visual representations with LLMs through methods such as Q-Former and instruction-following data.The cited related work includes BLIP-2, MiniGPT-4, and LLaVA.
- Video large language models: Video LLMs extend LLM-based multimodal reasoning to video through video-centric instruction data and annotation frameworks.Examples include VideoChat, Video-ChatGPT, and Video-LLaMA.
- Fine-grained video understanding: Fine-grained video understanding requires precisely locating and understanding specific events, commonly through temporal grounding and dense video captioning.Temporal grounding identifies segment timestamps for a text query, while dense video captioning localizes and captions events.
3 Method
SlowFocus combines query-guided temporal grounding, dense sampling of relevant clips, and global low-frequency context to improve fine-grained video understanding. Its method includes mixed-frequency attention, temporal relationship modeling, and a three-stage training strategy.
- Preliminary: Vid-LLMs commonly use low-frequency frame sampling and visual-token compression, creating a trade-off between frame-level detail and video-level temporal coverage.Fine-grained tasks suffer when sparse sampling omits local temporal details or excessive compression degrades frame semantics.
- Relevant segment grounding: SlowFocus first uses the question and low-frequency global frames to identify relevant temporal segments within the video.The model reformulates the question as a temporal-grounding query and predicts segment τ containing question-relevant details.
- Mixed-frequency sampling: It then densely samples the grounded segment, dynamically adjusting the sampling interval to obtain high-frequency frames and visual tokens.The sampling interval depends on the segment length and target sample count, while the full video retains low-frequency context.
- Multiple-frequency mixing attention: Multiple-frequency mixing attention integrates global low-frequency context with detailed high-frequency features before the language model predicts the answer.This integration addresses the need to relate multiple events and enrich local features with long-term context.
- Temporal relationship modeling: SlowFocus models temporal relationships by encoding sampled-frame positions into discretized temporal embeddings added to visual features.The temporal encoder represents relative frame positions even when visual tokens are unevenly distributed over time.
- Training strategy: Training proceeds through modality alignment, boundary enhancement, and SlowFocus adaptation to improve temporal localization and fine-grained temporal reasoning.The stages respectively align visual features, enhance temporal comprehension, and adapt the Vid-LLM to mixed-frequency processing.
4 FineAction-CGR benchmark
FineAction-CGR is introduced to evaluate fine-grained video understanding, especially temporal reasoning, through a video-disjoint benchmark with multiple instruction-following tasks. Its construction combines clip preprocessing, generated and ground-truth annotations, and task-specific evaluation.
- FineAction-CGR addresses the lack of challenging benchmarks for temporal video understanding tasks.
- Benchmark construction: The benchmark splits FineAction videos into 75% training and 25% testing sets without overlap.
- Benchmark construction: Video preprocessing segments videos into clips, yielding 62,912 video clips for fine-grained information.
- Annotation construction: Annotations combine time segments, captions, and action labels, using GPT-4V and a fine-tuned Video Recaptioner Model for clip-level captions.
- Downstream tasks: Four downstream tasks are designed: segmented captioning, temporal video grounding, temporal video reasoning, and multi-turn QA.
- Evaluation: Temporal grounding is evaluated with mean Intersection over Union, alongside metrics for the benchmark’s other tasks.
5 Experiments
Experiments evaluate SlowFocus on fine-grained, coarse-grained, long-video, qualitative, and ablation settings. Results show strong fine-grained performance, competitive transfer across video benchmarks, and benefits from mixed-frequency sampling and staged training.
- Fine-grained results: SlowFocus achieves 66.68 mIoU for temporal grounding and 53.10% accuracy for reasoning on FineAction-CGR.
- Fine-grained results: Baseline fine-tuning improves performance, but the remaining gap supports the proposed explanations for competing models’ weaker results.
- Coarse-grained results: SlowFocus matches state-of-the-art performance on MSVD-QA, MSRVTT-QA, ActivityNet-QA, and a video-based generative benchmark.
- Long-video results: On MovieChat-1K, SlowFocus reaches 58.6% accuracy in global mode and 48.1% in breakpoint mode despite no long-video-specific training.
- Fine-grained results: SlowFocus accurately identifies question-relevant temporal details and answers posed questions in qualitative examples.
- Ablations: The low-frequency-only baseline obtains 32.54 mIoU and 30.25 accuracy, while temporal components and higher dense-sampling counts improve performance.
- Ablations: Integrating all training stages produces the highest performance; stage 2 alone improves grounding by 51.56 mIoU and stage 3 alone improves reasoning by 25.14 accuracy.
- Ablations: With constant total tokens, increasing frame tokens improves performance while reducing global sampling frequency has limited effect within SlowFocus.
6 Conclusion and limitations
SlowFocus improves temporal localization and fine-grained video understanding by combining high-frequency local details with low-frequency global context. However, ambiguity in spatial details can still cause inaccurate predictions.
- Conclusion: SlowFocus improves temporal localization by identifying relevant temporal segments from a query.Its temporal encoder and multi-frequency mixing attention module model temporal relationships and inter-frame context.
- Conclusion: SlowFocus combines high-frequency local details with low-frequency global contexts for fine-grained video reasoning.
- Conclusion: The method demonstrates superior performance on newly established fine-grained video understanding benchmarks.
- Limitations: Ambiguous spatial details may still lead to inaccurate predictions because maintaining high resolution in video remains challenging.
A Overview
The appendix provides implementation, data construction, statistics, instruction-generation prompts, and broader-impact discussion details.
- Appendix contents: The appendix includes more implementation details in Section B.
- Appendix contents: It documents data construction details in Section C.
- Appendix contents: It reports data statistics in Section D.
- Appendix contents: It presents prompts used for instruction generation in Section E.
- Appendix contents: It discusses broader impacts in Section F.
B More implementation details
The implementation uses a frozen CLIP visual encoder and a projector for visual-language alignment, while temporal segments are represented with normalized discrete time tokens.
- Visual encoding: The frozen CLIP-ViT-L-14 visual encoder accepts 224 × 224 image inputs.
- Visual encoding: A trained projector layer aligns visual features with the pretrained LLM word embedding.
- Temporal representation: Temporal segments are represented textually as “from s to e,” where s and e denote normalized start and end points.
- Temporal representation: The normalized temporal points range from 000 to 999 in a discretized temporal token space.
- Temporal representation: Training supervision in stages 2 and 3 enables the Vid-LLM to locate temporal segments.
C Data construction
FineAction-CGR is constructed through video splitting, annotation integration, and diverse temporal tasks designed for fine-grained video understanding. Its preprocessing preserves segment information density and action chronology, yielding 62,912 video clips.
- Data construction: Proper splitting keeps clip content semantically consistent, supporting chronological inference and downstream action recognition and temporal reasoning.
- Data construction: The splitting strategy detects lens switching and stitches adjacent semantically similar events, including by merging short clips.
- Data construction: 62,912 video clips were obtained after splitting and stitching videos to preserve information density and temporal consistency.
- Annotation construction: Annotations combine action labels, clip and ground-truth time segments, and captions generated for videos and clips.
- Downstream tasks: FineAction-CGR contains captioning, temporal video grounding, temporal video reasoning, and multi-turn question-answering tasks.
- Downstream tasks: Its temporal tasks include first/last-time grounding, action-sequence reasoning, action counting, and progressive multi-turn questions.
D Data statistics
FineAction-CGR contains 11,188 videos and 131,984 question-answer pairs spanning multiple fine-grained video-understanding tasks. The dataset materials include examples and task-specific prompts for generating these annotations.
- Task examples: Figures 8 and 9 provide examples of different tasks in FineAction-CGR.
- Annotation scale: 131,984 QA pairs are generated across captioning, temporal grounding, and temporal reasoning tasks.The task distribution is illustrated in Figure 6 right.
- Prompt design: Figures 10–13 show prompts used to generate instruction data for action recognition, temporal grounding, temporal reasoning, and multi-turn QA.Some in-context examples are omitted because of page-length constraints.
F Broader impacts
The paper describes potential applications of SlowFocus in video analysis, surveillance, content moderation, and education, while also presenting prompt templates for several video-understanding tasks. It acknowledges that the technology could be misused to analyze private videos and spread disinformation.
- Potential applications: SlowFocus is described as potentially useful for video analytics, surveillance, automated content moderation, and educational-video analysis.The educational application is framed as helping analyze complex video content.
- Risks: The paper identifies misuse risks involving analysis of private videos and the spreading of disinformation.
- Instruction data: The supplied prompt materials cover action recognition, temporal grounding, temporal reasoning, and multi-turn QA.The prompts use captions, action labels, time segments, and chronological clip information to generate task-specific dialogues.
- Instruction data: Multi-turn QA prompts generate dialogues from complete video annotations containing captions, time segments, and actions across chronological clips.