PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Understanding with Large Language Models: A Survey

Yolo Y. Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, Chenliang Xu

arXiv:2312.17432v8cs.CVcs.CL

TL;DR

The survey addresses the need to understand rapidly growing video content and the gap in comprehensive coverage of LLM-based video understanding. It synthesizes Vid-LLM architectures, functions, tasks, evaluation, applications, and limitations, finding advanced multi-granularity reasoning and performance patterns associated with stronger model components and bridging architectures.

  • Problem

    Rapidly growing video content creates demand for proficient understanding tools, while previous surveys did not comprehensively cover general video understanding based on large language models.

  • Method

    The survey categorizes Vid-LLMs by video processing into Video Analyzer × LLM, Video Embedder × LLM, and (Analyzer + Embedder) × LLM, and by LLM role into five sub-types.

  • Results

    Vid-LLM performance reflects a combination of stronger foundation models, better visual encoders, appropriate temporal modeling, and more sophisticated bridging architectures rather than one innovation.

  • Takeaways & Limitations

    Vid-LLMs support multi-granularity reasoning from abstract to spatiotemporal analysis across tasks including summarization, captioning, and question answering.

  • Takeaways & Limitations

    Existing Vid-LLMs remain limited in evaluation metrics, long-form video handling, and visual-textual modality alignment.

Abstract

from arXiv · show

With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly. Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding that harness the power of LLMs (Vid-LLMs). The emergent capabilities of Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity (general, temporal, and spatiotemporal) reasoning combined with commonsense knowledge, suggesting a promising path for future video understanding. We examine the unique characteristics and capabilities of Vid-LLMs, categorizing the approaches into three main types: Video Analyzer x LLM, Video Embedder x LLM, and (Analyzer + Embedder) x LLM. Furthermore, we identify five sub-types based on the functions of LLMs in Vid-LLMs: LLM as Summarizer, LLM as Manager, LLM as Text Decoder, LLM as Regressor, and LLM as Hidden Layer. Furthermore, this survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs. Additionally, it explores the expansive applications of Vid-LLMs across various domains, highlighting their remarkable scalability and versatility in real-world video understanding challenges. Finally, it summarizes the limitations of existing Vid-LLMs and outlines directions for future research. For more information, readers are recommended to visit the repository at https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding.

I. INTRODUCTION

Video understanding evolved from handcrafted and early neural methods toward self-supervised pretraining and Vid-LLMs as video production expanded and manual processing became impractical.

  • I. INTRODUCTION: Millions of videos are created every day, increasing the need for tools that automatically manage, analyze, and process video content.Manual processing is labor-intensive and time-consuming.
  • I. INTRODUCTION: Video understanding progressed from handcrafted feature extraction and temporal analysis to deep neural networks that improved task-solving capabilities.Early methods used SIFT, SURF, HOG, background subtraction, optical flow, IDT, and HMMs; later models introduced CNNs and 3D CNNs.
  • I. INTRODUCTION: The field’s development is summarized in four stages: Conventional Methods, Early Neural Video Models, Self-supervised Video Pretraining, and Vid-LLMs.The survey describes continuously improving task-solving capability across these stages.

3) Self-supervised Video Pretraining:

Self-supervised video pretraining improves transfer across downstream tasks with less task-specific labeling, while LLMs add prompt-based, general-purpose multimodal task solving. The survey situates these developments within a timeline and broader research gap.

  • 3) Self-supervised Video Pretraining: Self-supervised video pretraining enables generalization across diverse tasks with minimal additional labeling.VideoBERT tokenizes video features with hierarchical k-means and can be fine-tuned for action classification and video captioning.
  • 3) Self-supervised Video Pretraining: Pretrained video models vary in architecture and training strategy, including ActBERT, VideoMAE, MaskFeat, ALPRO, and CLIP-ViP.These approaches extend the pretraining–finetuning paradigm for video understanding.
  • 3) Self-supervised Video Pretraining: LLMs provide in-context learning through prompting, instruction tuning, tool use, and broader multimodal task-solving capabilities.The survey connects these capabilities to LLMs integrated with video understanding.
  • 3) Self-supervised Video Pretraining: Previous surveys often focused on specific video tasks or broader multimodal methodologies, leaving general video understanding with LLMs insufficiently surveyed.This survey addresses that gap through a comprehensive review of Vid-LLMs.
  • 3) Self-supervised Video Pretraining: The survey covers tasks, datasets, metrics, Vid-LLM categories, training strategies, evaluation, applications, challenges, and future research.A supporting GitHub repository aggregates resources for video understanding with Vid-LLMs.

A. Video Understanding Tasks

Video understanding spans classification, retrieval, summarization, captioning, and question answering, progressing from recognizing content to generating language-based descriptions and responses.

  • A. Video Understanding Tasks: Video understanding has developed from classification and action recognition toward natural-language comprehension and reasoning over complicated video content.Captioning requires logical, commonsense descriptions, while VQA requires answering questions using video content and external information.
  • A. Video Understanding Tasks: Video classification and action recognition assign class labels or activity and event categories to videos or their sequences.Common datasets include UCF-101, HMDB51, ActivityNet, Charades, Kinetics-400, Kinetics-600, and Kinetics-700.
  • A. Video Understanding Tasks: Text-video retrieval matches textual descriptions with relevant video clips using similarity, commonly evaluated by Recall at K.Kinetic-GEB, MSRVTT, DiDeMo, YouCook2, and Youku-mPLUG are listed datasets.
  • A. Video Understanding Tasks: Video-to-text summarization generates concise textual summaries by extracting and interpreting key visual and audio content.BLEU, METEOR, CIDEr, and ROUGE-L commonly evaluate this task.
  • A. Video Understanding Tasks: Video captioning generates descriptive, coherent, accurate, and contextually relevant text from visual and auditory video information.MSVD, MSR-VTT, TGIF, Charades, YouCook2, Youku-mPLUG, VAST-27M, and VideoInstruct-100K are notable datasets.
  • A. Video Understanding Tasks: Video question answering analyzes video content and context to generate answers to textual questions, commonly evaluated with Top-1 or Top-K accuracy.Listed datasets include MSVD-QA, MSRVTT-QA, TGIF-QA, TVQA, ActivityNet-QA, and NExT-QA.

2) Temporal Understanding Tasks: •

Temporal and spatiotemporal tasks localize events, moments, objects, and boundaries across video time and space, supporting increasingly fine-grained video understanding.

  • 2) Temporal Understanding Tasks: Video summarization condenses long videos into shorter versions while preserving essential content, using F1-score, Spearman, and Kendall metrics.Common datasets include SumMe, TVSum, Ads-1k, VideoXum, and Instrcut-V2Xum.
  • 2) Temporal Understanding Tasks: Video highlight detection identifies and extracts the most important or interesting video segments.Common datasets include YouTube Highlights, TV-Sum, and VTW.
  • 2) Temporal Understanding Tasks: Temporal action or event localization identifies when activities start and end within a video sequence.THUMOS’14, ActivityNet-1.3, and UnAV-100 are listed datasets.
  • 2) Temporal Understanding Tasks: Temporal action proposal generation produces candidate video segments likely to contain actions or events.Relevant datasets include THUMOS’14, ActivityNet, and Charades.
  • 2) Temporal Understanding Tasks: Video temporal grounding aligns a textual query with specific moments or intervals, often evaluated with R1@0.5 and R1@0.7.Charades-STA, ViTT, DiDeMo, and PU-VALOR are common benchmarks.
  • 2) Temporal Understanding Tasks: Moment retrieval identifies precise video segments corresponding to textual or visual queries by aligning query semantics with video frames.DiDeMo is listed as a dataset for this task.
  • 2) Temporal Understanding Tasks: Generic event boundary detection identifies frames where significant changes occur and divides videos according to events or activities.Kinetics-GEBD is a widely used dataset.
  • 2) Temporal Understanding Tasks: Generic event boundary captioning and grounding identifies and describes transition points between significant events.Kinetics-GEB+ is the listed dataset.

B. Background for LLMs

Language models model text sequences through joint and conditional token probabilities, while LLMs extend this framework with large-scale parameters, self-attention, autoregressive training, and emergent abilities.

  • Language models learn a joint probability distribution over text-token sequences and factor it into conditional probabilities using the chain rule.The sequence length is denoted by L.
  • LLMs contain a tokenizer and multiple self-attention layers, and are trained to predict the next token autoregressively using teacher forcing.The model is represented as M.
  • Decoding strategies select the next token from the vocabulary using the model’s next-token probabilities.The vocabulary includes special tokens such as <SOS>, <EOS>, and <PAD>.
  • Greedy decoding selects the token with the highest probability, whereas sampling strategies randomly select tokens to produce diverse outputs and support self-consistency.
  • LLMs commonly exhibit scaling laws and emergent abilities, including in-context learning, instruction following, and step-by-step reasoning.Scaling model size, training data, and computation is associated with regular performance growth.

III. VID-LLMS

The survey introduces a taxonomy of Vid-LLMs based on how video is processed and how information flows between video-processing components and LLMs.

  • Taxonomy: Vid-LLMs are categorized into Video Analyzer × LLM, Video Embedder × LLM, and (Analyzer + Embedder) × LLM.The third category combines analyzer-produced text with embedder-produced representations.
  • Video Analyzer × LLM: Video Analyzer × LLM systems use analyzers to convert video inputs into textual analysis that can be passed to an LLM.Figure 4 depicts textual information flow in blue.
  • Video Analyzer × LLM: Within Video Analyzer × LLM, the LLM acts as a Summarizer or Manager according to its role in processing analyzer outputs.
  • Video Analyzer × LLM: Summarizer systems usually transmit information unidirectionally from video to analyzer to LLM, while Manager systems can invoke analyzers, process outputs, and interact over multiple rounds.
  • Taxonomy: The taxonomy also organizes Vid-LLMs by supported understanding granularity and identifies four fine-tuning strategy types.Figure 5 distinguishes abstract, temporal, and spatiotemporal understanding; Figure 6 lists full fine-tuning, connective adapters, insertive adapters, and hybrid methods.

2) Video Embedder × LLM:

Video Embedder × LLM systems convert videos into embeddings that LLMs process through several functional roles, generally requiring fine-tuning and adaptation between semantic spaces.

  • Video embedders use visual backbones or video encoders such as ViT or CLIP to convert videos into vector representations called video embeddings or tokens.Audio encoders can also be included for other video modalities.
  • Because video embeddings are not directly usable by LLMs, adapters usually map them from visual or multimodal semantic space into the LLM’s text-token space.
  • Video Embedder × LLM systems are classified by LLM function as Text Decoder, Regressor, or Hidden Layer.
  • LLM as Text Decoder: As a Text Decoder, the LLM decodes video embeddings into text, typically for general question answering or captioning rather than fine-grained localization.
  • LLM as Regressor / Hidden Layer: As a Regressor, the LLM can predict continuous values such as timestamps and bounding-box coordinates, while the Hidden Layer connects to task-specific heads for such regression outputs.

3) (Analyzer + Embedder) × LLM:

Vid-LLMs can combine textual video analysis with video embeddings, while different adapter strategies determine how information enters and changes the language model. The survey also contrasts training-free analyzer-based systems with fine-tuned embedder-based systems and reviews evaluation approaches.

  • (Analyzer + Embedder) × LLM: The (Analyzer + Embedder) × LLM category jointly supplies textual video analysis and video embeddings to the LLM.These systems can use any of the Summarizer, Manager, Text Decoder, Regressor, or Hidden Layer subcategories.
  • Training-free Vid-LLMs: Training-free Vid-LLMs use zero-shot, in-context learning, and Chain-of-Thought capabilities without updating model parameters.Analyzer-based systems are often training-free because video information has already been transformed into text.
  • Fine-tuning Vid-LLMs: Most Video Embedder × LLM systems undergo fine-tuning through fully fine-tuned, connective-adapter, insertive-adapter, or hybrid-adapter approaches.These categories differ in which modules are updated and how video information is connected to the LLM.
  • Fine-tuning Vid-LLMs: Connective adapters bridge the Video Embedder and LLM externally while freezing both modules and updating only the adapter.Common connective adapters include MLP or linear layers and Q-formers.
  • Fine-tuning Vid-LLMs: Insertive adapters are added inside the LLM, commonly using LoRA, while the Video Embedder and LLM remain frozen during training.They alter the LLM’s behavior through additions to its existing parameters.
  • Fine-tuning Vid-LLMs: Hybrid adapters combine connective and insertive adapters, typically using multiple stages for modality alignment and target-task training.The connective adapter is commonly trained first and then frozen before later training.
  • Evaluation: Evaluation methods are grouped into closed-ended, open-ended, and specialized approaches for capabilities such as temporal and spatiotemporal reasoning.The survey presents these categories in Table III.

A. Closed-ended Evaluation

The survey organizes video question-answering evaluation into closed-ended, open-ended, and specialized methods. These methods range from direct scoring against predefined answers to LLM-based assessment and analyses of fine-grained temporal or spatiotemporal understanding.

  • Closed-ended Evaluation: Closed-ended evaluation uses predefined answers or structured formats, enabling direct scoring with accuracy or ground-truth comparison metrics.Multiple-choice tasks use accuracy, while structured outputs can use CIDEr, METEOR, ROUGE, and SPICE.
  • Open-ended Evaluation: Open-ended evaluation compares unconstrained model answers with references using more sophisticated scoring methods, often involving GPT-3.5 or GPT-4.Examples include MovieChat-1K, MLVU, NExT-QA, VELOCITI, and EAGLE.
  • Benchmark Coverage: The benchmark comparison covers datasets, video counts, clips, durations, question-answer pairs, and video content.These dimensions are summarized in Table III.
  • Other Evaluation Methods: Specialized benchmarks evaluate fine-grained temporal and spatiotemporal understanding through dense captioning and related metrics.Dense captioning assesses both temporal localization and descriptive accuracy using BLEU, METEOR, and CIDEr.
  • Other Evaluation Methods: Qualitative evaluation adds error analysis, output-to-ground-truth comparisons, attention visualization, and self-explanation to numerical metrics.These methods provide insight into model limitations and visual elements prioritized during response generation.

D. Analysis of Model Performance

The survey associates stronger Vid-LLM performance with improvements across foundation models, visual encoders, temporal modeling, and bridging architectures. It also surveys applications spanning retrieval, education, gaming, interaction, and robotics.

  • D. Analysis of Model Performance: Larger and more recent foundation LLMs consistently outperform smaller counterparts, particularly on zero-shot VideoQA tasks.IG-VLM with 34B parameters is cited as an example.
  • D. Analysis of Model Performance: EVA-CLIP and ViT-G visual embedders are associated with superior performance across dense captioning and question-answering benchmarks.PLLaVA, IG-VLM, and Video LLaMA 2 are named examples.
  • D. Analysis of Model Performance: Temporal-task high performers typically process more than 100 frames, whereas general-understanding models use fewer frames.VTimeLLM, AVicuna, and ST-LLM are cited as temporal-task examples.
  • D. Analysis of Model Performance: Performance gains stem from stronger foundation models, better visual encoders, appropriate temporal modeling, and more sophisticated bridging architectures rather than one innovation.The survey presents this as a combination of factors supporting better contextual understanding.
  • Applications: Vid-LLMs support video search, context-aware recommendations, subtitle generation, translation, summarization, and editing.These applications are described for online video platforms and multimedia retrieval systems.
  • Applications: Applications also include virtual tutoring, sign-language translation, dynamic gaming content, customer-service chatbots, AR/VR/XR narratives, and state-aware interaction.The survey further describes robot navigation using LLMs with 3D scene graphs.

3) Healthcare and Security Applications: •

Vid-LLMs are applied to healthcare, security, surveillance, cybersecurity, autonomous vehicles, and other resource-constrained or privacy-preserving settings. The survey also emphasizes that fine-grained video understanding remains difficult because of data, research, and computational limitations.

  • Healthcare Applications: Healthcare applications include interpreting medical literature, supporting diagnosis and education, assisting professionals, and powering symptom-assessment chatbots.These systems are described as improving patient care and access to health information.
  • Security Applications: Security and surveillance applications analyze communications, detect anomalous patterns, identify suspicious behaviors, and support cybersecurity forensics.The survey also mentions phishing detection and improved video crowd counting.
  • Autonomous Vehicles: Autonomous-vehicle applications process language inputs, interpret road signs and instructions, and improve vehicle-control interfaces.The stated goals include enhancing safety and user experience.
  • Additional Applications: Additional applications include video-generation evaluation, prompt refinement, edge computing, and privacy-preserving federated learning.These uses extend Vid-LLMs beyond the previously discussed application domains.
  • Challenges: Fine-grained video understanding remains challenging because datasets and research are limited while frame-by-frame spatiotemporal analysis is computationally demanding.Deeper semantics such as emotions and scene dynamics are also harder to understand.

2) Long-form Video Understanding:

Vid-LLMs offer sophisticated, versatile video processing and multi-granularity reasoning, but long-form analysis, multimodal alignment, hallucination, deployment, and ethical governance remain central challenges.

  • 2) Long-form Video Understanding:: Long videos make identifying key events and sustaining attention difficult, especially in content-rich or complex plots.Effective systems must detect and highlight important temporal segments.
  • 3) Multimodal Video Understanding:: Multimodal understanding must align visual, audio, and textual data across spatial and temporal dimensions.The field also faces scarce datasets, limited research, and challenges in producing high-quality annotations.
  • 4) Hallucination in Video LLMs:: Hallucinations produce responses disconnected from source videos because of feature extraction limits, video context, vision-language domain gaps, and inherent LLM hallucinations.Proposed remedies include post-training, stronger spatiotemporal context modeling, and visual-linguistic latent collaboration.
  • 5) Industrial Deployment and Scalability:: Industrial deployment requires balancing efficiency and performance through compression, token merging, domain-specific fine-tuning, modular architectures, caching, and standardized integration.These strategies target scalable operation in industrial systems.
  • C. Ethical Implications: Vid-LLMs raise privacy, data-security, surveillance, misleading-content, and bias risks because their applications may require access to sensitive video.The survey calls for data governance, consent mechanisms, and ethical deployment prioritizing privacy and fairness.
  • VI. CONCLUSION: The survey categorizes Vid-LLMs by architecture and LLM role while highlighting abstract, temporal, and spatiotemporal reasoning across video tasks.It identifies three architecture types and five functional roles, and notes applications including summarization, captioning, and question answering.
Loading 2312.17432v8…