PaperlayerView exact revision on arXiv

Source-linked AI summary

TempCompass: Do Video LLMs Really Understand Videos?

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou

arXiv:2403.00476v3cs.CV

TL;DR

Existing benchmarks provide limited evidence about Video LLM temporal perception because they often omit fine-grained temporal aspects or diverse task formats. TempCompass addresses this with a multifaceted benchmark, conflicting videos, human-LLM instruction generation, and automated response evaluation. Across evaluated models, Video LLMs show notably weak temporal perception, while the benchmark remains limited by residual single-frame or language-prior influence and caption-evaluation errors.

  • Problem

    Existing benchmarks inadequately distinguish temporal aspects and task formats, limiting comprehensive evidence about Video LLM temporal perception.

  • Method

    TempCompass combines five basic and ten fine-grained temporal aspects, four task formats, conflicting videos, human-annotated meta-information with LLM instruction generation, and automated evaluation.

  • Results

    Video LLMs demonstrate deficient temporal perception and fail to surpass Image LLM counterparts, while performance varies substantially across task formats.

  • Takeaways & Limitations

    Comprehensive assessment of Video LLMs requires diverse temporal aspects and task formats rather than relying on limited benchmark designs.

  • Takeaways & Limitations

    Single-frame and language-prior influence persists, and automatic evaluation can misjudge captions containing hallucinated content despite matching ground-truth information.

Abstract

from arXiv · show

Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable to distinguish between different temporal aspects (e.g., speed, direction) and thus cannot reflect the nuanced performance on these specific aspects. On the other hand, they are limited in the diversity of task formats (e.g., only multi-choice QA), which hinders the understanding of how temporal perception performance may vary across different types of tasks. Motivated by these two problems, we propose the \textbf{TempCompass} benchmark, which introduces a diversity of temporal aspects and task formats. To collect high-quality test data, we devise two novel strategies: (1) In video collection, we construct conflicting videos that share the same static content but differ in a specific temporal aspect, which prevents Video LLMs from leveraging single-frame bias or language priors. (2) To collect the task instructions, we propose a paradigm where humans first annotate meta-information for a video and then an LLM generates the instruction. We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that these models exhibit notably poor temporal perception ability. Our data will be available at https://github.com/llyx97/TempCompass.

1 Introduction

TempCompass addresses benchmark gaps in distinguishing temporal aspects and task formats, enabling broader evaluation of Video LLM temporal perception. Using conflicting videos, human-LLM instruction generation, and automated evaluation, the study finds weak temporal perception across current models.

  • Existing benchmarks often conflate temporal aspects and restrict task formats, limiting comprehensive assessment of Video LLM temporal perception.
  • TempCompass evaluates five basic temporal aspects, ten fine-grained sub-aspects, and four task formats.
  • Conflicting video pairs or triplets share static content while differing in one temporal aspect, reducing reliance on single-frame bias and language priors.
  • Human-annotated meta-information and LLM generation produce 7,540 task instructions across the four task types.
  • Tailored ChatGPT prompts and rule-based methods automatically evaluate free-form Video LLM responses while balancing evaluation cost and accuracy.
  • Evaluation covers 11 SOTA multimodal LLMs, including 8 Video LLMs and 3 Image LLMs, and finds Video LLMs deficient in temporal perception.
  • Temporal perception varies substantially across task formats, supporting diverse formats in benchmark assessment.

2 Related Work

Related work spans multimodal LLM paradigms and temporal-perception benchmarks. Existing video benchmarks either underrepresent temporal aspects or provide insufficient task-format diversity, while TempCompass emphasizes both dimensions and open-domain videos.

  • 2.1 Multi-Modal Large Language Models: Pipeline multimodal LLMs convert visual information into text with vision experts before downstream language-task processing.
  • 2.2 Temporal Perception Evaluation: Temporal perception distinguishes video-centered from image-centered applications because videos convey dynamic information over time.
  • 2.2 Temporal Perception Evaluation: Earlier video-language evaluation often neglected distinctions among temporal aspects, motivating finer-grained assessment approaches.
  • 2.3 MLLM Benchmarks: Several multimodal LLM benchmarks target Image LLMs, while newer benchmarks address Video LLMs.
  • 2.3 MLLM Benchmarks: SEEDBench, VLM-Eval, and Video-Bench do not distinguish temporal aspects, whereas AutoEval-Video and MVBench include temporal aspects but lack diverse task formats.
  • 2.3 MLLM Benchmarks: TempCompass emphasizes diverse temporal aspects, task formats, and open-domain videos in comparison with representative benchmarks.

3 TempCompass Benchmark

TempCompass is a benchmark for testing Video LLM temporal perception across five temporal aspects, diverse static contents, and four task formats. It uses conflicting videos, annotated meta-information, and LLM-generated instructions to build a varied evaluation set.

  • 3.1.1 Temporal Aspects: TempCompass evaluates five basic temporal aspects—Action, Speed, Direction, Attribute Change, and Event Order—alongside ten fine-grained sub-aspects.Action includes coarse- and fine-grained actions; Speed includes absolute and relative speed; Direction includes object and camera direction; Attribute Change covers four sub-aspects.
  • 3.1.2 Task Formats: Four task formats—Multi-Choice QA, Yes/No QA, Caption Matching, and Caption Generation—enable comparisons of temporal perception across task types.Each format tests temporal information through a different response mode, including selecting choices, judging statements, matching captions, or generating captions.
  • 3.1.3 Static Contents: The benchmark covers nine static-content categories, with each video assigned one or multiple categories according to its visual content.Categories include people, animals, plants, food, natural objects, vehicles, artifacts, buildings, and abstract content.
  • 3.2 Data Collection: The collection pipeline selects temporal aspects and content categories, gathers videos and annotated meta-information, then uses ChatGPT to generate task instructions.Human annotation and manual refinement are interleaved with automatic instruction generation, producing 7,540 instructions from 410 videos and 500 meta-information entries.
  • 3.2.1 Video Collection: Raw videos come from ShutterStock, are distributed across content categories, and are excluded from WebVid to increase diversity and avoid overlap with a common pre-training dataset.The collection also controls static-content distribution so each category contains an adequate number of samples.
  • 3.2.1 Video Collection: Conflicting video pairs or triplets preserve static content while changing one temporal aspect, reducing reliance on single-frame bias and language priors.Reversal targets Direction and Attribute Change; spatial concatenation targets Speed; temporal concatenation targets Event Order.

4 Experiments

Experiments show that current MLLMs have weak and task-dependent temporal perception, with Video LLMs failing to consistently outperform Image LLMs. The results also assess conflicting-video robustness and automatic evaluation reliability.

  • Overall Performance: Five Video LLMs fail to convincingly surpass the random baseline across all tasks, while stronger models remain below human performance and do not consistently outperform Image LLMs.The five models are LLaMA-VID, Panda-GPT, Valley, Video-ChatGPT, and Video-LLaMA; Video-LLaVA and VideoChat2 perform better but still fall significantly short of humans.
  • Performance Across Temporal Aspects: Action is the strongest temporal aspect, with the best model achieving near 90 accuracy on Multi-Choice QA and Caption Matching, whereas the other four aspects perform significantly worse.Action can often be inferred from static visual cues, while Speed, Direction, Event Order, and Attribute Change depend more on temporal information across frames.
  • Performance Across Tasks: Temporal perception varies substantially across task formats, with Caption Generation generally performing worse than Multi-Choice QA and individual models showing task-specific strengths.Video-LLaVA leads on Speed in Caption Matching but performs no better than random on the same aspect in Yes/No QA and Caption Generation.
  • Ability to Respond in Desired Format: Rule-based matching is limited when MLLMs do not follow the desired response format, supporting the need for LLM-based evaluation of free-form answers.The answer prompt design also has a non-negligible impact on match rate.
  • Qualitative Results: On conflicting Direction videos, three MLLMs answer the original video correctly but fail on the reversed version, indicating difficulty perceiving movement direction.The example uses SPHINX-v2, Video-LLaVA, and Video-ChatGPT, whose responses are assessed automatically.
  • Automatic Evaluation Accuracy: The automatic evaluation method agrees highly with human judgments for Multi-Choice QA, Yes/No QA, and Caption Matching, but roughly 20% of Caption Generation evaluations are inconsistent.The inconsistency is attributed to hallucinated content that pure-text GPT3.5-Turbo has difficulty detecting.
  • Effect of the Conflicting Videos: Image LLMs outperform random on raw videos but move closer to random performance when conflicting videos are included, reducing single-frame and language-prior effects.The comparison averages results over temporal aspects other than Action, where conflicting videos were not constructed.

5 Conclusions

TempCompass evaluates Video LLM temporal perception across diverse temporal aspects and task formats. Its conflicting videos, human–LLM instruction process, and ChatGPT-based evaluator support broad evaluation, which reveals weak temporal perception in current Video LLMs.

  • 5 Conclusions: TempCompass introduces ten temporal aspects and four task formats to provide a comprehensive view of Video LLM temporal perception.The benchmark evaluates 8 SOTA Video LLMs and 3 Image LLMs.
  • 5 Conclusions: Conflicting videos mitigate single-frame bias and language priors, while human annotation combined with LLM generation produces high-quality task instructions efficiently.The benchmark also uses a ChatGPT-based automatic evaluator for free-form Video LLM responses.
  • 5 Conclusions: Evaluation results reveal a pressing need to enhance the temporal perception ability of Video LLMs.

6 Limitations

TempCompass remains limited by persistent single-frame and language-prior effects, while its automatic evaluation can misjudge some generated captions containing hallucinated content.

  • Single-frame and language priors persist despite conflicting videos, with Image LLMs performing above random baselines on specific tasks and temporal aspects.
  • The automatic evaluator struggles with some generated captions that align with ground-truth candidate information but include hallucinated content.
  • TempCompass constructs task instructions through human annotation, ChatGPT generation, and manual review across four task formats.
  • For Multi-Choice QA, ChatGPT generates five instructions per meta-information item and options are randomly shuffled to reduce positional bias.
  • Yes/No QA generates equal numbers of Positive and Negative questions from manually rectified Multi-Choice QA questions.
  • Caption Matching creates one True and three False captions from manually rectified questions, then randomizes their displayed order.

A.3 Data Statistics

TempCompass balances task instructions and answer options across temporal aspects while focusing primarily on videos shorter than 30 seconds and collecting human reference answers.

  • Each task type contains at least 1,500 instructions, and every basic temporal aspect receives a balanced number of instructions.
  • Ground-truth answers are balanced across options, except option D appears less often because some Multi-Choice questions have fewer than four options.
  • Among 675 four-option questions, option D is correct 157 times, aligning with the other options’ frequency.
  • The benchmark primarily contains short and medium-length videos within 30 seconds.
  • Human evaluation samples 200 instructions, with 10 instructions per temporal aspect across each task and 50 instructions per task.
  • Three human annotators’ results are averaged, with generated captions treated as incorrect when annotators answer “Cannot Answer.”

A.6 Comparison With Related Benchmarks

Compared with related benchmarks, TempCompass combines broader temporal coverage, varied task formats, and open-domain videos to assess temporal perception more holistically.

  • Most existing benchmarks lack comprehensive temporal-aspect categorization, whereas several benchmarks introduce complementary varieties of temporal aspects.
  • Most current benchmarks cannot reflect performance variation across different task formats.
  • Perception Test includes multiple task formats and temporal aspects but is constrained to indoor videos focused on people and artifacts.
  • TempCompass emphasizes rich temporal dimensions, diverse task formats, and open-domain videos for a more holistic temporal-perception assessment.
  • Related task formats include Free-form QA, Action Recognition, Grounded Video QA, Object Tracking, Point Tracking, and Retrieval, with some lacking textual instructions.
  • Temporal aspects in the comparison target basic temporal perception while excluding aspects requiring reasoning skills.

B.1 Rule-based Evaluation

TempCompass maps structured Video LLM responses to answer options using rules tailored to each task format before comparing selections with ground truth.

  • Multi-Choice QA responses map to an option when they match either the complete option text or its indicator.
  • Caption Matching accepts matches to the complete caption, its sentence, or the option indicator.
  • Yes/No QA evaluation checks whether the response starts with “yes” or “no” before comparing the mapped answer with ground truth.

B.2 LLM-based Evaluation

The benchmark evaluates responses with LLM-based judging when outputs do not match predefined options, while human annotators validate response correctness on sampled data. Caption Generation receives a task-specific evaluation procedure and explicit error criteria.

  • ChatGPT judges Multi-Choice QA, Yes/No QA, and Caption Matching responses against their instructions and ground-truth answers.
  • Caption Generation is evaluated by converting the generated caption into a corresponding Multi-Choice QA task, with in-context examples and an added reasoning step.
  • 400 responses from SPHINX-v2 and Video-LLaVA were sampled for human evaluation, with 100 samples per task across four tasks.
  • Three human annotators assign binary correctness labels using the video, response, task instruction, and ground-truth answer.
  • Caption Generation responses are incorrect when they describe the wrong candidate, omit candidate information, hallucinate inconsistent content, or fail to generate a description.

C.1 Model Architecture

The evaluated models use the prevalent multimodal architecture of a visual encoder, vision-language connector, and LLM, with varied temporal aggregation and training strategies. Inference settings and answer prompts are standardized or selected from each model’s recommended configuration, while Image LLMs receive a middle video frame.

  • Model Architecture: All evaluated MLLMs contain a visual encoder, a vision-language connector, and an LLM.
  • Video LLMs: Video-LLaMA aggregates individual frame representations with a trainable video Q-Former while freezing its vision encoder and LLM during training.
  • Video LLMs: Video-ChatGPT combines spatial and temporal pooling with a frozen CLIP image encoder and frozen Vicuna-v1.1-7B LLM.
  • Video LLMs: Valley adds a temporal modeling module to its vision encoder and fine-tunes LLM parameters during instruction tuning.
  • Video LLMs: mPLUG-Owl-video-7B introduces additional temporal query tokens into its visual abstractor for temporal modeling.
  • Inference: Image LLMs receive the middle frame of each video, while video-frame sampling and decoding follow the models’ recommended inference scripts.
  • Inference: Answer prompts specify output formats such as “Best Option:”, “Please answer yes or no: ”, and “Generated Caption:”.

D.3 Qualitative Results

Qualitative examples show that models often fail on temporal aspects when conflicting videos preserve static content while changing temporal information. The examples also support the automatic evaluator’s general reliability, alongside a small but recurring caption-evaluation error.

  • Temporal Perception: Models show deficient temporal perception for speed, direction, event order, and attribute change, especially on corresponding conflicting videos.
  • Temporal Perception: The examples indicate that models can answer accurately on some videos but falter when confronted with conflicting videos.
  • Automatic Evaluation: The automatic evaluation method provides accurate assessments in most cases despite the arbitrary forms of MLLM responses.
  • Automatic Evaluation: The LLM-based evaluator mistakenly marks a small portion of incorrect captions as correct, particularly in the caption examples.
  • Data Release: TempCompass data are released under CC-BY 4.0, with videos and textual annotations restricted to research use.
Loading 2403.00476v3…