PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Models

Chenglin Li, Qianglong Chen, Zhi Li, Feng Tao, Yin Zhang

arXiv:2411.09105v2cs.CVcs.AI

TL;DR

Existing video benchmarks offer limited control over content and difficulty, while it remains unclear whether LVLMs can handle symbolic and abstract cognition. VideoCogQA uses programmatically generated, adjustable video scenes to evaluate these abilities; reported results show performance declines as task complexity rises.

  • Problem

    Existing LVLM benchmarks rely on web-crawled data with limited content control and scalability, while symbolic understanding and abstract reasoning are often evaluated implicitly.

    Source: Introduction · p. 1

  • Method

    VideoCogQA uses a Python-based pipeline to generate game-inspired videos with adjustable complexity and paired question templates for cognitive evaluation.

    Source: Introduction · p. 1, Automated Video and QA Generation · pp. 4–5

  • Results

    GPT-4o performance drops 4% when additional objects are introduced in Action Arena and 10% at the highest difficulty level.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The results underscore the need to improve LVLM generalization of cognitive capabilities as video difficulty increases.

    Source: Conclusion · p. 7

    The authors hypothesize that poor temporal-task performance stems from visual encoders’ limitations in perceiving high-level abstract and symbolic concepts.

    Source: Model Performance Analysis · p. 6

Abstract

from arXiv · show

Recent advancements in Large Video-Language Models (LVLMs) have led to promising results in multimodal video understanding. However, it remains unclear whether these models possess the cognitive capabilities required for high-level tasks, particularly those involving symbolic and abstract perception. Existing benchmarks typically rely on real-world, annotated videos, which lack control over video content and inherent difficulty, limiting their diagnostic power. To bridge this gap, we propose VideoCogQA, a scalable and fully controllable benchmark inspired by game-world environments, designed to evaluate the cognitive abilities of LVLMs. By generating synthetic videos via a programmatic engine, VideoCogQA allows fine-grained control over visual elements, temporal dynamics, and task difficulty. This approach enables a focused evaluation of video cognitive abilities, independent of prior knowledge from visual scene semantics. The dataset includes 800 videos and 3,280 question-answer pairs, featuring tasks related to abstract concepts, symbolic elements, and multimodal integration, with varying levels of difficulty. Experimental results show that even state-of-the-art (SOTA) models, such as GPT-4o, achieve an average performance of 48.8% on tasks involving abstract concepts. Additionally, performance drops by 15% as task complexity increases, highlighting the challenges LVLMs face in maintaining consistent performance. Through this work, we hope to show the limitations of current LVLMs and offer insights into how they can more effectively emulate human cognitive processes in the future.

Introduction

VideoCogQA addresses the limited control and scalability of existing video benchmarks by using programmatically generated scenes to evaluate symbolic, abstract, and other cognitive abilities. Its evaluations report that model performance declines as task complexity increases.

Introduction

  • Figure 1 uses symbolic icons for players, bullets, and enemies, with “destroy” represented abstractly, to test counting in a game scene.Difficulty varies with the number and speed of enemies.Source: Figure 1 · p. 1
  • Existing LVLM benchmarks rely on web-crawled videos and emphasize semantic understanding, leaving symbolic understanding and abstract reasoning often evaluated only implicitly.VideoCogQA instead uses controllable, programmatically synthesized video to assess cognitive abilities.Source: Introduction · p. 1
  • VideoCogQA evaluates object, action, spatial, temporal, game-environment, and full-modal perception using controllable video scenes.Its scenes include symbolic objects and abstract actions, attributes, and relationships.Source: Introduction · pp. 1–2
  • Table 1 compares video benchmarks by understanding, reasoning, audio, synthesis, controllability, and difficulty-level distinctions.The supplied passage gives the table’s comparison scope, not its individual benchmark entries.Source: Table 1 · p. 2
  • GPT-4o performance drops 4% when additional objects are introduced in Action Arena and 10% at the highest difficulty level.The paper also reports that temporal-task performance drops stem from visual encoders’ insufficient grasp of high-level abstract and symbolic concepts.Source: Introduction · p. 2

Related Work

Prior video benchmarks often emphasize human behavior and contextual understanding, while synthetic data offers a route to controlled, scalable evaluation. VideoCogQA uses game-inspired scenes and programmatic video synthesis to target cognitive tasks that contextual cues may make easier to infer.

Video-LMMs and Benchmark

  • Many video benchmarks focus on human behavior and contextual understanding, which can let models infer actions from scene cues rather than perceive motion directly.VideoCogQA’s abstract objects performing actions such as bouncing or rotating are intended to require motion perception.Source: Video-LMMs and Benchmark · p. 2
  • VideoCogQA introduces a scalable, controllable dataset for cognitive abilities, addressing limited scalability and neglect of abstract cognitive tasks in existing video benchmarks.Figure 2 presents task scenes, aligned questions, and six core cognitive abilities.Source: Video-LMMs and Benchmark · p. 2, Figure 2 · p. 2

Synthetic Dataset

  • Synthetic datasets can avoid practical challenges of manual annotation and support controlled, scalable evaluation; VideoCogQA adopts Python-based video synthesis for LVLM assessment.The passage also positions the approach alongside prior synthetic image and video benchmarks.Source: Synthetic Dataset · pp. 2–3
  • Figure 3 depicts a pipeline in which scene-complexity variables are logged, GPT-4 generates scene-related question templates, and human filtering refines them before videos and QA pairs are collected.The figure caption identifies the steps but does not report their evaluation outcomes.Source: Figure 3 · p. 3

VideoCogQA

VideoCogQA organizes cognitive evaluation around six abilities and game-inspired synthetic scenes with controllable visual, spatial, and temporal complexity. Its Python-based pipeline generates videos and aligned question templates for evaluation.

Dataset Design

  • The dataset covers object, action, temporal, spatial, game-environment, and full-modal perception, using symbolic objects, abstract attributes and actions, and spatial and temporal relationships.Full-modal perception concerns integrating visual, textual, and auditory information.Source: Dataset Design · pp. 3–4

Automated Video and QA Generation

  • A Python pipeline renders variable game-inspired scenes in batches, with code parameters controlling spatial and temporal complexity and scene logs paired with question templates.GPT-4 generates scene-related QA templates; the evaluation uses multiple-choice questions with three to five shuffled options.Source: Automated Video and QA Generation · pp. 4–5

Dataset Design

  • Scene parameters vary task difficulty, including grid size, object count and action diversity, timing intervals, maze design, and abstract motion speed or path complexity.The scenes span object recognition, motion, temporal tracking, spatial reasoning, game mechanics, and audio-visual association.Source: Automated Video and QA Generation · pp. 4–5, Table 2 · p. 4

Automated Video and QA Generation

Experiments

Across tested models, abstract visual tasks remain difficult and accuracy declines as video difficulty rises. The experiments also examine human comparison, visual input, model scale, and maze navigation.

Setup

  • The evaluation uses ten open-source LVLMs, three Qwen2-VL sizes, Gemini-1.5-Flash, and GPT-4o, with default inference settings.Musical notes are converted to text to standardize input across video-language models.Source: Setup · p. 5

Main Results

  • Qwen2-VL-72B leads the tested open-source models with 53.7% overall accuracy, including 51.8% OP, 59.1% AP, 56.8% TR, 51.3% SR, 44.0% GP, and 76.7% FP.Most models struggle with OP, SR, and GP, which involve more visual abstract concepts.Source: Main Results · p. 5, Table 3 · p. 6
  • A 200-sample comparison reveals a significant performance gap between humans and LVLMs.Figure 6 compares human performance with GPT-4 and Qwen2-72B, including runs that substitute code-log video descriptions.Source: Main Results · p. 6, Figure 6 · p. 6

Performance Across Difficulty Levels

  • Accuracy declines for every model as video difficulty rises, with roughly a 10-point Easy-to-Medium drop and a further 5-point decline at Difficult.LLaVA-NEXT-Video-34B begins to outperform Video-LLaMA2 as tasks become more challenging.Source: Performance Across Difficulty Levels · p. 6, Table 4 · p. 7

Model Performance Analysis

  • Replacing videos with full temporal descriptions from code logs substantially improves performance, especially for Qwen2-72B and GPT-4.The authors suggest models struggle more to extract abstract information from raw visual input than to reason over temporal content.Source: Model Performance Analysis · p. 6, Figure 6 · p. 6
  • Both models frequently misidentify object size in a single-frame test, indicating limitations in fine-grained perception.The test sampled 100 frames each from S1 and scored descriptions of symbolic elements by grid location.Source: Model Performance Analysis · pp. 6–7, Figure 7 · p. 6

Impact of Model Size on Performance

Case Study

  • In the maze case, only Qwen2-VL-72B selects the correct path; other models choose partially understood or incorrect options.Correctly answering requires perceiving the green block’s trajectory and retaining the path structure.Source: Case Study · p. 7, Figure 9 · p. 7

Conclusion

VideoCogQA offers a controllable, scalable evaluation of LVLM cognitive abilities across video tasks. Its experiments find challenges with symbolic elements and worsening performance as video difficulty increases, underscoring the need to improve generalization.

  • VideoCogQA is a controllable, scalable dataset for evaluating LVLM cognitive abilities across diverse video tasks.Source: Conclusion · p. 7
  • Even SOTA models such as GPT-4o and Qwen2-VL-72B face significant challenges with symbolic elements, and performance drops sharply as video difficulty increases.Source: Conclusion · p. 7
  • The results underscore the need to improve LVLM cognitive-capability generalization.Source: Conclusion · p. 7
Loading 2411.09105v2…