PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Hongbo Liu, Peixian Chen, Sihan Liu, Peiyuan Zhang, Kai Zou, Dian Zheng, Xiaoxing Hu, Yuhao Dong, Mengdan Zhang, Yunhang Shen, Haoyu Cao, Wei Liu, Weibo Gu, Xing Sun, Shengjie Zhao

arXiv:2608.25529v1cs.CV

TL;DR

Video understanding benchmarks have largely emphasized response correctness, leaving instruction adherence to semantic, format, and conditional constraints underexplored. Video-IFBench introduces a broad benchmark with four instruction templates, 32 task types, 39 constraint categories, and a hybrid evaluation protocol; experiments show that current MLLMs still struggle, especially with constraint-rich and conditional instructions.

  • Problem

    Instruction following in video understanding is underexplored because existing benchmarks primarily assess response correctness rather than adherence to diverse user-specified constraints.

  • Method

    Video-IFBench combines four instruction templates, 32 task types, 39 semantic or format constraints, semi-automatic sample construction, and hybrid LLM-as-Judge and programmatic verification.

  • Results

    Current MLLMs struggle with many constraints, semantic constraints, and conditional structures, with the best evaluated model achieving 54.5% overall score.

  • Takeaways & Limitations

    Instruction following should be treated as a first-class evaluation dimension for video MLLMs because strong video understanding accuracy does not automatically translate into faithful adherence to user intent.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding, where models must satisfy diverse user-specified constraints, including those grounded in visual and audio content. We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.

1 Introduction

Video-IFBench addresses the limited evaluation of instruction adherence in video understanding, where models must satisfy semantic, format, and structurally complex constraints. It introduces a broad benchmark and hybrid evaluation protocol, finding that current MLLMs still struggle with strict instruction following.

  • Instruction design: The benchmark evaluates whether models can satisfy diverse requirements grounded in video content, including conditional requests that require selecting the correct branch or path.These requirements reflect practical scenarios such as describing only events after a specified video condition and listing them chronologically with timestamps.
  • Motivation: Existing video benchmarks primarily measure response correctness rather than faithful execution of complex user instructions.Prior instruction-following evaluations largely target text-only or narrower multimodal settings.
  • Benchmark: Video-IFBench covers single-task, multi-task, selection, and nested instructions across 32 task types and 39 semantic or format constraints.The benchmark contains 1.5K samples collected through a semi-automatic pipeline with MLLM extraction, programmatic processing, and human verification.
  • Evaluation: The evaluation combines LLM-as-Judge with programmatic verification to assess instruction adherence across more than 20 recent MLLMs.The protocol is presented as a hybrid assessment of model performance on video instruction following.
  • Findings: Current MLLMs struggle especially with many constraints, semantic constraints, and complex conditional instructions.The strongest evaluated model achieves only 54.5% overall score, and strong video understanding accuracy does not automatically ensure faithful adherence to user intent.

2 Related Work

Prior work evaluates instruction following mainly in text-only and narrower multimodal settings, while video benchmarks primarily assess understanding accuracy. Video-IFBench extends this line of work to broader video understanding tasks and more complex instruction structures.

  • Text-only instruction following: Text-only instruction-following benchmarks evaluate explicit, structured, compositional, multi-constraint, sequential, and agentic requirements.Examples include FollowEval, FollowBench, IFEval, InFoBench, and later studies of more realistic instruction settings.
  • Multimodal instruction following: Multimodal instruction-following benchmarks extend evaluation across image, speech, dialogue, and crosslingual settings.The cited benchmarks include MIA-Bench, MM-IFEngine, VC-IFEval, MMMT-IF, and MCIF.
  • Video instruction following: Prior video-domain instruction-following work studies constrained video captioning, whereas Video-IFBench targets broader video understanding tasks and more complex instruction structures.The contrast concerns both task breadth and instruction complexity.
  • Video understanding benchmarks: Existing video understanding benchmarks cover temporal understanding, long-video comprehension, knowledge-intensive reasoning, robustness, and streaming interaction.These benchmarks primarily measure response correctness and provide limited evaluation of faithful adherence to complex user instructions.

3 Video-IFBench

Video-IFBench evaluates video instruction following through diverse task compositions, video-grounded conditions, and response constraints. Its construction combines curated videos, structured annotations, generated instructions and checklists, human verification, and hybrid response evaluation.

  • Instruction taxonomy: Four instruction templates—Single, Multi, Selection, and Nested—cover task composition and conditional structure.Single and Multi vary the number of tasks, while Selection and Nested require following video-grounded branches or paths.
  • Instruction taxonomy: Single instructions vary constraint counts from 1 to 15, whereas Multi instructions combine at least two tasks with mostly no more than three global or task-specific constraints.
  • Instruction taxonomy: Selection chooses one correct branch, while Nested follows the correct root-to-leaf path through tree-structured conditions.
  • Dataset construction: The benchmark pipeline curates public videos, constructs task and constraint pools, extracts global and timestamped facts, generates instructions and checklists, and applies human verification.Long videos are split into one-minute segments for sequential fact extraction with shared entity memory.
  • Dataset construction: The constraint pool contains 39 types, including 22 semantic and 17 format constraints, while tasks span six high-level video-understanding categories.
  • Evaluation paradigm: Evaluation combines LLM judging with programmatic verification, first checking task coverage for the active instruction and then assessing associated checklist constraints.For conditional samples, only the instruction under the correct branch or path is active; deterministic verification handles constraints such as length, format, and keywords.

4 Experiments

Experiments show that current MLLMs often satisfy some video-instruction constraints but fail to satisfy all of them, with conditional, semantic, and constraint-rich instructions posing particular difficulty. Performance also varies by model group, scale, thinking mode, and branch configuration.

  • 4.2 Main Results: Gemini-3-Pro achieves 76.5 TCSR and 54.5 TISR overall, while TISR remains lower than TCSR across models.Qwen3.5-397B-A17B-Think is the strongest open-source model at 69.6 TCSR and 46.1 TISR.
  • 4.2 Main Results: Selection and Nested instructions are harder than Single and Multi instructions because models must select the correct video-grounded branch or path before answering.Nested instructions yield only 53.7 TCSR and 46.0 TISR for Gemini-3-Pro, while the best open-source model reaches 33.0 TCSR and 28.2 TISR.
  • 4.3.1 Constraint-level Performance Analysis: Temporal grounding and condition-dependent selection constraints, including Transition Boundary, Temporal Anchor, Temporal Scope, and Action-Relative Selection, score low across most models.These constraints require aligning responses with fine-grained visual or audio evidence.
  • 4.3.1 Constraint-level Performance Analysis: Models perform substantially worse on semantic constraints than on format constraints, reflecting the need for precise video understanding rather than response formatting alone.Proprietary models generally score higher across challenging constraints, while open-source models show greater variation and lower performance.
  • 4.3.2 Task and Constraint Scaling Analysis: Increasing constraint counts lowers performance within Single instructions, whereas capable models remain relatively stable as task counts increase.The controlled analysis therefore identifies constraint adherence, rather than multiple tasks alone, as the primary bottleneck.
  • 4.3.4 Branch Number and Position in Selection Instructions: More candidate branches reduce Selection task pass rates, and later correct-branch positions expose a preference for earlier branches.Gemini-3-Flash drops from 66.7% with two branches to 56.5% with four, while Qwen3.5-397B-A17B-Instruct drops from 58.0% to 39.1%; Qwen3.5-397B-A17B-Think falls from 76.5% at position 1 to 52.9% at position 3.

5 Conclusion

The paper introduces Video-IFBench and a hybrid evaluation protocol for video instruction following. Experiments show that current MLLMs struggle with constraint-rich and conditional instructions, motivating further research on faithful instruction following.

  • 5 Conclusion: Video-IFBench evaluates video instruction following with a hybrid protocol combining LLM-as-Judge and programmatic verification.The benchmark is intended to support research on complex, constraint-rich instructions in practical video applications.

A.1 Main results using Qwen3.5-35B-A3B as Judge Model.

Using a smaller judge generally raises absolute scores while preserving the main relative trends, supporting the robustness of the evaluation protocol.

  • A.1 Main results using Qwen3.5-35B-A3B as Judge Model.: A Qwen3.5-35B-A3B judge generally increases absolute scores, but relative model trends remain largely consistent with the main results.The appendix re-evaluates all model responses with the smaller judge and reports differences from Table 1.

A.2 Effect of Thinking Mode on Video-IFBench.

Thinking mode improves semantic-constraint performance for some model families but has model-dependent effects. Its gains are generally larger for semantic than format constraints.

  • A.2 Effect of Thinking Mode on Video-IFBench.: Thinking mode brings clear gains for Qwen3 and Qwen3.5 models but little improvement or performance drops for InternVL3.5.The authors hypothesize that these model-dependent effects may relate to instruction-following reward design during reinforcement learning.
  • A.2 Effect of Thinking Mode on Video-IFBench.: Thinking-style inference generally helps more on semantic constraints than on format constraints.The pattern suggests greater benefit when constraint following requires visual or audio understanding.

B Task Taxonomy

Video-IFBench organizes video understanding into task types spanning temporal, spatial, relational, logical, audio, and captioning capabilities. The taxonomy includes both recognition questions and reasoning over events, entities, viewpoints, and audio-visual alignment.

  • The taxonomy covers temporal understanding, spatial understanding, relationship and interaction, logical reasoning, audio change, and captioning tasks.
  • Audio and Captioning: Audio and captioning tasks cover speaker changes, audio-visual temporal alignment, segmented and dense captioning, and whole-video summarization.
  • Spatial Understanding: Spatial tasks include relative distance, spatial relationships, perspective-based judgments, and relative direction from a viewpoint.
  • Relationship & Interaction: Relationship tasks test entity interactions and binding actions to their agents and targets, such as identifying who passed to whom.
  • Logical Reasoning: Logical reasoning includes causal, counterfactual, multi-hop, and knowledge-transfer questions that connect events or apply video-demonstrated principles to new scenarios.

C Constraint Taxonomy

The constraint taxonomy combines semantic restrictions on what video content may be selected or described with format requirements governing how answers are produced. It covers temporal, spatial, entity, action, ordering, representation, language, length, and structural constraints.

  • Semantic Constraints: Semantic constraints regulate temporal scope, viewpoint, spatial relations, categories, attributes, actions, counts, and transition boundaries.
  • Semantic Constraints: Temporal constraints can require event ordering, intrinsic event stages, duration thresholds, temporal granularity, or specified shot-based references.
  • Semantic Constraints: Entity and action constraints can filter targets, require or avoid attributes and actions, and limit how many entities or events are explicitly mentioned.
  • Format Constraints: Format constraints specify ordering, normalized values, JSON, tables, ordered lists, closed-set answers, language, titles, keywords, perspective, and text types.
  • Format Constraints: Additional format constraints control response length, paragraph and list-item counts, titles, keyword inclusion or exclusion, pronouns, perspective, capitalization, labels, and source restrictions.

D Example Visualization

The examples visualize instruction-following failures across single, multi, selection, and nested instructions. Errors include violating semantic and temporal constraints, formatting requirements, and conditional branch selection.

  • Single Instructions: Single-instruction examples show failures involving forbidden color mentions, inaccurate temporal spans, outside-window content, capitalization, and other output restrictions.
  • Multi Instructions: The multi-instruction examples illustrate violations of duration, bullet-count, and forbidden-state constraints.
  • Selection and Nested Instructions: Selection and nested examples show models choosing the wrong branch or following the wrong path in conditional structures.
Loading 2608.25529v1…