PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning

Jiahao Wang, An Ping, Yanghai Wang, Yuanxing Zhang, Shihao Li, Hanyan Bian, Yichi Ren, Yize Zhang, Han Wang, Haowen Chen, Junze Li, Jiaqi Wang, Yiyang Hu, Zhuze Xu, Zijie Zhang, Jiaheng Liu

arXiv:2606.08572v1cs.CV

TL;DR

Omni-modal models can process audio and visual streams, but existing benchmarks do not adequately measure adherence to complex, cross-modal instructions. OmniCap-IF introduces a 50-constraint benchmark with separate format and content evaluation, temporal grounding, and a 54K tuning dataset; its analyses identify a format-content tradeoff and its tuned model improves across metrics.

  • Problem

    Existing evaluations do not adequately capture complex instruction following across omni-modal audio-visual content and user constraints.

  • Method

    OmniCap-IF evaluates 50 constraint types across format, Visual, Audio, and Audio-Visual content using decoupled assessment with Temporal Grounding.

  • Results

    OmniCaptioner-IF outperforms its base model across all metrics and performs on par with Gemini-3.1-Pro on format metrics.

  • Takeaways & Limitations

    The benchmark exposes a format-content tradeoff, while targeted instruction tuning improves complex instruction adherence and general omni-modal perception.

  • Takeaways & Limitations

    The benchmark primarily covers 30-to-90-second videos, and content evaluation partially relies on LLM-as-a-judge whose hallucinations cannot be entirely eliminated.

Abstract

from arXiv · show

While Omni-modal Large Language Models (OLLMs) have demonstrated impressive capabilities in jointly processing audio and visual streams, their ability to strictly adhere to complex, multi-faceted user instructions remains largely unexplored. Existing benchmarks primarily focus on holistic video understanding or text-only instruction following, failing to capture the intricate interplay between modalities and user constraints. To bridge this gap, we introduce OmniCap-IF, the first comprehensive benchmark specifically designed to evaluate instruction-following capabilities in omni-modal captioning. OmniCap-IF incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness. Our benchmark encompasses 50 distinct constraint types across pure visual, pure audio, and audio-visual modalities, while integrating Temporal Grounding to assess spatio-temporal precision. Extensive evaluations of prominent models on 1,920 high-quality samples reveal significant performance disparities. Furthermore, our analysis uncovers a critical "format-content tradeoff", demonstrating that increasing formatting complexity directly degrades models' omni-modal reasoning abilities. Finally, to advance the field, we curate a 54K instruction-tuning dataset, OmniCap-IF-54K and present OmniCaptioner-IF, which achieves notable improvements in both complex instruction adherence and general omni-modal captioning performance.

1 Introduction

OmniCap-IF addresses the lack of comprehensive evaluation for complex instruction following in omni-modal captioning. It combines 50 constraint types, temporal grounding, decoupled evaluation, and instruction tuning to improve controllable generation.

  • Motivation: Existing benchmarks do not adequately test compositional constraints across jointly processed audio-visual content.They emphasize semantic richness or question answering, or remain limited to single-modality instruction following.
  • Benchmark: OmniCap-IF is introduced as the first benchmark dedicated to instruction following in omni-modal captioning.Its framework covers format and content dimensions across Visual, Audio, and Audio-Visual modalities.
  • Benchmark: Temporal Grounding adds quantitative assessment of precise timestamp localization to the benchmark.This is intended to align evaluation more closely with real-world scenarios.
  • Findings: The evaluation reveals a format-content tradeoff: stricter structural constraints substantially degrade models’ audio-visual reasoning.The protocol separates formatting difficulty from content performance to expose this effect.
  • Training resources: OmniCap-IF-54K and OmniCaptioner-IF are presented as a training dataset and model for controllable omni-modal generation.The reported results indicate improvements in instruction adherence and general omni-modal perception.

2 Related Work

Prior instruction-following evaluations expanded from text to vision-language settings but remain incomplete for complex omni-modal scenarios. OmniCap-IF extends this line of work with omni-modal constraints and fine-grained temporal localization.

  • Instruction-following benchmarks: Earlier instruction-following benchmarks evaluated programmatic constraints, structural formatting, and complex logical rules primarily in text-based settings.Later efforts extended the paradigm to vision-language tasks.
  • Instruction-following benchmarks: Existing evaluations remain confined to partial modalities and do not meet the requirements of complex multi-modal applications.The related-work discussion identifies this as a remaining limitation despite recent advances.
  • Omni-modal captioning: OmniCap-IF addresses this gap by introducing omni-modal constraints and fine-grained temporal localization.These additions target comprehensive omni-modal instruction following.
  • Omni-modal captioning: Recent omni-modal captioning benchmarks generally prioritize semantic accuracy and descriptive richness over arbitrary user-specified instruction following.Their evaluation paradigms include question-answer pairs, cloze assessments, and holistic captioning.

3 OmniCap-IF

OmniCap-IF organizes omni-modal instruction following around 50 constraint types, a diverse 480-video benchmark, human-refined annotations, and separate structural and semantic evaluation. It also introduces a decoupled 54K instruction-tuning dataset.

  • Constraint taxonomy: The taxonomy contains 50 constraint types spanning format requirements and Visual, Audio, and Audio-Visual content.Format constraints include structural and stylistic requirements, while content constraints require modality-specific factual comprehension.
  • Benchmark construction: The test set contains 480 videos selected for audio-visual richness, alignment, and broad domain coverage.The collection spans domains ranging from comedy to technology.
  • Annotation pipeline: Annotations combine automated instruction–checklist generation with professional human refinement and verification.This two-stage pipeline is designed to combine scalability with annotation quality.
  • Dataset statistics: The dataset spans varied video durations, categories, and instruction complexities, supporting evaluation across content coverage and instructional difficulty.The reported statistics emphasize diversity and cross-domain generalization.
  • Evaluation: The benchmark reports structural and semantic fidelity separately through rule-based format and temporal checks plus QA-based content evaluation.Its metrics include Constraint Satisfaction Rate and Instruction Satisfaction Rate, with content broken down by modality.
  • Instruction tuning: OmniCap-IF-54K comprises 54K curated video-instruction-response triplets generated through a pipeline decoupled from benchmark evaluation.The dataset is intended to support generalizable instruction-following capabilities while preventing data leakage.

4 Experiments

Experiments evaluate 14 omni-modal models across format and modality-specific content constraints, external captioning benchmarks, complexity conditions, and cross-modal analyses. Results show stronger format than content adherence, persistent audio-visual integration difficulties, and improvements from OmniCaptioner-IF fine-tuning.

  • Main Results: 14 leading omni-modal models are evaluated on OmniCap-IF, whose content CSR is decomposed into Visual, Audio, and Audio-Visual modalities.The evaluation also uses hierarchical metrics and compares automated assessments with human judgment.
  • Main Results: Models generally perform better on independent Audio and Visual constraints than on joint Audio-Visual constraints.This pattern highlights the difficulty of integrating audio and visual information simultaneously.
  • Main Results: Models exhibit stronger format control than content adherence, while human annotators significantly outperform all evaluated models in format control.The paper attributes this gap to the greater multimodal reasoning demands of content requirements and deliberate human verification.
  • Main Results: OmniCaptioner-IF outperforms its Qwen2.5-Omni base model across all metrics and matches Gemini-3.1-Pro on format metrics.The series is produced by fine-tuning Qwen2.5-Omni on OmniCap-IF-54K.
  • Existing Benchmarks: OmniCaptioner-IF-3B surpasses Qwen2.5-VL-Instruct-3B across all IF-VidCap metrics, while OmniCaptioner-IF-7B doubles total accuracy on Omni-Cloze and performs comparably to Gemini-2.5-Pro on UGC-VideoCap.These evaluations cover visual-only instruction following and broader omni-modal captioning benchmarks.
  • Further Analysis: As formatting level increases while content constraints remain fixed, content CSR drops continuously and noticeably.The controlled experiment uses 1,200 curated samples and three format levels, from loose natural language to syntactic machine-readable rules.
  • Further Analysis: Increasing frame sampling rates steadily reduce Format CSR, whereas Content CSR first rises and then falls, with the turning point varying across models.The initial content gain is linked to richer visual evidence, while excessive frame density adds redundant visual tokens.
  • Further Analysis: Cross-modal gains are strong for Gemini-3.1-Pro and MiniCPM-o-4.5 but minimal for Qwen models, with some cases showing cross-modal interference.Adding visual context significantly boosts Audio CSR for the stronger-synergy models, while Qwen2.5-Omni declines in Overall CSR.

5 Conclusion

The paper introduces OmniCap-IF as a benchmark for instruction following in omni-modal video captioning and evaluates models with format and content constraints. It also releases OmniCap-IF-54K and OmniCaptioner-IF, which improve structural control and omni-modal captioning performance.

  • Conclusion: OmniCap-IF defines 50 constraints across format, visual, audio, and cross-modal dimensions and uses a dual evaluation protocol.The benchmark is designed as a comprehensive diagnostic testbed for omni-modal instruction following.
  • Conclusion: Evaluations reveal limited deep cross-modal synergy in open-source models compared with proprietary counterparts.The conclusion identifies this as a key limitation of current omni-modal large language models.
  • Conclusion: OmniCap-IF-54K contains 54K instruction-tuning examples, and OmniCaptioner-IF improves complex structural adherence and general omni-modal captioning.The model is presented as mastering complex structural constraints while retaining strong omni-modal captioning ability.

Limitations

The paper identifies limitations in its evaluation methodology, reasoning robustness, and benchmark scope. These include reliance on LLM-based judging, degradation under strict formatting, and concentration on 30–90-second videos.

  • Evaluation limitations: LLM-as-a-judge is used partially for content constraints, so evaluator hallucinations cannot be entirely eliminated.The authors mitigate this with factual QA and rule-based format and temporal verification.
  • Reasoning limitations: Strict syntactic constraints such as deeply nested JSON can degrade current OLLMs’ deep cross-modal reasoning.Instruction tuning alleviates the issue, but rigid formatting and complex multi-step reasoning remain challenging.
  • Scope limitation: The benchmark primarily covers videos ranging from 30 to 90 seconds.The authors identify ultra-long videos with dense, multi-constraint instructions as an important next evaluation setting.

C Temporal Grounding Evaluation Scheme

The temporal grounding evaluation uses separate procedures for event intervals and instantaneous trigger points. Interval predictions are assessed by overlap, while point predictions receive a duration-dependent tolerance.

  • Evaluation design: Temporal grounding constraints are evaluated with distinct schemes for time intervals and precise time points.The scheme applies across Visual, Audio, and Audio-Visual modalities.
  • Time intervals: t-IoU measures overlap between a predicted event interval and its ground-truth interval, with t-IoU ≥0.5 defining success.This procedure is used when instructions request an event duration.
  • Precise time points: Precise trigger-point predictions use a dynamic tolerance margin based on total video duration.The margin accounts for video sampling and human annotation characteristics.
  • Tolerance design: The tolerance uses a 1.0-second floor and scales to 5% of total video length.For a 60-second video, the scaling component gives a 3.0-second tolerance.

D Dataset Samples

The dataset samples combine structural, temporal, visual, audio, and audio-visual constraints in concrete captioning prompts. Examples require prescribed output formats alongside multimodal content extraction and temporal localization.

  • Audio-visual samples: One defensive-play sample requires a Markdown table combining timestamps, visual action, audio content, and character state.The prompt also requests the ball-blocking moment, the spoken line, the subsequent landing, and any squishing sound.
  • Audio-visual samples: The defensive-play sample decomposes evaluation into visual temporal grounding, audio temporal grounding, and omni-specific landing-sound reasoning.Separate checks ask when the character blocks the ball, starts speaking, and whether the landing process explains a squishing sound.
  • Structured output samples: Another sample requires a JSON object containing exactly two fields: magician_request and participant_compliance.The fields separately test speech recognition and detailed description of the participant’s actions.
  • Structured output samples: A third sample combines ALL CAPS output, a pipe delimiter, timestamp formatting, visual grill-mark description, and voice-based speaker inference.The timestamp interval where the overlay matches the spoken phrase is 00:18 - 00:20.

E Error Analysis

Error analyses show that baseline models frequently violate both structural and multimodal content constraints, whereas OmniCaptioner-IF improves formatting, temporal grounding, and audio-visual fidelity.

  • Overall comparison: Figure 9 provides the complete CSR heatmap across models, formatting categories, and audio-visual constraint types.It serves as a reference for the overall performance landscape.
  • Overall comparison: Figure 10 compares OmniCaptioner-IF with Qwen2.5-Omni baselines and reports consistently high CSR across formatting tasks after tuning.The largest improvement is described for strict structural formats.
  • Overall comparison: OmniCaptioner-IF improves strict-format CSR, including Timestamp from 10.8% to 91.0% for 7B and from 4.3% to 88.8% for 3B.Other gains include Markdown, Delimiter, and JSON constraints.
  • Overall comparison: OmniCaptioner-IF raises Temporal Grounding CSR from 10.6% to 33.9% for the 7B model while also improving Perspective constraints.The gains extend beyond format adherence to complex audio-visual requirements.
  • Baseline failures: In the Padel example, the baseline summarizes commentary and overcomplicates the JSON schema, while the target requires direct transcription and exact fields.The comparison highlights simultaneous format and content errors.
  • Baseline failures: In a perspective example, the baseline uses Arabic numerals and malformed time ranges instead of the required Roman-numeral list and [MM:SS] timestamps.The baseline also misattributes an interrupting character.
  • Baseline failures: In another example, the baseline hallucinates timestamps and unrelated audio, while OmniCaptioner-IF grounds the event at [00:20-00:28] and transcribes synchronized speech.The OmniCaptioner-IF output also links lip movement with the audio source.
  • Baseline failures: For audio-triggered extraction, the baseline repeats 00:00:00.00, whereas OmniCaptioner-IF retrieves distinct trigger-linked actions at [00:21], [00:31], and [00:35].The example demonstrates improved audio anchoring and temporal localization.

F Prompts

The appendix specifies the system prompts, extraction and judging procedures, constraint checks, and benchmark-prompt construction rules used to evaluate captions.

  • F.1 Test: The test system prompt requests accurate controlled omni-modal captions from visual and audio inputs without extra opening or closing remarks.It specifies a nominal sampling rate of 1 frame per second, while noting that model input settings may vary.
  • F.2 Judge: Format extraction uses a strictly rule-following program that receives the response, original prompt, and check item, then populates the check item's content field.The extracted content is passed to the corresponding format-checking function.
  • F.2 Judge: Each format check returns the logical AND of checks over content elements, with extraction rules preserving complete checkable content and applicable scope.List markers may be excluded for length or language checks, while structural syntax must be retained when required.
  • F.2 Judge: Ordered-list extraction treats a continuous list as one element, while JSON-object extraction passes the entire relevant object rather than only its keys.The examples contrast correct and incorrect extraction behavior.
  • F.2 Judge: Count checks extract all semantic items regardless of requested bounds, and absent required structures produce an empty extraction.These rules support accurate evaluation of overcounts and missing tables.
  • F.2 Judge: Content evaluation receives structured inputs and judges captions using specified checks against the question and reference video description.The appendix distinguishes this evaluator from the separate format-content extraction program.
  • F.3 Construction of Prompts for The Test Set: Prompt construction distinguishes table structure from Markdown decoration, top-level JSON object from array checks, and semantic count from textual length.These distinctions determine which constraint type is generated or evaluated.

H Evaluation Settings

Evaluation settings document model configurations, prompt provenance, frame sampling metadata, and the parameter-efficient fine-tuning setup for OmniCaptioner-IF.

  • Model settings: Most open-source models use default settings, while closed-source models are accessed through APIs with default configurations.Detailed evaluated-model settings are listed in Table 10.
  • Model settings: The model-testing system prompts are documented in Section F.1.This links the evaluation configuration to the appendix prompt specification.
  • Fine-tuning: OmniCaptioner-IF-7B and 3B are fine-tuned from Qwen2.5-Omni-7B and 3B using LoRA applied to all linear layers.The LoRA rank is 16 with alpha 32.
  • Fine-tuning: Training runs for one epoch on OmniCap-IF-54K with AdamW learning rates of 2 × 10^-5 for 7B and 3 × 10^-5 for 3B.These settings differ by model size.
  • Fine-tuning: Training uses eight H200 GPUs, effective global batch size 16, bf16 mixed precision, and 1-FPS video preprocessing.The passage also describes per-device batching and gradient accumulation.

I.2 Convergence Analysis and Dataset Sufficiency

The 54K instruction-tuning dataset supports rapid convergence within one epoch, while scaling beyond 40K samples yields only marginal additional gains. Automatic judging also shows high consistency with human experts, especially for format constraints.

  • Convergence Analysis: After approximately 1,000 steps, training loss flattens and stabilizes around 0.9 with minor end-of-epoch fluctuations.The reported trajectory is interpreted as smooth convergence within one epoch.
  • Dataset Sufficiency: 20K samples raise Overall CSR from 49.19% to 68.50% and Format ISR from 34.17% to 75.80% relative to Qwen2.5-Omni-7B.The largest improvement occurs with the first 20K training samples.
  • Dataset Sufficiency: Overall CSR increases only from 70.35% to 70.73% between 40K and 54K samples, indicating diminishing returns from further scaling.The gain is described as approaching saturation.
  • Dataset Sufficiency: Format CSR changes slightly from 90.52% at 40K samples to 90.39% at 54K, consistent with convergence rather than continued improvement.The passage characterizes this small oscillation as a typical convergence hallmark.
  • Evaluation Reliability: gpt-5-mini shows high consistency with human experts on 1,000 samples, particularly for format constraints.This comparison supports the reliability of the automatic evaluation protocol.

K Results on Omni-VideoQA Benchmarks

The paper evaluates generalizable omni-modal perception and reasoning on DailyOmni and WorldSense under Caption-to-QA and Direct QA settings.

  • Evaluation Settings: Evaluations on DailyOmni and WorldSense use two settings: Caption-to-QA and Direct QA.These settings provide distinct evaluation routes for omni-modal capabilities.

K.1 Caption-to-QA Performance

In Caption-to-QA, models first generate detailed captions and then answer questions from those captions. OmniCaptioner-IF-7B achieves strong performance on both benchmarks.

  • Evaluation Procedure: Caption-to-QA first generates detailed captions incorporating visual and audio details, then answers multiple-choice questions using only those captions.The question-to-prompt constructor specifies the caption-generation instruction.
  • Results: OmniCaptioner-IF-7B achieves 60.2% on DailyOmni and 43.2% on WorldSense, surpassing most open-source counterparts and matching Gemini-2.5-Pro.The reported results are based on generated captions evaluated by an LLM judge.

K.2 Direct QA Performance

In Direct QA, models answer questions directly from videos without generating intermediate captions. OmniCaptioner-IF-7B significantly outperforms its base model on both benchmarks.

  • Evaluation Procedure: Direct QA feeds the video and question directly to the model, testing native end-to-end multimodal understanding without an intermediate caption.This setting isolates direct video-question answering from caption generation.
  • Results: OmniCaptioner-IF-7B achieves 68.4% on DailyOmni and 49.4% on WorldSense, significantly outperforming Qwen2.5-Omni-7B.These results are reported for the Direct QA setting.
Loading 2606.08572v1…