PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding

Lin Fu, Zheyuan Yang, Yang Wang, Tingyu Song, Arman Cohan, Yilun Zhao

arXiv:2606.05259v1cs.CV

TL;DR

Models still struggle to move from surface-level video perception to reasoning that requires domain knowledge and multi-step inference. VideoKR pairs a skill-oriented, human-audited training corpus with an expert-annotated benchmark; post-trained models outperform prior approaches on knowledge-intensive reasoning while remaining competitive on general video reasoning.

  • Problem

    Current models remain limited in moving from surface-level video perception to tasks requiring domain knowledge and multi-step inference.

    Source: §1 · p. 1

  • Method

    VideoKR uses a human-audited, skill-oriented pipeline to generate reasoning examples from 145K CC-licensed videos across 82 professional subjects.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Models post-trained on VideoKR achieve the best knowledge-intensive performance among similar-sized models while remaining competitive on general video benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support a corpus-centric approach to advanced video reasoning that does not rely on sophisticated RL reward engineering.

    Source: §7 · p. 9

    Existing knowledge-intensive benchmarks include examples answerable without continuous video understanding, with single-frame answerability exceeding 35% on MMVU and VideoMMMU.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

We introduce VideoKR, the first large-scale training corpus specifically designed to strengthen knowledge- and reasoning-intensive video understanding. It comprises 315K video reasoning examples over 145K newly collected, CC-licensed, expert-domain videos. We develop a human-in-the-loop, skill-oriented example generation pipeline that targets progressively deeper video reasoning capabilities while ensuring the difficulty, diversity, and reliability of both the examples and their CoT rationales. We also curate VideoKR-Eval, a new expert-annotated benchmark where questions require genuine video understanding and knowledge-intensive reasoning rather than textual shortcuts. Our experiments show that, under a standard SFT$\rightarrow$GRPO pipeline, models post-trained on VideoKR outperform prior post-training approaches on knowledge-intensive video reasoning while remaining competitive on general video reasoning, highlighting data design as a key driver of progress in video reasoning. We further conduct comprehensive ablations to isolate the contributions of VideoKR, providing actionable insights for future work.

1. Introduction

VideoKR addresses the gap between progress in video perception and models’ continuing difficulty with domain-aware, multi-step video reasoning. It pairs a skill-oriented training corpus with an expert-annotated evaluation benchmark and reports strong knowledge-intensive performance under standard post-training.

Introduction

  • Existing video datasets emphasize perceptual tasks and everyday activities, leaving specialized domains and knowledge-intensive reasoning undercovered (Feng et al., 2025a; Wang et al., 2025a; Meng et al., 2025; Chen et al., 2025b; 2026).The paper links this gap to models’ struggles with multi-hop inference, scientifically grounded explanations, and events governed by non-observable principles.Source: §1 · p. 2
  • VideoKR provides a large-scale corpus for knowledge- and reasoning-intensive video understanding, built from 145K CC-licensed videos across 82 professional subjects.Its skill-oriented QA framework covers basic video reasoning, knowledge-enhanced video perception, and knowledge-intensive video reasoning; expert-involved quality control produces VideoKR-SFT-201K and VideoKR-RL-114K.Source: §1 · p. 2
  • The authors use a standard SFT→GRPO pipeline and standardized evaluation to isolate data design, then ablate CoT supervision, skill-based composition, and comparisons with prior corpora.These controlled studies are presented as a way to disentangle VideoKR’s contributions and inform future work.Source: §1 · p. 2
  • VideoKR-Eval addresses single-frame answerability in prior benchmarks through multi-model probing and expert re-annotation of filtered videos.The benchmark was developed after a manual audit found that many existing knowledge-intensive video reasoning examples could be solved with little video understanding.Source: §1 · p. 2
  • VideoKR-post-trained models achieve the best knowledge-intensive performance among similar-sized models while remaining competitive on general video benchmarks.The authors report that Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct post-trained on VideoKR outperform prior post-training approaches.Source: §1 · p. 2

2. Related Work

Related work has broadened video benchmarks toward domain-aware reasoning, while video post-training commonly combines SFT with RL. The paper identifies limitations in existing data and presents VideoKR as a newly collected, expert-domain alternative.

Video Understanding Datasets

  • Video benchmarks have expanded from general perceptual, spatiotemporal, and cross-modal evaluation toward domain-aware knowledge and reasoning tasks, including specialized-domain and scientific video benchmarks.Examples include Video-MME, MVBench (Li et al., 2024b), MMVU (Zhao et al., 2025b), VideoMMMU (Hu et al., 2025), Video-MMLU (Song et al., 2025), and SciVideoBench (Deng et al., 2025).Source: §2 · pp. 2–3

Post-training for Video Understanding

  • Video reasoning post-training typically combines SFT on datasets with chain-of-thought, temporal, or spatial signals and RL with verifiable rewards targeting spatial, temporal, or integrated spatiotemporal reasoning.The SFT→RL pipeline and its common signals and reward targets frame the post-training approaches discussed in the section.Source: §2 · p. 3
  • Existing post-training corpora often target basic perception, reuse older short-video datasets, or rely on single-model synthesis, whereas VideoKR uses newly collected CC-licensed expert-domain videos and human-in-the-loop generation.The framework is skill-oriented and aims to ensure data difficulty, diversity, and reliability; under SFT→GRPO, VideoKR-trained models outperform prior post-training approaches.Source: §2 · p. 3

3. VideoKR Training Corpus Construction

VideoKR is built through a semi-automated, quality-controlled pipeline that uses domain experts to audit model outputs, construct domain coverage, collect videos, generate skill-oriented examples, and validate the resulting training data.

3.1. Domain Knowledge Bank Construction

  • The corpus contains 63,745 knowledge points organized through 82 subjects and a Subject → Course → Lecture → Knowledge Point hierarchy.Experts selected four to eight core courses per subject and compiled syllabi; LLMs identified knowledge points with terms and paragraph-length definitions for each lecture.Source: §3.1 · pp. 3–4

3.2. Knowledge-Driven Video Collection

  • Scenario-based search retrieves CC-licensed videos depicting real-world applications of knowledge, rather than relying only on instructional recordings.LLMs generated one to three scenarios per knowledge point, which were converted into search keywords; candidates were screened using metadata and visual content, videos over 30 minutes were excluded, and four sampled frames were checked for safety. The process collected 146,567 videos.Source: §3.2 · p. 4

3.3. Skill-Oriented Example Generation

  • VideoKR’s skill-based pipeline distinguishes direct video reasoning (VIDR), knowledge-enriched perception (KNOWVID), and multi-hop knowledge-intensive reasoning (KNOWVIDR).Experts created 1,800 seed examples across the three skills and four disciplines, with independent review; frontier MLLMs then generated two examples per skill for each video, using sampled frames and seed examples. Generated examples passed self-consistency, video-dependency, and rationale-support checks.Source: §3.3 · pp. 4–5

3.4. VideoKR Data Quality Control

  • A human-validated selection protocol assigns qualified models to pipeline stages, while end-to-end assessment measures residual example and reasoning-trace errors.Seven frontier models were considered; experts assessed 100 real-input instances per model and step, and only models below a predefined error threshold were eligible. In an 800-example sample, annotators flagged 52 potentially non-visual-solvable questions and 32 reasoning-trace errors, 17 of which changed the answer.Source: §3.4 · p. 5
  • A two-stage decontamination process removed benchmark-matching and near-duplicate videos from VideoKR.YouTube-ID filtering removed 131 videos, and frame-level perceptual hashing with windowed sequence matching removed 877 near-duplicates.Source: §3.4 · p. 5

3.5. VideoKR-SFT-201K & VideoKR-RL-114K

  • The 315,537 examples were split by video into VideoKR-SFT-201K and VideoKR-RL-114K for supervised fine-tuning and RLVR training, respectively.SFT examples retain validated CoT rationales as supervision, while RLVR examples retain questions and verifiable answers.Source: §3.5 · p. 5

4. VideoKR-Eval Evaluation Benchmark

VideoKR-Eval addresses single-frame shortcuts in existing knowledge-intensive video benchmarks by retaining examples that require continuous video understanding and adding expert-reannotated questions grounded in video evidence.

4.1. Limitations of Existing Benchmarks

  • >35% single-frame answerability was observed on MMVU and VideoMMMU among frontier models, motivating a benchmark that demands continuous video understanding.Probing gave each model only the question, answer options, and one randomly sampled frame, with three independent trials per example.Source: §4.1 · p. 6

4.2. VideoKR-Eval Benchmark Construction

  • 1,254 original examples were retained only when each of three frontier models failed to answer correctly in at least one of three single-frame trials.The models were Qwen3-VL-235B-A22B, Claude-4.5-Sonnet, and GPT-5.2; the retained set is their intersection of examples requiring continuous video understanding.Source: §4.2 · p. 6
  • 746 expert-reannotated examples were added from filtered videos, bringing VideoKR-Eval to 2,000 examples.Experts wrote questions grounded in clearly observable video evidence, requiring domain knowledge and uniquely determined answers.Source: §4.2 · p. 6

5. Experiment Setup

The experiments use a standard SFT→GRPO pipeline to isolate the contribution of VideoKR data, then evaluate models on general and knowledge-intensive video reasoning benchmarks using a standardized protocol.

5.1. Post-Training on VideoKR

  • The study uses a controlled SFT→GRPO scaffold to test whether data design limits knowledge-intensive video understanding, training two base models on VideoKR-SFT-201K and VideoKR-RL-114K.Each stage runs for one epoch; Qwen3-VL-8B-Instruct also receives a one-epoch Zero-RL run. Batch size is 32, with ROUGE for open-ended QA and Exact Match for multiple-choice QA.Source: §5.1 · p. 6

5.2. Evaluation Setup

  • Evaluation covers seven benchmarks in general video reasoning and knowledge-intensive video reasoning, including the expert-domain VideoKR-Eval.General benchmarks are Video-MME, MVBench, and LongVideoBench (Wu et al., 2024); knowledge-intensive benchmarks are VideoMMMU (Hu et al., 2025), MMVU (Zhao et al., 2025b), SciVideoBench (Deng et al., 2025), and VideoKR-Eval.Source: §5.2 · pp. 6–7
  • The authors attribute cross-paper result inconsistencies partly to prompt misalignment and standardize evaluation with model-specific prompts and recommended frame configurations.They run each model three times with independent sampling and report the mean using LMMs-Eval.Source: §5.2 · p. 7

6. Experiment Results and Analysis

VideoKR post-training improves knowledge-intensive video reasoning, while the analyses show benefits from richer inference-time visual evidence, skill-composition and CoT supervision, and challenging training data.

6.1. Main Results

  • 46.6 (+4.7) and 51.5 (+3.0) are the knowledge-intensive averages for VideoKR-post-trained Qwen2.5-VL-7B and Qwen3-VL-8B, respectively.These rise from 41.9 and 48.5; Qwen3-VL-8B's 51.5 exceeds the strongest competing 7/8B-scale model, Qwen3-VL-8B-Thinking, at 50.0. Starting from SFT, adding RL consistently beats SFT-only, and RL-only generally outperforms SFT-only. Post-training improves Qwen2.5-VL-7B across all evaluated benchmarks and gives Qwen3-VL-8B clear gains, especially on knowledge-intensive benchmarks.Source: §6.1 · pp. 7–8

6.2. Case Study

  • In a 100-example VideoKR-Eval case study, post-trained Qwen3-VL-8B integrates visual evidence with domain knowledge and exhibits “aha-moment” reasoning patterns.Examples are provided in Appendix D.3.Source: §6.2 · p. 8

6.3. Analysis of Inference-Time Frame Scaling

  • 65.5% versus 60.1% on general reasoning and 46.6% versus 44.2% on knowledge-intensive reasoning are Qwen2.5-VL-7B SFT+RL results at 128 versus 16 frames.The model was trained with 128 frames and evaluated at 16, 32, 64, and 128 while other inference settings were fixed; increasing frame count consistently improves both the base and post-trained models. Figure 3 presents category-wise average accuracies across frame budgets.Source: §6.3 · p. 8, Figure 3 · p. 8

6.4. Ablations on VideoKR-SFT-201K

  • 42.4% knowledge-intensive performance results from combining all three skill components, while CoT supervision raises the knowledge-intensive average from 39.4% to 42.4%.Across VIDR, VIDR +KNOWVID, and VIDR +KNOWVID +KNOWVIDR, knowledge-intensive performance is 41.4%, 41.3%, and 42.4%, while VideoKR-Eval accuracy rises from 35.3 to 35.9 to 36.8. In the CoT comparison, the gain over Direct Output is 3.0 points; the reported ablation uses Qwen2.5-VL-7B-Instruct and 128 input frames.Source: §6.4 · pp. 8–9, Table 4 · p. 9

6.5. VideoKR vs Prior Post-Training Corpus

  • 42.4 under SFT makes VideoKR-SFT the only tested corpus to surpass the 41.9 base-model knowledge-intensive average, while VideoKR-RL reaches 43.0 (+1.1) under zero-RL.The zero-RL result is ahead of next-best VideoAuto-R1 at 42.7; Video-R1 and VideoRFT yield 36.2 and 38.4 under SFT. On sampled QA training examples, Qwen3-VL-8B scores 49.1%–57.1% on prior corpora but 42.3% on VideoKR, indicating prior corpora are saturated for these base models while VideoKR remains more challenging.Source: §6.5 · p. 9, Table 5 · p. 9

7. Conclusion

VideoKR offers a corpus-centric approach to post-training for advanced video reasoning: integrating structured domain concepts with visually grounded examples improves reasoning, especially on knowledge-intensive video tasks, without sophisticated RL reward engineering.

7. Conclusion

  • Post-training on VideoKR produces strong improvements, especially in knowledge-intensive video reasoning.Source: §7 · p. 9
  • Integrating structured domain concepts with visually grounded examples yields stronger reasoning performance without sophisticated RL reward engineering.Source: §7 · p. 9
  • The work presents a corpus-centric perspective on post-training for advanced video reasoning, treating perception, domain knowledge, and reasoning as integrated rather than loosely linked.Source: §7 · p. 9

Impact Statement … A.2. Annotator Information

The supplied passages describe VideoKR’s legal and annotation safeguards, its four-discipline knowledge-bank structure, and the annotators’ roles in dataset construction and evaluation. They also report an inference-cost estimate.

Impact Statement

  • CC-licensed videos allow reuse with attribution and provide clear provenance for VideoKR’s materials.The passage says these restrictions address provenance ambiguity in prior training corpora.Source: Impact Statement · p. 9
  • Annotators were paid an average of $13 USD per hour, above prevailing rates for comparable local work.Source: Impact Statement · p. 9
  • All expert annotators gave informed consent and explicitly authorized public release and redistribution of their annotations.Source: Impact Statement · p. 9
  • VideoKR and VideoKR-Eval data construction incurred approximately 70.4K US dollars in model inference costs.Source: Impact Statement · p. 9

A.1. Domain Knowledge Bank Construction

  • A manual review of leading universities’ undergraduate curricula identified 82 representative subjects across four major disciplines.The disciplines are Natural Sciences, Healthcare, Humanities and Social Sciences, and Engineering.Source: §A.1 · p. 13, Table 6 · p. 13
  • The subjects form the top-level index of a four-layer knowledge base—subject, course, lecture, and knowledge point—supporting broad coverage and balanced sampling.Source: §A.1 · p. 13

A.2. Annotator Information

  • The VideoKR construction pipeline involved 34 annotators across knowledge-bank construction, seed curation, model validation, quality assessment, and VideoKR-Eval construction.Source: Table 7 · p. 14

A.3. VideoKR Data Example · A.4. Human-Validated Model Selection Protocol · A.5. Data Contamination Mitigation

The appendices illustrate VideoKR examples across expert domains, describe human validation for assigning models to pipeline stages, and specify a near-duplicate video filtering procedure. The selection protocol uses stage-specific qualification rather than relying on a single foundation model.

A.3. VideoKR Data Example

A.4. Human-Validated Model Selection Protocol

  • The qualification protocol maintains a dynamic pool of eligible models for each synthesis stage to avoid imprinting biases from relying on a single model.The protocol is intended to preserve corpus quality while maximizing data diversity.Source: §A.4 · p. 17
  • Seven frontier models were piloted on 100 representative inputs per pipeline stage, and domain experts deemed a model eligible only when its total error rate was ≤3%.Evaluation distinguished hard compliance failures, such as JSON violations, from soft content failures, such as hallucinations or weak reasoning.Source: §A.4 · p. 17
  • Table 8 summarizes eligible models by construction step; “-” denotes a modality mismatch, while “✗” denotes failure of human validation.The passage does not provide the table’s individual model assignments.Source: Table 8 · p. 17, §A.4 · p. 17

A.5. Data Contamination Mitigation

  • The final near-duplicate filter flags a match when at least 70% of aligned frames in the best 20-second window have Hamming distance ≤30, then removes the matched training video.Benchmark and VideoKR videos are sampled at 1 fps; training videos are indexed in overlapping 20-second windows with a 1-second stride, and each benchmark video retrieves its top 10 candidate windows for verification.Source: §A.5 · p. 17
  • For scalable retrieval, training videos are divided into overlapping 20-second windows with a 1-second stride, indexed by 64-bit frame hashes, and searched for each benchmark video’s top 10 candidates.Videos are uniformly sampled at 1 fps, and candidate windows are verified using aligned-frame Hamming distance.Source: §A.5 · p. 17

B. VideoKR-Eval Benchmark · B.1. Detailed Statistics of VideoKR-Eval · C. Experiment Setup

VideoKR-Eval combines retained examples from three source benchmarks with expert-reannotated, visually grounded questions to create a 2,000-example benchmark for genuine video understanding and knowledge-intensive reasoning.

B.1. Detailed Statistics of VideoKR-Eval

  • VideoKR-Eval draws on VideoMMMU, MMVU, and SciVideoBench.Source: §B.1 · p. 17
  • Original examples were retained only when all three single-frame probing models judged them to require continuous video understanding.Source: §B.1 · p. 17, Table 9 · p. 17
  • The unanimous probing filter retained 1,254 original examples.Source: §B.1 · p. 17
  • Domain experts re-annotated filtered videos with 746 additional visually grounded question-answer examples.Source: §B.1 · p. 17, Table 9 · p. 17
  • The resulting benchmark contains 2,000 high-quality examples.Source: §B.1 · p. 17
  • Its examples are designed to require genuine video-level understanding and knowledge-intensive reasoning.Source: §B.1 · p. 17

C.1. Post-Training Details … D.1. Performance with Different Frames

The appendices specify GRPO rewards and training settings, standardize evaluation inference while varying prompts across models and benchmarks, and report a frame-count comparison table. The supplied passages describe the table’s scope but do not provide its results.

C.1. Post-Training Details

  • GRPO combines format and accuracy rewards at weights 0.1 and 0.9, respectively, using ROUGE for open-ended QA and exact match for multiple-choice tasks.The format reward is 1.0 when output follows the required <think>...</think><answer>...</answer> structure.Source: §C.1 · p. 18
  • Training uses AdamW, learning rates of 1 × 10^-5 for SFT and 5 × 10^-6 for RL, and GRPO rollouts of size 8 with temperature 1.0.Training uses up to 8 NVIDIA A800 GPUs, a 2,048-token maximum response length, and KL coefficient β = 0.01; SFT uses LLaMA-Factory and RL uses verl.Source: §C.1 · p. 18

C.2. Evaluation Setup

  • Evaluation uses temperature 0.1 and a maximum response length of 8,192 tokens for standardized inference.The stated configuration is intended to ensure fair and reproducible comparisons.Source: §C.2 · p. 18
  • Evaluation prompts vary by model and benchmark, ranging from direct multiple-choice answers to step-by-step reasoning that identifies visual content and timestamps.The VideoKR and OneThinker prompt requests reasoning and an answer in tags, while VideoAuto-R1’s Video-MME prompt requests only an option letter.Source: §C.2 · pp. 18–19
  • VideoKR-Eval prompts ask thinking models to reason step by step, while instruct and other models provide direct choice letters or single-word or phrase answers.The open-ended thinking-model prompt also asks for a step-by-step answer.Source: §C.2 · p. 21

D.1. Performance with Different Frames

  • Table 10 presents detailed accuracy on general and knowledge-intensive video reasoning benchmarks for post-trained models across different input-frame counts.The supplied passage gives the table’s scope but no accuracy values or comparative outcomes.Source: Table 10 · p. 22

D.2. Ablation Studies · D.3. Case Study

D.2 describes post-training-data ablations using a fixed base model and frame count; D.3 presents model-response comparisons on a knowledge-intensive video reasoning sample.

D.2. Ablation Studies

  • All Table 11 post-training-data ablations use Qwen2.5-VL-7B-Instruct as the base model with 128 input frames.Source: Table 11 · p. 23

D.3. Case Study

Related papers

Loading 2606.05259v1…