PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding

Lin Fu, Zheyuan Yang, Yang Wang, Tingyu Song, Arman Cohan, Yilun Zhao

arXiv:2606.05259v1cs.CV

TL;DR

Knowledge-intensive video reasoning remains difficult because existing models struggle with domain knowledge and multi-step inference. VideoKR addresses this gap with a quality-controlled, skill-oriented corpus and benchmark, and models trained on it achieve leading knowledge-intensive performance while remaining competitive on general video benchmarks.

  • Problem

    Current video models remain limited in domain knowledge and multi-step inference, while existing benchmarks often permit answers without continuous video understanding.

  • Method

    VideoKR combines 145K CC-licensed videos with human-audited, skill-oriented reasoning examples and VideoKR-Eval, an expert-annotated benchmark designed to require genuine video understanding.

  • Results

    Models post-trained on VideoKR achieve the best knowledge-intensive performance among similar-sized models while remaining competitive on general video benchmarks.

  • Takeaways & Limitations

    The results support data design as a fundamental driver of knowledge-intensive video reasoning progress without sophisticated reinforcement-learning reward engineering.

  • Takeaways & Limitations

    Existing knowledge-intensive video benchmarks contain substantial single-frame-answerable examples, limiting their measurement of continuous video understanding.

Abstract

from arXiv · show

We introduce VideoKR, the first large-scale training corpus specifically designed to strengthen knowledge- and reasoning-intensive video understanding. It comprises 315K video reasoning examples over 145K newly collected, CC-licensed, expert-domain videos. We develop a human-in-the-loop, skill-oriented example generation pipeline that targets progressively deeper video reasoning capabilities while ensuring the difficulty, diversity, and reliability of both the examples and their CoT rationales. We also curate VideoKR-Eval, a new expert-annotated benchmark where questions require genuine video understanding and knowledge-intensive reasoning rather than textual shortcuts. Our experiments show that, under a standard SFT$\rightarrow$GRPO pipeline, models post-trained on VideoKR outperform prior post-training approaches on knowledge-intensive video reasoning while remaining competitive on general video reasoning, highlighting data design as a key driver of progress in video reasoning. We further conduct comprehensive ablations to isolate the contributions of VideoKR, providing actionable insights for future work.

1. Introduction

VideoKR addresses the lack of specialized, reasoning-oriented video data by providing a large-scale corpus built from professional-domain videos and skill-oriented QA generation. Its evaluation and experiments show improved knowledge-intensive video reasoning, competitive general performance, and actionable findings from ablations.

  • Data Generation: VideoKR converts videos into skill-oriented reasoning data, with CoT examples additionally paired with high-quality reasoning traces.The pipeline grounds every example in one of three core skills essential for advanced video reasoning.
  • Motivation: Current video datasets emphasize perceptual objectives and everyday activities, offering limited specialized-domain coverage for knowledge- and reasoning-intensive understanding.This mismatch is identified as a key limitation in transitioning from surface-level perception to domain knowledge and multi-step inference.
  • VideoKR Corpus: 145K CC-licensed videos across 82 professional subjects form VideoKR, the first large-scale corpus targeted at knowledge- and reasoning-intensive video understanding.Videos are collected through a knowledge-driven protocol targeting real-world manifestations of domain knowledge.
  • Evaluation: VideoKR-Eval mitigates single-frame answerability through multi-model probing and expert re-annotation of filtered videos, while standardized evaluation enables fair and reproducible comparisons.These components are designed to assess genuine video understanding rather than textual or single-frame shortcuts.
  • Results and Analysis: Models post-trained on VideoKR achieve the best knowledge-intensive performance among similar-sized models while remaining competitive on general video benchmarks.Experiments use a standard SFT→GRPO pipeline, and ablations examine CoT supervision, skill-based composition, prior corpora, and controlled SFT and RL studies.

2. Related Work

Prior video benchmarks broaden evaluation from multimodal perception to spatiotemporal and cross-modal reasoning, while newer benchmarks target knowledge-, science-, and domain-aware reasoning. Video reasoning post-training commonly uses SFT followed by RL, but existing corpora often repurpose older short-video datasets and single-model synthesis, motivating VideoKR’s newly collected, expert-domain, CC-licensed data and human-in-the-loop generation.

  • Video Understanding Datasets: General-purpose benchmarks evaluate perceptual skills, spatiotemporal comprehension, and cross-modal reasoning in video understanding.Examples include Video-MME, MVBench, VSI-Bench, and VideoVista.
  • Video Understanding Datasets: Newer evaluation benchmarks extend video understanding toward knowledge-, science-, and reasoning-intensive, domain-aware capabilities beyond surface-level perception.The supplied passage identifies MMVU as one example of this trend, although its description is truncated.
  • Post-training for Video Understanding: Current video reasoning models typically use a two-stage post-training pipeline that combines supervised fine-tuning and reinforcement learning.The SFT stage is commonly initialized with video reasoning datasets containing explicit chain-of-thought annotations.
  • Post-training for Video Understanding: Existing post-training approaches largely repurpose video understanding datasets designed for basic perception.Table 1 indicates that open-source corpora largely rely on short videos from datasets released years ago.
  • Post-training for Video Understanding: Synthesis-based efforts typically depend on a single model, which can introduce systematic biases.VideoKR addresses this gap with newly collected expert-domain videos released under CC licenses and a human-in-the-loop, skill-oriented example-generation framework.

3. VideoKR Training Corpus Construction

VideoKR uses a quality-controlled, semi-automated pipeline that combines expert-curated domain knowledge, scenario-guided CC-video collection, skill-based QA generation, and human validation. The resulting 315,537 examples are partitioned into supervised fine-tuning and RLVR subsets with different supervision formats.

  • Pipeline Overview: 34 domain experts support a semi-automated construction pipeline in which model outputs are audited and validated by human experts.The pipeline addresses the infeasibility of exhaustive manual construction while mitigating systematic artifacts from model-based generation.
  • Domain Knowledge Bank: The Domain Knowledge Bank covers 82 subjects across four disciplines using a Subject → Course → Lecture → Knowledge Point hierarchy.Expert annotators provide 4 to 8 core undergraduate courses for each subject.
  • Video Corpus Collection: Scenario-guided search and filtering collects 146,567 CC-licensed videos after metadata screening and safety checks based on sampled frames.LLM-generated realistic scenarios are used to seek videos where domain knowledge is implicitly applied rather than relying only on knowledge-point terms.
  • Skill-Based Example Generation: The example-generation pipeline defines three complementary skills—perception, knowledge, and reasoning—and creates multi-choice and open-ended QA with verifiable supervision.Human experts curate seed examples with step-by-step reasoning, while frontier MLLMs generate two examples per skill for each video.
  • Example Validation: Self-consistency verification and video-dependency filtering reduce generation errors and reasoning bias before examples are retained.The verification-derived reasoning trace becomes the final trace when the independently re-derived answer matches the original.
  • Dataset Partitioning: 315,537 generated examples are split by video-level grouping into VideoKR-SFT-201K and VideoKR-RL-114K, retaining validated CoT for SFT but only questions and verifiable answers for RLVR.The differing formats reflect supervised fine-tuning’s use of CoT rationales versus RLVR’s optimization against verifiable answers.

4. VideoKR-Eval Evaluation Benchmark

VideoKR-Eval addresses shortcut-driven evaluation in existing knowledge-intensive video reasoning benchmarks by filtering for continuous video understanding and adding expert-reannotated questions grounded in observable evidence and domain knowledge. The resulting benchmark contains 2,000 examples.

  • Motivation: Existing benchmarks contain many examples solvable without continuous video understanding, as shown by single-frame probing with questions, options, and one sampled frame.Each example is evaluated in three independent trials.
  • Construction: 1,254 original examples are retained because they require continuous video understanding under multi-model single-frame probing.The retained examples come from VideoMMMU, MMVU, and SciVideoBench.
  • Construction: 2,000 examples comprise VideoKR-Eval, combining 1,254 retained original examples with 746 expert-reannotated examples.Detailed benchmark statistics are provided in Appendix B.1.
  • Construction: VideoKR-Eval applies multi-model single-frame filtering to VideoMMMU, MMVU, and SciVideoBench using Qwen3-VL-235B-A22B, Claude-4.5-Sonnet, and GPT-5.2.An example is single-frame-solvable for a model when it is answered correctly in all three trials.
  • Construction: 746 expert-reannotated examples replace discarded original QA pairs from filtered videos.Experts write questions grounded in clearly observable video evidence, requiring relevant domain knowledge and uniquely determined ground-truth answers.

5. Experiment Setup

The experiments use a controlled SFT→GRPO post-training scaffold to isolate data design, then evaluate models across general and knowledge-intensive video reasoning benchmarks. Evaluation is standardized around appropriate prompts, model-recommended frame settings, repeated sampling, and LMMs-Eval.

  • Post-training setup: The study deliberately adopts a standard SFT→GRPO pipeline so algorithmic complexity does not confound analysis of data design.The pipeline is used as a controlled scaffold for studying knowledge- and reasoning-intensive video understanding.
  • Post-training setup: Experiments use Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct, fine-tuning on VideoKR-SFT-201K for one epoch and then applying GRPO on VideoKR-RL-114K for one epoch.For Qwen3-VL-8B-Instruct, Zero-RL additionally runs GRPO directly on VideoKR-RL-114K for one epoch.
  • Evaluation benchmarks: Evaluation covers seven benchmarks spanning general video reasoning and knowledge-intensive video reasoning.General benchmarks include Video-MME, MVBench, and LongVideoBench; knowledge-intensive benchmarks include VideoMMMU, MMVU, SciVideoBench, and VideoKR-Eval.
  • Evaluation protocol: Prior post-training results show substantial cross-paper inconsistencies, primarily attributed to prompt misalignment between base models and their intended inference modes.The setup highlights that Qwen2.5-VL-Instruct is not a reasoning model, yet some studies evaluate it under mismatched prompt conditions.
  • Evaluation protocol: For fair comparisons, evaluations use official paper prompts when available, otherwise LMMs-Eval defaults, model-recommended frame settings, three independent sampling runs, and mean scores.All evaluations are performed with the LMMs-Eval framework.

6. Experiment Results and Analysis

VideoKR post-training improves knowledge-intensive video reasoning, with gains from integrated skill supervision and explicit CoT rationales. Ablations and analyses show that stronger performance is associated with challenging data, richer visual evidence, and domain-informed reasoning.

  • Main Results: 51.5 is the knowledge-intensive average achieved by post-trained Qwen3-VL-8B, exceeding the strongest competing Qwen3-VL-8B-Thinking model at 50.0.VideoKR raises Qwen2.5-VL-7B from 41.9 to 46.6 (+4.7) and Qwen3-VL-8B from 48.5 to 51.5 (+3.0); Qwen2.5-VL-7B also gains +4.8 and +8.5 points on MMVU and VideoKR-Eval.
  • Frame Scaling: 60.1% to 65.5% is the general-reasoning improvement for Qwen2.5-VL-7B (SFT+RL) when inference frames increase from 16 to 128.Knowledge-intensive performance also rises from 44.2% to 46.6%, while all other inference settings remain fixed.
  • Skill-Oriented Data Composition: 42.4% is the knowledge-intensive performance from combining VIDR, KNOWVID, and KNOWVIDR, compared with 41.4% for VIDR alone and 41.3% for VIDR plus KNOWVID.VideoKR-Eval accuracy rises monotonically from 35.3 to 35.9 to 36.8 across the same cumulative skill variants.
  • CoT vs. Direct Output: 3.0-point gain: CoT supervision lifts knowledge-intensive average performance from 39.4% for Direct Output to 42.4%.The ablation uses 80K sampled VideoKR-SFT-201K examples and otherwise matches the one-epoch, batch-size-16 fine-tuning setup.
  • Comparison with Prior Corpora: 43.0 is VideoKR’s strongest zero-RL knowledge-intensive average, +1.1 over the base model and ahead of VideoAuto-R1 at 42.7.Under SFT, VideoKR reaches 42.4 and is the only evaluated corpus above the 41.9 base model; Video-R1 and VideoRFT reach 36.2 and 38.4.
  • Training-Data Difficulty Analysis: 42.3% is Qwen3-VL-8B’s accuracy on VideoKR examples, versus 49.1%–57.1% on prior corpora, indicating a more challenging distribution and stronger learning signal.The analysis evaluates 3,000 randomly sampled QA examples with 128 frames and interprets high prior-corpus accuracy as saturation for current frontier base models.

7. Conclusion

The work presents a corpus-centric approach to post-training foundation models for advanced video reasoning. It shows that combining structured domain concepts with visually grounded examples improves reasoning performance without sophisticated reinforcement-learning reward engineering.

  • Conclusion: VideoKR frames advanced video-reasoning post-training around a corpus-centric perspective on foundation models.The approach treats data design as central to post-training rather than focusing solely on reward engineering.
  • Conclusion: Integrating structured domain concepts with visually grounded examples yields stronger reasoning performance.This integration connects visual perception, domain knowledge, and advanced reasoning within the training data.
  • Conclusion: The gains do not rely on sophisticated reinforcement-learning reward engineering.The conclusion attributes the stronger reasoning performance to the integration of structured concepts and visually grounded examples.

Impact Statement … A.2. Annotator Information

VideoKR emphasizes legally reusable, clearly sourced data and broad cross-domain knowledge coverage, supported by structured construction procedures and documented annotator participation. Its corpus uses CC-licensed videos, an 82-subject knowledge bank, and 34 annotators spanning multiple construction and evaluation roles.

  • Impact Statement: CC licensing enables redistribution, remixing, adaptation, and reuse with attribution while clarifying legal provenance.The dataset restricts videos to CC-licensed material to address provenance ambiguity in prior training corpora.
  • Impact Statement: Human expert annotators were compensated at an average rate of $13 USD per hour.The supplied passage states that this rate exceeds th…
  • A.1. Domain Knowledge Bank Construction: Table 6 organizes the knowledge bank across four major disciplines and their subfields.The table presents the complete subject list by major disciplines.
  • A. VideoKR Data Construction: VideoKR’s construction pipeline draws on 82 representative subjects spanning Natural Sciences, Healthcare, Humanities and Social Sciences, and Engineering.The subjects were identified through manual review of undergraduate curricula from leading universities worldwide.
  • A.1. Domain Knowledge Bank Construction: The 82-subject index supports broad cross-domain coverage and balanced sampling in a four-layer knowledge base.The layers are subject, course, lecture, and knowledge point.
  • A.2. Annotator Information: Table 7 documents the biographies of 34 annotators involved in VideoKR construction.Their roles cover Domain Knowledge Bank Construction, Seed Example Curation, Human-Validated Model Selection, Manual Quality Assessment, and VideoKR-Eval Construction.

A.3. VideoKR Data Example · A.4. Human-Validated Model Selection Protocol · A.5. Data Contamination Mitigation

The appendices illustrate VideoKR’s diverse, reasoning-intensive examples, describe human-validated model selection across synthesis stages, and outline near-duplicate filtering for contamination mitigation.

  • A.3. VideoKR Data Example: VideoKR examples require extracting visual evidence and applying domain knowledge, such as identifying a calculator intercept, computing Mg(OH)₂ Ksp, or interpreting software interfaces.The examples span natural science, healthcare, and engineering domains, with concise reasoning processes presented for readability.
  • A.3. VideoKR Data Example: 17.29% is computed from heap-analysis counts of 585 GeneralPath instances among 3384 new objects, corresponding to answer choice B.The calculation uses (585 ÷ 3384) × 100 and rounds to two decimal places.
  • A.3. VideoKR Data Example: Five helmeted figures are counted in the warehouse illustration, making answer choice D correct.The figures appear in a row under “Step 1: Define Problem.”
  • A.3. VideoKR Data Example: The examples also include quantitative aggregation and historical interpretation, including an average laptop inbound volume of 164 and classification as militant protest.These questions combine information extracted from video displays with arithmetic or contextual reasoning.
  • A.4. Human-Validated Model Selection Protocol: The qualification protocol maintains a dynamic pool of eligible foundation models rather than relying on a single model, reducing exposure to model-specific biases.Model assignments are determined for each synthesis stage after human-in-the-loop qualification.
  • A.4. Human-Validated Model Selection Protocol: Seven frontier models are evaluated for each pipeline step through controlled pilots sampling 100 representative instances, with experts separating hard compliance from soft content failures.The evaluated models are GPT-5.2, GPT-5-mini, Claude-4.5-Sonnet, Gemini-3-Flash, DeepSeek-V3.2, Qwen3-VL-235B-A22B, and GLM-4.6V.
  • A.5. Data Contamination Mitigation: Near-duplicate filtering samples benchmark and training videos at 1 fps, hashes frames with 64-bit perceptual hashes, and indexes overlapping 20-second windows with a 1-second stride.For each benchmark video, the system retrieves the top-10 candidate training windows and verifies overlap using aligned-frame Hamming distance.

B. VideoKR-Eval Benchmark · B.1. Detailed Statistics of VideoKR-Eval · C. Experiment Setup

VideoKR-Eval is constructed from three source benchmarks by filtering for examples requiring continuous video understanding and adding expert-reannotated, visually grounded questions. This process yields 2,000 benchmark examples.

  • B. VideoKR-Eval Benchmark: VideoKR-Eval draws from VideoMMMU, MMVU, and SciVideoBench.
  • B.1. Detailed Statistics of VideoKR-Eval: All three single-frame probing models had to judge retained original examples as requiring continuous video understanding.
  • B.1. Detailed Statistics of VideoKR-Eval: 1,254 original examples were retained after multi-model single-frame probing.
  • B.1. Detailed Statistics of VideoKR-Eval: 746 additional visually grounded QA examples were contributed through domain-expert re-annotation of the remaining filtered videos.
  • B.1. Detailed Statistics of VideoKR-Eval: The final VideoKR-Eval benchmark comprises 2,000 examples.

C.1. Post-Training Details … D.1. Performance with Different Frames

The paper specifies GRPO-based post-training and standardized evaluation prompts, then reports benchmark accuracy across general and knowledge-intensive video reasoning settings, including different frame configurations.

  • C.1. Post-Training Details: SFT uses a 1 × 10−5 learning rate, whereas RL uses 5 × 10−6, with AdamW optimization and 2,048-token maximum responses.GRPO uses rollout size G = 8, temperature 1.0, and KL penalty coefficient β = 0.01 on up to 8 NVIDIA A800 GPUs.
  • C.2. Evaluation Setup: Inference is standardized across evaluations with temperature 0.1 and a maximum response length of 8,192 tokens.Prompt formats vary by benchmark and model family, including reasoning-required, direct-answer, multiple-choice, and open-ended configurations.
  • C.2. Evaluation Setup: The evaluation setup covers VideoKR, OneThinker, Video-Auto-R1, Video-R1, VideoRFT, Qwen variants, and other models through benchmark-specific prompts.Prompting distinguishes thinking from instruct models and adapts answer formatting across Video-MME, MVBench, LongVideoBench, VideoMMMU, MMVU, and SciVideoBench.
  • C.2. Evaluation Setup: VideoKR-Eval prompts require step-by-step reasoning for designated thinking models, while instruct and other models answer directly.The benchmark includes separate multiple-choice and open-ended prompt formats for these model groups.
  • D. Experiment Results and Analysis: The experiments evaluate post-trained models on general and knowledge-intensive video reasoning benchmarks.Table 10 reports detailed accuracy across models under different frame settings.
  • D.1. Performance with Different Frames: Table 10 presents detailed accuracy for post-trained models across different frame configurations.The table covers both general and knowledge-intensive video reasoning benchmarks.

D.2. Ablation Studies · D.3. Case Study

The ablation studies evaluate post-training data using Qwen2.5-VL-7B-Instruct with 128 input frames, while the case study compares model responses on knowledge-intensive video reasoning samples. Three figures provide separate response comparisons for such samples.

  • D.2. Ablation Studies: The ablation setup fixes Qwen2.5-VL-7B-Instruct as the base model.This condition is stated in the Table 11 caption.
  • D.2. Ablation Studies: The ablation experiments use 128 input frames.The frame count is specified as a shared experimental condition.
  • D.3. Case Study: D.3 compares model responses on a knowledge-intensive video reasoning sample.Figure 9 presents one such comparison.
  • D.3. Case Study: A second case-study figure compares model responses on a knowledge-intensive video reasoning sample.This comparison appears in Figure 10.
  • D.3. Case Study: A third case-study figure compares model responses on a knowledge-intensive video reasoning sample.This comparison appears in Figure 11.
Loading 2606.05259v1…