PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenCoF: Learning to Reason Through Video Generation

Xinyan Chen, Ziyu Guo, Renrui Zhang, Dongzhi Jiang, Hongsheng Li

arXiv:2607.08763v1cs.CVcs.AI

TL;DR

Existing video generators lack dedicated, diverse temporal supervision and mechanisms for organizing intermediate states in Chain-of-Frame reasoning. OpenCoF introduces a broad reasoning-video dataset, fine-tunes Wan-CoF, and explores visual and textual reasoning tokens, yielding substantial gains across four external benchmarks.

  • Problem

    Video reasoning research lacks diverse temporal supervision and dedicated designs for modeling dynamic transitions and multi-step visual consequences.

  • Method

    OpenCoF builds a 17,312-video dataset across 11 task families, fine-tunes Wan-CoF, and explores visual and textual reasoning tokens for organizing intermediate reasoning states.

  • Results

    Wan-CoF shows substantial gains over the base model across four external video-reasoning benchmarks, while both reasoning-token variants further improve performance.

  • Takeaways & Limitations

    The findings support combining broad temporal supervision with explicit mechanisms for organizing intermediate reasoning states in video reasoning.

  • Takeaways & Limitations

    Visual and textual reasoning tokens are investigated separately, leaving their effective combination for future work.

Abstract

from arXiv · show

Reasoning has become a core capability for large models, especially when reliable decisions require understanding logical consequences. Recent video generation models offer a reasoning path distinct from previous Chain-of-Thought (CoT): reasoning can unfold through temporally connected frames, known as Chain-of-Frame (CoF) reasoning. However, existing video generators are primarily trained on general video corpora, still lacking diverse supervision and dedicated designs for CoF reasoning. To address this gap, we introduce OpenCoF, a framework comprising the OpenCoF-17K dataset, a reasoning video dataset spanning 11 task families, and Wan-CoF, a fine-tuned video model for studying whether diverse temporal supervision improves CoF behavior. Across four video reasoning benchmarks, Wan-CoF achieves considerable gains over the Wan2.2-I2V-A14B baseline. Building on this, we empirically explore more advanced designs for CoF capabilities, i.e., equipping the model with visual and textual reasoning tokens. This mechanism respectively captures low-level visual cues and high-level semantic priors for spatial and temporal reasoning. Through performance comparisons and attention analysis, we examine how these tokens contribute across model depth, denoising steps, space, and time. Our results suggest that stronger video reasoning requires both broad temporal supervision and explicit mechanisms for organizing intermediate reasoning state. We open-source the dataset, model, and code to facilitate future research on reasoning-oriented video generation.

1 Introduction

OpenCoF addresses the limited reasoning capability of video generators by providing broad temporal supervision through a 17,312-video dataset and a fine-tuned model. It further explores visual and textual reasoning tokens for organizing intermediate states in Chain-of-Frame reasoning.

  • Motivation: Static visual Chain-of-Thought pipelines remain limited, while video models still struggle with long-term temporal coherence, physical and spatial consistency, and logical continuity.Video reasoning uses temporally evolving frames, formalized as Chain-of-Frame reasoning.
  • Dataset: OpenCoF-17K contains 17,312 videos across 11 task families, curated through four complementary approaches for diverse temporal supervision.The approaches are instance-based rendering, expert-guided rendering, procedural scene synthesis, and repurposed existing videos.
  • Model and evaluation: Fine-tuning Wan2.2-I2V-A14B on OpenCoF-17K produces Wan-CoF, which achieves substantial gains over the base model across four external video-reasoning benchmarks.The benchmarks are MME-CoF, Gen-ViRe, VIPER, and RULER-Bench.
  • Reasoning-token mechanisms: OpenCoF explores Visual Reasoning Tokens and Textual Reasoning Tokens as complementary mechanisms for organizing intermediate reasoning states in video generation.Visual tokens are inserted into the visual latent sequence, while textual tokens operate at a different model level to capture complementary reasoning information.

2 OpenCoF-17K

OpenCoF-17K is a 17,312-sample video reasoning dataset spanning 11 task families, designed to provide dedicated temporal supervision for Chain-of-Frame capabilities. It unifies procedurally rendered, expert-guided, synthesized, and repurposed videos in a standardized conditional-generation format.

  • Dataset overview: The dataset addresses video generators’ lack of dedicated temporal reasoning supervision by assembling diverse reasoning-centric data for robust CoF capabilities.The collection is explicitly structured around video reasoning rather than general video generation.
  • Dataset overview: OpenCoF-17K contains 17,312 samples across 11 task families, pairing a conditioning image and textual prompt with a target reasoning video.All videos use 480p resolution, 15 fps, and 81 frames.
  • Curation pipelines: Its four curation pipelines are instance-based rendering, expert-guided rendering, procedural scene synthesis, and external video repurposing.The first three construct data generatively, while the fourth integrates high-quality demonstrations from existing datasets.
  • Curation pipelines: Structured assets such as puzzles and board configurations are sampled and deterministically translated into videos, including legal-checkmate chess moves and step-by-step 4 × 4 Sudoku solving.These pipelines provide rule-based and algorithmic reasoning supervision through sequential visual execution.
  • Curation pipelines: Expert-guided rendering uses human expertise, LLM-generated concepts, or T2I outputs to define structures before code converts intermediate structures into sequential videos.Examples include auxiliary-line construction in 2D geometry and dot-to-dot animation generated from minimalist line drawings.
  • Curation pipelines: Procedural synthesis and external repurposing broaden coverage through physics- or graphics-rendered scenes and standardized demonstrations from existing video datasets.Tasks include tangram assembly, cube folding, polycube rotation, physics motion, maze navigation, VBVR, and embodied manipulation.

3 Training of Wan-CoF

Wan-CoF is obtained by LoRA fine-tuning Wan2.2-I2V-A14B on the diverse OpenCoF-17K supervision and improves headline metrics across four video reasoning benchmarks. Its gains, including transfer to independent externally curated benchmarks, suggest broader CoF reasoning rather than task-domain overfitting.

  • Training setup: Wan-CoF is created by directly fine-tuning the general-domain pretrained Wan2.2-I2V-A14B with LoRA on OpenCoF-17K.Evaluation on four video reasoning benchmarks separates the effect of OpenCoF-17K from later reasoning-token designs.
  • Comparison with the baseline: 1.30 vs 1.00 on MME-CoF Overall, 0.391 vs 0.304 on Gen-ViRe average, 7.5 vs 3.3 on VIPER POC, and 56.8 vs 55.8 on RULER-Bench overall average.These comparisons are against the Wan2.2-I2V-A14B baseline and correspond to gains of +0.30, +0.087, +4.2, and +1.0, respectively.
  • Comparison with other models: Wan-CoF exceeds HunyuanVideo-I2V and Wan2.2-TI2V-5B on MME-CoF, reaches 0.391 Gen-ViRe average, and attains 7.5 VIPER POC.It outperforms Kling-v1 and Seedance-1.0-Pro on Gen-ViRe, while its VIPER POC is on par with Hunyuan-1.5 and Seedance-1.5.
  • Comparison with other models: Under a LoRA fine-tuning budget on a single open backbone, Wan-CoF ranks among the strongest open-source I2V models on CoF reasoning.The comparison positions it relative to open-source and closed-source competitors.
  • Generalization: Testing on independent, externally curated benchmarks suggests that OpenCoF-17K fosters transferable CoF reasoning skills across suite boundaries rather than overfitting to specific training domains.The external benchmarks have distributions distinct from the training set.

4 Exploration of CoF Techniques

The section introduces visual and textual reasoning tokens as complementary mechanisms for organizing intermediate CoF state, instantiated in Wan2.2-I2V-A14B. Fine-tuned variants improve benchmark performance, while attention analyses reveal distinct roles across depth, denoising, space, and time.

  • Visual Reasoning Tokens: Visual reasoning tokens are learnable tokens prepended to the visual sequence, enabling bidirectional self-attention that aggregates global context and redistributes features to visual tokens.They are discarded after the final DiT block, and predictions are read from the visual-token portion.
  • Textual Reasoning Tokens: Textual reasoning tokens are prepended to text conditioning and provide prompt-independent task-level context through cross-attention without being refreshed by visual tokens.They remain shared across spatial patches and temporal frames, complementing visual tokens’ spatially grounded channel.
  • Benchmark Comparison: Wan-CoFvt reaches the highest aggregate score on Gen-ViRe and RULER-Bench, whereas Wan-CoFtt leads on MME-CoF and VIPER.Both variants improve over Wan-CoF on the headline metric of every benchmark; vt specializes in persistent-plan dimensions, while tt leads elsewhere.
  • Attention Analysis: Both token types receive stronger attention early in denoising, while vt is especially selective across depth and can remain active into later blocks in the high-noise model.The patterns suggest reasoning tokens participate in specific computation stages rather than acting as constant global biases.
  • Attention Analysis: Spatial attention shows vt forming sparse axis-aligned bands near task-relevant objects, while tt attends more broadly as a high-level semantic prior.The reported visualization concerns an embodied manipulation prompt involving a robot arm, table, and beans.
  • Attention Analysis: Temporally, vt is more visible at sequence boundaries, whereas tt supplies a steadier semantic constraint over the subsequent reasoning process.This indicates complementary temporal emphases rather than simple object tracking.

5 Related Work

Related work spans scalable video-generation architectures, multimodal reasoning paradigms, CoF benchmarks, and early reasoning-focused training efforts. OpenCoF positions itself around dedicated temporal supervision and reasoning-token exploration to address limitations in prior approaches.

  • Video generation models: Video generation has progressed from U-Net diffusion to DiT and flow-matching architectures, enabling scalable training and higher-resolution synthesis.HunyuanVideo, HunyuanVideo-1.5, and Wan provide open backbones for image-to-video and text-to-video generation.
  • Multimodal reasoning: Conventional multimodal chain-of-thought uses text, static visual observations, external tools, or auxiliary image generation, whereas CoF reasoning unfolds through temporally connected frames.This establishes video generation as an alternative reasoning path for multimodal models.
  • CoF benchmarks: Benchmarks including MME-CoF, VideoThinkBench, V-ReasonBench, Gen-ViRe, RULER-Bench, MMGR, TiViBench, and VIPER evaluate CoF reasoning from different perspectives.OpenCoF builds on these benchmarks while focusing on the OpenCoF-17K dataset and reasoning-token exploration.
  • Reasoning-focused training: Prior reasoning-training efforts study maze-solving, tangram puzzles, test-time scaling, or cognitive task suites, but are often domain-specific or coupled between training and evaluation.These efforts suggest targeted supervision improves video reasoning; OpenCoF-17K instead combines four data sources, though the supplied passage truncates before listing them.

6 Conclusion and Limitations

The paper introduces OpenCoF-17K and Wan-CoF to improve Chain-of-Frame reasoning through diverse temporal supervision, while analyzing visual and textual reasoning tokens. It also identifies combining these token types as future work.

  • OpenCoF-17K is a diverse video reasoning dataset designed to improve Chain-of-Frame reasoning in video generation models.
  • Fine-tuning Wan2.2-I2V-A14B on OpenCoF-17K produces Wan-CoF, whose diverse temporal supervision improves CoF behavior across four external benchmarks.
  • The study explores Visual and Textual Reasoning Tokens and analyzes their organization of reasoning state through benchmark results and attention patterns.
  • Combining visual and textual reasoning tokens remains future work because they were investigated separately.

Appendix · Overview

The supplementary material is organized around additional implementation details and an additional ablation study, including backbone/LoRA setup and reasoning-token-count ablation.

  • Overview: The supplementary material is presented as an organized sequence of additional sections and subsections.The overview explicitly introduces the organization of the supplementary material.
  • Overview: The supplement includes a section on additional implementation details.This section is listed first in the supplementary-material overview.
  • Overview: The overview separates implementation details from ablation analysis.The two top-level entries are additional implementation details and additional ablation study.
  • Overview: The implementation-details section covers the backbone and LoRA setup.Backbone and LoRA setup is identified as a subsection of the implementation details.
  • Overview: The supplement also includes an additional ablation study.This study follows the implementation-details section in the overview.
  • Overview: The ablation study includes reasoning-token-count ablation.Reasoning token count ablation is listed as a subsection of the additional ablation study.

A Additional Implementation Details · A.1 Backbone and LoRA setup. · A.2 Model variants.

The implementation uses the Wan2.2-I2V-A14B backbone with LoRA applied to the DiT denoiser, while the text encoder and VAE remain frozen. Three variants isolate dataset training from visual- and textual-reasoning-token experiments, using different training durations and Nr = Nt = 16 for the token models.

  • A.1 Backbone and LoRA setup.: Wan2.2-I2V-A14B uses a Mixture-of-Experts denoising architecture with separate high-noise and low-noise experts across diffusion timesteps.
  • A.1 Backbone and LoRA setup.: The backbone combines an umT5 text encoder, a Wan-VAE for visual encoding and decoding, and a mainstream DiT denoiser.
  • A.1 Backbone and LoRA setup.: Wan-CoF, Wan-CoFvt, and Wan-CoFtt are all fine-tuned from the same Wan2.2-I2V-A14B backbone using LoRA.
  • A.1 Backbone and LoRA setup.: LoRA is applied only to the DiT denoiser, keeping the umT5 text encoder and Wan-VAE frozen.
  • A.1 Backbone and LoRA setup.: Within each DiT block, LoRA targets the q, k, v, and o attention projections together with the ffn.0 and ffn.2 MLP linear layers.
  • A.2 Model variants.: Wan-CoF keeps the architecture unchanged and is fine-tuned on OpenCoF-17K to attribute gains to the dataset alone.
  • A.2 Model variants.: Wan-CoFvt and Wan-CoFtt are independently fine-tuned with LoRA using only vt or only tt, respectively, with Nr = Nt = 16.
  • A.2 Model variants.: Wan-CoF is trained for 2 epochs, whereas Wan-CoFvt and Wan-CoFtt are trained for 5 epochs.

A.3 Compared systems.

Wan-CoF is compared against a broad set of closed-source and open-source video generation models, including multiple state-of-the-art commercial systems and HunyuanVideo variants.

  • Closed-source systems: The comparison covers closed-source systems including Kling-v1, Seedance-1.0-Pro, Seedance-1.5, Veo-3 variants, and Sora-2.The Veo-3 series includes Veo-3.0-Fast, Veo-3.0-Preview, Veo-3.0, and Veo-3.1.
  • Open-source systems: The open-source comparison set includes HunyuanVideo-I2V and HunyuanVideo-1.5.These models are included alongside Wan-CoF in the open-source comparison set.

A.4 Evaluation protocol.

The evaluation uses four video reasoning benchmarks spanning categories beyond OpenCoF-17K’s 11 task types, with RULER-Bench restricted to image-to-video cases and official judge models applied throughout.

  • Benchmark suite: Models are evaluated on four benchmarks: MME-CoF, RULER-Bench, VIPER, and Gen-ViRe.These benchmarks collectively cover reasoning categories beyond the 11 task types in OpenCoF-17K.
  • Evaluation conditions: RULER-Bench evaluation is restricted to image-to-video cases because the baseline does not support text-to-video generation.The restriction applies specifically to RULER-Bench; all benchmarks use their official judge model.
  • Evaluation conditions: Every benchmark is evaluated using its official judge model.This protocol standardizes judging across MME-CoF, RULER-Bench, VIPER, and Gen-ViRe.

B Additional Ablation Study

The ablation retrains Wan-CoFvt and Wan-CoFtt with n = 32 reasoning tokens instead of the Section 4 setting n = 16, while keeping other settings identical. Doubling the token count does not uniformly improve either variant: n = 16 outperforms n = 32 on three of four benchmarks for both variants.

  • Reasoning token count ablation: The study retrains Wan-CoFvt and Wan-CoFtt with n = 32 reasoning tokens under otherwise identical settings to test whether a larger token budget changes the conclusions.Section 4 fixes Nr = Nt = 16; the ablation compares this setting with n = 32.
  • Reasoning token count ablation: Doubling the token count does not produce a uniform improvement for either variant, with each benchmark moving in a different direction.The comparison reports headline metrics for every benchmark in Table 6.
  • Reasoning token count ablation: At the benchmark level, n = 16 outperforms n = 32 on three of four benchmarks for both vt and tt.n = 32 leads on only a single benchmark for each variant.
  • Reasoning token count ablation: The results indicate that reasoning tokens do not act as generic capacity that scales with n.This interpretation follows from the non-uniform benchmark movements and the three-of-four advantage for n = 16.
Loading 2607.08763v1…