PaperlayerView exact revision on arXiv

Source-linked AI summary

CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

Joowon Kim, Seungho Shin, Joonhyung Park, Eunho Yang

arXiv:2605.08735v1cs.CV

TL;DR

VGMs provide useful short-horizon visual simulation but struggle with long-horizon reasoning and execution errors. CollabVR couples VLM planning and verification with step-level VGM generation, improving open- and closed-source models across two benchmarks over established baselines at matched compute.

  • Problem

    VGMs offer visual coherence and short-horizon simulation but remain limited in abstract reasoning, long-range consistency, and task-level reasoning correctness.

  • Method

    CollabVR uses closed-loop, step-level VLM planning and clip verification to revise VGM action prompts and repair execution failures.

  • Results

    CollabVR consistently improves open- and closed-source VGMs over single-inference, Pass@k, and prior test-time scaling baselines on Gen-ViRe and VBVR-Bench.

  • Takeaways & Limitations

    Step-level VLM supervision improves video reasoning across generator types and remains beneficial when paired with reasoning-fine-tuned VGMs.

  • Takeaways & Limitations

    Test-time orchestration cannot overcome VGMs lacking the underlying capability, while weak instruction-following and imperfect verification limit repair effectiveness.

Abstract

from arXiv · show

Recent "Thinking with Video" approaches use Video Generation Models (VGMs) for visual reasoning by producing temporally coherent Chain-of-Frames as reasoning artifacts. Even strong VGMs, however, exhibit two recurring failure modes on goal-directed tasks: long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. Both stem from the absence of explicit reasoning built upon the VGM's short-horizon visual prior, a role naturally filled by Vision-Language Models (VLMs), but where to place the VLM is non-trivial: upfront plans commit before any frame is generated and post-hoc critiques over whole videos intervene too late. We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level granularity: the VLM plans the immediate next action, inspects the clip the VGM generates, and folds the verifier's diagnosis directly into the next action prompt to repair detected failures. On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute, with the largest gains on the hardest tasks. It also yields further improvements on top of a reasoning-fine-tuned VGM, indicating that step-level VLM supervision is orthogonal to and stackable with reasoning-oriented fine-tuning. We provide video samples and additional qualitative results at our project page: https://joow0n-kim.github.io/collabvr-project-page.

1 Introduction

CollabVR addresses VGM failures in goal-directed video reasoning by coupling VLM planning and verification with VGM generation at step-level granularity. Its progressive planning and failure-aware repair improve performance across VGMs and benchmarks at matched compute, including reasoning-fine-tuned models.

  • Motivation: VLMs provide logical decomposition and planning, while VGMs provide visual detail, physical coherence, and short-horizon simulation but struggle with abstract reasoning and long-range coherence.This complementarity motivates collaboration between the two model types.
  • Motivation: VGMs fail through overloaded prompts that cause long-horizon drift and localized mid-clip execution errors that can compound across a trajectory.These failures arise when tasks are not decomposed into coherent sub-goals or when generated clips contain incorrect intermediate actions.
  • CollabVR Framework: CollabVR uses step-level closed-loop coupling: the VLM plans the immediate next action, inspects the generated clip, diagnoses failures, and conditions the next prompt on that diagnosis.Step-level intervention matches recovery to the diagnosed state and contains errors before they contaminate an entire trajectory.
  • Progressive Planning: Progressive Planning adaptively selects the step count and emits only the next action from prior frames, mitigating long-horizon drift without fixed upfront decomposition.The module targets overloaded-prompt failure by conditioning planning on previously generated frames.
  • Collaborative Reasoning: Collaborative Reasoning verifies each VGM clip and folds diagnosed failures into the next action prompt, repairing local execution errors before they compound.The intervention module targets execution failure through failure-aware, clip-level repair.
  • Results: CollabVR improves open- and closed-source VGMs over single-inference, Pass@k, and VideoTPO on Gen-ViRe and VBVR-Bench, with further gains on reasoning-fine-tuned VGMs.A human-annotated benchmark also finds that VLM-predicted task complexity and failure localization align with expert judgments.

2 Related Work

Related work progresses from static visual artifacts to video-based reasoning and test-time scaling, but existing methods often optimize perceptual quality without localized task-failure repair. CollabVR addresses this gap through progressive planning, step-level verification, and recovery in a training-free loop compatible with off-the-shelf VGMs.

  • Visual reasoning artifacts: Thinking with Images uses sketches, diagrams, and sub-images as intermediate reasoning artifacts, while Thinking with Video uses temporally coherent generated videos to represent dynamic solution trajectories.Static images cannot capture dynamic processes or causal unfolding over time; high-fidelity VGMs enable video reasoning through Chains-of-Frames.
  • Test-time scaling: Video-specific test-time scaling applies frame-level search, evolutionary sampling, and self-refinement along the temporal axis, but primarily optimizes visual quality rather than task correctness.The cited reasoning failures include wrong solution paths, skipped sub-goals, and incorrect physical outcomes.
  • Closed-loop generation: Iterative refinement and LLM/VLM-guided generation place language models in closed loops with generators, extending from image systems to video systems.This line of work casts the LLM as a verifier or planner.
  • CollabVR: CollabVR progressively plans, verifies, diagnoses, and recovers at each step, treating clips as actionable units rather than an indivisible video and requiring no training.Its explicit failure diagnosis is folded into subsequent action prompts, and the framework works with any off-the-shelf VGM.

3 CollabVR: Closed-Loop Step-Level Video Reasoning

CollabVR treats video reasoning as stepwise trajectory construction through closed-loop VLM planning, VGM execution, and clip verification. Progressive planning adapts actions to generated clips, while verification supplies repair guidance for rejected attempts.

  • Framework overview: CollabVR constructs the target video trajectory stepwise with a VLM planner/verifier and an image-to-video generator.The VLM plans and verifies, while the generator maps the latest conditioning frame and action prompt to a short clip.
  • Framework overview: Each iteration plans an action, generates a clip, verifies it, accepts successful clips into history, and evolves rejected prompts using diagnostic feedback.Accepted clips update the latest frame and are concatenated into the final video; the loop stops when the task is complete or planning limits are reached.
  • Progressive planning: +13% relative gain over pre-planning is achieved by progressive planning on Gen-ViRe with VBVR-Wan2.2 at matched cost.The comparison isolates Module 1, progressive planning versus pre-planning.
  • Progressive planning: Progressive planning chooses only the immediate next action after observing prior generation, avoiding pre-planning’s commitment before any VGM output exists.Pre-planning decomposes the task into milestone prompts upfront, while progressive planning adapts to realized generation and avoids fixing the milestone count from the prompt alone.
  • Collaborative reasoning: The verifier returns accept or reject plus a textual reason and actionable repair suggestion, checking whether each planned action was executed.It flags wrong direction, wrong target, or scene collapse while not treating partial-but-correct progress as failure.

4 Experiments

CollabVR is evaluated on Gen-ViRe and VBVR-Bench against single inference, Pass@k, and VideoTPO under matched compute, with ablations and analyses of module effectiveness and VLM supervision. It improves reasoning performance and cost efficiency, while module contributions vary with task structure and decomposition can hurt atomic symbolic transformations.

  • Main results: CollabVR improves Gen-ViRe performance over single inference for VBVR-Wan2.2 and Veo 3.1, with the largest baseline margins on Planning and Algorithmic categories.VBVR-Wan2.2 rises from Pass@1 0.391 to 0.531, while Veo 3.1 rises from 0.481 to 0.550.
  • Main results: CollabVR achieves higher accuracy than Pass@k at lower per-sample generation cost and further improves the reasoning-fine-tuned VBVR-Wan2.2.These results support adaptive progressive planning with failure-aware recovery as an effective test-time scaling strategy whose supervision stacks with reasoning-oriented fine-tuning.
  • Module ablations: Verification and failure-aware recovery contributes more than progressive planning in the ablation, with gains of +0.063 versus +0.035.The dominant module shifts by benchmark: recovery corrects single-clip failures on VBVR-Bench, while progressive decomposition helps complex multi-step Gen-ViRe tasks.
  • Category effectiveness and limitations: CollabVR is weaker on symbolic transformations, where Analogy and Abstract gains are only +0.083 and +0.090, and combined decomposition can underperform either module on Analogy.Forcing decomposition on atomic transformations creates contrived intermediates that verification rejects.
  • VLM supervision: Gemini 2.5 Pro aligns most closely with human annotators across all three evaluated reliability axes, supporting its use as the default planner and verifier.The same axes are used to compare Qwen3.5-27B and Qwen3.5-9B independently of final task accuracy.

5 Conclusion · Appendix

CollabVR couples a VLM and video generation model in a closed loop, planning and verifying one sub-action at a time while adaptively repairing the constructed video. Its benefits depend on the generator’s underlying capability and verifier reliability, with limitations arising from weak instruction-following, difficult transformations, and imperfect verification.

  • 5 Conclusion: CollabVR pairs a VLM with a video generation model at step-level granularity.The framework forms a closed loop between planning, generation, and verification.
  • 5 Conclusion: The VLM plans one sub-action at a time before inspecting the clip produced by the VGM.This ordering enables verification after each generated step.
  • 5 Conclusion: After inspection, the VLM adaptively accepts, regenerates, or further decomposes the action.These are the framework’s stated step-level intervention choices.
  • 5 Conclusion: CollabVR redirects test-time compute from sampling more videos toward refining the one being constructed.The compute allocation follows from the framework’s step-level coupling.
  • 5 Conclusion: Abstract or symbolic transformations remain difficult when the generator never approximates them.Test-time orchestration cannot overcome a VGM lacking the underlying capability.
  • 5 Conclusion: Gains diminish on lower-capability VGMs because weak per-step instruction-following compounds errors across sub-clips faster than regeneration can repair them.The limitation is attributed to error accumulation outpacing repair.
  • 5 Conclusion: An imperfect verifier allows a fraction of failed clips to propagate downstream.Verification errors can therefore affect later steps.

A Implementation Details … B.3 Per-VLM Human-Alignment Breakdown

The implementation uses step-level VLM planning and verification with bounded generation attempts, explicit prompt protocols, and recovery mechanisms. Additional evaluations examine human preferences, attempt-budget scaling, and per-VLM alignment across planning, verification, and repair quality.

  • A.1 Hyperparameters: CollabVR defaults to Nmax=3 planning steps and M=3 generation attempts, using Gemini 2.5 Pro at temperature 0.2 for all VLM roles and reward-based baselines.The verifier samples one frame per second, except Gemini 2.5 Pro receives raw video for internal sampling.
  • A.2 Prompt Templates: The progressive planner emits only the next immediate visual action and target state, while the verifier judges execution and returns accept/reject plus a structured repair diagnosis.Verifier rejection tolerates partial progress and uses good_fraction to support partial regeneration from the correctly executed portion.
  • A.3 Verifier Output Examples: A bookshelf trace demonstrates rejection-driven prompt evolution repairing Step 1, followed by accepted progressive planning that completes Step 2 without further evolution.The accepted clip is committed to history and its last frame conditions the next step.
  • A.4 First-Frame Fidelity for VGM Selection: First-frame fidelity is required for clip concatenation and mid-clip regeneration; VBVR-Wan2.2, Veo 3.1, and Cosmos-Predict 2.5 meet this requirement, unlike Sora 2.Sora 2 is excluded from the main CollabVR results because its input-reference setup causes first-frame drift.
  • A.5 Auxiliary Failure Router: The failure router intervenes after M rejected evolution attempts, choosing regeneration for execution slips, splitting for structurally multi-step failures, or fallback when decomposition is unsuitable.For Veo 3.1, the router first decides whether to accept the single-shot baseline or invoke multi-step orchestration.
  • B.1 User Study: In a blind Prolific study with 40 participants and 16 tasks, CollabVR was the dominant human preference across all three reported comparison views.The study compared CollabVR with Pass@1 and Pass@4 at matched VBVR-Wan2.2 generation compute.
  • B.2 Effect of Per-Step Attempt Budget M: +4.89% and +1.59% are the gains from increasing M from 1 to 2 and 2 to 3; later increments fall below 1% while cost continues increasing nearly linearly.Beyond M=3, the cost-quality profile approaches plain Pass@k resampling.
  • B.3 Per-VLM Human-Alignment Breakdown: Gemini 2.5 Pro leads all human-alignment axes, with the largest closed-versus-open gap on verifier reject recall: 65.6% versus 40–45%.The size increase from 9B to 27B contributes only +4 points on verifier alignment, while the evolution-quality gap to 27B is Δ=0.06.

B.4 Per-Category ∆Heatmap on VBVR-Bench

On VBVR-Bench, M2 is the larger single-module contributor, driven by Abstract tasks, while Knowledge gains remain marginal because neither decomposition nor re-sampling supplies missing world knowledge.

  • Module contributions: ΔM2 +0.063 exceeds ΔM1 +0.035 as the larger single-module contribution on VBVR-Bench.This reverses the module-dominance pattern reported for Gen-ViRe.
  • Module contributions: Abstract shows the largest M2-alone gain at +0.131, as verifier-driven re-sampling recovers atomic perceptual targets without exposing genuine sub-goals.The passage attributes this category-specific advantage primarily to recoverable perceptual targets and the absence of genuine sub-goals.
  • Knowledge limitation: Knowledge remains marginal across all configurations, with Δ+0.010 for M1+M2.Neither decomposing nor re-sampling supplies the missing world knowledge, and the verifier can only redraw from the VGM’s existing distribution.

B.5 Veo 3.1 Module Ablation

On Gen-ViRe with Veo 3.1, verifier-and-regeneration alone improves single-shot performance but cannot solve multi-step tasks, while adding planner decomposition unlocks further gains. Unlike on VBVR-Wan2.2, planner-only operation hurts Veo 3.1, reflecting differences in the models’ design priors.

  • Ablation setup: The Veo 3.1 ablation mirrors the M1/M2 analysis previously conducted for VBVR-Wan2.2 on Gen-ViRe.The ablation evaluates which modules are enabled and includes an additional fixed-N=3 condition that bypasses adaptive planner selection.
  • M2 verifier-and-regen: 0.527 with N=1 and M=2 gives Veo 3.1 a +0.046 gain over Pass@1, nearly matching VBVR-Wan2.2’s +0.045 gain.Regeneration alone remains limited to tasks Veo 3.1 can complete in one clip; multi-step tasks remain out of reach regardless of regeneration count.
  • M1 decomposition: 0.550 from full M1+M2 is a further +0.023 over the M2-only ceiling.M1 emits multi-step decompositions with N>1, breaking tasks into sub-actions that Veo 3.1 can satisfy individually.
  • Cross-model asymmetry: M1 alone falls 0.035 below Pass@1 on Veo 3.1, unlike VBVR-Wan2.2, where it delivers +0.120 of the total +0.140 gain.The asymmetry is attributed to the VGMs’ design priors: VBVR-Wan2.2’s visual-reasoning fine-tuning places the planner’s artificial intermediate states inside its training distribution.

B.6 Cosmos-Predict-2.5 Detailed Results · C Module Diagnostics

CollabVR improves Cosmos-Predict 2.5 on VBVR-Bench but degrades on Gen-ViRe, except for the Analogy category. The results show that decomposition depends on reliable per-step instruction following and complements rather than replaces stronger VGM training.

  • B.6 Cosmos-Predict-2.5 Detailed Results: 0.308 →0.403: CollabVR improves Cosmos-Predict 2.5 (14B) on VBVR-Bench.VBVR-Bench is dominated by single-step tasks, where Cosmos handles single-clip verification and a few re-generations competently.
  • B.6 Cosmos-Predict-2.5 Detailed Results: 0.287 →0.182: CollabVR degrades Cosmos-Predict 2.5 on Gen-ViRe, with ∆= −0.105 vs. Pass@4.Gen-ViRe is dominated by multi-step reasoning tasks, requiring Cosmos to execute decomposed steps.
  • B.6 Cosmos-Predict-2.5 Detailed Results: 0.042 →0.375: Analogy is the only Gen-ViRe category where CollabVR helps Cosmos, with ∆= +0.333.Pass@1 is essentially zero in Analogy, so any successful sub-step can accumulate into a positive end-state.
  • B.6 Cosmos-Predict-2.5 Detailed Results: Decomposition helps a weak VGM when single-shot performance is so low that any partial progress beats the baseline.This regime is illustrated by Cosmos on Gen-ViRe’s Analogy category.
  • B.6 Cosmos-Predict-2.5 Detailed Results: CollabVR’s gain on a VGM is bounded below by that generator’s per-step instruction-following reliability.The framework requires a minimum per-step instruction-following floor before decomposition becomes profitable.
  • B.6 Cosmos-Predict-2.5 Detailed Results: CollabVR is framed as orthogonal to, rather than a substitute for, training stronger VGMs.Its effectiveness depends on using a generator that meets the required per-step instruction-following floor.

C.1 Pipeline Statistics … D.1 Examples by Step Count N

CollabVR’s runtime behavior varies substantially by benchmark and step complexity, while verification repairs many but not all generated clips. The VGM dominates compute, and qualitative examples show that step-level decomposition mainly improves structurally multi-step tasks.

  • C.1 Pipeline Statistics: Gen-ViRe samples average 2.56 planning steps, 1.46 regenerations, 4.01 generated clips, and 6.79 VLM calls, versus 1.48, 0.90, 2.38, and 3.85 on VBVR-Bench.The shorter VBVR-Bench trajectories reflect its reasoning-heavy but visually constrained tasks.
  • C.1 Pipeline Statistics: On Gen-ViRe, 72.2% of samples reach the N=3 cap, whereas VBVR-Bench has mean N=1.48 versus 2.56 on Gen-ViRe.Gen-ViRe terminates at N=1 for 16.7% and N=2 for 11.1% of samples.
  • C.1 Pipeline Statistics: VBVR-Bench regeneration counts are bimodal, with 58.6% requiring zero regenerations and 26.4% requiring two, while Gen-ViRe averages 1.46 regenerations.The VBVR-Bench multi-step failures contribute 451 extra clips overall.
  • C.2 Verifier Run-time Behavior: The verifier accepts 49.5% of clips initially and recovers 14.7% through regeneration, reaching 64.1% final acceptance at average retry depth 0.57.Another 35.9% remain rejected after the maximum two regenerations and are propagated as best-of-attempts clips.
  • C.2 Verifier Run-time Behavior: Final-reject rates rise sharply at step 3, reflecting cumulative visual drift because each step begins from the previous step’s last frame.The same pattern appears as diminishing returns beyond N=3 on Gen-ViRe and more sharply on Cosmos-Predict 2.5.
  • C.3 Cost Decomposition: Veo 3.1 generation costs $3.21 per Gen-ViRe sample versus $0.026 for the VLM, a ∼125× ratio.For the open-source backbone, VGM wall-clock time is approximately 980 s versus ∼68.5 s for the VLM, a ∼14× ratio.
  • C.3 Cost Decomposition: Because the VGM dominates compute by an order of magnitude or more in both deployment regimes, VGM-generated seconds per sample serve as a compute proxy.The Gen-ViRe VLM aggregate is ∼6.6 calls, ∼68.5 s of API wall-clock, ∼13 K input tokens, and $0.026 per sample.
  • D.1 Examples by Step Count N: For N=1 tasks, CollabVR produces visually similar outputs with incidental fixes, while for N=2–3 tasks it decomposes actions and recovers structurally correct trajectories.Examples include correcting the predicted next color from red to green and laying out all circles before sorting them by circumference.

D.2 Examples by VGM … D.5 Partial vs. Full Re-generation: Maze Case Study

CollabVR generalizes across video generators and verifier models, with stronger verifiers producing better recoveries. Residual failures arise from verifier detection limits or VGM execution limits, while partial regeneration can improve maze navigation by preserving correct prefixes.

  • D.2 Examples by VGM: CollabVR consistently improves intended in-canvas actions on Cosmos-Predict-2.5 across all four evaluated tasks.The alone model hallucinates external hands or pens, whereas pairing with CollabVR drives actions onto the canvas.
  • D.2 Examples by VGM: +CollabVR produces visibly more faithful executions than both open-source VBVR-Wan2.2 and closed-source Veo 3.1 across Gen-ViRe task types.The comparison spans dashcam, refrigerator, mirror-reflection, and Raven’s-matrices tasks.
  • D.3 Examples by VLM: Qwen3.5-9B false-accepts a mis-localized target, while Qwen3.5-27B partially recovers and Gemini 2.5 Pro identifies the precise distractor.The trace concerns finding a pentagon and drawing a red circle around it.
  • D.3 Examples by VLM: Final +CollabVR output proximity to the GT last frame increases monotonically with verifier capability across multiple VBVR-Bench tasks.This aggregate pattern mirrors the score ordering in Table 4.
  • D.4 Failure Cases: CollabVR has two residual ceilings: the verifier may fail to detect an issue, or the VGM may fail to execute a correctly diagnosed fine-grained operation.In the latter case, additional retries cannot help because the bottleneck has moved from supervisor to generator.
  • D.4 Failure Cases: The symbolic-category gap largely reflects verifier detection failures, whereas residual Spatial and Transformation gaps reflect VGM execution limits.The passages identify stronger VLM grounding and reasoning-oriented VGM training as complementary directions.
  • D.5 Partial vs. Full Re-generation: Maze Case Study: Full re-generation fails to reach the maze goal across four independent attempts because each resampled trajectory discards accumulated progress.Partial regeneration begins from the first failing frame fτ and preserves the correctly produced prefix.
  • D.5 Partial vs. Full Re-generation: Maze Case Study: Partial re-generation successfully converges on the maze goal by targeting the failed suffix rather than rerolling the entire trajectory.It is positioned as an auxiliary mode when the prefix carries useful information, with precise failure detection left as future work.

E Broader Impact

CollabVR situates video reasoning as a capability for dynamic, temporally grounded processes with applications in education, science, procedural guidance, navigation, and embodied-agent simulation. It inherits the misuse risks of generative video while adding no new generative capability beyond composing existing models under their licenses and safeguards.

  • Positive impacts: Video reasoning supports dynamic, temporally grounded applications that static-image or pure-text reasoning cannot express, including education, science, procedural walkthroughs, navigation, and embodied-agent simulation.The passage identifies these application areas as positive impacts of video reasoning.
  • Potential negative impacts: CollabVR inherits risks from its underlying generative models, including higher-fidelity synthetic video used for deceptive or non-consensual content.The framework composes existing models for closed-loop reasoning rather than training new generators.
  • Mitigations: The project uses off-the-shelf VGMs and VLMs under their original licenses and existing safeguards, while releasing artifacts without novel high-risk generative capability.The listed models include Wan2.2, Cosmos-Predict-2.5, Veo 3.1, Gemini 2.5 Pro, and Qwen3.5.
Loading 2605.08735v1…