PaperlayerView exact revision on arXiv

Source-linked AI summary

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

Dongjie Cheng, Ruifeng Yuan, Ran Wei, Ning Song, Yongqi Li, Wenjie Li

arXiv:2610.21910v1cs.AI

TL;DR

VLMs are expected to reason about motion, actions, and viewpoint changes that go beyond observed images, but VWM-based approaches often rely on fixed visual-imagination pipelines. CoWorld Reasoner uses a VWM-driven harness and feedback-guided co-evolution to support varied visual operations and their interpretation. It improves over direct reasoning across seven benchmarks, with its strongest variant also outperforming evolving baselines on five.

  • Problem

    VLMs are increasingly expected to reason about unobserved motion, actions, and viewpoint changes, while existing VWM approaches typically use fixed pipelines centered on predefined forms of visual imagination.

    Source: §1 · pp. 1, 3

  • Method

    CoWorld Reasoner exposes multiple VWM operations through tools and skills, and co-evolves tools, skills, and execution code from feedback while keeping both model weights fixed.

    Source: §1 · p. 3

  • Results

    The strongest variant improves all seven benchmarks, including relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and outperforms five evolving baselines on five benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CoWorld Reasoner improves over direct reasoning with the same VLM across seven benchmarks spanning spatial, temporal and causal, and perceptual reasoning.

    Source: §1 · p. 3, §5 · p. 11

    The method is framed for questions with an ordered sequence of input images, using a VLM, a VWM, and an editable harness.

    Source: §3 · p. 4

Abstract

from arXiv · show

As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyond what the input images show. Video World Models (VWMs) provide a natural unified engine for visual imagination by generating task-relevant visual content as additional evidence. However, existing approaches typically invoke a VWM through fixed pipelines centered on a predefined form of visual imagination, such as novel-view synthesis. To support flexible visual imagination for diverse reasoning tasks, we introduce CoWorld Reasoner, an agentic framework that uses a VWM as a unified visual reasoning engine. Its Video World Model-Driven Harness exposes multiple visual operations through tools and uses skills to guide the VLM in deciding whether to imagine, selecting an operation, constructing a generation request, and interpreting the resulting visual evidence. We further introduce VLM-VWM Harness Co-Evolution, which jointly revises tools, skills, and execution code based on execution feedback while keeping both model weights fixed. Across seven benchmarks spanning spatial intelligence, temporal and causal reasoning, and perceptual reasoning, CoWorld Reasoner outperforms direct reasoning with the same VLM. The strongest variant improves on all seven benchmarks, including relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and outperforms all five evolving baselines using the same backbone across five benchmarks.

1 Introduction

VLMs are increasingly expected to reason about motion, actions, and changing viewpoints beyond observed images, raising the question of how to generate and use task-relevant visual imagination. CoWorld Reasoner addresses this with a VWM-driven harness and co-evolution, and improves over direct reasoning across seven benchmarks.

Motivation

  • VLMs must reason about unobserved transitions and relations, such as hidden motion, future events, or what another viewpoint would reveal.Visual imagination generates task-relevant predictions to use as additional evidence.Source: §1 · pp. 1–2
  • Existing VWM approaches often use fixed pipelines centered on one form of imagination, such as novel-view synthesis, limiting their applicability to diverse reasoning problems.Different tasks may call for viewpoint changes, interpolation, continuation, or intervention.Source: §1 · p. 2

Approach

  • CoWorld Reasoner uses a VWM as a unified visual reasoning engine, with tools for diverse visual operations and skills guiding tool choice, request construction, and video interpretation.Co-evolution jointly revises tools, skills, and execution code from feedback while keeping both model weights fixed.Source: §1 · p. 3

Evaluation

  • The strongest CoWorld Reasoner variant improves all seven benchmarks, with relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and beats five evolving baselines on five benchmarks.The evaluation spans spatial, temporal and causal, and perceptual reasoning.Source: §1 · p. 3

2 Related Work

Related work covers generated visual evidence for multimodal reasoning, video and world models, and agents that evolve their prompts, skills, or code. CoWorld Reasoner combines these directions by exposing multiple VWM operations and jointly revising the harness for evidence generation and interpretation.

2.1 Multimodal Reasoning

  • Prior multimodal reasoning work uses generated images or videos, while spatial approaches include generated-view exploration and controls on how much visual content is produced.The section also distinguishes training-data and supervision approaches from inference-time exploration.Source: §2.1 · pp. 3–4

2.2 World Models

  • Video generation models learn visual dynamics from video data, while other world models simulate environments, generate editable 3D scenes, or respond to user actions in real time.CoWorld Reasoner instead exposes a VWM through multiple tools for visual reasoning.Source: §2.2 · p. 4

2.3 Self-Evolving Agents

  • Self-evolving agent methods revise prompts, skills, code, harnesses, or verifiers; CoWorld Reasoner jointly revises its VWM-driven harness to improve evidence generation and interpretation.Examples include GEPA [27], SkillOpt [28], DGM [29], Meta-Harness [30], and CoEvoSkills.Source: §2.3 · p. 4

3 Method

CoWorld Reasoner pairs an editable VWM-driven harness with feedback-guided co-evolution. The harness supports task-dependent generation and interpretation of visual evidence, while co-evolution iteratively revises its components across reasoning stages and rounds.

3 Method

  • CoWorld Reasoner comprises a VWM-driven harness for requesting and interpreting visual evidence and a co-evolution procedure that improves the harness using execution feedback.The method considers a question with an ordered sequence of input images, a VLM, a VWM, and an editable harness.Source: §3 · pp. 4–5

3.1 Video World Model-Driven Harness

  • The harness combines tools for view change, interpolation, continuation, or intervention with reusable skills, a verifier, and execution code.The agent can skip generation; when it generates video, it compares it with the inputs, interprets question-relevant evidence, and may request verification.Source: §3.1 · p. 5

3.2 VLM-VWM Harness Co-Evolution

  • Co-evolution revises tools, skills, verifier or stage instructions, and execution code using feedback while keeping the VLM and VWM weights fixed.Across four stages and repeated rounds, candidates run from selected incoming histories; the supervisor uses stage metrics, traces, and available answer-correctness feedback to select candidates, weighing gains against regressions.Source: §3.2 · pp. 6–7

4 Experiments

The experiments evaluate CoWorld Reasoner across seven multimodal benchmarks against direct GPT-5.6 reasoning and evolving baselines. Analyses examine task-dependent tool use, VWM choice, error recovery, and transfer across VLMs.

4.1 Experimental Setup

  • The evaluation covers spatial, temporal, causal, and perceptual reasoning, using 120 development examples excluded from testing and an item-weighted accuracy average.The seven benchmarks include MMSI, SAT, Spin, SpatiaLab MCQ and Open, ERQA, CausalSpatial, and PerceptionBench.Source: §4.1 · p. 7

4.2 Main Results

  • CoWorld Reasoner improves over direct GPT-5.6 on all seven benchmarks, leads all five evolving baselines on five, and achieves the highest aggregate accuracy.Gains reach 4.54 percentage points on MMSI and 6.75 points on ERQA; gains also occur on SAT, SpatiaLab, and ERQA, which have no development examples.Source: Table 1 · p. 8, §4.2 · p. 8
  • MMSI gains span all four categories and reach 6.56 points for positional relationships; SAT perspective taking reaches 84.85%, highest among compared methods.Spin improves in View, Mental Rotation, and Perspective; SAT goal reasoning improves while object motion declines.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Analysis

  • Disabling VWM tools and the verifier yields the largest tool gains on CausalSpatial occlusion and collision, where generated video exposes intermediate interactions.SAT perspective taking already improves without tools and gains no further benefit from generation.Source: §4.3 · pp. 8–9, Figure 3 · p. 10
  • On MMSI, cross-view and temporal skills pair with view-change and interpolation tools; on CausalSpatial, intervention skills pair with intervention and continuation tools.Skills and tools are rarely used on PerceptionBench, where direct perception often suffices.Source: §4.3 · p. 9, Figure 3 · p. 10
  • Both H3 Max Turbo and LTX-2.3 improve overall accuracy and performance on MMSI, Spin, and SpatiaLab, while H3 yields larger gains and improves SAT.LTX-2.3 falls below GPT-5.6 on CausalSpatial; the harness reuses spatial workflows across VWMs.Source: §4.3 · pp. 9–10, Figure 3 · p. 10
  • On GPT-5.6 errors, CoWorld Reasoner leads on MMSI, SpatiaLab, CausalSpatial, and PerceptionBench; VWM tools improve CausalSpatial recovery, leave SAT unchanged, and reduce ERQA recovery.CausalSpatial recovery gains align with occlusion and collision improvements, while overall accuracy remains close to the GPT-5.6 baseline.Source: Figure 4 · p. 10, §4.3 · p. 10
  • After one additional co-evolution round, the transferred harness raises Grok-4.5 item-weighted accuracy from 32.86% to 39.78%, with the largest gains on PerceptionBench and Spin.MMSI and CausalSpatial improve, SAT declines slightly, and gains span tasks using input images and tasks requiring reasoning about unseen views or outcomes.Source: §4.3 · p. 10

5 Conclusion and Future Work

CoWorld Reasoner uses a Video World Model-Driven Harness of tools, skills, and a verifier, with co-evolution using execution feedback to refine visual evidence requests and interpretation. It improves over direct reasoning on all seven benchmarks and beats all five evolving baselines on five; future work will explore test-time training to co-evolve the harness, VLM, and VWM.

5 Conclusion and Future Work

  • CoWorld Reasoner treats a VWM as a unified visual reasoning engine through a harness comprising tools, skills, and a verifier.Harness co-evolution uses execution feedback to improve how the agent requests, evaluates, and integrates visual evidence.Source: §5 · pp. 10–11
  • CoWorld Reasoner improves over direct reasoning with the same VLM on all seven benchmarks and outperforms all five evolving baselines on five.Agentic tool use and harness co-evolution extend VWM-based reasoning beyond fixed novel-view synthesis pipelines.Source: §5 · p. 11
  • Future work will explore test-time training to co-evolve the harness, VLM, and VWM for visual generation and multimodal reasoning across diverse tasks.Source: §5 · p. 11

A.1 Evaluation Data

The appendix describes development and evaluation data, baseline mechanisms, and the provenance of published comparison results.

A.1 Evaluation Data

  • The method uses 120 development examples to evolve reusable components and evaluates selected variants on benchmarks excluding those examples.The five evolving baselines were adapted to the same visual inputs and scoring interface.Source: §A.1 · p. 13, Table 3 · p. 13
  • SAT evaluation comprises 300 items from 150 questions under two option permutations, while SpatiaLab MCQ and Open use the same 1,400 questions in two answer formats.Source: §A.1 · p. 13
  • The evolving-agent comparisons use five GPT-5.6-adapted baselines that operate on input images and evolve prompts, memories, or code, while the proposed harness co-evolves tools, skills, and execution code.Published comparison rows include specified model versions and protocol notes; averages are reported only for methods evaluated on all benchmarks.Source: §A.1 · pp. 13–14, Table 4 · p. 14

A.2 Scoring and Usage Statistics

The appendix defines accuracy, recovery, skill-loading, and tool-use statistics, then records agent settings, visual operations, and candidate selection criteria.

A.2 Scoring and Usage Statistics

  • Accuracy is computed over benchmark items; recovery is accuracy on items GPT-5.6 direct reasoning answered incorrectly, and SpatiaLab formats are aggregated by correct answers and item counts.A separate GPT-5.6 judge evaluates semantic equivalence for open-ended SpatiaLab and applicable PerceptionBench answers.Source: §A.2 · p. 14
  • Skill-loading rate counts items with a skill loaded, while tool-use rate counts items with a VWM-driven tool call for an operation; an item can contribute to multiple columns.The Any column counts items with at least one skill load or tool call in the group.Source: §A.2 · p. 14

A.3 Agent and Generation Configuration

  • The agent carries interaction history through four stages, and the verifier receives generated video; the no-VWM variant disables tools and the verifier while retaining skills and execution.The experiments use MiniMax H3 Max Turbo image-to-video generation, with requests describing visual transformations rather than prescribing answers; Grok-4.5 transfer adds one co-evolution round.Source: §A.3 · p. 14, Table 5 · p. 15

A.4 VWM-Driven Skills and Visual Operations

  • The experiments define three skills and four visual-operation categories, and the supervisor selects candidates using stage assessments, sampled traces, and answer-correctness feedback.Selection checks skill choice, evidence coverage, request validity, image order, tool execution, interpretation, reference frames, correctness, and cross-benchmark gains against regressions.Source: §A.4 · p. 15, §A.5 · p. 15, Table 6 · p. 16

A.6 Development and Test Performance

The final evolved agents were compared on development and held-out test splits, with aggregate accuracy weighted by split size. CoWorld Reasoner outperformed GPT-5.6 on all four test benchmarks and achieved higher aggregate test accuracy than ReasoningBank.

A.6 Development and Test Performance

  • Figure 6 compares final evolved agents on 120 development examples and 8,044 held-out test examples, weighting aggregate accuracy by split size.Table 7 reports Spin accuracy by task category.Source: §A.6 · p. 16, Table 7 · p. 16
  • 62.88% test accuracy for CoWorld Reasoner exceeded 60.95% for ReasoningBank and 61.62% for GPT-5.6 across the four benchmarks.Both CoWorld Reasoner and ReasoningBank reached 55.83% aggregate development accuracy.Source: §A.6 · p. 16
  • CoWorld Reasoner improved over GPT-5.6 on each of the four test benchmarks.ReasoningBank improved over GPT-5.6 on three benchmarks during development but fell below it on all three corresponding test sets.Source: §A.6 · p. 16

A.7 Results by Task Category

The appendix reports category-level accuracy across several benchmarks and illustrates evolved guidance for constructing visual requests and interpreting generated evidence. The examples include binding image references to source order, distinguishing object motion from view changes, and revising an answer after a simulated outcome.

A.7 Results by Task Category

  • Tables 7–12 report category accuracies for Spin, SpatiaLab, ERQA, CausalSpatial, and PerceptionBench; Table 2 covers MMSI and SAT.All denotes overall benchmark accuracy, and Gain (pp) is the difference from GPT-5.6 in percentage points.Source: §A.7 · p. 16, Table 8 · p. 17, Table 9 · p. 17

A.8 Examples of Evolved Guidance

  • Request instructions align image references with source order, while interpretation guidance checks object identity and expresses relations in the question’s reference frame.The example maps the second input image to Picture 1 in the generation request.Source: §A.8 · p. 18
  • Comparing a subject with background landmarks across frames helps distinguish object rotation from camera motion.The SUV example tracks motion relative to a stationary chair, plant, and vase.Source: §A.8 · p. 18
  • In the plate-removal example, the generated video shows the other plates staying in place, prompting the agent to revise its initial answer.Boldface highlights key cues in the trajectories, and [...] indicates omitted skill text.Source: §A.8 · p. 18

A.9 Qualitative Case Studies

The case studies show how generated videos supply evidence for mapping room viewpoints, interpreting object rotation, and assessing an intervention. The examples also expose limits: the room sequence does not establish exact camera displacement, and the plate simulation cannot guarantee real physical behavior.

Room viewpoint mapping

  • The room sequence preserves furniture identities and shifts the visible landmarks, supporting a northward mapping of Photo 1 relative to Photo 2 while leaving exact translation uncertain.The gray-chair row and cabinet anchor Photo 1 to Photo 3, whose stipulated east-facing view makes image-left north.Source: §A.9 · p. 21

Object rotation

  • The generated table transition depicts apparent in-place rotation, but the white background provides no stationary landmarks and the final top-down direction must be interpreted against the ordered input views.The second view’s table orientation provides a cue opposite to a clockwise turn from the initial front-facing orientation.Source: §A.9 · pp. 23–24

Plate-removal intervention

  • After the lowest plate is removed, the remaining plates stay grouped without visible displacement, but the simulation’s approximate mechanics cannot guarantee the real outcome.The generated sequence shows the removed plate sliding toward the foreground while the others do not visibly tip, fall, float, or break.Source: §A.9 · p. 27

C. Nothing will happen

The static scene supports a no-risk answer because the SUV faces open floor and nearby obstacles appear offset from its path. A generated forward-motion sequence likewise shows it passing the plant and vase without visible contact, but only for the depicted path.

Static assessment

  • The static scene supports “B. No”: the SUV faces open floor, with the plant and vase offset from its apparent path, so no additional visual-world operation is judged useful.The decision is based on the visible path and nearby obstacles already appearing together in the input.Source: C. Nothing will happen · p. 28

Assessment boundary

  • The risk assessment is limited to the generated forward path; unshown hazards and exact 3D motion remain unresolved, and the apparent clearance does not guarantee a real-world outcome.The SUV passes close to the plant, so the apparent lack of contact depends on the generated path.Source: C. Nothing will happen · p. 30

Generated motion assessment

  • In the generated sequence, the headlight-marked front leads the SUV past the plant and gray vase without visible contact.The camera viewpoint and floor scene remain stable as the SUV advances.Source: C. Nothing will happen · p. 30

B. No

The examples use generated video to assess a bathroom transition and a toy-truck intervention. The bathroom view change does not establish travel direction, and the truck’s full trajectory remains unresolved after it leaves the visible area.

B. No

  • The generated bathroom transition is continuous, but its rightward view reorientation cannot distinguish lateral travel from camera rotation or forward/backward movement.The towel rack leaves the upper-left as the toilet and bathtub come into view; the generated frames do not establish the camera’s signed direction of travel.Source: B. No · pp. 32–33
  • The intervention request specifies removing only the white oval vase while preserving the room and other objects, then showing the truck moving forward.The verifier focuses on whether the truck contacts the remaining gray vase or another object.Source: B. No · pp. 35–36
  • After the white oval vase is removed, the truck passes left of the gray vase without visible contact, so no additional removal is indicated.The truck leaves the visible floor area, leaving its complete trajectory and any later collision unresolved.Source: B. No · p. 36

Related papers

Loading 2610.21910v1…