PaperlayerView exact revision on arXiv

Source-linked AI summary

$\Psi$-0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model

Khai Loong Aw, Klemen Kotar, Baihan Zhang, Yifan Liu, Lilian Chen, Atlas Kazemian, Rahul Venkatesh, Daniel Yamins

arXiv:2610.32177v1cs.AIcs.CVcs.LG

TL;DR

Physical-world AI capabilities often remain fragmented across specialized models, making it difficult to compose and jointly adapt them. Ψ-0.6 unifies perception, language, geometry, and simulation in streams of thought, and reports zero-shot spatial-reasoning gains alongside task-feedback improvements that transfer across unseen benchmarks.

  • Problem

    Physical-world AI capabilities often remain split across specialized models, while unified multimodal models and systems that invoke external tools lack a shared reasoning process that jointly optimizes predictions and their use.

    Source: §1 · p. 1

  • Method

    Ψ-0.6 uses a shared token interface to compose images, text, geometry, and motion, and applies reinforcement learning to adapt intermediate-representation choices from task feedback.

    Source: §1 · p. 3

  • Results

    Ψ-0.6 outperforms leading open-weight models zero-shot on MindCube, with further gains from geometric context and composed reasoning, while reinforcement-learning gains transfer across four unseen benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results demonstrate that task feedback can shape both what a model predicts and how it reasons when perception, geometry, language, and imagination are unified.

    Source: §1 · p. 3

    Ψ-0.6 was developed at substantially smaller training-data, compute, research, and engineering scale than contemporary frontier multimodal systems, so the paper does not establish the framework’s performance at frontier scale.

    Source: §7.2 · p. 15

Abstract

from arXiv · show

Humans reason about the physical world by flexibly combining different forms of inference as a problem unfolds. Achieving this flexibility is a major open challenge in AI: capabilities often remain separated across specialized models, limiting how they can be composed and jointly adapted to a task. We introduce $\Psi$-0.6, a unified vision-language world model that adaptively composes perception, language, geometry, and simulation within a shared reasoning process we call streams of thought. These streams interleave images, text, and explicit representations of geometry and motion, all of which the model can generate and interpret through a shared token interface. $\Psi$-0.6 surpasses leading open-weight models zero-shot on multi-view spatial reasoning, with further gains from reasoning through geometric intermediates. It also supports counterfactual simulation: generating and interpreting imagined scenes to determine, for example, whether moving an object would obstruct a pathway. Reinforcement learning from task feedback jointly optimizes the selection, generation, and use of intermediate representations, improving visual question-answering accuracy with gains that transfer to unseen benchmarks. $\Psi$-0.6 offers a path toward adaptive physical reasoning -- not only predicting the physical world but also choosing the representations through which to reason.

1 Introduction

Ψ-0.6 unifies perception, language, geometry, and simulation in a shared reasoning process, then learns to select and use intermediate representations from task feedback. It reports gains in spatial reasoning, counterfactual simulation, and visual question answering.

Motivation

  • Current physical AI capabilities often remain fragmented across specialized models, while unified models and tool-using systems do not integrate or jointly optimize them in a shared reasoning process.The paper frames its question as how an AI system can learn which representations are useful for physical-world problems.Source: §1 · p. 1

Unified model

  • The model represents images, text, camera pose, depth, optical flow, and point tracks as tokens in a shared transformer, allowing one representation’s output to condition the next.An image pair can prompt a camera-pose estimate, while an image and camera pose can prompt a new view.Source: §1 · p. 3

Streams of thought

  • Streams of thought interleave images and text with geometry and motion, composing perception, language, and imagination within one reasoning process.For counterfactual questions, the model can translate a specified displacement into optical flow, generate the resulting scene, and answer using its own prediction.Source: §1 · p. 3

Adaptive reasoning

  • Task feedback can jointly shape which intermediate representations the model generates and how it uses them, because those representations are actions of the same policy.Unlike systems that invoke fixed external tools, Ψ-0.6 generates intermediates within the reasoner itself.Source: §1 · p. 3

Results

  • Ψ-0.6 outperforms leading open-weight models zero-shot on MindCube, gains from geometric context and composed reasoning, and transfers reinforcement-learning gains across four unseen benchmarks.It also surpasses dedicated expert models on FlowBench with specified counterfactual motion and transfers to real robot data after training only on simulation examples.Source: §1 · p. 3

B Fine-grained control

The available passage fragments indicate a patch-editing operation that moves a pointer while preserving content.

Fine-grained control

Q A question answering

Figure 2 presents Ψ-0.6’s multimodal architecture, prompting interface, and patch-level control. It also highlights interactive view generation and probabilistic predictions.

Prompting interface

  • Prompts specify available information and the requested inference pathway; a generated view can then serve as context for subsequent reasoning.Source: Figure 2 · p. 4

Fine-grained control

  • Each patch uses pointer and content tokens, with pointers specifying modality and spatial and temporal locations for targeted conditioning and scene edits.Source: Figure 2 · p. 4

Probabilistic prediction

  • The model represents compatible continuations probabilistically, so sampling can yield alternative predictions from the same prompt.Source: Figure 2 · p. 4

2 A Unified Model with Adaptive Streams of Thought

Ψ-0.6 combines a shared multimodal model and flexible prompting with reinforcement learning that adapts reasoning strategies from task feedback. Its training stages build predictive capabilities, align them with language, and teach task-directed use.

2 A Unified Model with Adaptive Streams of Thought

  • Ψ-0.6 extends a unified-model, generic-prompting, and reinforcement-learning recipe to visual, geometric, and linguistic capabilities and task-directed reasoning.Source: §2 · p. 4

2.1 Unified Multimodal Model

  • Images, camera pose, depth, optical flow, point tracks, and text are tokenized for one 8B-parameter autoregressive transformer, making perception, description, and simulation conditional predictions.Source: §2.1 · p. 5

2.2 Generic Prompting Interface: Streams of Thought

  • Prompts specify inputs and requested outputs, enabling different token orderings to express conditional predictions and compose them into multistep reasoning.A generated view can become context for a spatial answer; fine-grained patch control and probabilistic prediction further support flexible prompting.Source: §2.2 · p. 5

2.3 Reinforcement Learning: Adaptive Stream-of-Thought Reasoning

  • Reinforcement learning treats representation choices as policy actions, rewarding selection of visual intermediates such as depth, camera motion, and optical flow according to downstream answer quality.Source: §2.3 · pp. 5–6

2.4 Training Recipe: Pretraining, Mid-training, and Post-training

  • Training progresses from video-based visual and geometric prediction through language and vision-language alignment to instruction tuning for user-directed reasoning and task-feedback-based adaptation.Pretraining mixes RGB-only and interleaved geometric sequences; post-training retains earlier sequence types while adding spatial, counterfactual, and question-answering tasks.Source: §2.4 · p. 6

3 Related Work

Prior unified multimodal models and tool-augmented systems support compositional reasoning, but Ψ-0.6 integrates vision-language reasoning with physical world modeling through a shared token interface.

3 Related Work

  • Ψ-0.6 extends unified multimodal reasoning by composing text, images, camera pose, depth, optical flow, and point tracks as intermediate steps.Generated representations can be recursively consumed, and task feedback can shape which representations the model selects.Source: §3 · pp. 6, 8
  • Figure 3 contrasts six related-model classes, tool-using frontier assistants, and Ψ-0.6’s unified pathways.Each pathway in panel C contrasts with a class in panel A.Source: Figure 3 · p. 7
  • Unlike systems that delegate intermediate operations to external tools, Ψ-0.6 generates them through the reasoner’s shared interface.This makes representation choice and content part of the model’s computation and enables end-to-end optimization.Source: §3 · p. 8

4 Capability and Compositional Unification: Spatial Intelligence

Ψ-0.6 combines a visual world-model backbone with language and explicit geometric representations to compose spatial reasoning operations. On MINDCUBE, it performs strongly zero-shot and gains further accuracy from compositional prompting and fine-tuning.

4 Capability and Compositional Unification: Spatial Intelligence

  • A shared transformer interleaves RGB, camera pose, depth, optical flow, point tracks, and text, enabling outputs such as pose estimates or synthesized views to condition later reasoning.The paper proposes visual world modeling as a foundation for spatial reasoning, with language supplying an interface to these capabilities.Source: §4 · pp. 8–9
  • Frozen Ψ-0.6 reaches 49.7–51.3% across RGB-only, RGB-depth, and RGB-camera configurations and is state-of-the-art among the compared open-source generalist VLMs and spatial specialists.MINDCUBE tests questions involving counterfactual camera motions and large viewpoint changes.Source: §4 · p. 9
  • Inverse-dynamics prompting reaches 57.7%, compared with 51.3% zero-shot, by predicting relative camera poses, selecting a matching input view, and reading its answer.Novel view synthesis instead generates an RGB frame at the target pose, including viewpoints absent from the inputs.Source: §4 · p. 9
  • 76.3% accuracy with RGB+camera inputs on the held-out Tiny test split exceeds InternVL3’s map-then-reason and Qwen2.5-VL with SFT+RL, is comparable to 3DThinker with SFT+RL (76.0%), and trails GeoThinker (83.0%).The RGB-only result is 68.7% and exceeds the displayed SFT-only Qwen2.5-VL and 3DThinker baselines.Source: §4 · p. 9

5 From Compositions to Counterfactuals

Ψ-0.6 uses world modeling to reason about hypothetical object motions, predicting their consequences and answering from the imagined scene. On FlowBench, it shows strong flow prediction, improved question answering, and transfer to real data not used in training.

5 From Compositions to Counterfactuals

  • FlowBench questions ask about consequences of specified object displacements, such as occlusion, relative distance, contact, or whether an object can move further.The model intervenes on a scene, predicts the consequence, and reads the answer from its prediction.Source: §5 · pp. 9–10, Figure 5 · p. 10
  • For text-to-flow prediction, Ψ-0.6 estimates motion probability per patch and decodes likely-moving patches first, then uses classifier-free guidance to amplify the text instruction’s effect.The model then predicts the future frame and reasons to an answer.Source: §5 · pp. 10–11, Figure 5 · p. 10
  • 56.4% accuracy for Ψ-0.6’s best configuration exceeds 16.4% for the strongest image-editing baseline; its rEPEmove is 0.454 versus 0.915 for FOFPred.Every evaluated Ψ-0.6 configuration improves moving-region next-frame fidelity over the zero-motion reference by +0.25 to +1.60 dB, while image editors decrease it by −0.96 to −4.00 dB.Source: §5 · p. 11, Table 1 · p. 11
  • Fine-tuning on FLOWBENCH-SIM raises Ψ-0.6’s overall accuracy from 24.5% to 73.1% and category-averaged accuracy from 24.6% to 73.6%.The benchmark evaluates questions about specified counterfactual object motions across simulated and real-data splits.Source: §5 · pp. 10–11, Table 2 · p. 11
  • On FLOWBENCH-REAL, Ψ-0.6 reaches 51.4% overall and 48.9% category-averaged accuracy, gains of 13.9 and 11.9 percentage points over the base model.No FLOWBENCH-REAL examples are used in training; “Closer” rises from 23.3% to 63.0%, and “Near” reaches 74.6%.Source: §5 · p. 12, Table 3 · p. 12

6 Optimization Unification: RL for Adaptive Streams of Thought

Ψ-0.6 learns streams of thought that are self-generated, representation-adaptive, and unfold over time, using task feedback to optimize intermediate choices and generation end to end.

6 Optimization Unification: RL for Adaptive Streams of Thought

  • The same policy chooses and generates intermediate representations, allowing reinforcement learning on task feedback to optimize the reasoning process end to end.The figure describes scoring notations by ground-truth answer probability relative to RGB-only and training on beneficial notations.Source: §6 · p. 12, Figure 6 · p. 13
  • Because intermediates are generated within the reasoner, they can co-adapt through gradient-based credit assignment and include future or counterfactual states unavailable through existing tool calls.Task-dependent intermediate forms may themselves be ambiguous.Source: §6 · p. 12
  • 5.6% mean relative gain over video-only question answering on held-out training-distribution examples, and 9.0% mean relative gain across four unseen benchmarks.The learned policy also shows higher best-of-k than single-rollout performance.Source: Figure 6 · p. 13, §6 · p. 14
  • Representation-adaptive computation makes the representational medium itself part of test-time computation, with depth, optical flow, camera pose, or predicted scenes invoked as situations evolve.The paper identifies this unfolding computation as particularly relevant to embodied agents and robotics.Source: §6 · p. 13
  • For visual question-answering, the policy emits notations selecting depth, camera-motion, and optical-flow streams, with sampled notations scored by ground-truth loss reduction.The training data combines videos from EgoDex, SSv2, PE, and OpenVid-1M, whose captions are converted into question-answer pairs.Source: §6 · p. 14
  • Training first uses filtered behavior cloning on notations that outperform RGB-only, then on-policy GRPO refines stream selection using rewards relative to group samples.GRPO updates the policy’s LoRA parameters.Source: §6 · p. 14

7 Discussion

Ψ-0.6 makes visual, geometric, linguistic, and world-model predictions composable reasoning operations. The discussion connects geometric intermediates to physical reasoning and learning, while identifying limits and open directions for the framework.

7.1 Implications for physical reasoning and learning

  • Geometric intermediates can decompose visual problems, focus learning on physical factors, and provide compact, interpretable tokens grounded in recognizable properties.Multiple aligned representations from one video may improve sample efficiency when data quantity and diversity are limited.Source: §7.1 · p. 15

7.2 Limitations

  • The framework’s results establish unification’s conceptual and functional consequences, not its performance ceiling; Ψ-0.6 used substantially fewer training resources than frontier multimodal systems.The authors report preliminary gains with scale and expect further scaling to strengthen capabilities.Source: §7.2 · p. 15
  • The paper does not isolate positive or negative transfer among capabilities during training, despite showing functional unification and inference-time gains from composition.A sparse mixture-of-experts architecture is proposed as a next step for managing specialization and model capacity.Source: §7.2 · p. 16

7.3 Key open problems

  • Adaptive-stream experiments optimize visual question answering, not physical control; extending the framework to embodied agents would require action tokens and rewards that jointly shape action and reasoning.The proposed action modalities include robot joint commands, end-effector poses, and gripper states.Source: §7.3 · p. 16
  • The discussion proposes physical-consistency signals as a route to generic verifiers that could score intermediate reasoning without human labels for every trajectory.Examples include checking whether inferred camera motion generates a compatible second view.Source: §7.3 · p. 16
  • Ψ-0.6 uses deliberately chosen abstractions; future systems could propose, evaluate, and retain useful intermediate representations, learning the vocabulary of reasoning itself.The proposed loop evaluates abstractions by predictive consistency or downstream task performance.Source: §7.3 · pp. 16–17

A.1 Spatial Intelligence Reasoning: MindCube

  • Structured prompting improves the displayed MindCube Rotation and Among results most, while direct RGB+depth prompting is strongest on Around and near chance on Rotation.The SFT configuration is strong on Around and Among but remains near chance on Rotation; the authors hypothesize limited scene diversity in Rotation training data impedes generalization.Source: §A.1 · p. 25

A.2 Details of FlowBench

  • FlowBench evaluates answers to specified counterfactual-motion questions across simulated indoor scenes and real robot datasets spanning five embodiments.Its questions include whether an object’s movement would occlude a view or change which objects are closest.Source: §A.2 · p. 25

A.4 Definition of metrics for flow prediction

FLOWBENCH VQA fine-tuning raises Ψ-0.6’s overall accuracy from 24.5% to 73.1%. The flow metrics focus on moving pixels and normalize errors by true motion so background pixels do not dominate evaluation.

A.4 Definition of metrics for flow prediction

  • rEPE averages each moving pixel’s endpoint error relative to its own motion distance, excluding static pixels that cover most of each frame.Moving pixels cover approximately 0.6% of the frame; their displacements average 100.2 px and have a median of 83 px.Source: §A.4 · p. 26
  • Whole-frame endpoint error is dominated by static background, while absolute error on moving pixels conflates mistakes across different motion magnitudes.The relative moving-pixel error is reported as the primary quantity.Source: §A.4 · p. 26
  • Acc counts the fraction of samples whose relative error falls below tolerance t, with correctness defined as average moved-object error within 40% of true displacement.The tolerance scales with motion magnitude; a blank prediction scores rEPE 1 and is never counted as correct.Source: §A.4 · pp. 26–27

A.5 VQA on FLOWBENCH.

  • 24.5% to 73.1%: FLOWBENCH fine-tuning raises Ψ-0.6’s overall VQA accuracy, yielding the second-highest overall accuracy among evaluated models.It trails Qwen3-VL-8B fine-tuned on the same dataset by 3.2 points.Source: §A.5 · p. 27

A.6 Flow prediction ablation

The flow-prediction ablation favors pMotion with moderate classifier-free guidance (CFG): it improves accuracy and flow metrics without longer decoding, while stronger guidance trades reconstruction gains for flow improvements.

A.6 Flow prediction ablation

  • pMotion orders flow generation from patches most likely to move, while CFG amplifies the difference between text-conditioned and reference logits.These decoding techniques use Ψ-0.6’s fine-grained control and probabilistic predictions.Source: §A.6 · p. 27
  • 31.8% to 44.2%: pMotion raises accuracy and reduces directional error from 14.1° to 6.6°; CFG at scale 1.5 further raises accuracy to 54.0%.At CFG 1.5, rEPEmove is 0.463 and blank rate is 5.6%.Source: §A.6 · p. 27, Table 4 · p. 27
  • CFG scale 2.5 eliminates blank predictions but reduces reconstruction gains from +1.49 to +0.25 dB, despite further flow-metric improvements.Decoding beyond 32 tokens yields no further improvement, likely because FlowBench motions are sparse and involve a single object.Source: §A.6 · p. 27, Table 4 · p. 27

A.7 Question–answer data for stream selection

The appendix describes how stream-selection training data are constructed, outlines PSI’s patch-based and probabilistic prompting properties, and contrasts Ψ-0.6’s unified imagination with systems that separate reasoning from generation.

A.7 Question–answer data for stream selection

  • Stream-selection training uses videos from EgoDex, SSv2, PE, and OpenVid, converting their captions or labels into open-ended question–answer pairs.Formulaic labels and task descriptions are converted deterministically; longer PE and OpenVid captions are rephrased into one to three pairs without video input.Source: §A.7 · p. 28
  • Transfer is evaluated on four unseen multiple-choice video QA benchmarks, with stream selection based on video and question alone, not answer options.The benchmarks are HanDyVQA, TGIF-QA, STAR, and EgoPlan-Bench2.Source: §A.7 · p. 28

B.1 Preliminaries on Probabilistic Structure Integration (PSI)

  • PSI gives Ψ-0.6 fine-grained patch control and probabilistic prediction, so prompts can target or relocate regions and partially specified scenes can yield alternatives.Patch content and location are represented separately, and prompts constrain rather than prescribe a single deterministic target.Source: §B.1 · p. 29

B.2 Supplementary Discussion

  • Unlike approaches that couple a reasoner to a separate visual generator or world model, Ψ-0.6 generates and interprets imagined states within one model and token interface.The cited approaches include Mindjourney, SandboxVLM, Thinking with Novel Views [86], APC [87], and AVIC [88].Source: §B.2 · p. 29

C Model and Training Details for Reproducibility

The appendix specifies Ψ-0.6’s shared token interface, sequence-based inference and training, visual and geometric representations, and adaptive stream-selection procedure. It also documents data mixtures, optimization, and context-length extension for reproducibility.

C.1.1 A common token and position interface; C.1.3 Text representation; C.1.4 Transformer architecture

  • One token sequence represents images, depth, flow, camera pose, and text, using spatial-temporal coordinates and modality channels; pointers locate visual patches and condition their content.The decoder-only transformer has approximately 8.05 billion parameters, and the appendix summarizes its token representation and architecture.Source: §C.1.1 · p. 30, Table 5 · p. 30, §C.1.4 · p. 32

C.1.2 Visual quantization and geometric labels; C.1.3 Text representation

  • Visual quantizers encode RGB, depth, and flow patches, while preprocessing estimates depth and camera geometry and computes optical flow and discretized relative poses.Text uses byte-pair encoding with pointers; the appendix also describes quantizer training and geometric labels.Source: §C.1.2 · p. 31

C.2.1 Notation and sequence construction

  • Sequence notation orders supplied context and requested outputs, allowing generated elements to condition later predictions; training samples notations, allocates tokens, and masks conditioning elements from loss.The appendix gives examples for geometric video modeling, captioning, and question answering with depth.Source: §C.2.1 · pp. 32–33

C.2.2 Generating and composing elements

  • Generation fills content slots in modality-specific templates, then may decode a sequential patch prefix and the remaining patches in parallel to reduce sequential steps.Inference also exposes sampling settings, classifier-free guidance, and motion-based patch ordering as controls.Source: §C.2.2 · p. 33

C.3 Training Adaptive Streams-of-Thought

  • An adaptive policy selects and orders depth, camera-motion, and flow representations from a video and question; an answer scorer’s loss supplies its training signal.Initialization scores candidate notations against the reference answer, and subsequent on-policy optimization rewards lower answer loss relative to RGB-only answering.Source: §C.3 · p. 34

C.4.1 Supervised objective and sampling; C.4.2 Training stages and data mixtures

  • Supervised training samples weighted dataset groups and applies next-token cross-entropy, with instruction training giving each sequence equal weight despite different supervised-token counts.The stages progress from RGB prediction through geometry, language and caption alignment to instruction training, while retaining earlier predictive tasks.Source: §C.4.1 · pp. 34–35, §C.4.2 · p. 35

C.4.2 Training stages and data mixtures; C.4.3 Data sources and text construction; C.4.4 Optimization and infrastructure; C.4.5 Context-length extension

  • The final instruction mixture raises video-question sampling from approximately 5% to 15% and lowers local-caption sampling from approximately 15% to 10%, alongside documented video, caption, question-answer data construction and supervised-training settings.Source: §C.4.2 · p. 35, §C.4.3 · pp. 35–36, §C.4.4 · p. 37

C.4.5 Context-length extension; C.5 Technical Glossary and Hyperparameter Reference

  • The appendix’s technical glossaries consolidate architecture, representation, optimization, decoding, and adaptive-policy settings for the implementation.The context-extension phase retains the architecture, tokenizers, position representation, and next-token objective while accommodating longer streams.Source: §C.4.5 · p. 37, §C.5 · p. 38, Table 9 · p. 38

Related papers

Loading 2610.32177v1…