Source-linked AI summary
Flow-of-Thought: A Framework for Visual Reasoning
Mariia Baidachna, Nicolas Pugeault
TL;DR
Foundation models still struggle with spatial reasoning that requires tracking continuous geometric changes. Flow-of-Thought generates inspectable visual trajectories and compares competing hypotheses, reaching strong locked rotation results and improving over an endpoint-only control on BLINK Multi-view.
Problem
Foundation and multimodal models struggle with spatial tasks requiring continuous geometry and step-by-step tracking of visual changes.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
FoT’s intermediate images make endpoint evidence inspectable and expose errors in geometric structure, branch reconstruction, or maze-path timing.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
Mental imagery, ``seeing with the mind's eye'' is an essential aspect of human cognition. Despite rapid progress Large Language Models (LLMs) and Vision Transformers (ViTs) still underperform on tasks requiring spatial understanding. To address this, we introduce Flow-of-Thought (FoT), a framework that integrates the generation of visual sketches as intermediate reasoning steps, mimicking mental imagery in humans. We train coordinate-aware trajectory flow fields on $SO(2)$ group orbits and cumulative shortest paths, then freeze the learned dynamics; same vs. different decisions compare competing generative hypotheses using foreground-weighted reconstruction energy. On locked tests FoT reaches 100.0% accuracy on Tetris and 99.0% on colored shapes. Under frozen transfer, the orbit-trained 2D flow improves over its endpoint-only control on BLINK Multi-view (72.2% vs. 63.9% on 133 public validation pairs), supporting continuous visual traces as an effective and interpretable representation for spatial reasoning in some out-of-distribution settings.
1 Introduction
Foundation models and multimodal systems still struggle with spatial reasoning that humans can solve by mentally simulating transformations or using intermediate visual traces. Flow-of-Thought (FoT) models continuous visual dynamics, achieving strong locked rotation results and a positive but qualified BLINK transfer result.
1 Introduction
- Humans can mentally simulate incremental rotations, while intermediate path sketches can make maze navigation easier to solve.The introduction frames mental imagery as an internal visual aid for spatial reasoning.Source: §1 · p. 1
- Text-based reasoning struggles with visual tasks because discrete one-dimensional representations do not capture continuous geometry, topology, and spatial relationships.The passages describe failures in mental rotation and maze tasks, including spatial hallucinations or logical collapse in multimodal models.Source: §1 · pp. 1–2
- FoT learns continuous visual dynamics from trajectory supervision and uses competing generated hypotheses to decide same-versus-different rotation cases.A coordinate-, time-, and action-conditioned spatial U-Net predicts dense transport fields; frozen trajectories generate the endpoint evidence used for decisions.Source: §1 · p. 2
- 100.0% accuracy on Tetris and 99.0% on colored shapes; BLINK Multi-view reached 72.2% versus 63.9% for the endpoint-only control on 133 public validation pairs.Maze traces achieved 97.5% endpoint IoU and 100% goal reach, though later segments sometimes activated prematurely; FoT did not exceed the endpoint-only control on 3D Blocks.Source: §1 · p. 2
2 Related Work
Related work connects intermediate reasoning, visual sketching, and cognitive accounts of spatial imagery. FoT draws on these strands to motivate reasoning through evolving visual states.
Chain-of-Thought (CoT) and iterative reasoning
- Diffusion of Thought integrates reasoning into diffusion models, refining intermediate steps globally over time rather than using only sequential tokens.The related-work discussion also cites Chain-of-Thought models [DeepSeek-AI, 2025, OpenAI, 2024] and AlphaGeometry as structured-reasoning approaches.Source: §2 · pp. 2–3
Visual Reasoning and Chain-of-Sketch
- Visual Sketchpad [Hu et al., 2024] and Inductive CoS [Lotfi et al., 2024] use generated visual aids or frame-by-frame traces for geometric and high-globality tasks such as mazes.The discussion notes that models may memorize puzzle heuristics rather than adapt dynamically [Shojaee et al., 2025].Source: §2 · p. 3
Cognitive Neuroscience of Spatial Logic
- Mental-rotation experiments support continuous internal simulation, while spatial navigation has been linked to continuous mental scanning [Shepard and Metzler, 1971; Kosslyn et al., 1978].FoT’s continuous ODE transition operator is presented as modeling this psychological process.Source: §2 · p. 3
3 Preliminaries
The preliminaries cast visual reasoning as continuous dynamics over an initial image and task condition. Supervision follows task-specific visual trajectories, and the inference adapter supplies hypotheses and allowed actions.
3 Preliminaries
- A visual-reasoning sample contains an input condition, an initial visual state, and either a target trajectory or a final label.The framework formalizes datasets as D with samples containing c, x0, and either γ(t) or o.Source: §3 · p. 3
- Rotation supervision follows rendered SO(2) orbits, while maze supervision follows cumulative shortest-path prefixes, exposing intermediate states rather than only final predictions.The field is conditioned on a task action, such as a requested rotation.Source: §3 · p. 3
- The deterministic adapter maps each item to its initial state, condition, candidate hypotheses, and allowed actions; rotation uses same and reflected-source hypotheses.Figure 1 separates learning from frozen inference.Source: §3 · p. 3
4 The Flow-of-Thought Framework
FoT learns a coordinate-aware visual flow field for rotation and maze trajectories, then freezes it for inference. Dataset-specific adapters supply hypotheses and actions; decisions compare their integrated endpoints.
4.1 Trajectory Flow Field
- A coordinate-aware, time- and action-conditioned spatial U-Net predicts transport fields; training also uses group normalization and SiLU activations.For rotation, the target is a tangent field along rendered rotation orbits; maze dynamics use an additive field over cumulative path masks.Source: §4.1 · pp. 3, 5
- For rotation, single-source rendering samples each requested state from the original image, avoiding blur from repeatedly resampling intermediate images.The learned field is integrated with 12 fixed Heun steps, using a smooth ℓ1 loss between predicted and target fields.Source: §4.1 · p. 5
- Maze training uses cumulative shortest-path prefixes as targets, with foreground-balanced tangent regression, differentiable multi-step binary-cross-entropy, and Dice losses.Eight Heun steps produce the final path and intermediate trace; the map encodes walls, start, and goal.Source: §4.1 · p. 5
4.2 Frozen Hypothesis Competition
- Frozen inference integrates each hypothesis/action under the same field; rotation branches minimize endpoint distance over a predeclared 36-angle set and predict SAME when Esame ≤ Ediff.The 2.5° offset keeps candidates off both manifest angle grids; no pair labels, fitted threshold, target-task adaptation, or analytic image transformation enters the decision.Source: §4.2 · p. 5
5 Data and Experiments
The experiments cover locked synthetic rotation decisions, maze navigation, and frozen transfer to 3D Blocks and BLINK Multi-view. The supplied passages describe dataset construction and evaluation setup.
5 Data and Experiments
- The evaluation tests the framework on representative rotation tasks in Figure 2 and on maze navigation; training details are provided in Appendix A.1.Source: §5 · p. 6
- The locked Tetris test has 100 pairs asking whether each target is a rotation of the source or its mirror.The flow is trained on J/L/S/Z sources and selected on held-out F/P shapes, with random initial pose, reflection, and continuous angular displacement.Source: §5 · p. 6
- The colored-shapes decision set contains 100 locked pairs and tests multi-object compositional rotation with additional RGB channels.Training and validation use disjoint deterministic procedural samples.Source: §5 · p. 6
- Frozen transfer applies the 2D colored-shape flow without target-task training to 78 3D Blocks pairs and 133 public BLINK Multi-view pairs.The 3D branches use the supplied signed angle pair; BLINK branches minimize over positive or negative object rotations on a fixed 10° grid up to 170°. BLINK is exploratory because its sign convention was finalized after a two-item smoke check.Source: §5 · p. 6
- Maze evaluation uses perfect 9 × 9 cell mazes generated by depth-first backtracking, with breadth-first search supplying the shortest path.The learning target is a sequence of cumulative path masks from the fixed start to the goal, resized to 64 × 64.Source: §5 · p. 6
6 Results and Discussion
FoT scores highly on locked synthetic rotation tasks and shows suggestive BLINK transfer; maze paths are accurate but can activate future segments prematurely. Qualitative audits show identity-preserving rotations, contrasting with viewpoint-conditioned morphing on 3D Blocks.
6 Results and Discussion
- FoT scores 100.0% on Tetris and 99.0% on colored shapes; BLINK reaches 72.2% [64.0, 79.1] versus endpoint-only 63.9% [55.5, 71.6], an 8.3-point suggestive gain.The BLINK intervals overlap, so the improvement is suggestive rather than conclusive.Source: Table 1 · p. 7, §6 · p. 7
- Maze endpoint IoU is 97.5% [94.6, 100.0], prefix IoU 84.2% [82.2, 86.3], goal reach 100%, and obstacle violations zero, but premature activation is 31.7%.Mean future-path intensity is 0.164; strong endpoint and prefix metrics do not imply perfectly causal drawing.Source: §6 · p. 7
6.1 Qualitative Analysis
- Synthetic-task intermediate frames preserve object identity during rotation, whereas 3D Blocks often shows viewpoint-conditioned morphing.This qualitative 3D behavior is consistent with weak, unstable transfer and a stronger endpoint-only control.Source: §6.1 · p. 7
- On held-out mazes, predicted endpoints are nearly exact, but later path segments activate faintly too early.Figure 5 aligns ground-truth and predicted states at matched times, making the timing limitation inspectable.Source: §6.1 · p. 7, Figure 5 · p. 8
- Figure 3 reports 95% confidence intervals, but overlapping intervals are not pairwise significance tests; maze is omitted because its dense trajectory metrics are not accuracy-equivalent.Open markers indicate published point estimates without compatible reported uncertainty.Source: Figure 3 · p. 8
6.2 Interpretability
- Intermediate images expose whether rotation failures involve lost structure or implausible branch reconstruction, and whether maze errors concern wall avoidance, endpoint completion, or path timing.These traces are tied directly to endpoint evidence.Source: §6.2 · p. 9
7 Conclusion and Future Work
FoT uses trajectory-supervised transport fields as an interpretable scratchpad and frozen hypothesis competition to make decisions. The authors report success on locked synthetic rotation tasks and positive BLINK transfer, while identifying limits in 3D and maze timing.
7 Conclusion and Future Work
- FoT combines trajectory-supervised transport fields with frozen hypothesis competition over generated endpoints.The paper presents the transport field as an interpretable mental scratchpad.Source: §7 · p. 9
- The framework solves locked synthetic rotation tasks and transfers positively to BLINK, identified as the primary OOD evidence.The paper says the small, seed-sensitive 3D evaluation does not yet substantiate the trajectory claim.Source: §7 · p. 9
- Future work targets physically faithful 3D dynamics, stricter causal path generation, and integration as an external tool for multimodal foundation models.Source: §7 · p. 9
A.1 Task-Specific Formulations
The appendix specifies training and inference formulations for rotation and maze trajectories, alongside the evaluation protocols for synthetic and transfer decisions.
A.1 Task-Specific Formulations
- Primary models use three independent seeds, 3,000 training trajectories per seed, and 400 fixed validation trajectories for 30 epochs.Checkpoints are selected by validation silhouette IoU for rotation and endpoint path IoU for maze.Source: §A.1 · p. 11
- Rotation models train on randomly pre-rotated sources, with optional mirroring, and rendered SO(2) orbits supervised from a common base image.Inference uses Heun integration and a backward sampling map to render requested frames from the original source.Source: §A.1 · p. 11
- Maze trajectories use a cumulative trace state conditioned on walls, start, and goal, with added-path pixels weighted 20 and foreground pixels weighted 3.Every second training batch also receives a multi-step binary-cross-entropy plus soft-Dice loss, weighted by 2, over predicted prefixes.Source: §A.1 · p. 11
- Tetris and colored-shape decisions use locked 100-item sets and a normalized reconstruction-energy margin, while 3D Blocks and BLINK use 78 and 133 items, respectively.Wilson intervals quantify item-level uncertainty, and seed-level geometric metrics use two-sided Student-t intervals across three runs.Source: §A.1 · p. 11
A.2 Ablation Studies
The ablations compare path supervision with matched controls and test frozen transfer where intermediate states are unavailable. Results favor orbit supervision on BLINK, but not on 3D Blocks, where the comparison is unstable.
A.2 Ablation Studies
- Matched rotation-orbit supervision improves decisions and geometric endpoint quality over the endpoint-only and linear controls.The ablation holds source examples, architecture family, optimizer schedule, update count, seeds, frozen reconstruction decision, and action grid fixed.Source: §A.2 · p. 11
- 72.2% vs. 63.9% on BLINK favors orbit flow over endpoint-only, while 61.5% vs. 66.7% on 3D Blocks shows no advantage for orbit flow.The BLINK gain is 8.3 points; its intervals overlap, while the 3D comparison is unstable at n=78 and measured too coarsely to quantify the deficit.Source: §A.2 · pp. 11, 13
- Figure 6 shows three-seed distributions with raw points, seed means, and 95% intervals; accuracy panels also show seed-ensemble decisions with Wilson intervals.Source: Figure 6 · p. 12
- The frozen consistency audit describes the model as approximately group-consistent, although equivariance is not built into its architecture.Table 3 reports identity, inverse/cycle, composition, and non-grid endpoint defects, with lower defects better.Source: Table 3 · p. 13, Figure 7 · p. 14