PaperlayerView exact revision on arXiv

Source-linked AI summary

MindTopo: Can Foundation Models Reason in Topological Space?

Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Jianwen Lyu, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li

arXiv:2609.11900v1cs.AIcs.CLcs.CV

TL;DR

Foundation-model evaluations largely underrepresent topological spatial relations, despite their cognitive importance. MINDTOPO benchmarks five such properties through reasoning and closed-loop planning across procedurally generated tasks, finding that models reason about topology better than they act on it and remain well below human performance.

  • Problem

    Spatial reasoning benchmarks largely emphasize metric or viewpoint-dependent relations, leaving broad evidence about foundation models’ topological intuition limited.

  • Method

    MINDTOPO evaluates continuity, separation, order, enclosure, and knots at reasoning and planning levels across 13 procedurally generated task types.

  • Results

    Every MLLM performs better on reasoning than planning; GPT-5.6-Sol reaches 61.42% overall versus 97.87% observed human performance.

  • Takeaways & Limitations

    Topological intuition remains a blind spot: models can recognize static relations but struggle to maintain or operate on them across action sequences.

  • Takeaways & Limitations

    Scenes are procedurally rendered and lack the visual variability of real-world photographs, while video-policy evaluation covers only three video generative models.

Abstract

from arXiv · show

Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots. MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We benchmark 14 MLLMs and study agent configurations augmented with image and video generation, including 3 video generative models in planning settings. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions. Our website is at https://mind-topo.github.io/

1. Introduction

MINDTOPO frames topological intuition as a distinct component of spatial reasoning, evaluating whether models can recognize and act on relations preserved under continuous deformation. Its benchmark spans five properties and two cognitive levels, revealing a persistent gap between static reasoning and action.

  • The benchmark addresses a gap in prior evaluations, which largely test metric or viewpoint-dependent relations rather than a broad range of topological relations and actions.
  • MINDTOPO evaluates continuity, separation, order, enclosure, and knots because these relations remain invariant under continuous deformation and are foundational to spatial understanding.
  • Reasoning asks models to identify or predict topological relations, whereas planning places them in closed-loop environments where actions build, preserve, or alter structures.
  • Across 14 MLLMs, every model performs better on reasoning than planning, and the best model remains far below observed human performance.
  • Training Qwen3-VL-2B-Instruct improves reasoning more than planning, while audited generated-video rollouts do not reliably follow dynamics or preserve topology across transitions.

2. MINDTOPO Benchmark

MINDTOPO formalizes topological properties as values of embedded structures that remain invariant under continuous deformation, then evaluates them through matched reasoning and closed-loop planning tasks. Its automated, procedurally controlled pipeline produces a large benchmark spanning diverse topological properties, task types, and difficulty levels.

  • 2.1. Problem Formulation: MINDTOPO defines a topological property as a value assigned to an embedded structure that remains unchanged under ambient isotopy.The benchmark treats continuous deformation as disallowing cutting, joining, or passing one part through another.
  • 2.1. Problem Formulation: Reasoning predicts relations from rendered observations, while planning evaluates closed-loop policies whose actions update simulator states until a goal predicate is satisfied.
  • 2.2. Topological Properties: The suite covers five properties through matched task pairs, including maze connectivity, separable assemblies, bead order, enclosure, and knot reasoning with corresponding planning environments.
  • 2.3. Data Collection and Statistics: The automated pipeline generates scenes, derives annotations and success conditions programmatically, controls topological difficulty, and converts accepted states into reasoning questions or planning episodes.
  • 2.3. Data Collection and Statistics: Procedural rendering and templated questions may introduce visual, language, or answer-prior shortcuts, motivating matched-text, answer-prior, appearance-only, and symbolic-input controls.
  • 2.3. Data Collection and Statistics: 11,030 instances span 13 task types, with 73% reasoning and 27% planning, while task-specific acceptance checks reject invalid, ambiguous, or unsolvable configurations.

3. Experiments

MINDTOPO shows that current MLLMs recognize topological relations more successfully in static reasoning than in interactive planning, with substantial variation across models, primitives, and environments. Training improves performance, especially on reasoning, while generated visual rollouts often preserve local cues without reliably maintaining valid topology across actions.

  • 3.2. Benchmark Results: 61.42% is the leading task-macro average, versus 97.87% for humans; GPT-5.6-Sol falls from 66.83% on reasoning to 52.75% on planning.Other models show similar reasoning–planning gaps, including Gemini-3.1-Pro at 52.24% versus 19.23%.
  • 3.2. Benchmark Results: The action gap widens across model tiers, with the best open-weight planning average reaching only 5.27% and every open-weight model scoring 0% on Pipe and One Stroke.The results indicate that open-source scaling does not close the planning bottleneck under this evaluation.
  • 3.2. Benchmark Results: No single model leads across topological primitives: GPT-5.6-Sol leads continuity, separation, order, and enclosure, while Gemini-3.1-Pro leads knots.Leadership can also differ between reasoning and planning within one property, as illustrated by Sheep and Chat Noir.
  • 3.2. Benchmark Results: Pipe, One Stroke, and Untangle remain difficult, with best scores of 50.50%, 35.00%, and 29.17%, respectively, across the 14 MLLMs.These environments require sequences of legal actions toward structural goals rather than single-scene relation identification.
  • 3.3. Training: 31.44% is the strongest nine-task average for SFT followed by GRPO, compared with 8.00% for the base policy and 18.69% for GRPO alone.The combined policy performs best on Knots, Bead, Sheep, and Untangle among the listed tasks.
  • 3.3. Training: Training raises average reasoning accuracy from 14.24% to 51.53% but planning success from only 0.20% to 6.33%; Pipe remains at 0% under every condition.Held-out transfer is selective: Bead and Knots improve, whereas 2D Maze, Assembly, and Sheep do not.
  • 3.4. Error Analysis: Reasoning errors are dominated by perception grounding, whereas planning errors concentrate in action planning, dynamics, and feature-state prediction.Across the audited models, 58.1% of reasoning errors are perception-grounding failures, while planning errors include 43.3% action planning and 24.5% dynamic violations.
  • 3.5. Probing Experiment: Image prediction averages 0% across One Stroke tiers and 5.71% on hard Untangle, showing that local visual cues do not solve trajectory-wide invariants.With GPT-5.6-Luna, the same condition reaches 68.6% on Sheep and 61.9% on 2D Maze.

4. Related Work

Prior work studies topological cognition, spatial interaction, and world modeling, but these lines of research address different structures, responses, and interaction settings.

  • Cognitive studies motivate treating topological relations as a foundation of spatial understanding.
  • Spatial benchmarks evaluate scene relations and multi-view integration, while interactive work studies exploration, camera-movement composition, puzzles, and simulation environments.
  • World-modeling research connects spatial evaluation with predicting action consequences and training agents across multiple turns.

5. Conclusion and Limitations

MINDTOPO benchmarks topological intuition across five properties and two cognitive levels, finding that models recognize static relations more readily than they maintain them during action. The benchmark is procedurally controlled but limited in visual realism, model coverage, topological scope, and audit size.

  • Conclusion: MINDTOPO covers five topological properties across reasoning and planning, with 11,030 instances spanning 13 procedurally generated task types.
  • Benchmark scope: The benchmark’s five properties are continuity, separation, order, enclosure, and knots, grounded in cognitive-developmental and formal-topological accounts.
  • Conclusion: Across 14 MLLMs, static recognition exceeds action performance, while video-policy rollouts can reach plausible endpoints without preserving valid topological transitions.
  • Limitations: The benchmark uses procedurally rendered scenes and only three video generative models, excludes several topology-flavored relations, and audits a limited sample of generated outputs.
  • Mathematical foundations: The formalization models scenes as embedded structures with ambient space, compact structure, and marked elements, while ambient isotopy preserves the marked structure through continuous homeomorphic deformation.
  • Mathematical foundations: Continuity and separation are operationalized through path components and component partitions, with task-specific derived targets used when a category does not directly test its invariant.

A.5. Continuity Tasks

Continuity tasks test whether models can identify connected components from visual scenes and preserve or construct connectivity through actions. They span 2D and 3D mazes, pipe rotations, and procedural difficulty controls.

  • 2D Maze: 2D Maze reasoning uses rendered grid scenes to ask reachability or which wall removals reconnect disconnected points, with connectivity computed by a BFS oracle.The renderer includes axial, diagonal, and partial wall variants, while exact set equality determines answer correctness.
  • Difficulty and scoring: Continuity tasks use controlled difficulty tiers defined by scene structure, wall properties, network complexity, and rejection rules that suppress trivial instances.The 2D Maze tables document wall taxonomy and per-tier rejection constraints, while Pipe contains 200 episodes per tier.
  • 3D Maze: 3D Maze reasoning integrates five views of furnished houses, including occlusions and door states, to determine connectivity among labeled navigation points.The benchmark samples room, door, and point configurations and computes ground truth with a navigation connectivity oracle.
  • Pipe: Pipe planning requires sequentially rotating local pipe cells until every non-empty segment connects to the source within the action budget.Each puzzle starts from a scrambled connected tree-shaped network, and difficulty varies grid size, density, branching, and oracle solution length.

A.6. Separation Tasks

Separation tasks examine whether models can decompose objects or plan paths that preserve color-region constraints. They include image-based assembly reasoning and closed-loop One Stroke planning.

  • Assembly: Assembly reasoning asks models to select the valid two-part decomposition of a complete 3D object from five candidate options.Distractors use component replacements, missing components, and extra components, with difficulty determined by primitive part count.
  • Assembly: Assembly instances are built from 84 objects across six categories, yielding 1,035 reasoning questions with repeated seeds that vary distractors and option order.The underlying correct decomposition remains exact across the generated variants.
  • One Stroke: One Stroke planning requires a continuous path that separates differently colored regions while keeping same-color cells connected on the appropriate side.The action sequence is evaluated as a full trajectory under color-separation constraints and an action budget.
  • One Stroke: One Stroke difficulty varies board size, color count, and shortest solution length, with 200 planning episodes per tier.The stored oracle is the shortest valid path for each accepted puzzle rather than necessarily the initially sampled construction path.

A.7. Order Tasks

Order tasks test whether models recover invariant sequences despite changing viewpoints, folds, or constrained permutations. They cover bead-string traversal, origami-point transformations, and blank-mediated grid swaps.

  • Bead String: Bead String reasoning asks for bead colors in traversal order or classifies two sequences as identical, reversed, cyclically rotated, or different.Scenes use open and closed 3D curves, including tangled torus-knot loops, with difficulty controlled by occlusion, color similarity, topology, and viewpoint.
  • Origami Point: Origami Point reasoning distinguishes invariant point identity from changing Euclidean location as paper models fold and rotate.The task uses eight origami bases, preset trajectories, labeled points, and three difficulty tiers.
  • Swap 2D Puzzle: Swap 2D Puzzle planning transforms an initial grid into a target grid by swapping selected blocks with the blank cell.The benchmark computes exact shortest-path distance in the state graph and sets the action budget to 1.2 times that distance.

A.8. Enclosure Tasks

Enclosure tasks test inside–outside reasoning over nested boundaries, through-holes, and partitions, plus planning that progressively blocks escape routes. Their scenes vary enclosure geometry, distractors, and adversarial movement policies.

  • Hole: Hole reasoning counts only openings that pass through a top board into open space, excluding pits, shadows, and shallow depressions.Procedural layouts use rectangular, circular, and polygonal boards with difficulty-controlled distractor types.
  • Scale and difficulty: Enclosure datasets include 999 Hole reasoning questions, 1,000 Sheep questions, and 600 Chat Noir planning episodes.Difficulty varies nesting, partitions, gaps, board geometry, cat policy, and other task-specific parameters.
  • Sheep: Sheep reasoning asks models to identify bounded sheep, escape routes through outer-fence gaps, and repairs needed to close the outermost enclosure.The task uses nested-fence and partitioned pasture layouts with four question types.
  • Scoring: Evaluation uses exact match for scalar and region answers, set comparison for unordered sheep identifiers, and episode success for planning trajectories.Planning succeeds only when the environment reaches the task-defined terminal condition within its action budget.
  • Chat Noir: Chat Noir planning blocks a hex-grid board so a moving cat loses every route to the boundary before escaping.The cat follows tier-specific movement policies, and accepted setups are solvable while requiring multi-step enclosure planning.

A.9. Knots Tasks

The Knots tasks evaluate topological structure through static recognition questions and sequential planning to eliminate rope crossings. They use procedurally generated rope scenes and pegboard environments with difficulty controlled by scene complexity, crossing structure, and scramble depth.

  • Knots, Reasoning: Knots reasoning classifies rendered rope scenes by knot, link, component, and counterfactual structure across five question types.The benchmark distinguishes rings, knots, open ropes, links, unlinked ropes, and mixed scenes, and asks models to count or reason about linked components.
  • Knots, Reasoning: The reasoning benchmark contains 1,000 questions distributed across T01–T05, with 387 T01, 386 T02, 93 T03, 40 T04, and 94 T05 instances.
  • Untangle, Planning: The Untangle planning task requires legal endpoint relocations that reduce projected rope crossings to zero within a 15-action budget.Each action selects an occupied source hole and an unoccupied target hole; illegal actions consume a step without changing the state.
  • Untangle, Planning: Untangle difficulty increases grid size, rope count, scramble depth, crossing participation, and graph structure, with hard instances requiring cyclic multi-rope components.The hard tier requires at least three ropes to cross more than one other rope and a crossing graph containing a cycle.
  • Evaluation: Across the broader benchmark, reasoning tasks use static parsed answers, while planning tasks execute complete action sequences and score success from the environment state.Primary task accuracy is complemented by parse and format-hit rates, while planning additionally reports mean over-optimality.

B.2. Training

Training experiments compare base, supervised fine-tuned, reinforcement-learned, and sequentially fine-tuned-plus-reinforced Qwen3-VL-2B-Instruct policies under fixed evaluation protocols. Held-out transfer is uneven, and the reported transfer results remain low or below base performance on several target relations.

  • Data splits: Training uses deterministic 8:1:1 splits, with reasoning stratified by question type and difficulty and planning stratified by difficulty.The experiments start from Qwen/Qwen3-VL-2B-Instruct and separate reasoning tasks from planning tasks.
  • Supervised fine-tuning: SFT trains answer-only JSON targets for reasoning and complete action sequences for planning, using exact or solver-derived trajectories where available.Planning targets include shortest or minimum-action solutions for One Stroke, Pipe, Swap, and Untangle.
  • Reinforcement learning: RL applies GRPO with reward 1.0 for correct answers or successful action sequences and 0.0 for parseable but incorrect responses.Reasoning samples eight completions per prompt, whereas planning samples 32, with temperature 0.8 and top-p 0.95.
  • Evaluation protocol: Evaluation decodes one greedy completion per example and counts planning success only when execution reaches the task-defined terminal state.Single-task comparisons keep input format, scorer, and test examples fixed across Base, SFT, RL, and SFT + RL.
  • Held-out transfer: Held-out transfer is uneven: 3D Maze reaches 18.57%–20.28%, Origami Point stays below the 19.60% base result, and Hole reaches at most 8.41% from 0.10%.The results indicate that transfer depends on the target relation rather than arising automatically from multi-environment training.

B.4.1. Annotation Interface and Human Performance Evaluation

Human evaluation uses the same visual inputs, answer schemas, and interactive environments as model evaluation. Annotators achieved high accuracy and agreement across the benchmark, with coverage explicitly separated from the full instance count.

  • Annotation interface: Reasoning annotators receive rendered inputs, task-specific answer widgets, and optional comments, matching the model-facing evaluation interface.
  • Annotation interface: Planning annotators play the same browser-based environment as the model runner, with success determined from the terminal state.
  • Annotation procedure: Five trained annotators completed the production evaluation after written instruction, video training, 300 trials, and qualification, with two auditors monitoring quality.
  • Human performance: Humans answered 10,732 of 11,008 evaluated instances correctly, achieving 97.49% sample-micro accuracy while leaving 22 benchmark instances unevaluated.The missing instances are one Bead example and 21 Origami Point examples.
  • Human performance: Inter-annotator agreement is 0.89 on a shared 200-example subset evaluated under the benchmark’s canonicalized scoring procedures.Reasoning uses exact-match agreement, while planning requires the same evaluated outcome.

B.5. Full Results by Difficulty

The benchmark reports results by topological property and difficulty tier, separating reasoning accuracy from planning episode success. Human results are likewise stratified by difficulty and retain the benchmark’s limited annotation coverage.

  • Difficulty breakdown: Difficulty-stratified results are organized into adjacent Easy, Medium, and Hard columns for each task and topological category.The category tables cover Continuity, Separation, Order, Enclosure, and Knots.
  • Human results: Table 25 reports human reasoning accuracy and planning episode success as correct/evaluated counts with percentages, covering 11,008 of 11,030 instances.
  • Category results: Tables 26–30 report accuracy for reasoning and episode success for planning across difficulty tiers for Continuity, Separation, Order, Enclosure, and Knots.Blue shading uses a shared 0–100 scale, while lavender marks the best and second-best result in each task–tier column.

B.6. Reading the Per-Difficulty Tables

The tables require careful interpretation because some apparent difficulty trends or low scores arise from scoring design, constant answers, formatting failures, or generated-rollout limitations rather than capability alone.

  • B.6. Reading the Per-Difficulty Tables: Hard Hole’s near-zero exact-match rate reflects exact enumeration under distractor structures, not a broken difficulty tier.GPT-5.5’s mean absolute count error rises from 0.05 on Easy to 2.95 on Hard, while invalid responses remain absent.
  • B.6. Reading the Per-Difficulty Tables: Chat Noir is intentionally non-monotonic: stronger cat policies lower Medium performance, while denser initial boards partly raise Hard performance.For GPT-5.5, scores are 60.50, 7.00, and 40.50 across Easy, Medium, and Hard.
  • B.6. Reading the Per-Difficulty Tables: Some open-weight scores are constant-answer artifacts rather than independent evidence of task competence.BAGEL-7B and ThinkMorph-7B always answer option E on Assembly, while Gemma-4-31B-IT always answers 0 on Hole.
  • B.6. Reading the Per-Difficulty Tables: Two Hole cells are format failures because the strict parser rejects object-wrapped integers, not because the models necessarily miscounted.A lenient recovery would produce 6.71 for BAGEL and 13.91 for ThinkMorph instead of the reported 0.00 and 0.60.
  • B.6. Reading the Per-Difficulty Tables: Generated rollouts can appear locally plausible while violating dynamics, topology, or cross-frame consistency.The audit tracks dynamics, invariance, and consistency rather than treating plausible endpoints as valid simulations.

B.7.1. Computer-Vision Video Diagnostics

The video diagnostics combine task-specific parsing, static and dynamic checks, terminal-state gates, and human validation to distinguish valid process rollouts from merely plausible endpoints.

  • B.7.1. Computer-Vision Video Diagnostics: The verifier evaluates One Stroke, Pipe, and Untangle using task-specific static and cross-frame dynamic metrics across generated videos.The audit covers LTX-2.3, Wan2.2-I2V-A14B, MiniMax-H3, and Veo 3.1 Lite, with 1,245 videos total.
  • B.7.1. Computer-Vision Video Diagnostics: A readable degraded frame can remain scorable, whereas failed or grid_not_found parser statuses are unreadable for terminal-state evaluation.Parser labels describe detector completeness and are converted to a common binary scorability gate only after parsing.
  • B.7.1. Computer-Vision Video Diagnostics: Process-valid success requires terminal-state success plus every applicable static and dynamic check, making it stricter than endpoint success.Static checks evaluate sampled frames, dynamic checks evaluate adjacent frames, and explicitly not-applicable rows are excluded from denominators.
  • B.7.1. Computer-Vision Video Diagnostics: All 180 audited video-level metric decisions agreed with manual review, supporting downstream metric reliability on the sampled set.The audit nevertheless found weak rejection of human-unscorable terminal states: 38 of 40 gate errors were false acceptances.
  • B.7.1. Computer-Vision Video Diagnostics: The diagnostic evidence should not be interpreted as perfect state reconstruction because exact parsing is lower, especially for Untangle.Manual validation assessed verifier decisions rather than generator quality, and the terminal scorability gate achieved 54.0% balanced accuracy.

B.9. Shortcut and Input Representation Controls

Shortcut and representation controls test whether performance depends on rendered structure, prompt or answer priors, appearance cues, or explicit symbolic state descriptions.

  • B.9. Shortcut and Input Representation Controls: Sheep exhibits strong prompt and answer priors: Gemini 3.1 Flash-Lite scores 61.0 with full input and 56.0 under both text-only and answer-prior conditions.InternVL3.5-241B scores 28.0 with full input and 28.0 with answer-prior input.
  • B.9. Shortcut and Input Representation Controls: Appearance-only input reaches 36.0 on the matched Sheep subset, exceeding Gemini 3.1 Flash-Lite’s 61.0 full-input result only in the stated control comparison.The authors treat these controls as diagnostics rather than exact decompositions of score.
  • B.9. Shortcut and Input Representation Controls: Symbolic input improves Gemini on Swap by 33.3 points and Chat Noir by 15.0 points, but not InternVL3.5-241B.The effect depends on both the environment and the model rather than following from symbolic input alone.
  • B.9. Shortcut and Input Representation Controls: Pipe and One Stroke remain at zero or near zero under every tested input condition, so representation alone does not remove the planning bottleneck.Their failures persist even after replacing rendered scenes with explicit state descriptions.
  • B.9. Shortcut and Input Representation Controls: Error analysis separates failures into instruction following, task understanding, perception grounding, topological invariance, state prediction, dynamics, and action planning.This taxonomy avoids collapsing distinct failure sources into a single perception-versus-reasoning split.

C.3. Cross-Model and Cross-Task Distributions

The appendix organizes projected failures by environment and model while situating MINDTOPO among topology-focused, cognitively grounded, and broader spatial or world-model benchmarks.

  • C.3. Cross-Model and Cross-Task Distributions: Failure distributions are projected from sampled incorrect predictions and reported separately for reasoning and planning tasks.Figures 12 and 13 provide environment-level and model-level breakdowns, while Tables 39 and 40 report projected category counts.
  • C.3. Cross-Model and Cross-Task Distributions: Environment-specific breakdowns preserve differences among perception, invariance, state-prediction, dynamic, and planning failures.This prevents distinct failure mechanisms from being collapsed into one aggregate error rate.
  • C.3. Cross-Model and Cross-Task Distributions: MINDTOPO is positioned against TopoBench, which contains 900 grid-puzzle instances spanning six topology-focused families at three difficulty levels.TopoBench uses complete-solution verifiers and analyzes 750 reasoning traces.
  • C.3. Cross-Model and Cross-Task Distributions: Unlike most MLLM spatial benchmarks centered on Euclidean or viewpoint-centric relations, MINDTOPO targets topological invariance across cognitive properties and action settings.Related work includes static metric benchmarks, interactive spatial evaluation, and world-model studies focused on physical fidelity or dynamics realism.
  • C.3. Cross-Model and Cross-Task Distributions: The benchmark’s Hole and enclosure prompts operationalize holes as through-openings and enclosures as closed regions formed by connected fence segments.These task definitions distinguish holes from pits or shallow depressions and count repairs needed to close an outermost fence.
Loading 2609.11900v1…