PaperlayerView exact revision on arXiv

Source-linked AI summary

MindTopo: Can Foundation Models Reason in Topological Space?

Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Jianwen Lyu, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li

arXiv:2609.11900v1cs.AIcs.CLcs.CV

TL;DR

Foundation-model spatial benchmarks largely test metric or viewpoint-dependent relations, leaving broader topological reasoning and action underexamined. MindTopo evaluates five topological properties through reasoning and planning, finding that models perform better at reasoning than planning.

  • Problem

    Foundation-model benchmarks largely test metric or viewpoint-dependent relations, while evidence on reasoning across broader topological relations and acting on them remains limited.

    Source: §1 · p. 2

  • Method

    MindTopo benchmarks five topological properties at reasoning and planning levels across 13 procedurally generated task types.

    Source: §1 · pp. 2–3

  • Results

    61.42% task-macro average for GPT-5.6-Sol, versus 97.87% observed human performance, with MLLMs consistently performing better in reasoning than planning.

    Source: §3.2 · p. 7, §5 · p. 14

  • Takeaways & Limitations

    MindTopo offers a controlled diagnostic for future foundation models targeting the qualitative structure of the physical world.

    Source: §5 · p. 14

    Procedurally rendered scenes provide clean ground truth but lack the visual variability of real-world photographs.

    Source: §5 · p. 14

Abstract

from arXiv · show

Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots. MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We benchmark 14 MLLMs and study agent configurations augmented with image and video generation, including 3 video generative models in planning settings. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions. Our website is at https://mind-topo.github.io/

1. Introduction

MINDTOPO evaluates whether foundation models can perceive, reason about, and act on topological relations that persist under continuous deformation. It tests five properties at reasoning and planning levels, addressing spatial benchmarks’ emphasis on metric or viewpoint-dependent relations.

Introduction

  • Topology concerns spatial relations invariant under continuous deformation, and prior cognitive findings motivate treating these relations as a distinct layer of spatial reasoning.Existing foundation-model benchmarks largely test metric or viewpoint-dependent relations, while prior work on path connectivity or knot reasoning does not establish broader topological reasoning and action [6] [7] [8] [9] [10].Source: §1 · p. 2
  • MINDTOPO organizes evaluation around continuity, separation, order, enclosure, and knots, using cognitive science to define tested properties and formal topology to generate and evaluate structural targets.The taxonomy draws on Piaget’s classification and later cognitive work, while formal topology supplies invariants and related targets.Source: §1 · p. 2
  • Each property is tested through reasoning about rendered scenes and planning by a closed-loop agent selecting actions in an interactive environment.Parametric generators provide controllable difficulty and ground truth from scene state; the benchmark contains 11,030 instances across 13 task types, including 8,030 reasoning questions and 3,000 planning episodes.Source: §1 · p. 2, Figure 2 · p. 3
  • Every MLLM scores higher on reasoning than on planning, while the best-performing model remains far below observed human performance.The findings expose a gap between reasoning about topological relations and using them to guide valid actions.Source: §1 · pp. 2–3
  • On Qwen3-VL-2B-Instruct, supervised fine-tuning followed by reinforcement learning gives the strongest average performance, with larger gains in reasoning than planning and uneven held-out reasoning gains from reinforcement learning.The training gains do not close the reasoning–planning gap.Source: §1 · pp. 2–3
  • Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions.The study includes agent configurations augmented with image and video generation, including 3 video generative models in planning environments.Source: §1 · pp. 2–3

2. MINDTOPO Benchmark

MINDTOPO evaluates topological intuition through reasoning and closed-loop planning tasks across five properties, using an automated, difficulty-controlled pipeline to generate 11,030 instances.

2.1. Problem Formulation

  • Reasoning asks models to identify or predict topological relations; planning evaluates whether a closed-loop agent can use them to reach a goal through actions.Planning policies may use an MLLM alone or include image or video generation.Source: §2.1 · p. 4

2.2. Topological Properties

  • Continuity pairs maze-reachability judgments with pipe-connection planning, while separation pairs furniture-component identification with one-stroke grid partitioning.Reasoning tasks include what-if edits to walls or connectors; planning tests constructing continuity or separating contiguous regions.Source: §2.2 · pp. 4–5
  • Order tasks test bead-sequence and cyclic-order judgments under string transformations, alongside sliding-block planning toward a target color permutation.The reasoning task includes what-if folding or rotation questions.Source: §2.2 · p. 5
  • Enclosure tasks test inside/outside judgments and through-hole counts, while Chat Noir planning requires enclosing a moving cat before it reaches the boundary.The reasoning tasks include what-if changes to enclosure material or solid objects.Source: §2.2 · p. 5
  • Knots tasks distinguish knotted ropes from visual tangles, test linked loops and ring removal, and pair these questions with a rope-untangling planning task.Prior cognitive evidence distinguishes knot understanding from perception and domain-general physical reasoning.Source: §2.2 · p. 5

2.3. Data Collection and Statistics

  • A four-stage automated pipeline generates scenes, derives labels programmatically, controls difficulty, and converts accepted states into reasoning questions or planning episodes.Difficulty parameters include wall count, grid size, and crossing count; reasoning and planning scenes are generated separately.Source: §2.3 · p. 5, Figure 3 · p. 6
  • Procedural rendering and templated questions may create visual, language, or answer-prior shortcuts, which the authors evaluate with matched-input controls.The paper separately notes limited visual variation from real-world scenes.Source: §2.3 · p. 6
  • 11,030 instances cover 13 task types across five topological properties, with 73% for reasoning and 27% for planning.The eight reasoning tasks contribute roughly 1,000 instances each, while the five planning tasks contribute 600 each.Source: §2.3 · p. 6

3. Experiments

Experiments benchmark 14 MLLMs against human performance on reasoning and planning, then examine task-specific training, error patterns, and image- and video-generation probes. Planning success requires executed actions to reach a task-defined terminal state; the probes test whether generated observations preserve state across actions.

3.2. Benchmark Results

  • GPT-5.6-Sol leads at 61.42% task-macro average, versus 97.87% for humans, while its reasoning accuracy (66.83%) exceeds planning success (52.75%).The prose reports the same reasoning-to-planning drop for Gemini-3.1-Pro (52.24% to 19.23%), GPT-5.5 (53.77% to 24.33%), and Qwen3.5-397B-A17B (26.54% to 5.27%).Source: §3.2 · p. 7, Table 1 · p. 7
  • The reasoning–planning gap varies by model tier: the best open-weight planner scores 5.27% on average, and every open-weight model scores 0% on Pipe and One Stroke.The strongest proprietary models retain more reasoning performance in planning than open-weight models; the supplied results say current open-source scaling does not close the planning bottleneck.Source: §3.2 · pp. 7–8
  • Leadership varies by primitive and environment: GPT-5.6-Sol leads four properties, Gemini-3.1-Pro leads knots, and the best scores remain low on Pipe, One Stroke, and Untangle.GPT-5.6-Sol scores 50.50% on Pipe and 35.00% on One Stroke; Gemini-3.1-Pro scores 29.17% on Untangle. On Swap, GPT-5.6-Sol scores 87.30%, versus 47.67% for GPT-5.5 and 17.50% for Gemini-3.1-Pro. The passage cautions that results do not isolate causes such as state representation, task difficulty, or action horizon.Source: §3.2 · p. 8

3.3. Training

  • SFT followed by GRPO achieves the strongest nine-task average at 31.44%, while reasoning accuracy rises to 51.53% from 14.24% but planning success reaches only 6.33%.The combined policy scores 79.60% on Knots, 52.95% on Bead, 58.20% on Sheep, and 17.42% on Untangle. Leave-one-task-out RL improves held-out Bead (9.31% to 26.53%) and Knots (11.20% to 15.50%), but not 2D Maze, Assembly, or Sheep; Pipe remains at 0% under every training condition.Source: Table 2 · p. 8, §3.3 · p. 9

3.4. Error Analysis

  • Reasoning errors are mainly perception-grounding failures (58.1%), whereas planning errors concentrate in action planning (43.3%) and dynamic violations (24.5%).For reasoning, perception grounding accounts for 64.2% of Gemini-3.1-Pro’s errors and 54.6% of InternVL3.5-241B’s. In planning, Gemini’s largest category is action planning (56.6%), while InternVL’s is dynamic violations (41.0%). The analysis assigns one primary failure category in a fixed causal order.Source: §3.4 · p. 10, Figure 6 · p. 10

3.5. Probing Experiment

  • Interleaved image prediction retains local cues but fails on trajectory-wide constraints: GPT-Image-2 scores 0% across One Stroke tiers and 5.71% on hard Untangle.With GPT-5.6-Luna, GPT-Image-2 averages 68.6% on Sheep and 61.9% on 2D Maze across the three displayed tiers. Wan2.2-I2V-A14B scores 54.29%, 20.00%, and 0.00% on easy, medium, and hard Untangle, and 0.00% across One Stroke tiers; it violates dynamics in 116 of 119 audited rollouts.Source: §3.5 · p. 11, Table 3 · p. 11, Figure 7 · p. 11
  • Across 1,560 audited rollouts, no generator achieves process-valid task success on One Stroke, Pipe, or Untangle, even though some terminal states satisfy the task oracle.For example, the terminal oracle accepts 32.4% of LTX-2.3 Untangle videos and 35.2% of planner-free LTX-2.5 Pipe clips, but every parsed success still violates at least one applicable static or dynamic constraint along the way. A human audit agrees with sampled video-level metric decisions, though strict conjunction failures limit evidence about false rejections.Source: §3.5 · pp. 12–13, Table 4 · p. 12

4. Related Work

Related work spans cognitive and topological evaluation, spatial reasoning and interaction, and world modeling and agent training; MindTopo connects topological properties to both reasoning and planning.

Cognitive and Topological Evaluation

  • MindTopo groups five topological properties into one suite spanning reasoning and interactive planning, extending work from visual abilities and topology-specific benchmarks.BabyVision tests visual abilities; CurveBench, TopoBench, and KnotGym respectively evaluate nested-curve containment, symbolic grid puzzles, and interactive rope manipulation.Source: §4 · p. 13

Spatial Reasoning and Interaction

  • MindTopo complements scene, viewpoint, active-exploration, camera-planning, puzzle, simulation, and embodied-action benchmarks by organizing reasoning and planning around topological properties.The paper points to Table 5 for a summary of this complementary coverage.Source: §4 · p. 13

World Modeling and Agent Training

  • MindTopo’s generated-rollout diagnostics complement world-modeling and agent-training work by checking whether predicted transitions preserve task topology.Related work includes CausalSpatial’s generated evidence about object motions, ENACT’s forward and inverse world modeling, and RAGEN, RAGEN-2, and VAGEN on multi-turn agent training and world-model reasoning.Source: §4 · p. 13

5. Conclusion and Limitations · Appendix

MINDTOPO evaluates topological intuition across five properties and two cognitive levels, and the results identify a gap between recognizing relations in static scenes and maintaining them through actions. The conclusion also outlines benchmark limitations and directions for expansion.

5. Conclusion and Limitations

  • MINDTOPO covers five topological properties at reasoning and planning levels across 13 procedurally generated task types.The benchmark is grounded in Piaget’s classification and evaluates topological intuition in foundation models.Source: §5 · p. 13
  • Models recognize topological relations in static scenes but struggle to maintain or operate on them across action sequences.The authors characterize topological intuition as a blind spot across 14 MLLMs; training improves Qwen3-VL-2B-Instruct performance, but planning success remains low and held-out-task gains are uneven.Source: §5 · p. 14
  • Three video generative models produced plausible endpoints, but their rollouts did not preserve valid topological transitions.The paper evaluates these models as policies in planning environments.Source: §5 · p. 14
  • Procedurally rendered scenes provide clean ground truth but lack the visual variability of real-world photographs.Scenes are rendered via Three.js or task-specific simulators.Source: §5 · p. 14
  • MINDTOPO’s five Piagetian primitives leave orientation, composed continuous transformations, and higher-genus surfaces out of scope.The authors identify these topology-flavored relations as possible additions in future releases.Source: §5 · p. 14
  • The video-policy evaluation covers three models, so broader evaluation is needed to assess how general the observed failures are.The authors also note that the annotation audit covers 464 generated images and 119 scored videos from 120 targeted videos; one Untangle video was unavailable.Source: §5 · p. 14

A. MINDTOPO: Benchmark Details · A.1. Cognitive-Science Grounding of the Five Properties · A.2. Mathematical Foundations of the Five Properties

MindTopo grounds five topological properties in cognitive science and formalizes their invariants and task-specific targets, including cases where a task uses a related structural target rather than the invariant itself.

A.1. Cognitive-Science Grounding of the Five Properties

  • MindTopo adapts Piaget’s five classes by absorbing proximity into Separation and promoting Knots to a standalone property.The taxonomy draws on Piaget’s account, later mathematical and educational analyses, and algebraic topology.Source: §A.1 · p. 24
  • Continuity concerns perceiving an unbroken whole, while Separation distinguishes neighboring elements; their tasks test path-connectedness and connected units.Continuity tasks include 2D MAZE, 3D MAZE, and PIPE; Separation tasks include ASSEMBLY and ONE STROKE.Source: §A.1 · p. 24
  • Order captures spatial succession, tested through sequence read-out, tracking points across folds, and recovering permutations after elementary operations.The tasks are BEAD, ORIGAMI POINT, and SWAP 2D PUZZLE.Source: §A.1 · p. 24
  • Enclosure concerns boundaries separating interior from exterior, while Knots combines topological phenomena such as crossing parity, linking number, and isotopy.SHEEP and HOLE test enclosure judgments, CHAT NOIR plans under a closing-boundary constraint, and KNOTS and UNTANGLE test knot perception and intervention.Source: §A.1 · pp. 24–25

A.2. Mathematical Foundations of the Five Properties

  • Each scene is modeled as an embedded structure in R2 or R3 with a compact structure and marked elements; continuity and separation use component-based targets.The generators are locally path-connected, so connected and path components coincide; some tasks use derived targets rather than directly evaluating the category’s invariant.Source: §A.2 · pp. 25–26
  • Order formalizes the sequence of marked points along a decorated arc or circle, but ORIGAMI POINT and SWAP 2D PUZZLE use correspondence and grid-arrangement targets.BEAD fixes a unique sequence with a start marker and direction arrow; string-relation queries distinguish identical, reversed, cyclically rotated, and different orders.Source: §A.2 · p. 26
  • Enclosure labels points by bounded components of a planar complement, while 3D HOLE counts through-handles using the first Betti number.CHAT NOIR succeeds when the cat occupies an unblocked component containing no boundary cell; the generated solids have no internal cavities.Source: §A.2 · p. 26
  • Knot classification uses ambient isotopy, splitness, and linking number, but UNTANGLE instead targets a zero-crossing overhead projection.A nonzero linking number certifies a non-split link, whereas zero does not certify splitness; KNOTS also queries component counts and loop removal.Source: §A.2 · pp. 26–27

A.3. Task Catalog Overview … A.9.2. Untangle (Knots, Planning)

The catalog covers MindTopo’s topological tasks across visual reasoning and closed-loop planning, spanning connectivity, separation, order, enclosure, and knots. Procedurally generated tasks assess reasoning through exact answers and planning through successful goal-reaching actions, with seed-driven generators supporting reproducible labels and simulator-based success checks.

A.3. Task Catalog Overview; A.4. Shared Generation and Quality Control

  • 13 task types span five topological properties and two cognitive levels, with deterministic generators producing reference answers or simulator-defined planning success conditions.Generators store seeds and configuration metadata; human annotators do not create benchmark labels.Source: §A.3 · p. 27, §A.4 · p. 27

A.5.1. 2D Maze (Continuity, Reasoning); A.5.2. 3D Maze (Continuity, Reasoning)

  • The two maze tasks test visual connectivity: 2D scenes ask about reachable points or wall-removal bars, while 3D scenes require integrating five views of furnished rooms and closed doors.The 2D maze uses a connectivity oracle; the 3D oracle accounts for walls, doors, furniture, and other obstacles.Source: §A.5.1 · p. 28, §A.5.2 · p. 29

A.5.3. Pipe (Continuity, Planning)

  • Pipe planning requires rotating individual cells to connect every pipe to a source, with success determined over the full trajectory before the action budget expires.Puzzles begin as scrambled versions of a known connected, tree-shaped network; difficulty varies by grid size, density, branching, and oracle solution length.Source: §A.5.3 · p. 30

A.6.1. Assembly (Separation, Reasoning)

  • Assembly asks models to identify the valid two-part decomposition of a rendered object among five options, including distractors with missing, extra, or substituted components.The split contains 1,035 questions from 84 objects across six categories; difficulty is based on primitive part count.Source: §A.6.1 · p. 31

A.6.2. One Stroke (Separation, Planning)

  • One Stroke planning requires a continuous, non-looping path from the lower-left cursor to the upper-right target that separates different colors while keeping same-color cells connected.Accepted puzzles are filtered by shortest valid solution length, and episode success is evaluated on the completed trajectory.Source: §A.6.2 · pp. 31–32

A.7.1. Bead (Order, Reasoning)

  • Bead String reasoning asks models to recover a marked traversal’s color sequence or classify two sequences as identical, reversed, cyclically rotated, or different.Its 1,000 questions vary bead count, curve topology, occlusion, color similarity, and viewpoint; sequence answers require exact order and multiplicity.Source: §A.7.1 · pp. 32–33

A.7.2. Origami Point (Order, Reasoning)

  • Origami Point tests whether models track labeled point identity as paper folds and rotates, using forward and reverse image sequences across eight bases and three difficulty tiers.The 1,000 instances use mesh-based point labels and visibility checks; Hard cases include underside points revealed only after rotation. The task draws on Piaget’s conservation under transformation.Source: §A.7.2 · pp. 33–34

A.7.3. Swap 2D Puzzle (Order, Planning)

  • Swap 2D Puzzle planning transforms an initial grid into a goal grid by swapping selected blocks with the blank cell until the arrangements match exactly.The generator computes shortest-path distance in the state graph and sets the action budget to 1.2 times that minimum, rounded up.Source: §A.7.3 · p. 34
  • Swap 2D Puzzle planning is scored on whether the executed episode reaches the goal arrangement within its step budget.Actions specify a grid position; Fig. 35 contains the complete prompt and qualitative examples.Source: §A.7.3 · p. 34

A.8.1. Hole (Enclosure, Reasoning)

  • Hole reasoning asks models to count only through-holes in a top-down board image, excluding pits, depressions, and lower-board openings; the split contains 999 questions across three tiers.Answers are scalar integers scored by exact match; ground truth counts only openings passing through the board to open space.Source: §A.8.1 · p. 35

A.8.2. Sheep (Enclosure, Reasoning)

  • Sheep reasoning tests enclosure and escape judgments in nested-fence and partitioned pastures, using four question types and a released split of 1,000 questions.Questions count sheep unable to escape, identify escapees or enclosing cells, and determine repairs needed to close a fence; answers use exact-match or set scoring.Source: §A.8.2 · pp. 35–36

A.8.3. Chat Noir (Enclosure, Planning)

  • Chat Noir is enclosure planning in which models block cells on a hex grid to trap a moving cat before it reaches the boundary, across 600 episodes.Difficulty varies the cat’s movement policy, and episodes require trapping the cat within the action budget.Source: §A.8.3 · pp. 36–37

A.9.1. Knots (Knots, Reasoning)

  • Knots reasoning uses single rendered images to test knot and link recognition, rope counts, link structure, and counterfactual ring removal across 1,000 questions.The task catalog covers 24 generated scene types, including deceptive unknots, knots, links, and mixed scenes; answers are scored by exact match, with T04 lists compared as sets.Source: §A.9.1 · pp. 37–38

A.9.2. Untangle (Knots, Planning)

  • Untangle planning asks models to relocate rope endpoints on a pegboard until simulated rope paths have no projected crossings, with 600 episodes and a 15-action budget.A symbolic endpoint solver supplies a plan-length proxy and local oracle policy, but does not compute an exact shortest plan for the simulated rope physics.Source: §A.9.2 · pp. 38–39

A.10. Dataset Statistics

Table 21 summarizes MindTopo’s dataset composition across its topological properties, cognitive levels, and difficulty tiers.

A.10. Dataset Statistics

  • Planning instances are distributed uniformly across the three difficulty tiers in a breakdown spanning five topological properties and the Reasoning and Planning levels.Source: Table 21 · p. 39

A.11. Evaluation Design · B. Experiments and Analysis · B.1. Models and Setup

MindTopo evaluates reasoning through single-response visual questions and planning through budgeted, executed trajectories, using accuracy as its primary metric. The experiments apply shared prompts and deterministic protocols across the assessed models and generative backends.

A.11. Evaluation Design

  • Reasoning models answer one visual query without rollout, using set, sequence, scalar-integer, or categorical-label response formats scored by exact task-specific matching.A shared layered extractor first attempts strict JSON and then applies fallback phrase, legal-value, and response-tail checks.Source: §A.11 · pp. 39–40
  • Planning models act through validated environment actions under per-task budgets, and an episode fails when its budget expires or an action violates the success conditions.The runner scores executed trajectories, not textual plans; reference policies include an oracle solver and a uniformly sampled legal-action random policy, with a symbolic endpoint proxy for knot untangling.Source: §A.11 · p. 40
  • Accuracy is the primary metric; results are also aggregated separately by cognitive level and as a macro mean weighting the five topological properties equally.Diagnostic measures include strict-JSON parse rate, answer-format hit rate, and planning-task mean over-optimality on solved episodes; the paper’s tables report accuracy.Source: §A.11 · p. 40
  • The scoring pipeline accepts any budget-compliant planning trajectory satisfying the final-state predicate, normalizes set answers, and rejects sequences of the wrong length.Set answers are converted to frozensets so order, case, and duplicates do not affect scoring.Source: §A.11 · p. 40

B.1. Models and Setup

  • Table 22 inventories the evaluated foundation models and generative backends, including their snapshots and assigned pipelines.The passage identifies Table 22 as the source of the complete model and backend list.Source: §B.1 · p. 40, Table 22 · p. 41
  • Reasoning and planning use deterministic decoding; planning runs in Hydra+Playwright without giving models symbolic state graphs, oracle connectivity, or hidden generator metadata.Reasoning uses temperature=0, do_sample=false, and at most 2048 new tokens; prompts request one JSON object, while planning supplies each turn’s observation, task rules, and legal answer schema.Source: §B.1 · pp. 40–41
  • Static images use native renderer resolution as PNGs, while planning screenshots use browser captures and configured fixed viewports where needed.Fixed viewports are 1440×960 or 1440×1080 before scene-only cropping.Source: §B.1 · p. 41
  • Each task type uses a fixed prompt template shared across models, with no task-specific tuning after outputs are seen.Representative property-level templates are provided in Appendix E.1, Figs. 14–18.Source: §B.1 · p. 42

B.2. Training · B.3. Illegal Actions · B.4. Human Evaluation

The appendix details Qwen3-VL-2B-Instruct training and evaluation, finding uneven held-out transfer, and specifies how illegal actions are handled during environment interaction.

B.2. Training

  • Training starts from Qwen3-VL-2B-Instruct, with deterministic 8:1:1 single-task splits and a five-policy leave-one-task-out setup.Reasoning splits are stratified by question type and difficulty, planning splits by difficulty; the held-out setup trains on four reasoning tasks and splits the fifth 1:4 for validation and test.Source: §B.2 · p. 42
  • SFT uses answer-only JSON targets for reasoning and complete JSON action sequences for planning, with each planning sequence replayed to verify task success.Planning targets are generated using shortest-path search or the task-specific minimum rotations or overlap-aware breadth-first search.Source: §B.2 · p. 42
  • GRPO samples 8 completions per reasoning prompt and 32 per planning prompt, rewarding correct answers or successful sequences with 1.0 and invalid outputs with −0.1.Planning reward is determined by executing the predicted sequence in the actual environment; no intermediate-action credit is given. The RL runs use the VAGEN/VERL stack [53].Source: §B.2 · p. 42
  • SFT + RL initializes GRPO from the base checkpoint merged with the task-specific SFT adapter, while leave-one-task-out results come from five separately trained held-out policies.The SFT + RL runs retain the corresponding RL-only split, reward, sampling, and optimization settings.Source: §B.2 · p. 42
  • Evaluation uses one deterministic greedy completion per example, and planning counts as correct only when executing the parsed sequence reaches the task-defined success state.Single-task Base, SFT, RL, and SFT + RL comparisons use fixed input formats, scorers, and test examples.Source: §B.2 · pp. 42–43
  • 3D Maze transfer stayed near the 18.57%–20.28% base-policy range, Origami Point stayed below its 19.60% base result, and Hole rose from 0.10% to at most 8.41%.These results indicate that cross-task transfer depends on the target relation rather than following automatically from multi-environment training.Source: Table 23 · p. 43, §B.2 · p. 43

B.3. Illegal Actions

  • An action outside an environment’s legal-action set is rejected before state transition; rejection leaves state unchanged, consumes a step, and normally returns only the unchanged observation.Malformed output becomes an invalid-action sentinel, whereas API failures are tracked separately and never reach the environment; legal-action lists are disabled in prompts by default.Source: §B.3 · pp. 43–44

B.4.1. Annotation Interface and Human Performance Evaluation · B.4.2. Inter-Annotator Agreement

Human annotators achieved 97.49% sample-micro accuracy across 11,008 evaluated examples, and inter-annotator agreement was 0.89 on a shared subset of 200 examples.

B.4.1. Annotation Interface and Human Performance Evaluation

  • Reasoning annotations use the model answer schemas and visual inputs, while planning annotations use the same browser environment and terminal-state success criterion as the model runner.The reasoning interface provides task-specific answer widgets and an optional comment field; the planning system resets from JSONL configuration and records each action.Source: §B.4.1 · p. 44, Figure 8 · p. 45, Figure 9 · p. 45
  • Five experienced annotators completed written and video training, 300 trial examples, and a qualification round; two auditors monitored quality throughout.Annotators were paid more than 1.5 times the applicable local minimum hourly wage and consented to research use of their answers and performance results.Source: §B.4.1 · p. 44
  • The evaluation used synthetic puzzle data, collected no personal or sensitive information, and did not involve institutional review.Human annotations cover 11,008 of 11,030 benchmark instances; the remaining 22 were one Bead and 21 Origami Point instances.Source: §B.4.1 · p. 44
  • 97.49% sample-micro accuracy was achieved on 11,008 examples, with 10,732 correct; 22 benchmark instances were not represented in this human result.The reported Human row uses the same parser and scorer as model evaluation; these results are an empirical human-upper-bound check, while ground-truth labels remain generator-derived.Source: §B.4.1 · p. 44

B.4.2. Inter-Annotator Agreement

  • IAA was 0.89 after canonicalization across 200 examples independently evaluated by three annotators.The shared subset included reasoning and planning tasks and used the same instructions as the models.Source: §B.4.2 · p. 45
  • Reasoning agreement is exact match after canonicalization, whereas planning agreement requires the same evaluated outcome under the deterministic task protocol.Disagreements were reviewed separately for possible ambiguity, interface issues, or annotation errors, not resolved by majority vote for IAA.Source: §B.4.2 · p. 45

B.5. Full Results by Difficulty

This section presents benchmark results by difficulty, retaining the model grouping and metrics used in the main table. It organizes results across the five topological categories and Easy, Medium, and Hard tiers.

B.5. Full Results by Difficulty

  • Tables 26–30 organize continuity, separation, order, enclosure, and knots results by Easy, Medium, and Hard difficulty tiers.They report accuracy for reasoning tasks and episode success rate for planning tasks.Source: §B.5 · p. 45, Table 26 · p. 46, Table 27 · p. 46
  • Table 25 reports human accuracy on reasoning tasks and episode success rate on planning tasks, stratified by benchmark difficulty.It covers 11,008 of 11,030 instances through completed human annotations, and its Total row matches the Human row in Table 1.Source: Table 25 · p. 46

B.6. Reading the Per-Difficulty Tables · B.7. Generative Models as Topological World Models

The appendix explains apparent per-difficulty scoring anomalies and evaluates generated rollouts as auxiliary world models. Exact-match scoring, answer-format failures, and rollout violations qualify how the reported results should be interpreted.

B.6. Reading the Per-Difficulty Tables

  • All models score at most 2.40% on Hard Hole, while GPT-5.5 and Gemini-3.1-Pro reach 81–97% on Easy and Medium.Hole uses exact integer equality; GPT-5.5’s mean absolute count error rises from 0.05 (Easy) to 0.08 (Medium) and 2.95 (Hard), with no invalid responses. Hard adds all four hole and pit types, so the near-zero exact-match rate reflects harder exact enumeration, not a broken tier.Source: §B.6 · p. 49
  • GPT-5.5 scores 60.50, 7.00, and 40.50 across the reported Chat Noir tiers, with Hard above Medium.Difficulty is set by the cat policy, but initial blocks also increase by tier: radius 3 uses 8–10, 10–12, and 12–14, while radius 4 uses 10–13, 13–16, and 16–19. Denser starting boards make enclosure easier, partly offsetting the stronger policy; the Easy-to-Medium drop reflects policy and the Medium-to-Hard rise reflects extra blocks.Source: §B.6 · p. 49
  • BAGEL-7B and ThinkMorph-7B answer option E throughout Assembly, while Gemma-4-31B-IT answers 0 on all 999 Hole questions.Option E’s 20.19% base rate on 1,035 Assembly questions explains the identical BAGEL and ThinkMorph scores; Gemma’s constant answer yields 0.00 in all three Hole tiers.Source: §B.6 · p. 49
  • BAGEL-7B and ThinkMorph-7B Hole responses are parser failures, not evidence about counting ability.The parser rejects {"answer":{10}}; 999 of 999 BAGEL and 950 of 999 ThinkMorph responses are invalid. Lenient recovery would give 6.71 and 13.91 rather than the reported 0.00 and 0.60, but the study retains a strict shared parser for comparability and classifies these as format failures.Source: §B.6 · p. 49

B.7. Generative Models as Topological World Models

  • In the world-model setup, a reasoner prompts a generator for a plausible future observation, then answers or acts using that artifact and the original observation.Image trials use edit-mode generation anchored to the current PNG, with gpt-image-2, BAGEL, or ThinkMorph; video trials use image-to-video with Wan2.2-I2V-A14B or CogVideoX-style replay. No-imagination and imagination controls keep the reasoner, initial observation, task prompt, and action budget fixed.Source: §B.7 · p. 49
  • Imagined rollouts depict task-specific changes to paths, connectivity, ordering, escape routes, holes, or knot crossings.The reasoner can call the image generator at each step, up to ten times per episode; end-to-end video uses one video per episode. Generated conditions use 35 episodes per difficulty tier (105 per task).Source: §B.7 · p. 50
  • The rollout analyses use distinct samples: 119 Wan2.2 videos for error annotation and 630 videos for the CV diagnostic audit.Figure 7 targeted 60 videos each for One Stroke and Untangle; one Untangle video was unavailable, leaving 60 and 59 scored. Table 4 uses 35 videos per difficulty tier for each LTX-2.3 or Wan2.2 task–generator pair (105 per cell). Planner-only baselines reuse 1,000 reasoning and 600 planning instances for GPT-5.4-mini and InternVL, while Luna planning baselines use 105 episodes per task, 35 per tier.Source: §B.7 · p. 50
  • 116 of 119 audited Wan2.2-I2V-A14B rollouts contain dynamics errors, 93 invariance errors, and 90 cross-frame consistency errors.Annotations include identity drift, silent crossings, hallucinated openings, and smooth but impossible transitions; the authors treat generated rollouts as diagnostics, not replacements for oracle state transitions.Source: §B.7 · p. 50

B.7.1. Computer-Vision Video Diagnostics · B.8. Camera and Viewpoint Controls

The video-diagnostic audit supports the reliability of automatic metric decisions on the reviewed sample, while exposing a weakness in rejecting unscorable terminal states. Camera and viewpoint controls vary the rendered view while keeping topological ground truth fixed.

B.7.1. Computer-Vision Video Diagnostics

  • The CV audit covered 1,245 videos from four generators across One Stroke, Pipe, and Untangle, using 41 evenly spaced frames per video and counting unscorable videos as failures.A video passes only if every applicable frame or transition passes; checks with no applicable instance are excluded from that check’s denominator.Source: §B.7.1 · p. 50
  • Terminal outcomes use the final sampled frame, dynamic checks compare adjacent frames, and process-valid task success requires a scorable successful terminal state plus valid static and dynamic checks.Degraded parser labels are readable and may remain scorable; failed and grid_not_found labels are unreadable. Figure 10 presents per-check video-level pass rates and the combined process-valid success row.Source: §B.7.1 · pp. 50–51, Figure 10 · p. 51, Table 32 · p. 53
  • Figure 11 illustrates sampled Pipe, Untangle, and One Stroke video frames alongside task-specific overlays for checking transitions between adjacent frames.The overlays depict the cross-frame correspondences defined for dynamic verification.Source: §B.7.1 · p. 51, Figure 11 · p. 52
  • The terminal-state scorability gate achieved 77.8% accuracy and 54.0% balanced accuracy, with 9.5% recall on human-unscorable videos and 38 of 40 errors accepting unscorable states.Its 98.6% recall on the more frequent human-scorable class dominates the overall accuracy; the audit identifies weak rejection of unscorable states as a limitation.Source: §B.7.1 · p. 52, Table 34 · p. 54

B.8. Camera and Viewpoint Controls

  • The benchmark rerenders the same seeded topology under top, oblique, wider, tighter-crop, or distractor-heavy views to separate viewpoint difficulty from topological difficulty.In planning environments, the viewport is fixed and scene-only screenshots are captured at each step to prevent resolution changes from becoming a hidden model-specific advantage.Source: §B.8 · pp. 52, 54, Table 35 · p. 54

B.7.1. Computer-Vision Video Diagnostics

  • All 180 audited video-level metric decisions agreed with manual review, and every sampled parse was usable, though exact parses were less common, especially for Untangle.The audit treats a partial parse as usable when it retains enough evidence for the diagnostic decision; it does not establish perfect state reconstruction or rank generator quality.Source: §B.7.1 · pp. 51–52, Table 34 · p. 54

B.9. Shortcut and Input Representation Controls

The section tests whether answers depend on scene structure or on residual cues, comparing full input with text-only, answer-prior, appearance-only, and symbolic conditions on matched instances. The symbolic condition replaces the rendered scene with a description of the same state.

Shortcut and Input Representation Controls

  • Sheep retains strong answer priors: Gemini scores 61.0 with full input and 56.0 with text-only and answer-prior input, while InternVL scores 28.0 with full and answer-prior input.The controls suggest answer distribution explains most measured Sheep performance on this matched subset.Source: §B.9 · p. 55, Table 36 · p. 55
  • On 2D Maze, removing the scene lowers Gemini from 21.0 to 8.0, but appearance-only input scores 36.0, above full input.Appearance-only cues therefore carry nontrivial answer signal; the controls are diagnostic, not an exact score decomposition.Source: §B.9 · pp. 55–56, Table 36 · p. 55
  • Symbolic input raises Gemini by approximately 33.3 percentage points on Swap (13.33 to 46.67) and 15.0 points on Chat Noir (25.00 to 40.0), while InternVL gains on neither task.The effect depends on both environment and model, rather than following from symbolic input alone.Source: §B.9 · pp. 55–56, Table 37 · p. 55
  • Pipe and One Stroke remain at zero or near zero under every tested input condition, including explicit state descriptions, so input representation alone does not account for these planning failures.The errors are consistent with limits in state-transition reasoning and action selection; symbolic input also lowers Sheep performance for both models and is not an oracle upper bound.Source: §B.9 · p. 56

B.10. Do Foundation Models Have Topological Biases? … C.2.7. 6. Action Planning Error

The analysis finds distinct topological-error profiles across tasks and classifies failures by where they arise, from visual grounding and task understanding to state prediction, dynamics, and action planning. The taxonomy separates these sources because a single perception-versus-reasoning split is too coarse.

B.10. Do Foundation Models Have Topological Biases?

  • Projected errors reveal connectivity-by-default in mazes, hole undercounting, and different failure profiles for static knot recognition and interactive untangling.Across the two Maze reasoning environments, 81.0% of projected failures are perception-grounding errors; Pipe planning errors are 61.4% action planning and 17.6% dynamic violations. Hole Detection errors are 80.8% perception grounding, 15.4% task understanding, and 3.8% instruction following, with none labeled topological invariance. Static Knots failures are 57.3% topological invariance and 35.1% perception grounding, whereas Untangle failures are 61.3% instruction following and 20.6% action planning; the table summarizes these environment-specific signatures.Source: §B.10 · p. 56, Table 38 · p. 57

C.1. Annotation Framework

  • The annotation framework rejects a single perception-versus-reasoning split and separates seven failure sources, from instruction following through action planning.The benchmark evaluates visual grounding, topological-relation identification, state-change prediction, and valid action planning. Its categories distinguish instruction following, task understanding, perception grounding, topological invariance, feature-state prediction, dynamic violations, and action planning.Source: §C.1 · pp. 56–57

C.2.1. 0. Instruction Following Error; C.2.2. 1. Task Understanding Error

  • Instruction-following errors concern unusable or noncompliant responses, while task-understanding errors concern solving the wrong objective, applying the wrong rule, binding the wrong entity, or drifting across turns.Category 0 is assigned after the fallback parser cannot recover a valid answer; it also includes rationale-answer inconsistencies and output-budget truncation before a structured answer. Category 1 responses may be parseable but target the wrong problem, and interactive task drift occurs when the model loses the original framing across turns.Source: §C.2.1 · pp. 57–58, §C.2.2 · pp. 58–59

C.2.3. 2. Perception Grounding Error

  • Perception-grounding errors arise when a model understands the task but relies on incorrect visual facts, including misidentified elements, misplaced objects, missed cues, or mismatched images.Examples include confusing a through-hole with a filled surface, misreading relative positions, hallucinating an obstacle or connection, and failing to align corresponding objects or states across images.Source: §C.2.3 · p. 59

C.2.4. 3. Topological Invariance Understanding Error

  • Topological-invariance errors occur when a grounded scene is assigned the wrong relation or the model mistakes preserved topology for change, or change for preservation.The category covers connectivity, enclosure, separation, overlap, crossing, linking, and inside/outside relations under transformations; examples include false invariance, false change, and assigning the wrong relation.Source: §C.2.4 · p. 59

C.2.5. 4. Feature State Prediction Error; C.2.6. 5. Dynamic Error

  • Feature-state prediction errors concern the resulting state after an action, whereas dynamic errors concern an impossible transition along the path to that state.State errors include wrong post-action topological or discrete features and incomplete updates to dependent features. Dynamic errors include moving through barriers, violating topological constraints, or assuming a final arrangement without a legal continuous path.Source: §C.2.5 · p. 60, §C.2.6 · p. 60

C.2.7. 6. Action Planning Error

  • Action-planning errors arise when the model understands and grounds the task and obeys its dynamics but selects a poor legal action or sequence.These include legal but irrelevant actions, locally reasonable short-horizon plans that induce later failure, and later decisions based on a stale state estimate.Source: §C.2.7 · pp. 60–61

C.3. Cross-Model and Cross-Task Distributions … F. The Use of Large Language Models

This appendix section documents supplementary materials for MindTopo, including error summaries, prior-work context, fixed prompts, and task examples. It also details order and enclosure task mechanics, illustrates knot tasks, and discloses LLM use.

C.3. Cross-Model and Cross-Task Distributions

  • Figures 12–13 and Tables 39–40 report environment- and model-level error-category distributions for the reasoning and planning splits.For each model–task pair, the authors sampled up to 35 incorrect predictions, assigned one primary category per sample, and projected category proportions onto the total error count; the splits contain eight reasoning tasks and five planning tasks.Source: §C.3 · p. 61, Figure 12 · p. 62, Figure 13 · p. 62

D. Extended Related Work Discussion

  • Prior work grounds topology in early spatial cognition, adult sensitivity to invariants, and knots as a distinct cognitive domain, while existing benchmarks cover adjacent or partial capabilities.TopoBench [38] uses symbolic grid puzzles and does not test visual perception or closed-loop action; MINDTOPO pairs rendered scenes with reasoning and interactive planning. Spatial and world-model benchmarks commonly emphasize metric relations, viewpoint, intervention outcomes, physical fidelity, or affordances rather than invariance under continuous deformation.Source: D. Extended Related Work Discussion · pp. 63–65

E.1. Property-Level Prompt Templates

  • One prompt template per task type is held fixed across models, with representative property-level templates shown in Figures 14–18.The figures provide templates for continuity, separation, order, enclosure, and knots.Source: §E.1 · p. 65, Figure 14 · p. 65, Figure 15 · p. 66

E.2. Task-Level Prompt Cards

  • The task-level prompt cards reproduce complete prompts and qualitative examples grouped by topological category and task, with full specifications in Appendix A.The cards document the instructions, questions, rules, and required answer formats for the illustrated tasks.Source: §E.2 · p. 67

E.2.1. Continuity

  • Continuity examples cover maze reachability and single-bar removal as reasoning tasks, and pipe rotation toward source connectivity as an interactive task.The pipe task requires rotating non-empty cells 90 degrees clockwise per turn until all pipes connect to the source.Source: §E.2.1 · pp. 67, 68, 70

E.2.2. Separation

  • The One-Stroke task asks an agent to extend a path from start to goal so same-colored cells share a region and different-colored cells remain separated.Each turn selects a legal move, and success requires reaching the goal while satisfying the color-separation rule.Source: §E.2.2 · p. 75

E.2.3. Order

  • Order examples ask models to classify bead-string pairs as identical, reversed, cyclically rotated, or different, alongside origami-point examples across difficulty tiers and an interactive swap puzzle.The bead-string definition specifies the four sequence relationships, and the origami examples span bird, open-sink, pinwheel, and waterbomb bases.Source: §E.2.3 · pp. 77, 82, Figure 31 · p. 78, Figure 32 · p. 79
  • Order is tested with a 2D grid-swapping puzzle in which one colored block swaps with the empty cell each turn until the grid matches the target.Actions use zero-based row and column coordinates and must be legal for the current state.Source: §E.2.3 · p. 82

E.2.4. Enclosure

  • Enclosure examples cover hole detection, sheep escape through fence gaps, identifying the region with the most sheep, repairing fence gaps, and interactively trapping a cat.The tasks distinguish openings that pass through a board from shallow pits, define escape as a continuous path to the exterior through gaps, and require blocking one hex cell per turn in Chat Noir.Source: §E.2.4 · pp. 83, 85, 86, 87, 88

E.2.5. Knots

  • The knots subsection includes qualitative examples for structure classification, component count, link topology, link properties, and untangling.The supplied passages identify the figures’ topics but do not describe their results.Source: Figure 42 · p. 89, Figure 43 · p. 90, Figure 44 · p. 91

F. The Use of Large Language Models

  • The authors used Gemini 3.1 Pro and GPT-5.5 as auxiliary tools for writing, editing, and literature review, then had human authors critically revise and fact-check all content.The authors state that they are responsible for the paper’s statements and conclusions.Source: F. The Use of Large Language Models · p. 93

Related papers

Loading 2609.11900v1…