PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoBench: Benchmarking LLMs on Hard Topological Reasoning

Mayug Maniparambil, Nils Hoehing, Janak Kapuriya, Arjun Karuvally, Ellen Rushe, Anthony Ventresque, Noel O'Connor, Fergal Reid

arXiv:2603.12133v1cs.AIcs.CL

TL;DR

LLMs remain weak at maintaining global spatial invariants in topological grid puzzles, a capability that existing benchmarks rarely isolate. TopoBench evaluates six puzzle families and diagnoses failures through trace annotation, causal interventions, and targeted mitigations. Hard-instance performance remains below 25%, and the evidence points to spatial-constraint extraction and maintenance—not reasoning over available constraints—as the primary bottleneck.

  • Problem

    Existing benchmarks rarely test whether LLMs can maintain global spatial invariants through sequential spatial updates or distinguish reasoning failures from spatial-representation failures.

    Source: §1 · p. 1

  • Method

    TopoBench evaluates six puzzle families across three difficulty tiers, annotates 750 reasoning traces, tests error patterns through causal interventions, and evaluates representation, tool, and prompt mitigations.

    Source: §1 · pp. 1–2, §4.2 · p. 6

  • Results

    Performance stays below 25% on hard instances, while premature commitment and constraint forgetting cause large accuracy drops and structured constraint information improves selected performance.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Models struggle more with extracting and maintaining constraints from spatial representations than with reasoning once structured constraints are available.

    Source: §1 · p. 2, §5 · p. 11

    The causal analysis covers one model and two puzzle families, tool augmentation covers one model and one puzzle family, and all mitigations use inference-time pass@1 evaluation.

    Source: §5 · p. 11

Abstract

from arXiv · show

Solving topological grid puzzles requires reasoning over global spatial invariants such as connectivity, loop closure, and region symmetry and remains challenging for even the most powerful large language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a benchmark of six puzzle families across three difficulty levels. We evaluate strong reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated reasoning is a benign effect of search. Finally we study mitigation strategies including prompt guidance, cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in reasoning over them. Code and data are available at github.com/mayug/topobench-benchmark.

1 INTRODUCTION

TopoBench targets global topological and geometric constraints that remain under-studied in LLM benchmarks, then diagnoses whether failures reflect reasoning or spatial-constraint handling. Across evaluation and interventions, models struggle on hard instances, while constraint extraction and maintenance emerge as the central bottleneck.

  • TopoBench comprises six topology-focused puzzle families spanning connectivity, loop closure, symmetry, reflection, and contiguity, with three difficulty tiers and rule-based verification.Source: §1 · p. 1, Figure 1 · p. 2
  • 0.24 accuracy is achieved by GPT-5-mini-high on hard instances, while the best open-weight model, DeepSeek V3.2, reaches 0.10.Source: §1 · p. 2
  • Premature commitment and constraint forgetting each cause approximately 11 percentage-point accuracy drops when injected on Bridges and Undead, whereas repeated reasoning has no measurable causal effect.Source: §1 · p. 2
  • Error frequency poorly predicts causal harm: constraint forgetting appears in only 2–7% of traces but ranks among the most damaging interventions, while repeated reasoning is frequent but benign.Source: §1 · p. 2
  • Cell-aligned representations improve accuracy across most families, and structured constraint information improves Bridges hard accuracy by 10%, whereas ASCII-grid tools and prompt guidance do not reliably help.Source: §1 · p. 2
  • The interventions indicate that models struggle more with extracting and maintaining structured constraints from spatial representations than with reasoning once those constraints are available.Source: §1 · p. 2

2 RELATED WORK

Prior reasoning benchmarks largely emphasize algebraic, logical, textual, or local spatial tasks rather than maintaining global topological invariants. TopoBench extends related work by combining topological evaluation with causal diagnosis of reasoning failures and representation-sensitive tool interventions.

  • Existing benchmarks rarely isolate global invariants such as path connectivity, loop closure, and region symmetry across an entire grid.Source: §2 · p. 3
  • TopoBench differs from grid and puzzle benchmarks that emphasize local pattern matching, cell-level arithmetic, or interactive visual reasoning by targeting topology-focused constraints.Source: §2 · p. 3
  • Unlike purely observational error taxonomies, the diagnostic pipeline combines LLM-judge trace annotation with controlled interventions on gold solution prefixes to test causal impact.Source: §2 · p. 3
  • Representation studies motivate testing evenly tokenized grid rows, while tool augmentation separates state tracking and constraint calculation from the model’s reasoning process.Source: §2 · p. 3

3 BENCHMARK AND METHODOLOGY

TopoBench evaluates textual multi-step reasoning over six families of global spatial constraints using controlled difficulty, diverse reasoning models, one-shot prompts, and dedicated verifiers. Its construction varies board size and deductive depth so harder tiers demand deeper spatial reasoning rather than merely larger grids.

Task Families

  • TopoBench evaluates six puzzle families covering network and path connectivity, rotational symmetry, reflection-based visibility, contiguity, and loop closure.Source: §3 · p. 4, Table 1 · p. 4

Dataset Construction

  • The dataset contains 900 instances from 50 puzzles per family and difficulty, with easy, medium, and hard tiers scaled by board size and generator-controlled deductive depth.Source: §3 · p. 4
  • Difficulty scaling combines larger boards with deeper deduction requirements, while de-duplication prevents identical board configurations or solutions within puzzle-by-tier splits.Source: §3 · p. 4

Input Representation and Prompting

  • Models receive ASCII grid inputs, complete rules, and one worked example in one-shot prompts, with no intermediate reasoning traces and up to 100k evaluation tokens.Source: §3 · p. 5

Evaluated Models and Execution

  • Nine models spanning closed and open source families, dense, mixture-of-experts, and diffusion architectures are evaluated with single-attempt pass@1 scoring and no external code execution.Source: Table 2 · p. 4, §3 · p. 5

Verification

  • Dedicated puzzle-specific verifiers reconstruct proposed grids and apply the full constraint set using binary correct/incorrect scoring without partial credit.Source: §3 · p. 5

4 EXPERIMENTS AND ANALYSIS

TopoBench exposes a sharp difficulty boundary in topology-heavy reasoning: frontier models perform well mainly on easy instances but broadly fail on hard ones. Diagnosis and intervention experiments indicate that extracting and maintaining spatial constraints, rather than reasoning over already structured constraints, is the primary bottleneck.

4.1 MAIN RESULTS

  • 0.24 hard-tier accuracy is achieved by the strongest tested model, while the best open-weight model reaches 0.10; performance collapses broadly on hard instances.Galaxies and Loopy remain near zero beyond easy, whereas Bridges and Pattern retain measurable medium- and hard-tier accuracy.Source: §4.1 · pp. 5–6
  • The benchmark evaluates six topology-focused puzzle families across three difficulty tiers with verifier-checked solutions and nine reasoning models.Families target connectivity, loop closure, symmetry, reflection, and contiguity, enabling controlled diagnosis across constraint types.Source: §4.1 · p. 5

4.2 DIAGNOSING FAILURE MODES

  • 750 traces are classified with an eleven-category error taxonomy, focusing on 455 incorrect traces to distinguish observable failure patterns from causal contributors.The analysis uses LLM-as-a-judge labeling and supporting excerpts, while Table 4 separates plausibly causal categories from symptomatic ones.Source: §4.2 · p. 6, Table 4 · p. 7
  • 76% of failed traces contain explicit surrender, while repeated reasoning, premature commitment, and representation drift each occur in about one-third of aggregate failures.On Bridges, state-tracking failure reaches 47%; on Undead, explicit surrender is 70%, representation drift 43%, and premature commitment 37%.Source: §4.2 · pp. 6–7, Figure 2 · p. 7
  • 20.8 pp and 11.3 pp accuracy drops follow premature commitment injection on Bridges and Undead, while repeated reasoning is indistinguishable from baseline.Constraint forgetting also causes large significant drops, whereas state-tracking failure is weaker on Bridges but significant on Undead; length-matched controls show added context alone is not harmful.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3.1 INPUT FORMAT INTERVENTION

  • Cell-aligned integer encodings improve Bridges and Galaxies by roughly +30 to +40 pp but can reduce Undead accuracy by 26 pp for DeepSeek.Image augmentation is unreliable, and prompt interventions do not significantly improve hard Bridges performance.Source: §4.3.1 · p. 9, §4.3.2 · p. 10

4.3.2 TOOL-AUGMENTED REASONING

  • 50% accuracy is reached on hard Bridges with the full constraint-query tool suite, compared with 40% without tools; adding spatial rendering can reduce the two-tool result from 46% to 42%.Structured tools eliminate board-validity errors, while renderings are called frequently and may interfere with algebraic reasoning.Source: §4.3.2 · p. 10, Table 6 · p. 10
  • Across format and tool interventions, the primary bottleneck is extracting constraints from spatial representations rather than reasoning over structured constraints.Cell-aligned formats ease compilation, while structured tools externalize authoritative state and constraint information.Source: §4.3.1 · p. 9, §4.3.2 · p. 10

5 CONCLUSION

TopoBench finds topology-heavy spatial reasoning remains difficult on hard instances, while mitigation gains are uneven and point to constraint extraction as the central bottleneck. The conclusion also bounds these findings by limited causal coverage, single-model tool tests, inference-time pass@1 evaluation, and trace-based error labels.

5 CONCLUSION

  • Performance stays below 25% on hard instances, while input representations and tool augmentation improve performance unevenly across puzzle families.The supplied passages identify this as the benchmark-level outcome and emphasize uneven mitigation gains.Source: §5 · pp. 10–11
  • Tool ablations indicate that structured constraint information, rather than repeated spatial renderings, accounts for improvements on Bridges.The conclusion attributes the gain to the information supplied by the tool, not repetition of visual state descriptions.Source: §5 · p. 11, Table 7 · p. 15
  • Tool-augmented Bridges experiments point to extracting and maintaining constraints from spatial representations as a larger bottleneck than reasoning once constraints are available.This is the paper’s stated interpretation of the tool evidence.Source: §5 · p. 11
  • The causal analysis covers one model and two puzzle families, while tool augmentation covers one model and one puzzle family.These scope limits constrain how broadly the causal and tool findings can be generalized.Source: §5 · p. 11
  • All mitigations are inference-time with pass@1, and trace-derived error labels may not faithfully reflect the model’s internal computation.The limitations section also notes that visible trace errors may not be the true causes of failure.Source: §5 · p. 11

B.2 ORTHOGONALITY TO EXISTING BENCHMARKS

TopoBench is evaluated against established puzzle and reasoning benchmarks to test whether it measures a distinct capability. Its weaker average correlations with general reasoning scores suggest topology-heavy tasks are complementary and less well predicted by standard benchmarks.

B.2 ORTHOGONALITY TO EXISTING BENCHMARKS

  • The comparison tests TopoBench against KORGym, Enigmata, ARC-AGI-1/2, AIME 2025, and AA Intelligence using model-score correlations.These benchmarks provide the reference set for evaluating overlap with existing capabilities.Source: §B.2 · p. 15
  • TopoBench shows weaker average correlations with general reasoning benchmarks than KORGym and Enigmata, which strongly correlate with those benchmarks.The comparison uses Spearman rank correlations across benchmark pairs.Source: §B.2 · p. 15, Figure 5 · p. 17
  • The results suggest TopoBench probes sustained state tracking, global constraint propagation, and representation-sensitive spatial inference that standard reasoning scores predict less well.These are the skills identified as less captured by existing benchmark performance.Source: §B.2 · p. 15
  • TopoBench therefore serves as a complementary evaluation tool because high scores on existing reasoning benchmarks may not imply high scores on topology-heavy tasks, and vice versa.The paper characterizes topological and spatial reasoning under high state-tracking load as a partially independent capability dimension.Source: §B.2 · pp. 15–16
  • The supplied table caption describes per-model accuracy changes by input format relative to each model’s Plain ASCII baseline, but gives no values here.Only the table’s comparison structure is supported by the supplied passage.Source: Table 8 · p. 16

D FULL ERROR TAXONOMY

The full taxonomy contains eleven error categories, while the main narrative focuses on seven categories and omits four low-frequency or trendless categories from the main figures.

D FULL ERROR TAXONOMY

  • The taxonomy contains eleven error categories, with seven used in the main narrative and four additional categories omitted from main-body figures.The omitted categories are UE, HV, TD, and OTH.Source: D FULL ERROR TAXONOMY · p. 20, Table 10 · p. 20
  • Categories above the first taxonomy divider are treated as plausibly causal, whereas categories below it are treated as symptomatic.The table caption defines this organizational distinction without claiming every category is causally validated.Source: Table 10 · p. 20

E PER-PUZZLE ERROR ANALYSIS

Per-puzzle error patterns vary sharply: FlowFree and Pattern show distinctive scaling or representation failures, while Bridges concentrates failures at hard difficulty. Early-surrender traces are longer and cluster near the token ceiling, but difficulty confounds that interpretation.

E PER-PUZZLE ERROR ANALYSIS

  • FlowFree’s premature-commitment rate rises from 14% to 36% to 48% across easy, medium, and hard tiers, while explicit surrender reaches 96% at hard.Repeated reasoning peaks at 52% on medium before declining to 48% on hard.Source: E PER-PUZZLE ERROR ANALYSIS · p. 20
  • Galaxies reaches 68% explicit surrender at easy and 100% at medium, while repeated reasoning falls from 52% at easy to 8% at hard.The passage characterizes this as early surrender before meaningful commitments.Source: E PER-PUZZLE ERROR ANALYSIS · p. 20
  • Pattern has representation-drift rates of 60%, 60%, and 46% across easy, medium, and hard tiers, with premature commitment jumping to 66% at hard.The passage links the drift pattern to simultaneous row and column constraint reasoning.Source: E PER-PUZZLE ERROR ANALYSIS · p. 20
  • Undead’s premature commitment reaches 46% at hard, while state-tracking failure rises from 4% to 14% to 36% across difficulty tiers.The passage relates this pattern to richer monster-type and visibility state.Source: E PER-PUZZLE ERROR ANALYSIS · p. 20
  • Bridges has the lowest explicit-surrender and repeated-reasoning rates, but hard instances concentrate state-tracking failure, premature commitment, and representation drift at 36%, 14%, and 14%.This profile motivates Bridges as a target for causal intervention.Source: E PER-PUZZLE ERROR ANALYSIS · p. 20, Figure 6 · p. 21

E.1 EARLY STOPPING AND TOKEN USAGE

  • Explicit-surrender traces use 1.23× more output tokens than other failed traces, with no such trace below 22,305 tokens and average lengths of 122k versus 96k characters.The low variance and token clustering are consistent with approaching a reasoning limit, but do not establish that interpretation alone.Source: §E.1 · p. 21, Table 11 · p. 21
  • The early-surrender analysis cannot fully separate puzzle difficulty from surrender tendency, and within-cell comparisons are limited by small samples of other failures.This caveat weakens causal interpretation of the token-length differences.Source: §E.1 · p. 21

F ERROR PREVALENCE BY OUTCOME

Outcome-associated error frequencies diverge from intervention-based causal effects: errors that co-occur with failure are not necessarily the errors that cause it. Across puzzle types, constraint forgetting and premature commitment emerge as robustly harmful, while repeated reasoning is benign.

F ERROR PREVALENCE BY OUTCOME

  • On Bridges, state-tracking failure is the strongest outcome discriminator (+44 pp), while on Undead, error surrender (+69 pp) and state-tracking failure (+27 pp) lead.These deltas aggregate incorrect-minus-correct prevalence across all three difficulty tiers.Source: F ERROR PREVALENCE BY OUTCOME · p. 21, Figure 7 · p. 22
  • The correctness-association ranking differs from the causal ranking: ES and RR correlate with failure, whereas PC and CF have the strongest intervention effects.CF is nearly non-discriminating observationally but causes an 10–11 pp accuracy drop when injected.Source: F ERROR PREVALENCE BY OUTCOME · p. 22, Figure 7 · p. 22

G INTERVENTION DETAILS

G.3 CROSS-PUZZLE COMPARISON: BRIDGES VS UNDEAD

  • The intervention results were replicated on Undead across Easy/4×4, Medium/5×5, and Hard/7×7 using 100 puzzles per tier and 300 puzzles per condition.The same six conditions and DeepSeek-v3.2 model were used, with CF subtypes adapted to Undead mechanics.Source: §G.3 · p. 23, Table 14 · p. 23

G.4 DETAILED PER-CONDITION ANALYSIS

Interventions identify premature commitment and constraint forgetting as consistently harmful, while repeated reasoning is not causal; effects vary with puzzle structure. Baseline calibration indicates that gold-prefix interventions operate in a comparable regime to standard evaluation.

G.4 DETAILED PER-CONDITION ANALYSIS

  • Premature commitment causes the largest Bridges effect, reducing accuracy by 20.8 pp and collapsing hard-tier accuracy to 8.0%.Easy accuracy is unaffected, while Medium falls from 82.0% to 43.0%, showing that the effect concentrates when recovery requires extensive backtracking.Source: §G.4 · p. 23
  • Constraint forgetting reduces accuracy by 10.6 pp on Bridges and 11.3 pp on Undead because internally consistent illegal states require semantic constraint verification.The model trusts and builds on an illegal state without checking move legality.Source: §G.4 · pp. 23–24

G.5 UNDEAD-SPECIFIC FINDINGS

  • State-tracking failure is puzzle-dependent: its effect is borderline on Bridges (−7.8 pp) but significant on Undead (−11.7 pp).Undead’s richer monster and visibility state may increase sensitivity to grid/text inconsistencies, whereas CF creates harder-to-detect semantic violations.Source: §G.4 · p. 24, §G.5 · p. 24
  • Repeated reasoning does not significantly differ from baseline on either puzzle, and length-matched controls show that added context alone has no causal effect.The pattern is therefore more consistent with search-related behavior than a failure mechanism.Source: §G.4 · p. 24, §G.5 · p. 24
  • Undead is a harder evaluation regime than Bridges, with 44.7% versus 66.2% baseline accuracy and hard Undead accuracy near the 9% floor.Repeated reasoning still trends negatively by 4–6 pp without reaching significance.Source: §G.5 · p. 24

G.6 BASELINE CALIBRATION AGAINST STANDARD EVALUATION

  • A methodological limitation is that presenting the first approximately 15% of a gold solution could alter the model’s solving distribution and interact with injected errors.The calibration comparison reduces, but does not eliminate, this concern.Source: §G.6 · p. 24
  • The gold-prefix intervention baseline does not systematically boost or degrade performance relative to standard evaluation, supporting interpretation of injected-error drops as causal.Undead totals differ by less than 1 pp, while Bridges shows a larger gap concentrated at medium and hard tiers.Source: §G.6 · pp. 24–25, Table 15 · p. 25

H PROMPT-BASED INTERVENTIONS TARGETING PREMATURE COMMITMENT

Prompt interventions targeting premature commitment do not improve full Bridges solving and often harm hard instances. The tested conditions suggest that prompt-level guidance cannot reliably activate the search and recovery behavior required by difficult puzzles.

H PROMPT-BASED INTERVENTIONS TARGETING PREMATURE COMMITMENT

  • The study evaluates DeepSeek-v3.2 on 50 medium and 50 hard Bridges puzzles per condition across ten strategy, example, and recovery conditions.The interventions target full non-partial puzzles and premature commitment.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 25
  • Conditions include backtracking, planning, self-correction, gold-path examples, worked examples, and concise or verbose PC-recovery demonstrations.The examples vary in puzzle size, reasoning verbosity, and whether they demonstrate avoidance or recovery.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 25
  • No prompt intervention significantly outperforms baseline: concise PC recovery reaches 61% versus the 60% baseline, within sampling noise.The baseline is 80% on medium and 40% on hard puzzles.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 25, Table 16 · p. 26
  • Hard accuracy declines as prompts lengthen: a single instruction lowers hard accuracy from 40% to 24–30%, while two-shot examples reach only 30%.Among one-shot examples, the compact gold path without natural-language reasoning performs best.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 25
  • All three text-only strategy instructions degrade hard accuracy by 10–16 pp, indicating that abstract advice is not followed effectively by the extended-thinking process.The result is consistent with prompt guidance failing to control the model’s internal strategy.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 26
  • Prompt-level changes alone do not elicit robust planning or recovery; the authors suggest training on puzzle-solving data, potentially with reinforcement learning, may be needed.Concise recovery is less harmful than avoidance-oriented guidance but still does not significantly improve performance.Source: H PROMPT-BASED INTERVENTIONS TARGETING… · p. 26

I TOOL-AUGMENTED REASONING: DETAILS

  • The tool appendix specifies five puzzle-engine tools divided between state mutation and information queries, while leaving move selection and search to the model.The engine validates moves, renders state, reports constraints, and identifies components or candidate links.Source: I TOOL-AUGMENTED REASONING: DETAILS · p. 26, §I.1 · pp. 26–27

I.2 FULL ABLATION RESULTS

Tool ablations show that structured constraint information, rather than spatial board rendering, drives accuracy gains. Across ablations, the evidence converges on spatial constraint extraction as the primary bottleneck.

I.2 FULL ABLATION RESULTS

  • State-mutation tools alone reduce hard Bridges accuracy to 26%, but adding state summary raises it by 16 pp to 42%.Neighbors, components, and candidate links add only 2, 4, and 2 pp respectively.Source: §I.2 · p. 27, Table 17 · p. 27
  • The full tool suite improves medium Bridges accuracy by 12 pp, from 80% to 92%, while reducing token consumption by 11%.Board-validity errors affecting 10% of baseline outputs are eliminated.Source: §I.2 · p. 27

I.3 SPATIAL VS. STRUCTURED INFORMATION

  • Spatial state tools return ASCII grids, whereas constraint-query tools return structured JSON containing pre-computed quantities such as remaining degree, joint capacity, and connected components.These derived quantities are not directly available in the raw grid and otherwise require algorithmic computation.Source: §I.3 · pp. 27–28
  • Although the ASCII grid is lossless, the model must scan directions, decode bridge characters, compute degree sums, and check intersections to extract actionable constraints.State summary instead shifts the task toward algebraic comparison using numeric fields such as remaining: 2.Source: §I.3 · pp. 27–28

I.4 TRACE-LEVEL STATISTICS

  • With state-mutation tools only, the model calls render board 12.5 times per puzzle and enters trial-and-error loops; state summary reduces calls to 8, and full tools reduce them to 6.Removing render board improves accuracy by 4 pp with state summary and has no effect with the full stack.Source: §I.4 · p. 28

I.5 CONVERGENCE OF EVIDENCE

  • Three lines of evidence converge on spatial constraint extraction as the bottleneck: cell-aligned formatting helps, structured tools recover accuracy, and removing the grid improves or preserves performance.The spatial representation is never beneficial when structured constraint tools are available.Source: §I.5 · p. 28

J FULL RESULT TABLES

The appendix documents benchmark result tables, input formats, evaluation prompts, and the procedures used to classify reasoning behaviors. It also defines the structural errors tracked in the analysis.

J FULL RESULT TABLES

K COST

  • Inference-cost estimates use output-token cost only, excluding marginal input-token cost.Source: K COST · p. 28

L INPUT FORMATS

  • The same Bridges puzzle is represented as ASCII, intformat, and intformat JSON for input-format comparison.Source: Figure 8 · p. 32

BEHAVIOUR CATEGORIES

  • Constraint forgetting is an explicit proposed action that violates structural rules, such as overcounting, crossing, invalid connections, or overwriting fixed cells.Speculative invalid moves later rejected and incorrect deductions from valid constraints are excluded.Source: BEHAVIOUR CATEGORIES · p. 33
  • Premature commitment is persistence for multiple steps after an early move or assumption is provably wrong, including wrong elements retained in the final answer.Repeated reasoning instead requires near-identical exploration of an already tried path, whereas different-path backtracking is normal search.Source: BEHAVIOUR CATEGORIES · p. 33
  • The taxonomy also covers topic drift, state-tracking failure, explicit surrender, incomplete output, unjustified elimination, hallucinated validation, and representation drift.State-tracking failure requires divergence between claimed board state and the cumulative effect of stated actions.Source: BEHAVIOUR CATEGORIES · p. 33

M ERROR-ANALYSIS JUDGE PROMPT

N EVALUATION PROMPTS

Related papers

Loading 2603.12133v1…