PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

Sumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Hammoud, Tavish McDonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Natasha Jaques, Ameya Prabhu, Brian Bartoldson, Bhavya Kailkhura, Christian Schroeder de Witt

arXiv:2604.14140v1cs.LGcs.AI

TL;DR

LongCoT addresses the limited direct measurement of models’ ability to reason reliably across long chains of interdependent steps. It benchmarks that ability with 2,500 expert-designed problems, and even the best evaluated frontier models achieve less than 10% accuracy. The benchmark is intended to measure long-horizon reasoning separately from tool use and scaffolding.

  • Problem

    Reliable reasoning over long chains becomes critical as models take on harder tasks, but existing benchmarks rarely stress-test reasoning across many interdependent steps.

    Source: §1 · p. 1

  • Method

    LongCoT is a scalable benchmark of 2,500 problems across five domains, using short prompts, verifiable answers, and graph-dependent reasoning chains that can span tens to hundreds of thousands of tokens.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Less than 10% accuracy is achieved by even the best evaluated frontier models on LongCoT, with GPT 5.2 reaching 9.83%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LongCoT directly measures whether models can sustain coherent reasoning across long dependent traces, separating that capability from tool use and scaffolding.

    Source: §5 · p. 9

    The three best closed-source frontier models average $13.66 per 1 million output tokens and $2.25 per 1 million input tokens, often about $1 per question.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.

1. Introduction

LongCoT targets the undermeasured ability to sustain reasoning across many interdependent steps, which existing evaluations often do not directly test. It introduces a scalable benchmark and finds that even the best evaluated frontier models achieve less than 10% accuracy.

1. Introduction

  • Current benchmarks rarely stress planning, long-term state maintenance, error monitoring, and credit assignment across many interdependent steps.Existing evaluations often focus on short reasoning, long inputs, or agentic workflows where tools and scaffolds contribute to performance.Source: §1 · pp. 1–2
  • LongCoT comprises 2,500 problems across chemistry, mathematics, computer science, chess, and logic, with short prompts, verifiable answers, and dependency graphs spanning tens to hundreds of thousands of reasoning tokens.The prompt has a median length of 2K tokens.Source: §1 · p. 2
  • Expert-designed parameterized templates scale problems through meaningful compositions or implicit search, while keeping atomic steps tractable so difficulty comes from dependencies and long-horizon control.The targeted demands include planning, error detection, context management, and backtracking.Source: §1 · pp. 2–3
  • The primary evaluation excludes tools and scaffolding; code execution improves some procedural domains, while compositional mathematics, chemistry, and computer science remain near zero.The unscaffolded track is intended to isolate planning, state maintenance, constraint propagation, and backtracking.Source: §1 · p. 3
  • Less than 10% accuracy is achieved by even the best evaluated frontier models, exposing a gap between existing benchmark performance and reliable long-horizon reasoning.The paper reports this result across evaluated open- and closed-source models.Source: §1 · p. 3

2. Related Work

Related benchmarks cover difficult reasoning, long-context inputs, or tool-using agentic tasks, but do not directly isolate very long chains of model reasoning. LongCoT is designed to address that distinction.

Frontier Reasoning Benchmarks

  • Frontier mathematics and science benchmarks typically induce fewer than 10K completion tokens and do not isolate reasoning behaviors needed at 100K+ tokens.The paper highlights planning, tracking variables, maintaining coherence, and suppressing error accumulation as long-horizon behaviors.Source: §2 · p. 4

Long-Context Benchmarks

  • Long-context evaluations often emphasize large inputs or moderately sized outputs, whereas LongCoT demands chains of thought often containing 100K+ tokens.The paper describes LongCoT as, to its knowledge, the first evaluation demanding such long outputs through individually tractable subproblems.Source: §2 · p. 4

Agentic Benchmarks

  • Agentic benchmarks measure whole-system success, but tool use and domain-specific scaffolding make it difficult to isolate the model’s underlying long-horizon reasoning.LongCoT separates reasoning from tool orchestration to extract signal about long-horizon reasoning effectiveness.Source: §2 · p. 4

3. Benchmark Construction

LongCoT is built from explicit dependency graphs and implicit search structures across five domains, using short prompts and verifiable answers to test extended reasoning over interdependent steps. Its expert-designed templates span different reasoning demands and difficulty levels.

3.1. Structure

  • Explicit templates expose dependency DAGs, while implicit templates encode them through rules or constraints; explicit DAGs use chains, multi-parent dependencies, conditionals, and forced backtracking.Source: §3.1 · p. 4
  • Together, these structures stress distant context retrieval, planning, branch exploration and execution, and detecting errors or dead ends late in a chain.Source: §3.1 · p. 5

3.2. Domains

  • Across chemistry, mathematics, chess, logic, and computer science, templates model molecule workflows, Olympiad DAGs, adversarial search, planning and constraint puzzles, and long deterministic simulations.Source: §3.2 · p. 6

3.3. Benchmark composition

  • The benchmark contains 2,500 problems across five domains (500 per domain): 500 easy LongCoT-mini items, plus 750 medium and 1,250 hard problems.Each domain uses 10 templates with 50 parameter settings each. Large combinatorial answer spaces make random guessing ineffective, while incorrect intermediate values rarely yield correct final answers, enabling reliable outcome-only evaluation.Source: §3.3 · pp. 6–7
  • 95.7%: GPT 5.2 accuracy on isolated math subproblems, yet accuracy on composed problems degrades beyond independent error compounding.Source: §3.3 · p. 6
  • 9.83%: GPT 5.2 is the top LongCoT model, while open-source models are near zero; LongCoT-mini differentiates performance across a wider range.Source: Figure 4 · p. 7

4. Evaluation

Frontier models perform poorly on LongCoT, and accuracy falls further as dependency graphs grow. Results and trace analyses point to planning, state management, and error control as persistent long-horizon challenges.

4.1. Frontier Models on LongCoT

  • The evaluation covers seven frontier models across five domains, using each model’s highest available CoT setting and maximum permitted output limit.The set includes four closed-source and three open-source models, evaluated in a single-shot setting.Source: §4.1 · p. 7
  • 9.83% was the highest LongCoT accuracy, achieved by GPT 5.2; Gemini 3 Pro reached 6.08% and Grok 4.1 Fast Reasoning 2.04%.On the easier 500-question LongCoT-mini, GPT 5.2 achieved 38.7%, while Kimi K2 and DeepSeek V3.2 reached 7.5% and 8.3%.Source: §4.1 · p. 7
  • For a given model, domain-specific results were mostly stable across all five domains.The authors interpret this as consistent with a benchmark designed to test long-horizon reasoning rather than deep domain knowledge.Source: Figure 5 · p. 8

4.2. Experimental Analysis

  • GPT 5.2 used an average of 62,046 tokens per problem, more than any other model, and performance rose with token usage.Models using more of their available reasoning budget did markedly better.Source: §4.2 · p. 8
  • All evaluated models degraded steeply as composed Omni-Math problems grew from 1 to 35 nodes, especially beyond 15; accuracy fell below the independent-error baseline.The model gap widened at longer horizons, while the figure caption reports that difficulties arise before context windows saturate.Source: §4.2 · p. 8, Figure 6 · p. 8
  • Reasoning-only RLM did not outperform GPT 5.2 alone, while code-enabled RLM gains appeared mainly on implicit domains: Logic 68.3% and Chess 30.6%.Explicit compositional domains remained near zero, and the authors attribute the implicit-domain gains to dependency structure that can be externalized to code.Source: §4.2 · pp. 8–9, Figure 7 · p. 8

4.3. Qualitative Analysis

  • Qualitative traces show that reasoning allocated to solving, planning, setup, verification, dead ends, and backtracking varies substantially by domain and model.The figure represents each cell as 1% of a trace, read left-to-right and top-to-bottom.Source: §4.3 · p. 9, Figure 8 · p. 9
  • Observed failure patterns include poor early planning, compounding errors, exhausted effective context, premature guessing or giving up, and failures to backtrack.These patterns are reported as general issues in long-horizon reasoning capabilities.Source: §4.3 · p. 9

5. Discussion

LongCoT isolates long-horizon reasoning by testing whether models sustain dependent reasoning over extended traces. Its low frontier-model accuracy points to persistent difficulties with planning, state maintenance, and error control.

5. Discussion

  • LongCoT targets coherent reasoning across tens to hundreds of thousands of tokens, separating this capability from tool use, retrieval, external memory, and task-specific orchestration.Its templates keep local steps tractable, so the authors interpret frontier accuracy below 10% as evidence of long-horizon-specific skill limits.Source: §5 · p. 9
  • The benchmark’s difficulty comes from maintaining plans, state, and error control over long traces, where drift and compounding mistakes can accumulate.The authors identify these abilities as relevant to autonomous deployment, where agents still need to maintain goals, constraints, and partial results over long trajectories.Source: §5 · p. 9
  • The authors propose targeting long-horizon stability in training and inference and using LongCoT to measure whether methods improve it.They describe reliable long-horizon reasoning as one of several necessary capabilities for ambitious applications such as scientific discovery, drug design, and complex engineering.Source: §5 · p. 9

A. Per-domain Results

LongCoT-mini offers a cheaper evaluation option while retaining signal across model classes.

A. Per-domain Results

  • LongCoT-mini costs approximately $150 to run with GPT 5.2, compared with approximately $2,500 for the full benchmark, while providing high signal across model classes.The appendix reports detailed performance by problem template and model for each domain.Source: A. Per-domain Results · p. 12

A.1. Logic

The listed logic-section problems span planning puzzles, state-tracking challenges, and algorithmic counting or optimization tasks.

A.1. Logic

  • BlocksWorld, RandomHanoi, and Sokoban require move planning, exploration, and recovery from dead ends or invalid moves.Their constraints differ: only top blocks move in BlocksWorld, Hanoi discs must obey size ordering, and Sokoban boxes can be pushed but not pulled.Source: §A.1 · p. 12
  • Dungeon and Sudoku emphasize tracking constraints across many steps, while PackagingMinWaste requires supplier selection and package assignments that minimize unused space.Dungeon seeks minimum starting health along a right-or-down path; Sudoku requires satisfying row, column, and subgrid rules.Source: §A.1 · pp. 12–13
  • TrapezoidCounting requires counting convex quadrilaterals with parallel sides, while WizardsTotalStrength sums subarray contributions based on each subarray’s minimum and sum.The supplied table caption identifies accuracy by Logic task template and difficulty.Source: §A.1 · p. 13, Table 2 · p. 13

A.2. Computer Science

The computer-science tasks test long-horizon reasoning through exact execution and state tracking across algorithmic traces and dependent subproblems. They span simulation, scheduling, optimization, type inference, and compiler analysis.

Other computer-science templates

  • Further tasks test long-context execution through Hindley-Milner type inference, VLIW scheduling, matrix-chain optimization, and chained LLVM IR analysis and transformations.The matrix-chain task also asks for traceable properties of the chosen decomposition; IR queries include post-pass instruction counts.Source: §A.2 · pp. 14–15

Max-Flow Min-Cut Gauntlet

  • The Max-Flow Min-Cut Gauntlet chains rounds of Edmonds-Karp flow computation, cut and route identification, and capacity updates into a final edge-capacity answer.Precise BFS ordering, DFS backtracking, and tie-breaking rules govern each round.Source: §A.2 · p. 14

Chained Scheduling Simulation

  • Eight dependent scheduling subproblems feed outputs forward, requiring precise algorithm execution, tie-breaking, and tracking of job states and completion times.The algorithms include EDF, DAG list scheduling, round-robin, and SJF.Source: §A.2 · p. 14

Turing Machine Simulation

  • A three-tape Turing machine is simulated across eight dependent problems, with each answer setting parameters for the next and requiring state and tape tracking.The final state encodes the state identifier, head positions modulo a prime, and symbols under each head.Source: §A.2 · p. 14

Program and value tracing

  • Other tasks require tracing MPI values or backpropagation values under interventions, and exactly executing loop programs to answer trace-dependent questions.The loop questions include locating iterations, counting predicate matches, and reporting run extrema.Source: §A.2 · pp. 14–15

A.3. Chemistry

The chemistry tasks combine molecular analysis and reaction prediction with chained choices that can affect later questions. They test reasoning over molecular properties, structure, representations, and substructures.

Chained molecule selection

  • Most chemistry subquestions select molecules for later questions by ranking a property or applying a threshold, making earlier selections consequential.Reaction prediction can also prompt backtracking when selected reactants are incompatible.Source: §A.3 · pp. 15–16

Molecule Properties

  • Molecular-property tasks require comparing quantities such as molecular weight, hydrogen counts, and heavy-atom counts to select molecules, rather than relying on surface heuristics.The property set also includes aromaticity, chirality, and Lipinski’s rule of 5.Source: §A.3 · pp. 15–16

Reaction Prediction

  • Reaction Prediction asks models to predict products from reactants and conditions, testing functional-group compatibility, mechanisms, and competing pathways.Reactants may be specified by prior subquestions, and the prompt permits using given conditions or other appropriate conditions.Source: §A.3 · p. 16

Structure and representations

  • Molecular-topology questions require reasoning over atom-graph distances, including diameter, Wiener index, and eccentricity, while ring tasks ask for ring counts and sizes.Substructure tasks test recognition of common motifs, and representation tasks test equivalence across distinct SMILES strings or adjacency matrices.Source: §A.3 · p. 16

Proteins Substructures

  • Protein Substructures tasks combine backbone geometry, secondary-structure prediction, missing-residue inference, and structure-aware residue ranking.The missing-residue predictions use local structural context, including compactness and backbone angles.Source: §A.3 · p. 16

A.4. Chess

The chess tasks cover move selection, reverse game reconstruction, long move-sequence simulation, and combinatorial or constrained pathfinding problems. Their settings require maintaining board state and reasoning over extended or globally constrained choices.

Templates 1&6: Next Best Moves

  • Move-selection tasks rank the top three continuations or identify one optimal puzzle move by integrating piece activity, king safety, material balance, and tactical threats.The positions are supplied in FEN notation.Source: §A.4 · p. 16

Template 2: Reverse Game Reconstruction

  • Reverse Game Reconstruction requires constructing a legal alternating SAN sequence consistent with the final board, side to move, and required move fragments.Early choices constrain later legality, while SAN notation requires exact board-state tracking.Source: §A.4 · p. 16

Templates 3&8: Move Sequence Simulation

  • Move Sequence Simulation applies hundreds of UCI moves to produce a final FEN, with one template also asking for the best next move.The templates test state tracking across extended sequential operations.Source: §A.4 · p. 17

Templates 4&5

  • The combinatorial tasks include a knight’s minimum-move route across a 100×100 board and a minimax pawn-capture game requiring anticipation of an opponent’s strategy.The knight route must visit all targets; in the game, Alice maximizes total moves and Bob minimizes them.Source: §A.4 · p. 17

Templates 7, 9&10

  • Other tasks count collision-free simultaneous piece moves, maximize rook placement after obstacle deletions, or find a knight path avoiding squares attacked by enemy pieces.The rook task relies on global structure after deletions, while constrained pathfinding first requires computing enemy attack coverage.Source: §A.4 · p. 17

A.5. Math

The Math section describes five templates for organizing interdependent problems and illustrates instantiated templates across domains, clarifying that diagrams are not model inputs. Its example traces a long-horizon chain in which molecule selection and reaction tasks feed into later graph-property calculations.

A.5. Math

  • Five math templates encode linear dependencies, DAG dependencies, conditional branching, inverse-function backtracking, or a DAG presented before its problems.The five templates are Linear, DAG, Conditional, Backtrack, and DAG-First.Source: §A.5 · p. 18

B. Example Templates

  • Example templates span domains and include a matrix-chain task asking for an efficient parenthesization and related structural and computational properties.The examples note that graphical depictions are illustrative only and are not provided to models; the matrix-chain questions cover operation count, nesting depth, and parenthesis span.Source: B. Example Templates · pp. 18, 22, 23

A.5. Math

  • The nine-subproblem template combines fingerprint-bit and SMILES selections, reaction predictions, formula and hydrogen checks, and topological-diameter calculations in one final answer.An accompanying illustration shows the molecular structures referenced in the questions.Source: B. Example Templates · pp. 27–28

A B C Target-2 2.1 2.2

The examples illustrate benchmark problems built from linked subproblems or strategic search, with final answers that depend on managing those structures. The chemistry example chains six protein-structure tasks; the template examples include a pawn-capture game and a chess-position puzzle.

A B C Target-2 2.1 2.2

  • The protein-structure example chains six subproblems, from locating missing residues and computing backbone features to predicting structure, residue identity, and a final ranking.The final ranking uses neighbor counts for predicted α-helices or distance from the protein geometric center for β-strand or loop regions.Source: A B C Target-2 2.1 2.2 · pp. 28–29
  • The chemistry example’s final ranked residue list is [PRO, PRO, ILE, TYR, ASN, GLU, GLY].The ranking criteria use a 12 Å neighbor-count radius for α-helices and distance to the protein geometric center for β-strand or loop regions.Source: A B C Target-2 2.1 2.2 · pp. 29–30

Template

  • The templates include a 30 × 30 knight-and-pawn game requiring optimal alternating choices, plus a chess-position prompt asking for the next best move.In the pawn game, Alice maximizes and Bob minimizes the total moves while each captures a selected pawn in the fewest possible knight moves.Source: Template · pp. 30–31

C. Additional Analysis

Additional analyses show that LongCoT-mini differentiates model performance, while composed dependencies—not comparable output length alone—drive a sharp accuracy drop. The authors also examine long-horizon failure patterns, scaffolding, and reasoning traces, with analysis scope limited in some cases.

LongCoT-mini

  • LongCoT-mini results show GPT 5.2 performs significantly better than all other models and help differentiate open-source models with near-zero full-benchmark performance.Its parameterized structure also allows finer-grained study of performance across horizon lengths.Source: C. Additional Analysis · p. 32

Long-Horizon Reasoning Challenges

  • Long-horizon failures include inefficient early plans, compounding intermediate errors, premature giving up, and repeated shortcuts instead of switching strategies.The authors report that these behaviors interact, and that models may continue along an incorrect path rather than backtracking.Source: C. Additional Analysis · p. 33
  • The authors expect LongCoT to support research on long-horizon reasoning and yield practical benefits for complex autonomous tasks.They describe this as an expectation based on consistently observed behaviors.Source: C. Additional Analysis · p. 33

Compositional Reasoning vs. Context Length

  • On medium questions, GPT-5.2 scores 58% on independent versus 8% on composed questions; on hard questions, it scores 55% versus 4%, despite comparable token usage.The comparison uses the same leaf nodes; the authors identify compositional dependency structure, rather than context length, as the primary failure driver.Source: C. Additional Analysis · pp. 33–34, Figure 10 · p. 33

Recursive Language Models

  • With code simulation enabled, performance is Logic: 68.3%, Chess: 30.6%, and CS: 26.7%, while Math and Chemistry remain at zero.Improvements occur primarily on implicit domains where parts of the dependency structure can be externalized to code; code execution does not remove LongCoT’s core difficulty.Source: C. Additional Analysis · p. 34
  • Recursive sub-agents struggle when dependency information, tracked variables, or failed exploration paths do not carry across calls; implicit tasks also expose planning gaps.The authors distinguish context decomposition from task decomposition and note that sequential or retrievable structures are better suited to context-folding approaches.Source: C. Additional Analysis · p. 34

Reasoning Trace Analysis

  • In LongCoT-mini traces, incorrect traces show roughly twice the backtracking rate, while correct traces devote more reasoning budget to setup and comprehension.The analysis uses open-source models because closed-source reasoning traces are unavailable, and draws 20 correct and 20 incorrect DeepSeek V3.2 traces plus 16 of each for Kimi K2 Thinking.Source: C. Additional Analysis · p. 34, Figure 11 · p. 35

D. Limitations and Future Work

The authors identify evaluation cost and provider inconsistency as practical constraints, note that task-specific training remains unexplored, and qualify how directly some tasks apply to real workflows.

D. Limitations and Future Work

  • Evaluation costs often exceed $1 per API call, preventing reporting of pass@k and majority-voting scores; GPT 5.2 results also varied across providers.The authors plan to rerun Mathematics and Computer Science evaluations directly with the OpenAI API for greater consistency.Source: D. Limitations and Future Work · p. 35
  • LongCoT models were not trained specifically for these tasks, and such training would require substantial compute because outputs regularly approach 100K tokens.The authors suggest training-free approaches as a more tractable direction and report a preliminary study of RLMs.Source: D. Limitations and Future Work · p. 35
  • Some questions may not apply directly to realistic workflows, although LongCoT isolates long-horizon reasoning from scaffolding and tool-calling.The authors release the benchmark publicly.Source: D. Limitations and Future Work · p. 35

Related papers

Loading 2604.14140v1…