PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Context Limits: Subconscious Threads for Long-Horizon Reasoning

Hongyin Luo, Nathaniel Morgan, Tina Li, Derek Zhao, Ai Vy Ngo, Philip Schroeder, Lijie Yang, Assaf Ben-Kish, Jack O'Brien, James Glass

arXiv:2507.16784v1cs.CL

TL;DR

LLMs’ context limits and the overhead of complex agent workflows constrain long-horizon reasoning and tool use. The paper introduces TIM, which reasons over recursive task structures, and TIMRUN, which manages working memory and tool integration. The authors report strong reasoning ability, efficient inference and tool use, and agentic-benchmark performance matching strong baselines.

  • Problem

    LLMs’ context limits constrain long-horizon reasoning, while multi-agent workflows add coordination and tool-processing overhead.

    Source: §1 · pp. 1–2

  • Method

    TIM decomposes complex tasks into recursive subtasks, while TIMRUN prunes unhelpful subtask memory and integrates tool responses during inference.

    Source: §1 · pp. 2–3

  • Results

    TIM without explicit agent-specific design matches strong baselines on agentic benchmarks, and TIM with TIMRUN delivers strong reasoning ability, efficient inference and tool use, and greater flexibility and scalability for agentic tasks.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    The combined system supports agentic tasks with a model and serving infrastructure designed for structured reasoning, memory management, and end-to-end tool integration.

    Source: §1 · p. 3, §5 · pp. 10–11

    The synthetic training dataset’s quality is questionable because tool responses were synthesized rather than produced by actually calling tools.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

To break the context limits of large language models (LLMs) that bottleneck reasoning accuracy and efficiency, we propose the Thread Inference Model (TIM), a family of LLMs trained for recursive and decompositional problem solving, and TIMRUN, an inference runtime enabling long-horizon structured reasoning beyond context limits. Together, TIM hosted on TIMRUN supports virtually unlimited working memory and multi-hop tool calls within a single language model inference, overcoming output limits, positional-embedding constraints, and GPU-memory bottlenecks. Performance is achieved by modeling natural language as reasoning trees measured by both length and depth instead of linear sequences. The reasoning trees consist of tasks with thoughts, recursive subtasks, and conclusions based on the concept we proposed in Schroeder et al, 2025. During generation, we maintain a working memory that retains only the key-value states of the most relevant context tokens, selected by a rule-based subtask-pruning mechanism, enabling reuse of positional embeddings and GPU memory pages throughout reasoning. Experimental results show that our system sustains high inference throughput, even when manipulating up to 90% of the KV cache in GPU memory. It also delivers accurate reasoning on mathematical tasks and handles information retrieval challenges that require long-horizon reasoning and multi-hop tool use.

1 INTRODUCTION

LLMs’ linear token generation and context limits constrain long-horizon reasoning, while multi-agent workflows add coordination and tool-processing overhead. The paper proposes recursive reasoning trees with selective working memory, implemented by TIM and TIMRUN, to support long-horizon reasoning and tool use in a single inference.

1 INTRODUCTION

  • LLMs’ linear token generation and context limits make it difficult to maintain long-horizon reasoning trajectories and coordinate complex workflows.Multi-agent designs add overhead because they require hand-built control flow, context management, exception handling, communication, and tool-processing modules.Source: §1 · pp. 1–2
  • The paper models reasoning as a recursive tree of subtasks, with complex tasks decomposed until leaves can be completed in one step.Its hypothesis is that an intermediate task need not attend to subtasks from previous steps.Source: §1 · p. 2
  • Selective working memory prunes irrelevant subtasks and KV entries, enabling long-horizon reasoning while retaining instructions and important context.The authors state this approach increases decoding throughput and reduces memory cost.Source: §1 · p. 2, Figure 1 · p. 2
  • TIM recursively decomposes complex tasks, aggregates subtask outputs, and uses external tools within a single language-model inference call.TIMRUN identifies reasoning structure, releases KV states of unhelpful subtasks, reuses memory, and integrates tool responses.Source: §1 · pp. 2–3
  • The paper presents TIM and TIMRUN as a concise route to agentic reasoning: provide TIM with a toolkit and launch one model inference.The introduction also describes this as enabling virtually unlimited long-horizon reasoning beyond output-token limits.Source: §1 · p. 3

2 THREAD INFERENCE MODEL (TIM)

TIM represents reasoning as recursive task trees and trains a transformer model to generate that structure. Thread-2 refines the task schema and working-memory handling, while the preview training uses synthetic data and reinforcement learning.

2.1 THREAD-2

  • Each reasoning task comprises a thought, optional tool use, optional subtasks, and a conclusion that supports future reasoning.Tasks share one schema, and conclusions aggregate subtask results.Source: §2.1 · p. 3
  • Thread-2 gives subtasks access to working memory containing the system prompt, user input, and unpruned tasks, addressing instruction gaps in the earlier Thread framework.This supports end-to-end reasoning in one language-model call.Source: §2.1 · p. 3
  • A fixed-size subtask stack prunes the earliest completed subtask list once the stack exceeds its threshold.The threshold is set among 0, 1, and 2; the authors note that reasoning can require redundancy and flexibility.Source: §2.1 · pp. 3–4
  • TIM can handle multiple tool calls in one decoding pass and extends its KV cache with tool responses encoded as new input tokens.This avoids manually appending responses and resubmitting the message list for each tool call.Source: §2.1 · p. 4

2.2 TRAINING (PREVIEW)

  • The preview post-trains Qwen3-8b on synthetic questions spanning math, research, and ToolBench, with question-specific tool availability.The synthetic dataset’s quality is questionable because language models synthesize tool responses instead of calling the tools.Source: §2.2 · pp. 4–5
  • GRPO training enforces the JSON structure and rewards agreement with annotated answers; the authors report that it can improve the fine-tuned model’s performance.The reported improvement holds despite noisy supervised data and constrained-format rollouts.Source: §2.2 · p. 5

3 TIMRUN, THE INFERENCE RUNTIME FOR TIM

TIMRUN is co-designed with TIM to support long-horizon reasoning despite output-window limits. It prunes completed subtasks and manages tool calls within the runtime to reduce memory use and redundant communication.

3 TIMRUN, THE INFERENCE RUNTIME FOR TIM

  • TIMRUN addresses TIM’s stricter output-window limit by reusing GPU memory and positional embeddings during long-horizon generation.For example, Qwen 2.5 supports 128k input tokens but only 32k output tokens.Source: §3 · p. 5

3.1 SUBTASK PRUNING

  • TIMRUN prunes completed subtasks’ KV states, optionally buffering them, then re-encodes remaining tokens to reuse positional embeddings and memory beyond the output limit.The re-encoding is parallelized, so the authors report throughput is not significantly impacted.Source: §3.1 · pp. 5–6, Figure 3 · p. 6

3.2 END-TO-END MULTI-HOP TOOL USE

  • TIMRUN routes tool calls directly and appends responses to ongoing reasoning, so each reasoning token is sent to TIMRUN only once.This reduces inter-module communication and avoids redundant token transmission.Source: §3.2 · p. 7, Figure 4 · p. 7

4 EXPERIMENTS

Across reasoning and research tasks, the experiments report that maintaining a pruned working memory does not hurt accuracy and can improve it, while TIMRUN sustains high throughput despite intensive memory manipulation. The evaluations cover STEM reasoning, multi-hop research, and serving efficiency.

4 EXPERIMENTS

  • Across reasoning and research tasks, maintaining working memory instead of attending to all context tokens does not hurt accuracy, and TIMRUN sustains high throughput despite intensive memory access and manipulation.The experiments also report that pruning irrelevant context can improve reasoning accuracy and reduce hallucination.Source: §4 · p. 7

4.1 REASONING

  • TIMRUN’s subtask pruning does not degrade overall reasoning performance and improves TIM on many tasks while using less than half the cache slots required for the full output sequence.The reported KV Pruned values are lower bounds on memory savings because peak cache length typically occurs only once during generation.Source: §4.1 · pp. 7–8, Table 1 · p. 8

4.2 RESEARCH

  • On Datacommons QA, TIM uses a concise system message rather than task-specific few-shot prompts, and TIMRUN processes tool responses after subtasks complete and leave the pruning buffer.The evaluation used 140 questions, and the model had not been trained on the Datacommons tool.Source: §4.2 · pp. 8–9, Table 2 · p. 8
  • On BrowseComp, TIM-large significantly outperforms GPT-4o with browsing and performs comparably to ReACT built on Deepseek R1; TIM-8b also outperforms GPT-4o end-to-end.These frontier-model experiments used no agent framework; TIM-large was not served on TIMRUN, while the TIM-8b result used subtask pruning for context efficiency.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 EFFICIENCY AND SCALABILITY

  • TIMRUN outperforms naive memory operations and SGLang in the reported throughput comparisons, while its throughput remains relatively stable as BrowseComp tool calls increase.With subtask pruning, TIMRUN supports more than 30 tool calls within a single inference; increasing pruning-buffer size can reduce throughput when memory-management overhead exceeds computation savings.Source: §4.3 · pp. 9–10, Figure 5 · p. 10
  • For multi-turn tool use, TIMRUN avoids client-side tool-parameter transmission and token-state management, and removes prior tool responses along with completed subtasks.The runtime directly invokes custom tools and encodes their results.Source: §4.2 · p. 8, §4.3 · p. 10

5 CONCLUSION

The conclusion presents TIM and TIMRUN as a co-designed system for recursive task decomposition and efficient inference. It reports that TIM matches strong agentic baselines without explicit agent-specific design, while the combined system offers efficient inference and tool use.

5 CONCLUSION

  • TIM without explicit agent-specific design matches strong baselines that rely on more complex agent frameworks and task-specific post-training.The paper also reports that concise KV caches increase throughput and enhance performance on certain tasks by helping the model focus on relevant context.Source: §5 · p. 10
  • The combined TIM and TIMRUN system delivers strong reasoning ability, more efficient inference and tool use, and greater flexibility and scalability for agentic tasks.Source: §5 · p. 11

Related papers

Loading 2507.16784v1…