PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Reasoning Agents in Evolving Environments

Yule Wen, Yixin Ye, Yanzhe Zhang, Diyi Yang, Hao Zhu

arXiv:2511.04898v1cs.AI

TL;DR

Real-time agents must make decisions while their environments continue to change, a setting existing approaches often do not capture. The paper introduces a dynamic evaluation gym and AgileThinker, which combines reactive and planning threads; it consistently outperforms single-paradigm agents as task difficulty and time pressure rise.

  • Problem

    Many LLM-agent evaluations assume environments change only when agents act, leaving timely decisions in continuously evolving environments an open question.

    Source: §1 · p. 2

  • Method

    The paper introduces Real-Time Reasoning Gym and AgileThinker, which runs planning and reactive language-model threads in parallel.

    Source: §1 · p. 2

  • Results

    AgileThinker consistently outperforms single-paradigm methods, with advantages growing as cognitive load increases and time constraints tighten.

    Source: §1 · p. 3, §8 · p. 11

  • Takeaways & Limitations

    Real-Time Reasoning Gym provides a testbed for evaluating agent designs in continuously evolving environments under time pressure.

    Source: §1 · p. 2, §8 · p. 11

    Experiments were conducted only on DeepSeek models, and the paper reports no empirical evidence that AgileThinker precisely models human dual systems.

    Source: §9 · p. 11

Abstract

from arXiv · show

Agents in the real world must make not only logical but also timely judgments. This requires continuous awareness of the dynamic environment: hazards emerge, opportunities arise, and other agents act, while the agent's reasoning is still unfolding. Despite advances in language model reasoning, existing approaches fail to account for this dynamic nature. We introduce real-time reasoning as a new problem formulation for agents in evolving environments and build Real-Time Reasoning Gym to demonstrate it. We study two paradigms for deploying language models in agents: (1) reactive agents, which employ language models with bounded reasoning computation for rapid responses, and (2) planning agents, which allow extended reasoning computation for complex problems. Our experiments show that even state-of-the-art models struggle with making logical and timely judgments in either paradigm. To address this limitation, we propose AgileThinker, which simultaneously engages both reasoning paradigms. AgileThinker consistently outperforms agents engaging only one reasoning paradigm as the task difficulty and time pressure rise, effectively balancing reasoning depth and response latency. Our work establishes real-time reasoning as a critical testbed for developing practical agents and provides a foundation for research in temporally constrained AI systems, highlighting a path toward real-time capable agents.

1. Introduction

Real-time agents must reason while their environments continue to evolve, but existing work often assumes the world changes only after an agent acts. The paper introduces a dynamic testbed and AgileThinker, which combines timely reactions with long-term planning.

1. Introduction

  • Existing evaluations often let environments wait for agents, leaving timely decision-making in evolving worlds an open question.Hazards, opportunities, and other agents can change the environment while reasoning is still underway.Source: §1 · p. 2
  • Real-Time Reasoning Gym introduces dynamic games in which states update during reasoning, enabling study of real-time agent designs.Its games include moving-car hazards, fleeting food opportunities, and coordination with partners acting independently.Source: §1 · p. 2
  • AgileThinker runs planning and reactive language-model threads in parallel, letting the reactive thread use partial planning traces without waiting for complete analysis.The planning thread reasons over frozen game states while the reactive thread makes timely decisions.Source: §1 · p. 2
  • AgileThinker consistently outperforms single-paradigm methods, with its advantage confirmed in wall-clock experiments and similar trends observed with proprietary models.The evaluation focuses on DeepSeek V3 and R1, whose transparent reasoning trajectories support AgileThinker.Source: §1 · p. 3

2. Real-Time Reasoning Gym

Real-Time Reasoning Gym evaluates agents in reproducible, cognitively challenging environments that continue stepping while agents reason. Its games and token-based time pressure expose different demands of acting in a changing world.

2. Real-Time Reasoning Gym

  • The games are designed to be dynamic, cognitively challenging, and reproducible, supporting evaluation of reasoning under changing conditions.The formulation avoids non-negligible hardware noise by using simulated environments.Source: §2 · p. 3
  • The gym advances at a fixed rate even when an agent is still reasoning, applying a default action if no action arrives in time.This differs from turn-based environments, which step only after the agent finishes thinking and acts.Source: §2 · p. 3
  • Its games test safety around moving hazards, seizing transient opportunities, and coordinating with independent partners.Freeway uses bidirectional traffic, Snake has briefly available apples, and Overcooked pairs the agent with a non-stationary scripted partner.Source: §2 · pp. 3–4
  • Token count serves as a hardware-independent time proxy, and the environment steps after a set number of generated tokens.The study uses four pressure levels: 32k, 16k, 8k, and 4k tokens per step, from low to high pressure.Source: §2 · pp. 4–5
  • Game-specific normalized scores measure timely task performance, but cross-model comparisons may be unfair because tokenizers and architectures differ.The metrics count Freeway crossings, Snake apples eaten before collision, and Overcooked orders completed with the partner.Source: §2 · p. 5

3. Real-time Reasoning Agents

Reactive agents respond within each environment step, while planning agents reason over multiple steps. AgileThinker combines these approaches through parallel threads and a time-sharing protocol.

3. Real-time Reasoning Agents

  • Reactive agents produce one action per step under a token budget, whereas planning agents use extended reasoning to generate multi-step actions or code plans.Code plans can adapt to potential changes, while multi-step action plans are often easier to generate.Source: §3 · p. 5
  • AgileThinker runs a continuous planning thread and a separate reactive thread that uses the latest observation and partial planning output to make timely decisions.Planning operates over frozen game states; reaction is constrained to fit within the environment update period.Source: §3 · p. 6
  • During each environment step, planning runs continuously while reaction activates only in the final TR time units.Increasing TR gives the reactive thread more time to adapt but leaves less planning output for it to use.Source: §3 · p. 6

4. Is Single Paradigm Enough for Real-Time Reasoning?

Reactive and planning agents each fail on one side of the decision-quality and latency tradeoff: reactive performance falls as cognitive load rises, while planning performance falls as time pressure rises. The tested budget-forcing and code-policy variants also do not reliably resolve this tradeoff.

4. Is Single Paradigm Enough for Real-Time Reasoning?

  • The experiments vary cognitive load at 8k tokens per step and vary time pressure from 4k to 32k tokens per step at medium cognitive load.Each agent is evaluated 32 times per setting, averaging eight game seeds and four LLM sampling seeds.Source: §4 · p. 6, Figure 5 · p. 7
  • Reactive-agent scores fall from 0.89 to 0.15 as cognitive load increases, versus 0.88 to 0.50 for AgileThinker.The reactive agent's restricted computation limits its consideration of future consequences.Source: §4 · p. 7
  • Planning-agent scores drop from 0.92 to 0.05 as time pressure increases, versus 0.90 to 0.58 for AgileThinker.Planning can rely on outdated observations; AgileThinker's reactive thread receives the latest state and can adjust the planning thread's output.Source: §4 · p. 7
  • At step 3, the reactive agent pursues nearby food into a collision, while AgileThinker uses planning-thread reasoning to choose a safer target.The planning agent reasons from an outdated step-1 state and defaults left, despite recognizing the future collision risk of the nearest food.Source: Figure 6 · p. 8
  • R1 with budget forcing scores 0.01, below 0.39 for the V3-based reactive agent, while forced truncation typically yields no-ops.Meaningful actions mainly occur when generation naturally fits within the budget; R1 code-policy helps in Freeway but underperforms direct action generation in the other games.Source: §4 · pp. 7, 9

5. How to Manage Resources Between Reaction and Planning?

AgileThinker's performance depends on balancing the reactive thread's token budget against the planning thread's ongoing reasoning. The best budget is near the reactive thread's natural token use, but the optimum differs by environment.

5. How to Manage Resources Between Reaction and Planning?

  • The experiment fixes NTE at 8k and varies NTR from 0.5k to 8k tokens, comparing scores with the untruncated reactive thread's token-use CDF.The CDF represents the reactive thread's token usage across game trajectories when generation is unconstrained.Source: §5 · p. 9
  • AgileThinker performs best when NTR approximates the reactive thread's natural token upper bound, avoiding both truncation and extended idling.Too little time prevents the reactive thread from using planning guidance; too much creates idle periods while the planning thread continues reasoning.Source: §5 · p. 9
  • The optimal NTR is approximately 5k tokens in Freeway and around 2k in Snake and Overcooked.AgileThinker outperforms single-system baselines across broad budget ranges, so rough upper-bound estimates can suffice.Source: §5 · p. 9

6. Performance Improvement Under Wall-Clock Time

Wall-clock experiments show that AgileThinker outperforms both single-paradigm agents in physical time. The paper also reports a strong linear relationship between generated-token count and inference runtime.

6. Performance Improvement Under Wall-Clock Time

  • Token count and physical inference time have a strong linear correlation: T = αN + β, with α = 0.0473 s/token, β = 334.55 s, and R2 = 0.9986.The relationship is fit by least squares over all experiment trajectories.Source: §6 · p. 10
  • With six-minute environment intervals, AgileThinker consistently outperforms Reactive and Planning Agents in wall-clock time.The comparison uses a derived TPOT of 0.047 s/token and an environment interval corresponding to approximately 8,000 tokens per step.Source: §6 · p. 10, Table 2 · p. 10

7. Related Work

Prior LLM-agent evaluations commonly use static environments or pause state changes during reasoning, while related delay-aware work has focused on traditional reinforcement learning. This paper frames real-time reasoning for LLM agents and distinguishes AgileThinker by letting its reactive thread access the planning thread's partial reasoning trace.

7. Related Work

  • Existing LLM-agent evaluations often use static environments or pause state changes during reasoning; prior delay models largely target traditional reinforcement learning.The paper measures elapsed token count as a hardware-agnostic temporal unit for fair, reproducible comparisons across agent systems.Source: §7 · p. 10
  • Current budget-control methods still struggle to precisely control token counts and can lose performance when budgets are inadequate.The paper reports that state-of-the-art budget-control methods cannot effectively balance reaction and planning, motivating dual LLM architectures for real-time environments.Source: §7 · pp. 10–11
  • AgileThinker differs from other dual-agent designs because its reactive thread can access the planning thread's partial reasoning trace.This allows the reactive thread to make informed decisions with minimal delay.Source: §7 · p. 11

8. Conclusion

The paper formalizes real-time reasoning and introduces Real-Time Reasoning Gym to evaluate agents in continuously evolving environments. AgileThinker combines reactive and planning reasoning, outperforming baselines with advantages that grow under greater cognitive load and tighter time constraints.

8. Conclusion

  • AgileThinker engages two reasoning paradigms in parallel and consistently outperforms all baselines, with advantages growing as cognitive load increases and time constraints tighten.The conclusion also identifies extending the gym to more realistic scenarios, improving coordination between threads, and training urgency-aware agents as future directions.Source: §8 · p. 11
  • Real-Time Reasoning Gym evaluates LLM agents in continuously evolving environments and supports independent control of cognitive load and time pressure using token count as a hardware-independent temporal measure..Source: §8 · p. 11
  • The paper identifies real-time reasoning as a fundamental challenge for agents in real-world deployment.Source: §8 · p. 11
  • The evaluation reveals shortcomings in both reactive and planning reasoning paradigms.Source: §8 · p. 11

9. Limitations

The authors report that experiments were conducted only on DeepSeek models and that they lack empirical evidence that AgileThinker precisely models human dual systems. The appendix describes the three game environments and a planning prompt that treats prior strategy as optional guidance.

9. Limitations

  • Experiments use only DeepSeek models, and the authors report no empirical evidence that AgileThinker precisely models human dual systems.They cite the lack of reasoning traces from commercial providers as a reason for the model scope.Source: §9 · p. 11

A. Environment Details

  • The three environments cover highway navigation, grid-based Snake, and cooperative Overcooked, where the second player is scripted in this single-agent setting.Source: A. Environment Details · p. 15

Prompt for the Planning Agent

  • The planning prompt treats a prior thinking-model strategy as optional guidance, retaining it only when current conditions still align with its assumptions.Source: Prompt for the Planning Agent · p. 18

C.1. Main Results

AgileThinker’s advantage over single-paradigm agents generally becomes statistically significant as cognitive load and time pressure increase. The analysis compares paired scores across configuration seeds and tests conditions spanning three load levels and three token budgets.

C.2. Significance Test

  • The validation spans Easy, Medium, and Hard cognitive loads and High, Medium, and Low pressures of 32k, 8k, and 4k tokens per step.Source: §C.2 · p. 24
  • The significance analysis pairs AgileThinker and reactive-agent scores by environmental configuration seed and applies a paired t-test to the score differences.Scores are averaged across runs within each seed, and p-values use n − 1 degrees of freedom.Source: §C.2 · pp. 24–25
  • AgileThinker’s advantage generally becomes statistically significant as cognitive load and time pressure increase.Figure 8 reports p-values for comparisons against reactive agent V3 and planning agent R1; significance is assessed at p < 0.05.Source: Figure 8 · p. 25, §C.2 · p. 25

C.3. Results of Other Models

Across model variants and throughput settings, combining reactive and planning reasoning improves performance, while code-as-policy examples reveal game-specific limits in search horizon and contextual coordination.

C.3. Results of Other Models

  • The primary experiments use DeepSeek-V3 and DeepSeek-R1 because AgileThinker requires transparent reasoning trajectories, with DeepSeek-V3.2 also evaluated.The paper also evaluates Gemini-2.5-Flash to assess generalizability.Source: §C.3 · p. 25
  • With Gemini-2.5-Flash, combining reactive and planning paradigms improves performance across all tested time-pressure levels despite using a reduced design.The reactive thread references the planning thread’s final output because Gemini reasoning traces are unavailable.Source: §C.3 · p. 25, Table 10 · p. 27
  • Gemini-2.5-Flash often exceeds its allocated token budget, showing that its built-in control cannot precisely regulate response length.Figure 9 presents response-length distributions for 4k, 8k, 16k, and 32k thinking budgets.Source: Overcooked · p. 27, Figure 9 · p. 27

Freeway

  • Freeway’s small time-by-position state space permits breadth-first search over movement sequences, enabling generated code to implement a simple strategy for optimal performance.The passage describes Freeway as having a simple, efficient solution.Source: Freeway · p. 28

Snake

  • In Snake, generated code typically uses limited-depth heuristic search, which misses distant food opportunities and collision risks beyond its horizon.The example searches five turns; the passage says this constrained horizon is insufficient for complex scenarios.Source: Snake · p. 28

Overcooked

  • In Overcooked, generated policies use limited context and can prioritize goals inconsistently, producing suboptimal or oscillatory actions instead of coherent long-term coordination.One example idles while holding an onion when pots are occupied; another alternates between adding an onion and serving ready soup.Source: Overcooked · p. 29

C.5. Performance under Limited Throughput

  • At equivalent token throughput, concurrent AgileThinker significantly outperforms reactive and planning agents, while parallel execution adds only modest gains.The reported advantage suggests cognitive specialization, rather than extra computational resources, is the primary benefit; AgileThinker remains effective for resource-constrained deployments.Source: Table 11 · p. 29, §C.5 · p. 30

D. Walltime Experiments

Figure 10 shows an almost linear relationship between generated token count and wall-clock time using the DeepSeek official API, supporting token count as a time abstraction.

D. Walltime Experiments

  • Generated token count and wall-clock time show an almost linear correlation using the DeepSeek official API.Figure 10 supports the validity of the token-as-time abstraction; labels such as 4k and 8k indicate environment time-pressure budgets.Source: Figure 10 · p. 30

Related papers

Loading 2511.04898v1…