PaperlayerView exact revision on arXiv

Source-linked AI summary

ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflection

Jeonghye Kim, Sojeong Rhee, Minbeom Kim, Dohyung Kim, Sangmook Lee, Youngchul Sung, Kyomin Jung

arXiv:2505.15182v2cs.CLcs.AIcs.LG

TL;DR

ReAct-based LLM agents can produce thoughts that are poorly grounded in history or task objectives, a concern in long-horizon, partially observable environments. ReflAct replaces next-action-focused reasoning with continuous reflection on state and goal. It outperforms ReAct across ALFWorld, ScienceWorld, and Jericho, and the paper identifies mathematics and coding as unexplored domains.

  • Problem

    ReAct-based methods can produce thoughts poorly grounded in historical context or the overarching objective, raising questions about extending the ReAct reasoning–action framework.

    Source: §1 · p. 1

  • Method

    ReflAct replaces the core reasoning–action cycle with continuous reflection on the agent’s current state relative to the task goal.

    Source: §1 · p. 2

  • Results

    ReflAct outperforms ReAct by 36.4% on ALFWorld, 8.5% on ScienceWorld, and 38.1% on Jericho.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments show that ReflAct consistently outperforms ReAct and other baselines in long-horizon reasoning tasks.

    Source: §7 · p. 8

    Applicability to mathematics and coding remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in LLM agents have largely built on reasoning backbones like ReAct, which interleave thought and action in complex environments. However, ReAct often produces ungrounded or incoherent reasoning steps, leading to misalignment between the agent's actual state and goal. Our analysis finds that this stems from ReAct's inability to maintain consistent internal beliefs and goal alignment, causing compounding errors and hallucinations. To address this, we introduce ReflAct, a novel backbone that shifts reasoning from merely planning next actions to continuously reflecting on the agent's state relative to its goal. By explicitly grounding decisions in states and enforcing ongoing goal alignment, ReflAct dramatically improves strategic reliability. This design delivers substantial empirical gains: ReflAct surpasses ReAct by 27.7% on average, achieving a 93.3% success rate in ALFWorld. Notably, ReflAct even outperforms ReAct with added enhancement modules (e.g., Reflexion, WKM), showing that strengthening the core reasoning backbone is key to reliable agent performance.

1 Introduction

ReAct-based agents can produce thoughts poorly grounded in their history or objective, and those errors can compound in partially observable settings. ReflAct replaces the reasoning–action cycle with continuous state-and-goal reflection; across three environments, it outperforms ReAct.

1 Introduction

  • ReAct-based methods often generate thoughts poorly grounded in historical context or the overarching objective, which can compound into divergent beliefs and failed execution.The paper argues that redesigning the reasoning backbone, rather than adding complementary modules, addresses this problem.Source: §1 · pp. 1–2
  • ReflAct shifts thought from predicting the next action to reflecting on the current state relative to the task goal.It replaces the core reasoning–action cycle rather than adding reflective or memory components to ReAct.Source: §1 · p. 2
  • ReflAct outperforms ReAct by 36.4% on ALFWorld, 8.5% on ScienceWorld, and 38.1% on Jericho.The evaluations used open-source and proprietary LLMs in long-horizon, partially observable text environments.Source: §1 · p. 2
  • ReflAct also surpasses enhancement modules layered on existing backbones.The paper presents this comparison as evidence for revising the reasoning process itself.Source: §1 · p. 2

2 Preliminaries

The paper models language-based tasks as partially observable decision processes and describes ReAct as alternating between language reasoning and action. Each thought is conditioned on interaction context and informs the next action.

2 Preliminaries

  • A language-based task is modeled as a POMDP with an instruction, hidden environment state, actions, observations, transitions, and rewards.In this setting, instructions, actions, and observations are expressed in natural language.Source: §2 · p. 2
  • ReAct extends the POMDP with a thought space, where each thought is a natural-language reasoning step interleaved with action.The agent’s policy alternates between generating thoughts and selecting actions.Source: §2 · p. 2
  • At each timestep, ReAct conditions thought generation on the task and interaction history plus the current observation, then uses the thought-enriched context to select an action.The action is executed and produces the next observation, continuing the loop until completion or a time limit.Source: §2 · p. 2

3 Revisiting the Efficacy of Thought

Thoughts directly shape action probabilities: relevant thoughts can favor appropriate actions, while conflicting thoughts can bias choices. Ungrounded thought can mislead decisions, and ReAct failures include incoherent state beliefs and short-sighted planning.

3.1 How Thought Impacts Action Selection

  • Relevant thought sharpens action probabilities toward appropriate choices, whereas incorrect or conflicting thought increases the likelihood of inappropriate actions.The authors report that structured thought can drive decisions predominantly, reducing reliance on broader historical context.Source: §3.1 · p. 3
  • ReAct’s mean action-distribution entropy is 0.30, versus 1.23 for NoThinking across 134 ALFWorld tasks.The authors interpret the lower entropy as more confident and focused behavior.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 The Problem of Ungrounded Thought

  • Ungrounded thought disconnected from the actual state can mislead the policy into suboptimal or incorrect actions.In the soap-bar example, the agent revisits a cabinet because its thought fails to capture the situation from its observation history, creating a loop and task failure.Source: §3.2 · pp. 3–4
  • Across 134 ALFWorld tasks, thought can help solve tasks but also introduces avoidable failures, including with GPT-4o.Figure 3 groups outcomes by whether only NoThinking failed, only ReAct failed, both failed, or both succeeded.Source: §3.2 · p. 3, Figure 3 · p. 3
  • The identified reasoning limitations are incoherent internal state and short-sighted planning that disregards the long-term goal.The paper illustrates the former with revisiting a location or falsely assuming an object is held.Source: §3.2 · p. 4
  • Changing guidance to assess the current state in relation to the task goal led the agent to better align its action with the overall objective in the failure case.The paper also reports this pattern for failure cases in ScienceWorld.Source: §3.2 · p. 4

4 Proposed Method: ReflAct

ReflAct changes the reasoning–action cycle by having the agent reflect on its current state in relation to the task goal. The reflection makes the agent’s internal belief and goal explicit in the context used for action selection.

Proposed Method: ReflAct

  • The agent’s thought is selected to maximize expected long-term return when it conditions the action policy.The return sums future rewards with discount factor γ; the thought should consider the task goal at every timestep, not just the next action.Source: §4 · p. 4
  • ReflAct uses a redesigned instruction to reflect on the agent’s state in relation to the task goal before outputting an action, without an additional module.The paper also uses one-shot examples modified to show this reflection format.Source: §4 · pp. 4–5
  • ReflAct defines reflection as a structured representation of the agent’s internal belief state and task goal, used to condition subsequent decisions.The belief state summarizes the agent’s understanding based on interaction history; the goal summarizes what it is trying to achieve.Source: §4 · p. 5

5 Related Works

Prior work extends ReAct with planning, post-task reflection, or external knowledge, while ReflAct redesigns the reasoning process itself to improve grounding and goal alignment.

Related Works

  • ReAct interleaves reasoning and action, while Plan-and-Solve adds initial planning and enhancement modules add post-task analysis, external knowledge, or trajectory prediction.The paper argues that these approaches can still suffer when the ReAct backbone grounds past observations incorrectly.Source: §5 · p. 5

6 Experiments

Experiments compare ReflAct with reasoning baselines and enhancement modules across three text-based environments, then examine its failure patterns and self-correction. ReflAct outperforms the baselines, while the analyses contrast state–goal reflection with verbalization and external guidance.

6.1 Experimental Settings

  • Evaluation covers ALFWorld, ScienceWorld, and Jericho, using GPT-4o variants and Llama-3.1-Instruct variants as agent models.ALFWorld uses binary task success; ScienceWorld and Jericho are evaluated with success rate and average reward.Source: §6.1 · p. 5

6.2 Comparison with Prior Methods

  • ReflAct improved over ReAct by 36.4%, 8.5%, and 38.1% in ALFWorld, ScienceWorld, and Jericho, respectively, and consistently outperformed baseline reasoning frameworks across models and tasks.The comparison also reports improvements over NoThinking of 77.9%, 25.9%, and 101.1% in those environments, respectively.Source: Table 2 · p. 6, §6.2 · p. 6
  • The experiments compare ReflAct with NoThinking, ReAct, and Plan-and-Act, where Plan-and-Act reasons only at the first timestep before acting without further thoughts.ReAct outperforms NoThinking; Plan-and-Act performs comparably or slightly better than ReAct with GPT-4o but underperforms it with Llama models.Source: §6.2 · pp. 5–6
  • Merely verbalizing state or goal underperforms ReAct, and adding next-action reasoning to state-and-goal verbalization still falls short of ReAct in ScienceWorld and Jericho.ReflAct’s explicit reflection on the relationship between state and goal yields a substantially greater performance gain; Figure 5 compares these strategies with Llama-3.1-8B-Instruct.Source: Figure 5 · p. 6, §6.2 · p. 7

6.3.1 ReflAct and Reflexion

  • Post-task Reflexion generally improves performance, but NoThinking and ReAct still fall short of ReflAct’s trial-0 performance; GPT-4o ReflAct with Reflexion reaches 94.8% success in ALFWorld.The study applies post-task reflection after each trial for three trials.Source: §6.3.1 · p. 7, Figure 6 · p. 7

6.3.2 ReflAct and WKM

  • Applying WKM to ReAct improves performance but remains below ReflAct, while replacing thought with external state knowledge degrades performance.The authors suggest internal state modeling may be more effective than relying on state knowledge generated by an external model.Source: §6.3.2 · pp. 7–8, Figure 7 · p. 8

6.4.1 Failure Ratio Comparison Between Reasoning Frameworks

  • Across 134 ALFWorld tasks, ReAct introduces failures not seen with NoThinking, whereas ReflAct has no failures unique to it and fails only on tasks where other agents also fail.The comparison uses GPT-4o and Llama-3.1-8B-Instruct.Source: §6.4.1 · p. 8, Figure 8 · p. 8

6.4.2 ReflAct Self-Corrects Automatically

  • A ReflAct example shows the agent recognizing that cooling lettuce failed because it was away from the fridge, then moving there and completing the action.The section describes this as recovery through maintaining an internal belief about state and aligning decisions with the goal.Source: §6.4.2 · p. 8

7 Conclusion

The conclusion presents ReflAct as a backbone that replaces next-action-focused reasoning with ongoing reflection on state and goal. The experiments show it consistently outperforms ReAct and other baselines.

Conclusion

  • ReflAct emphasizes ongoing reflection on state and goal to support more grounded, reliable decisions in complex, partially observable tasks.The conclusion contrasts this with ReAct’s difficulty maintaining coherent beliefs about the environment and goal.Source: §7 · p. 8
  • The experiments report that ReflAct consistently outperforms ReAct and other baselines in long-horizon reasoning.The authors hope the work encourages further research into more robust and informative reasoning guidance for intelligent agents.Source: §7 · p. 8

Limitations

The paper identifies three boundaries for ReflAct: its use beyond interactive agent environments is unexplored, it modestly increases token length, and external-module hybrids remain future work.

Limitations

  • ReflAct’s applicability to mathematics and coding remains unexplored, and adapting its reflection mechanism may be needed for those domains’ distinct demands.The paper distinguishes step-by-step deduction in mathematics from consistency in logic, state, and syntax in programming.Source: Limitations · p. 9
  • ReflAct modestly increases token length compared with ReAct, motivating more concise reflection formulations.Suggested directions include compressing reflections or adding summarization while retaining informativeness within a smaller token budget.Source: Limitations · p. 9
  • The paper leaves hybrid architectures combining ReflAct with external modules, such as verifiers, for future research.A verifier might evaluate reflected goals or suggest fixes for inconsistencies.Source: Limitations · p. 9

Ethical Statements

The paper frames grounded agent reasoning as relevant to reliability and safety, situates ReflAct among belief-state, prompting, and enhancement approaches, and illustrates how state-goal reflection can correct a ReAct failure.

Ethical Statements

  • Ungrounded reasoning can make agents unreliable or harmful in real-world contexts; the work aims to improve trustworthiness through better alignment between internal beliefs and environment state.The stated safety concern is most acute in complex, partially observable environments.Source: Ethical Statements · p. 9

A Expanded Related Works

  • ReflAct prompts LLMs to reason over goal-conditioned state representations without auxiliary modules, unlike extrinsic symbolic-state approaches and add-on reasoning enhancements.The related-work discussion contrasts belief-state modeling, symbolic representations, backbone frameworks, enhancement modules, and imitation-learning approaches; ReflAct requires neither gold trajectories nor parameter updates.Source: A Expanded Related Works · pp. 11–12

B Another Case of the Ungrounded Thought Problem

  • In the ScienceWorld boiling-lead case, ReAct falsely assumes it is holding the lead and fails, while replacing its thought with state-and-goal reflection leads to successful completion.The reflection helps the agent recognize its actual situation and make goal-aware long-horizon decisions.Source: B Another Case of the Ungrounded… · p. 12

C Benchmark Details

The appendix describes three text-based evaluation environments, the reasoning-framework and enhancement-module setups, a state-goal ablation, and resource use; it also contrasts ReflAct’s grounded single-call behavior with RAFA’s trajectory search.

C Benchmark Details

  • ALFWorld tests multi-step household instructions, ScienceWorld tests scientific and procedural tasks, and Jericho comprises adventure games with ordered implicit subgoals.The described tasks involve interactive language grounding; Jericho additionally requires agents to infer state through interaction.Source: C Benchmark Details · p. 12
  • ALFWorld success requires full completion; because ScienceWorld and Jericho success flags could be incorrect, the authors require progress reward of at least 0.7 and exactly 1.0, respectively.ScienceWorld and Jericho provide progress rewards from 0 to 1; the stated bug could mark an incomplete task successful.Source: C Benchmark Details · p. 12
  • The ALFWorld test set has 134 tasks, ScienceWorld has 211 tasks, and Jericho has 20 game-based tasks.The paper also describes a ScienceWorld lead-boiling failure case in which ReAct’s mistaken belief about holding the lead leads to failure.Source: C Benchmark Details · p. 13, Figure 9 · p. 13

D.1 Experimental Setup: Comparison of Backbone Reasoning Frameworks

  • The backbone comparison includes NoThinking, ReAct, and Plan-and-Act, with Plan-and-Act retaining only the first-timestep thought while NoThinking removes thought at every timestep.The setups use the same environment code base and test set, and the ReAct instruction is attributed to Song et al. (2024) and Xiong et al. (2025).Source: §D.1 · p. 13

D.2 Experimental Setup: Comparison of Enhancement Reasoning Modules

  • The enhancement-module experiments cover Reflexion, WKM, RAFA, and MPO, implemented using the original papers’ repositories; WKM uses its public dataset and world model.The listed environments use the MPO implementation for ALFWorld and ScienceWorld and AgentBoard for Jericho.Source: §D.2 · pp. 13–14

D.3 Experimental Setup: ReflAct Ablation Study

  • The ablation varies whether the agent states the current state, task goal, both, or state and goal plus ReAct-style thought before acting.The instructions specify state and goal text, including current location and inventory, in the State+Goal variants.Source: §D.3 · p. 14

E Resources

  • Inference used one A6000 GPU for Llama-3.1-8B and two A100 GPUs for Llama-3.1-70B; WKM world-model training used four A6000 GPUs.Source: E Resources · p. 14

F.1 ReflAct and RAFA

  • ReflAct uses one language-model query per step, whereas RAFA’s depth-2, branching-factor-2 tree requires 13 queries; the comparison also illustrates ReflAct selecting the state-appropriate action.RAFA uses three action-sampling, six observation-prediction, and four value-estimation queries per step; its overhead grows rapidly with depth.Source: §F.1 · pp. 14–15, Figure 10 · p. 14

F.2 ReflAct and MPO

MPO meta-plans improve NoThinking and ReAct but do not bring them to ReflAct’s performance; on Qwen3-8B, ReflAct’s gains vary by environment, remaining marginal in ScienceWorld and larger in ALFWorld and Jericho.

F.2 ReflAct and MPO

  • MPO meta-plans substantially improve NoThinking and ReAct, yet both remain below ReflAct; the plans yield slight GPT-4o gains but a minor Llama-3.1-8B-Instruct drop when applied to ReflAct.The authors suggest out-of-distribution meta-plans may mislead and note that ReflAct’s grounded decision-making may already support high-quality planning.Source: §F.2 · p. 15, Figure 11 · p. 15

G Evaluating ReflAct with Advanced Reasoning Models

  • With Qwen3-8B, ReflAct improves success rates significantly in ALFWorld and Jericho but only marginally in ScienceWorld compared with ReAct.Qwen3-8B’s inference-time scaling was disabled because excessively long reasoning traces could prevent proper Thinking and Action outputs.Source: G Evaluating ReflAct with Advanced… · pp. 15–16, Table 3 · p. 15

H Additional Benchmark Evaluation

Additional analyses report ReflAct outperforming ReAct across four BabyAI models and remaining robust to prompt paraphrasing. ReflAct uses fewer reasoning steps and has a lower hallucinated-action rate despite slightly longer token sequences; case studies also identify task-specific failure modes.

H Additional Benchmark Evaluation

I.1 ReflAct Prevents Suboptimality Through Goal-Oriented Reasoning

  • ReflAct succeeds in the swimming-items Jericho example where ReAct fixates on a shovel and fails.The comparison attributes ReflAct’s success to goal-oriented reasoning and reflection on the state and objective.Source: Figure 12 · p. 16, §I.1 · p. 16

I.2 Robustness of ReflAct across Prompt Variants

  • Instruction paraphrases caused only minor performance fluctuations; ICL-example variants likewise showed very small performance differences overall.Instruction 2, the most substantially changed revision, produced the largest drop; Llama-3.1-8B-Instruct improved slightly under ICL-example paraphrasing.Source: §I.2 · pp. 16–17

I.3 Token Length vs. Hallucination Action Rate in ReAct and ReflAct

  • ReflAct produces slightly longer token sequences than ReAct, yet its hallucination rate decreases.Token length is normalized by trajectory steps; hallucinated-action rate is the proportion of actions yielding erroneous observations.Source: §I.3 · pp. 17–18, Figure 13 · p. 17

I.4 Number of Reasoning Steps Comparison

  • Across ALFWorld, ScienceWorld, and Jericho, reasoning-step counts rank NoThinking > ReAct > ReflAct.ReflAct therefore uses the fewest reasoning steps on average across the three domains.Source: §I.4 · p. 18, Table 7 · p. 18

J ReflAct Success & Failure Case Study

  • GPT-4o ReflAct failures include not finding the target in time or substituting a mug for the required cup; Llama-3.1-8B failures include trying other cleaning tools instead of the clean action.The analyzed tasks require performing the appropriate action at the designated location before placing the object at its goal location.Source: J ReflAct Success & Failure Case Study · p. 18
  • ReflAct is designed to improve decision-making as a standalone backbone; the authors suggest trial-and-error memory learning as a possible further enhancement.The proposed memory enhancement is intended to help learn environment rules.Source: J ReflAct Success & Failure Case Study · p. 18

Related papers

Loading 2505.15182v2…