Source-linked AI summary
Agentic Knowledgeable Self-awareness
Shuofei Qiao, Zhisong Qiu, Baochang Ren, Xiaobin Wang, Xiangyuan Ru, Ningyu Zhang, Xiang Chen, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen
TL;DR
Agent planning methods often indiscriminately inject trajectories, feedback, and knowledge, overlooking the need to assess which resources a situation requires. KnowSelf trains agents to identify situations and selectively use reflection or external knowledge, achieving strong performance with minimal reflection and knowledge across simulated planning tasks and model scales. The authors position the work as an initial exploration, evaluated only on two simulation datasets and small-scale models.
Problem
Current agent-learning approaches indiscriminately provide planning trajectories, feedback, or knowledge, while language agents lack situational self-awareness for regulating these resources.
Method
KnowSelf labels self-explored trajectories with situation-specific special tokens and uses two-stage training so agents can selectively query knowledge based on the current situation.
Results
KnowSelf achieves superior performance with minimal reflection and knowledge across two simulated agent-planning datasets and two model scales, outperforming various baselines.
Takeaways & Limitations
KnowSelf provides an initial data-centric approach for agents to selectively self-correct and query knowledge during planning according to situations.
Takeaways & Limitations
Experiments cover only two simulation datasets and small-scale 7B and 2B models, leaving other agentic tasks and larger models unexplored.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have achieved considerable performance across various agentic planning tasks. However, traditional agent planning approaches adopt a "flood irrigation" methodology that indiscriminately injects gold trajectories, external feedback, and domain knowledge into agent models. This practice overlooks the fundamental human cognitive principle of situational self-awareness during decision-making-the ability to dynamically assess situational demands and strategically employ resources during decision-making. We propose agentic knowledgeable self-awareness to address this gap, a novel paradigm enabling LLM-based agents to autonomously regulate knowledge utilization. Specifically, we propose KnowSelf, a data-centric approach that applies agents with knowledgeable self-awareness like humans. Concretely, we devise a heuristic situation judgement criterion to mark special tokens on the agent's self-explored trajectories for collecting training data. Through a two-stage training process, the agent model can switch between different situations by generating specific special tokens, achieving optimal planning effects with minimal costs. Our experiments demonstrate that KnowSelf can outperform various strong baselines on different tasks and models with minimal use of external knowledge. Code is available at https://github.com/zjunlp/KnowSelf.
1 Introduction
Existing agent-learning methods often indiscriminately provide trajectories, feedback, or knowledge, while lacking situational self-awareness for deciding when resources are needed. KnowSelf trains agents to regulate reflection and knowledge use, achieving strong planning performance with minimal external resources.
- Current agent learning can make agents brittle by fitting explicit planning patterns and indiscriminately incorporating external feedback or knowledge.
- Human self-awareness supports assessing cognitive states and adapting strategies to decide when to rely on abilities, reflect, or seek additional knowledge.
- KnowSelf defines agentic knowledgeable self-awareness as judging whether the agent can provide the correct next action in the current situation.
- KnowSelf uses self-explored situations, heuristic special-token labels, and two-stage training to teach agents situational planning behavior.
- Experiments evaluate KnowSelf on two simulated planning datasets and two model scales, with further analyses of scaling, generalization, and mechanism.
- KnowSelf achieves optimal performance with minimal reflection and knowledge compared to various baselines.
2 Background
The paper frames agent planning as decision-making in partially observable environments and argues that rote trajectory fitting does not provide situational self-awareness. KnowSelf extends fast and slow thinking with knowledgeable thinking, enabling selective external-knowledge use.
- KnowSelf addresses agent planning in dynamic partially observable environments represented as task, state, action, transition, and observation components.
- Current methods often fit a decision equation through rote memorization rather than recognizing changing situations.
- Fast thinking means directly producing the correct action with little thought, whereas slow thinking requires multiple reasoning and reflection steps.
- Knowledgeable thinking applies when the agent cannot provide the correct action and must rely on external knowledge.
- KnowSelf extends fast or slow thinking by introducing external knowledge into the thinking system to improve knowledgeable self-awareness.
- Its framework marks self-explored trajectories with special tokens, trains the model in two stages, and uses generated tokens to identify situations during inference.
3 Method
KnowSelf teaches agents to judge whether a situation calls for fast action, reflection, or external knowledge, then select the corresponding behavior through special tokens and staged training.
- Knowledge System: KnowSelf builds a lightweight offline knowledge system and uses it to select situationally relevant knowledge rather than injecting knowledge indiscriminately.The knowledge base is formed offline from a minimal number of completed trajectories.
- Situation Judgement Criterion: KnowSelf classifies situations by whether the agent can act correctly immediately, needs rethinking, or requires additional knowledge.The criterion uses direct prediction, revised prediction after reflection, and failure after rethinking to distinguish the three cases.
- Self-awareness Learning: The resulting selective mechanism is designed to reduce the training and inference costs associated with excessive reflection and knowledge use.The paper also expands the model vocabulary to accommodate the added special tokens.
- Self-awareness Inference: Special tokens mark reflection and knowledge use, allowing the agent to switch among fast, reflective, and knowledge-supported behaviors during inference.The inference procedure directly continues correct actions, permits reflection after an initial action, or retrieves knowledge through the selection model R.
- Data Construction: The method constructs self-awareness data by augmenting history-action pairs with situation-specific outputs, including reflection traces or selected knowledge when needed.Incorrect first actions trigger rethinking; incorrect reflected actions trigger knowledge retrieval from the knowledge base.
- Self-awareness Learning: Training uses two stages: autoregressive learning first creates a reference agent, then DPO uses wrong-action negatives with an additional normalized SFT loss.The second stage combines pairwise awareness training with a reintroduced SFT objective to stabilize learning.
4 Experiments
KnowSelf is evaluated on ALFWorld and WebShop using two model scales and broad planning baselines. It outperforms competing methods while using less external knowledge, with additional analyses covering ablation, generalization, and scaling.
- Experimental Settings: The experiments use average reward on ALFWorld and WebShop, two simulated planning datasets with different reward structures, and compare two model scales against multiple planning baselines.ALFWorld provides binary completion rewards, whereas WebShop provides dense rewards from 0 to 1.
- Further Analysis: Figure 3 analyzes KnowSelf through ablations, out-of-distribution generalization, and scaling across model and data sizes on ALFWorld.For generalization, three simple tasks are used for training and three challenging tasks for testing.
- Main Results: KnowSelf consistently outperforms baselines without knowledge on both Llama-8B and Gemma-2B, with Gemma-2B surpassing GPT-4o’s REACT and Llama-8B comparable to GPT-4o’s Reflexion.The comparison uses Know%=0% and notes that GPT-4o’s Reflexion can attempt each task up to five times.
- Main Results: KnowSelf surpasses all 100% knowledge baselines while using only 15.01% and 17.12% knowledge on Llama-8B for ALFWorld and WebShop, respectively.On ALFWorld, Llama-8B KnowSelf uses less knowledge than Gemma-2B, at 26.41%.
- Main Results: The results indicate that excessive knowledge enhancement can hurt weaker models, while stronger models can complete tasks with less external knowledge.The paper frames selective knowledge use as a way to advance planning while reducing training and inference costs.
5 Analysis
KnowSelf improves planning by reducing pattern overfitting, generalizing to unseen tasks, and selectively invoking reflection or knowledge according to situations. Analyses examine its scaling behavior and the emergence of knowledgeable self-awareness in model layers.
- Ablation Study: KnowSelf outperforms full-knowledge and reflection-only variants with low knowledge rates of 15.01% on Llama-8B and 26.41% on Gemma-2B.On Gemma-2B, full knowledge performs worse than reflection-only, while self-reflection alone outperforms knowledge-only intervention on both models.
- Generalization: KnowSelf generalizes across unseen ALFWorld tasks, outperforming Reflexion while ETO reaches 5.88% on PutTwo and 0% on the other tasks.KnowAgent achieves no successes on the three unseen tasks, whereas KnowSelf retains selective reflection and knowledge use.
- Scaling Law: Performance improves with larger models and more self-awareness training data, while relative self-awareness below 40% causes fluctuations or decreases.Llama-8B generally outperforms Gemma-2B, and performance consistently improves as the absolute self-awareness data volume increases.
- Mechanism: Knowledgeable self-awareness emerges in the final Transformer layers, where situation-token probabilities distinguish fast and knowledgeable thinking.The analysis computes average probabilities for situation tokens across layers in both thinking settings.
- Prompting Case Study: Prompting alone does not teach state-of-the-art reasoning models reliable knowledgeable self-awareness in dynamic environments.O1 invokes knowledge without sufficient understanding in one case, while R1 omits knowledge and still fails to produce the correct action in another.
6 Related Work
Prior work situates LLM agents across diverse application domains and connects situational awareness to effective decision-making and emerging studies of LLM self-awareness.
- LLM-based agents have been developed for robotics, operating-system manipulation, software engineering, and data science applications.
- Situational awareness concerns understanding an environment, its elements, and how they change over time or other factors.
- Research has examined situational awareness in robotics and human-computer interaction, and more recently explored self-awareness or self-knowledge in LLMs.
7 Conclusion
The paper introduces agentic knowledgeable self-awareness and KnowSelf to let agents selectively self-correct and query knowledge according to situations during planning.
- KnowSelf is a data-centric approach that enables agents to selectively self-correct and query knowledge based on situations during planning.
- Experiments report the effectiveness and efficiency of KnowSelf, while the authors characterize the work as preliminary.
Limitations
The paper frames knowledgeable self-awareness as an initial exploration rather than a complete theory of AI self-awareness. Its empirical scope is limited to small language models and simulated language-agent tasks, while multimodal settings and alternative training approaches remain open.
- Scope of self-awareness: The work is an initial exploration of knowledgeable self-awareness for language agents, intended to stimulate further research rather than define general AI self-awareness.The authors note that broader self-awareness raises issues including delusions, robustness, safety, and controllability.
- Tasks and models: Experiments cover only two simulated datasets and small-scale 7B and 2B models, leaving larger models and tasks such as function calling and code generation unexplored.The authors specifically identify 30B and 70B models as future evaluation targets.
- Modality: The study examines language-agent scenarios only, while future work is intended to extend knowledgeable self-awareness to multimodal agents handling images, videos, and audio.The paper describes its current coverage as only a first step toward more complex multimodal situations.
- Methods: KnowSelf is primarily a data-driven approach, whereas reinforcement learning or new model architectures may be needed for broader solutions.The authors present these alternatives as directions for further exploration rather than evaluated components of the current method.
E Datasets
Evaluation uses ALFWorld and WebShop to test household manipulation and online shopping, alongside prompt-based and training-based baselines. The comparison includes methods using reflection, retrieved experience, failures, symbolic knowledge, and task/state knowledge.
- Datasets: ALFWorld evaluates household navigation and object manipulation across six task types with binary task-completion rewards.The tasks are Put, Clean, Heat, Cool, Examine, and Puttwo.
- Datasets: WebShop evaluates agents that browse an online shopping platform and purchase products according to user instructions, with rewards based on heuristic attribute-and-price matching.Gold trajectories for both datasets are collected from AgentBank.
- Baselines: The baseline set includes REACT, Reflexion, ExpeL, ETO, KnowAgent, and WKM, covering prompting, reflection, retrieved experience, failure-based training, symbolic knowledge, and task/state knowledge.REACT uses Thought-Action-Observation prompting, while Reflexion uses iterative verbal feedback and replanning.
- Evaluation setup: Prompt-based baselines are evaluated with two-shot examples, including task-specific examples for each of ALFWorld’s six tasks.The fine-tuning-based baselines use full-parameter training.
G Training Setups
The training setup fine-tunes Llama-8B and Gemma-2B with full parameters using a two-stage schedule and fixed inference temperature. Hyperparameters differ substantially between the first and second stages.
- Models and optimization: Llama-8B and Gemma-2B are fine-tuned with full parameters using DeepSpeed.The setup uses AdamW and fixes inference temperature at 0.
- Training schedule: Stage 1 uses learning rate 2e-5, batch size 8, and 3 epochs, while Stage 2 uses learning rate 5e-7, batch size 3, and 1 epoch.The second-stage DPO/RPO coefficient β is 0.5 and the balanced factor α is 1.
- Training schedule: The training configuration uses full-parameter optimization with fixed β, α, optimizer, epoch, and inference-temperature settings reported in the detailed hyperparameter table.Table 8 provides the paper’s consolidated training hyperparameters.
H Detailed Analysis of Training Stages
The training-stage analysis motivates replacing DPO with RPO and shows that NLL stabilization, SFT initialization, and length regularization are important to the final training procedure.
- Training-stage ablation: The paper compares SFT-only, RPO-only, SFT plus DPO, and complete SFT plus RPO training configurations on Llama-8B using ALFWorld.These variants are reported in Table 9.
- Training-stage ablation: DPO can produce negative gains relative to SFT, whereas RPO is adopted after unsatisfactory DPO results and is evaluated in the stage ablation.The authors report that Stage 1 (SFT) + DPO may perform worse than Stage 1 Only (SFT).
- Training-stage ablation: The NLL component stabilizes DPO-related training, while removing the length penalty causes non-convergence and disrupts the balance between SFT and DPO losses in RPO.The authors attribute DPO’s decline to divergence of the policy from the reference model.
- Training-stage ablation: RPO benefits from SFT initialization, as the large gap between Stage 2 Only (RPO) and SFT + Stage 2 (RPO) indicates that SFT supplies a strong reference model.The comparison is included in the training-stage ablation summarized in Table 9.
I The influence of Knowledge Retriever
The retriever analysis finds that better retrieval improves outcomes, but selects prompted DeepSeek-V3 because it is cheaper and avoids retraining under changing environments. The knowledge-selection prompts use trajectories and rules to choose targeted, generalizable guidance while learning from successes and failures.
- Better retrieval improves experimental outcomes, while Table 10 compares different knowledge retrievers.
- Prompted DeepSeek-V3 is selected as the retriever because Sentence-BERT lacks scalability for dynamically changing environments and DeepSeek-V3 has lower API costs than GPT-4o.
- Rule Updating and Consolidation: The prompts require comparing explored and correct actions, checking potential and existing rules, and updating or merging rules when new evidence warrants it.
- Rule Construction: Rule construction records failure causes and successful strategies, including successful process steps, to support future agent behavior.
- Rule Construction: New rules should be targeted, concise, general, universal, and expressed as situation-action recommendations rather than references to specific items or locations.
- Knowledge Selection for Training Data Construction: Knowledge selection analyzes the current trajectory and applicable rules before outputting the current state, analysis, and chosen rule.