PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?

Chuanyuan Tan, Junjie Yu, Yuxin Wang, Yining Zheng, Xipeng Qiu, Wenliang Chen

arXiv:2608.27924v1cs.CL

TL;DR

Reliable UAQ handling matters for trustworthy LLM agents, yet the role of memory in this setting remains unclear. The paper systematically studies four memory methods across three UAQ datasets and two base models, finding selective and dataset-shift-fragile gains, with decision guidance and procedural or rule-based representations providing the strongest support.

  • Problem

    Memory is widely used in agent systems, but its reliability for handling unanswerable questions remains unclear.

  • Method

    The paper evaluates four representative memory methods across three UAQ-related datasets and two base models within a unified agentic RAG framework.

  • Results

    Memory gains are selective and fragile under dataset shift, while cross-model reuse is often more feasible than cross-dataset transfer and decision guidance preserves gains better than trajectory shaping.

  • Takeaways & Limitations

    Reliable UAQ memory depends more on transferable behavioral guidance and suitable representations than on storing larger amounts of experience.

  • Takeaways & Limitations

    The conclusions are scoped to reliable UAQ handling and do not directly generalize to tasks with substantially different objectives.

Abstract

from arXiv · show

Reliable handling of unanswerable questions (UAQs) is critical for trustworthy LLM-based agents. Although memory is widely used in agent systems, its role in reliable UAQ handling remains unclear. We present a systematic study of agent memory for UAQ handling under a unified agentic RAG framework, evaluating four representative memory methods across three UAQ-related datasets and two base models. We find that memory can improve UAQ performance in some settings, but such gains are selective rather than universal and remain fragile under dataset shift. Interestingly, cross-model memory reuse is often more feasible than cross-dataset transfer, suggesting that shifts in answerability patterns pose a greater challenge to memory reuse than changes in the base model itself. We further find that UAQ gains are more strongly preserved through decision guidance than through trajectory shaping, and that memory effectiveness depends strongly on representation. In particular, procedural and rule-based memories often provide the most reliable support for UAQ handling, while memory composition is most effective when procedural guidance is combined with complementary behavioral signals. Overall, our findings suggest that reliable UAQ memory depends less on storing larger amounts of experience and more on preserving transferable behavioral guidance.

1 Introduction

Reliable UAQ handling is essential for trustworthy LLM agents, but memory’s role in agentic settings remains unclear. This study systematically evaluates memory methods and analyzes when and why they help.

  • Motivation: Memory is relevant to UAQ handling because agents must learn when to answer, clarify, or abstain while avoiding misleading responses.Stored experience may also propagate misleading heuristics or dataset-specific answerability patterns.
  • Study design: The study evaluates four representative memory methods across three UAQ-related datasets and two base models under a unified agentic RAG framework.The design covers in-distribution performance, human-hint interaction, cross-dataset and cross-model transfer, and controlled memory analyses.
  • Findings: Memory improves UAQ performance selectively rather than universally, with gains depending strongly on the base model and weakening under dataset shift.These findings motivate examining both effectiveness and transferability rather than assuming memory provides general reliability gains.
  • Findings: Memory helps UAQ handling mainly through reusable decision guidance rather than trajectory shaping alone.The study separates these two sources of memory gains to identify which mechanism preserves reliability.
  • Findings: Procedural memory is effective for reliable UAQ handling, while successful and failed experiences help differently depending on their representation.Failure-derived signals support principle-based guidance, whereas descriptive memories benefit more from successful trajectories.

2 Preliminaries

The paper formulates UAQ handling as a ReAct-style agent task with external memory. Memory is retrieved before interaction, then past trajectories are distilled into reusable guidance for future questions.

  • Task formulation: The agent receives a question, interacts with an external environment, and produces a response in a ReAct-style loop.Questions are categorized as answerable or unanswerable based on whether available evidence supports an answer.
  • Task formulation: UAQs arise from false premises, missing information, non-existent entities, or unresolved ambiguity, requiring refusal, clarification, or cautious responses.ABQs instead require correct answers supported by available evidence.
  • Task formulation: UAQ reliability is evaluated jointly with answer correctness because agents should maintain ABQ accuracy while responding appropriately to UAQs.The experiments quantify these dimensions using answer accuracy and acceptable response rate.
  • External memory: Retrieved memory is injected as initial guidance before environment interaction, and the resulting trajectory is distilled into memory for future reuse.This architecture allows memory to influence both decision behavior and subsequent environment interaction.
  • External memory: The framework distinguishes reusable behavioral guidance from memory’s influence on later interaction trajectories.This distinction supports the paper’s analysis of decision guidance versus trajectory shaping.

3 Experimental Setup

The experiments compare memory methods across three mixed answerability datasets using balanced samples and standardized evaluation. Performance is assessed through answer accuracy, acceptable UAQ responses, and their joint score.

  • Datasets: The evaluation uses KUQ, UAQFact, and RefuNQ, each containing answerable and unanswerable questions with different UAQ constructions.KUQ has six UAQ categories, UAQFact uses knowledge-graph-supported triples, and RefuNQ replaces target entities with non-existent entities.
  • Datasets: Each dataset contributes 400 stratified test samples with balanced answerability labels, while KUQ and UAQFact each provide 200 training examples for memory construction.RefuNQ is evaluation-only because its single synthetic UAQ type has limited diversity for memory construction.
  • Metrics: Accuracy measures correctness on answerable questions, while Acceptable Ratio measures reasonable refusal or clarification behavior on UAQs.Acceptable Ratio is computed with an LLM judge and validated against human annotations.
  • Metrics: Joint Score combines answer accuracy and acceptable response rate as JS = w1 · Acc + w2 · AR, with w1 = 0.7 and w2 = 0.3.The score integrates performance across answerable and unanswerable questions.
  • Baselines and protocol: The study compares no-memory and human-hint baselines with four representative memory methods under a fixed evaluation repository and retrieval budget.Repositories are built from the corresponding training data, frozen during testing, and limited to at most two retrieved memory items per query.
  • Implementation: Experiments use DeepSeek-V3.2 and Qwen3-235B agents in a Wikipedia-based ReAct environment with up to 10 interaction rounds, averaging three runs.The judge model is gpt-5.4-mini.

4 Are Existing Memory Methods Reliable for UAQ?

Memory improves UAQ reliability selectively rather than universally, with effects shaped by model, dataset, transfer setting, and functional channel. Gains primarily reflect reusable decision guidance, while guidance and trajectories can be complementary.

  • In-Distribution Effectiveness: Memory improves UAQ reliability selectively rather than universally across models and datasets.On DeepSeek-V3.2, all four methods improve JS on KUQ, while gains on Qwen3-235B are substantially less stable.
  • In-Distribution Effectiveness: +11.27 / -2.14 / +1.88 percentage points are the aggregate AR / Acc / JS changes for DeepSeek-V3.2, versus +2.93 / -2.54 / -0.90 for Qwen3-235B.These averages cover four memory methods and two in-distribution datasets relative to the No-memory baseline.
  • In-Distribution Effectiveness: Memory gains mainly come from improving AR rather than increasing Acc.Across stronger settings, memory consistently improves AR while leaving Acc largely unchanged, encouraging more cautious UAQ behavior.
  • Human Hint Interaction: Combining memory with Human Hint generally increases AR but often reduces Acc, producing mixed effects on JS.The authors caution that higher AR can reflect over-refusal or excessive caution, so memory design should preserve the AR-Acc balance.
  • Transfer: Cross-dataset transfer is limited because memory remains tied to dataset-specific answerability patterns.Transferred memory may increase caution in AR, but Acc and decision quality can deteriorate simultaneously, weakening JS.
  • Transfer: Cross-model transfer is substantially more stable than cross-dataset transfer.Transferred memory often continues to outperform the No-memory baseline, although some settings degrade relative to same-model memory.
  • Functional Channels: Guidance-only is generally stronger than trajectory-only, especially on KUQ, while combined guidance and trajectories often outperform either channel alone.The guidance advantage is clearest on DeepSeek-V3.2; the combined setting can recover performance when an individual channel falls below No-memory.

5 What Memory Should Store for Reliable UAQ Handling?

Memory effectiveness depends strongly on how experience is represented and composed, not merely on whether memory is available. Procedural and complementary mixed representations provide the most reliable support, while experience polarity interacts with representation.

  • Memory Representation: The memory categories are a practical taxonomy rather than strictly mutually exclusive classes.The paper provides illustrative examples in Appendix A.3.
  • Memory Representation: Memory effectiveness varies substantially across content types, making preserved experience more important than memory availability alone.The controlled ablation keeps the agent framework, retrieval, storage, and inference protocol fixed while changing only memory representation.
  • Memory Representation: Procedural memories, especially workflow and success trace, are generally the most effective, while principle is also competitive.These forms preserve reusable execution patterns or abstract decision rules rather than only raw interaction details.
  • Memory Representation: Compact trajectory remains competitive in some settings by preserving behaviorally useful contextual cues.Effective single-content memories preserve reusable behavioral patterns in compact and transferable forms.
  • Memory Composition: Mixed memory compositions often outperform individual memory types, but effectiveness depends on representation complementarity.Workflow appears in many strong combinations, while principle contributes reusable heuristics and compact trajectory contributes useful context.
  • Memory Composition: Success trace plus workflow is often weaker than combinations pairing workflow with principle or compact trajectory.Combining similar procedural memories does not always produce the largest gains.
  • Experience Polarity: The usefulness of successful and failed experience depends strongly on representation.Compact trajectory performs best with success-only memory, whereas principle can perform best with failure-only memory on DeepSeek-V3.2; insight is less sensitive to polarity.

6 Related Work

Prior work studies UAQs through benchmarks and standalone-model interventions, while memory research targets broader agent capabilities. This work connects the directions by analyzing memory’s transferability, functional mechanisms, and representations for UAQ reliability.

  • Prior Work: Prior UAQ research has addressed false premises, uncertainty expression, and abstention through benchmarks, prompting, training, and retrieval-based methods.These efforts mainly evaluate standalone UAQ handling or QA systems.
  • Prior Work: Memory research has become important for improving long-horizon reasoning, planning, and experience reuse in LLM-based agents.The paper positions agent memory as a complementary research direction to UAQ handling.
  • This Work: This work systematically analyzes memory transferability, functional mechanisms, and content representation for UAQ reliability under a unified agentic framework.It differs from prior work focused mainly on standalone UAQ handling or general agent performance.

7 Conclusion

The study finds that reliable UAQ memory depends on transferable behavioral guidance rather than simply storing more experience. Its conclusions are scoped to controlled UAQ handling, with important limits from the evaluated tasks, methods, and fixed environment.

  • Memory can improve UAQ performance, but gains are selective and fragile under dataset shift; cross-model reuse is often more feasible than cross-dataset transfer.
  • UAQ gains are more strongly preserved through decision guidance than trajectory shaping, with representation strongly affecting effectiveness.
  • Procedural and rule-based memories provide particularly useful support, while failure-derived signals can contribute to rule-based memory.
  • Reliable UAQ memory depends less on storing larger amounts of memory than on preserving transferable behavioral guidance.
  • The findings focus on reliable UAQ handling rather than general agent performance and therefore may not directly generalize to substantially different objectives.
  • Experiments use a fixed agentic RAG framework with Search and Lookup in a Wikipedia-based ReAct environment, excluding heterogeneous tools and richer long-horizon evidence aggregation.

A.2 Baseline Memory Methods

Four representative memory methods are implemented within a shared agentic RAG and Wikipedia environment, while analytical memory types are extracted and retrieved through a unified pipeline.

  • Four representative methods are implemented under the same agentic RAG framework and Wikipedia interaction environment.
  • Embedding, retrieval budget, interaction environment, and inference protocol are standardized while preserving each method’s core memory construction and reuse mechanism.
  • Expel stores reflective insights and successful execution patterns, while MemEvolve maintains decision-oriented summaries and abstract principles.
  • AWM uses workflow-style procedural memory, whereas AgentKB stores structured packages combining planning and execution information.
  • Generic methods Mem0 and MemOS showed no meaningful UAQ performance gains and were excluded from the main experiments.
  • Analytical memory types are derived from complete trajectories and designed to represent descriptive, rule-based, and procedural memory.
  • The representations range from compact trajectory notes and localized insights to abstract principles, success traces, and reusable workflows.

A.4 Reliability of LLM-as-judge Used in AR Evaluation

The study validates the LLM-as-judge used for Acceptable Ratio evaluation against human annotations. Agreement is high across the evaluated datasets.

  • 100 UAQs per dataset from the No-memory setting were independently annotated by three humans, with majority labels forming the final judgment.
  • Responses were marked acceptable when they explicitly refused, identified false premises, or asked clarification questions for ambiguous queries.
  • LLM-as-judge and human AR were compared using Cohen’s Kappa, which achieved high values across all datasets.

B.1 Main Results on Memory Effectiveness

The closed-source GPT-5.5 evaluation reports memory improvements over No-memory across the included dataset–method comparisons. These results suggest that memory effects vary with the base model and its AR–Acc trade-off.

  • GPT-5.5 was evaluated with the same memory-update and held-out-test protocol, using No-memory and three representative memory methods.
  • All three representative memory methods improve JS over No-memory on both datasets.
  • The GPT-5.5 results show more balanced AR and Acc improvement than some open-source models’ predominantly AR-driven gains.

B.2.1 Cross-Dataset Transfer

The appendix reports evaluations of memory effectiveness under in-distribution, transfer, guidance-versus-trajectory, memory-type, composition, and success–failure settings. Quantity-controlled results indicate that memory effects remain content-dependent and that the best setting becomes less stable.

  • Table 14 reports cross-model transfer results on KUQ using mean±std.
  • Table 15 compares decision guidance (G), interaction trajectories (T), and their combination (G&T) using JS scores.
  • Table 16 gives full mean±std results for guidance-only, trajectory-only, and combined memory.
  • Quantity-Controlled Setting: With equal sample quantities, compact trajectory benefits most from successful cases, Insight is relatively insensitive to source type, and Principle is more sensitive to success–failure distinctions, while the best setting becomes less stable.
  • Table 13 reports cross-dataset transfer results using mean±std.

C Prompts

The prompts define the evaluation pipeline, agent interaction format, and memory-content extraction procedures. They distinguish reusable workflows, concise notes, guiding principles, actionable insights, and structured step-by-step summaries, with appendix tables documenting corresponding evaluations.

  • C.1 Prompts for Evaluation Pipeline and Metrics: The evaluation agent follows a ReAct-style loop that interleaves Thought and Action, using Search, Lookup, and Finish[answer] actions.
  • C.1 Prompts for Evaluation Pipeline and Metrics: The Human Hint prompt instructs the agent to abstain when it cannot provide a valid answer and ask for clarification when crucial information is missing.
  • C.1 Prompts for Evaluation Pipeline and Metrics: The no-memory system prompt requires unsupported, nonsensical, unsafe, or insufficiently informed requests to be identified and refused.
  • C.1 Prompts for Evaluation Pipeline and Metrics: The unanswerable_check evaluator outputs verdict 0 for successful answering, 1 for rejection, and -1 for clarification requests, which are used to compute AR.
  • C.2 Prompts for Memory Content: Principle-style memory produces one-sentence guiding principles for successful cases and cautions derived from failed cases.
  • C.2 Prompts for Memory Content: Insight memory extracts 3-6 actionable lessons from successful or failed executions, focusing respectively on what worked or what should be avoided.
  • C.2 Prompts for Memory Content: Compact trajectory memory distills successful executions into a concise, reusable workflow emphasizing strategy, key steps, and decision flow.
  • C.2 Prompts for Memory Content: The appendix reports evaluations of memory types, content compositions, and successful-versus-failed experience ablations in Tables 17-20.
Loading 2608.27924v1…