Source-linked AI summary
RuleMem: Active Rule Memory for Long-Term Conversational Agents
Xingyuan Zeng, Zuohan Wu, Quanming Yao, Yue Wang, Wei Liu, Libin Zheng, Jiuke Wang, Jian Yin
TL;DR
Long-term conversational QA must bridge semantic gaps and support reliable reasoning over dispersed dialogue histories, while existing memory mainly passively stores facts. RuleMem induces and validates natural-language Horn-clause rules to actively guide evidence retrieval and deduction. Across evaluated benchmarks, the framework reduces recall and reasoning failures and outperforms mainstream memory baselines, though a documented case shows over-generalized rules can override contradictory facts.
Problem
Long-term conversational QA faces semantic gaps in evidence retrieval and reliability challenges when reasoning over retrieved evidence.
Method
RuleMem induces reusable natural-language Horn-clause rules from historical conversations and validates them with Rule Perplexity Consistency using internal and external consistency signals.
Results
RuleMem alleviates recall and reasoning failures and outperforms mainstream memory baselines across multiple long-term conversation QA benchmarks.
Takeaways & Limitations
RuleMem demonstrates that conversational memory can actively guide retrieval and reasoning, helping agents learn and deduce from experience.
Takeaways & Limitations
A documented failure shows that an over-generalized activated rule can override specific contradictory factual evidence and produce an incorrect answer.
Abstract
from arXiv · showhide
Question answering agents in long-term conversations must reason over massive, temporally dispersed dialogue histories. However, existing memory mechanisms primarily treat past information as \textit{passively} stored facts, leading to semantic gaps and unreliable reasoning. To address this limitation, we propose RuleMem, a rule-based memory framework that induces reusable logical rules from historical interactions to \textit{actively} guide both evidence retrieval and reasoning. Specifically, RuleMem constructs natural-language Horn clauses from conversations and validates them via a Rule Perplexity Consistency (RPC) mechanism. These induced rules enable the retrieval of semantically distant evidence while providing an explicit logical structure for answer generation. We conducted a comprehensive evaluation of RuleMem on two long-term conversational benchmarks, LoCoMo and LongMemEval_s*. In a rigorous comparison against 14 baselines on LoCoMo, RuleMem achieved the highest accuracy, exceeding the baseline average by 27.47 points (a 54.3% relative improvement).
Introduction
Long-term conversational QA must retrieve semantically indirect evidence and reason reliably across massive, temporally dispersed dialogue histories. RuleMem addresses both challenges by inducing reusable natural-language rules that actively guide retrieval and deduction, with RPC filtering unreliable rules.
- Long-term conversational QA requires reasoning across massive, unstructured, and temporally dispersed dialogue histories.
- Semantic gaps arise when relevant evidence does not explicitly use the query’s wording, such as inferring absence from a recorded vacation.
- Existing fact-memorization methods rely heavily on lexical or shallow semantic overlap between queries and stored memories.
- Rule induction abstracts reusable patterns from specific observations, helping identify clues and interpret evidence for new questions.
- RuleMem uses natural-language Horn clauses to guide retrieval of semantically distant evidence and provide explicit premises for logical deduction.
- RPC filters induced rules by evaluating consistency between internal language priors and external factual evidence.
Related Work
Prior memory systems store facts, organize instances, or record environment-specific experience, but conversational QA needs generalizable abstractions and deductive premises. RuleMem supplies generalized rules to complement retrieved concrete facts.
- LLM agent memory includes factual and experiential forms, with factual systems evolving from episodic streams toward read-write architectures and structured topologies.
- Experiential action traces are tightly coupled to specific environments, limiting their transferability to conversational QA.
- Standard retrieval-augmented generation supplies concrete facts but lacks abstract major premises needed for rigorous deductive reasoning.
- RuleMem extracts generalized logical rules from past conversations to serve as explicit major premises and guide inference.
The RuleMem Framework
RuleMem forms a closed loop: it converts dialogue into facts and reasoning paths, induces and validates reusable rules, then uses those rules to retrieve evidence and generate explicit answers.
- Bottom-Up Rule Memory Construction: RuleMem maintains a fact memory base of temporal dialogue facts and a rule memory base of induced natural-language rules.
- Bottom-Up Rule Memory Construction: Reasoning paths are sampled from graph-structured facts, filtered for logical validity, and linked to raw dialogue snippets before induction.
- Bottom-Up Rule Memory Construction: Rule induction groups similar relational paths and abstracts entity-specific facts into typed placeholders within Horn-clause structures.
- Rule Validation via RPC: RPC tests whether induced rules are supported by internal model priors and retrieved external facts, retaining only rules above a confidence threshold.
- Top-Down Rule-Guided Question Answering: At query time, RuleMem matches questions to rule heads before using rule bodies as cues for factual retrieval.
- Top-Down Rule-Guided Question Answering: An LLM filters recalled facts against question entities and typed constraints, after which rules and grounded evidence jointly structure answer generation.
Experiments
RuleMem is evaluated on long-term conversational benchmarks against memory and retrieval baselines, with analyses targeting recall, reasoning, robustness, and component contributions. It improves retrieval and reasoning through Guided Recall and Explicit Reasoning, while performance depends on reliable rule filtering and balanced RPC hyperparameters.
- Main Results: RuleMem achieves average BLEU of 36.90 and accuracy of 78.05 across four LoCoMo question types.The evaluated types are Single-hop, Multi-hop, Open-domain, and Temporal.
- Recall Analysis: Guided Recall raises average recall from 0.56 to 0.79 (+41.1%) across memory frameworks.It uses activated-rule bodies as supplementary search constraints to retrieve logically related facts lacking direct keyword overlap.
- Reasoning Analysis: Explicit Reasoning reduces average reasoning failures from 120.4 to 105.9 (-12.0%).It injects abstract rules as a logical skeleton when retrieved evidence is correct but contexts remain difficult to organize.
- Ablation Study: Removing rule abstraction or RPC filtering degrades performance, showing that both abstract rules and reliability checks contribute to RuleMem.Without RPC, erroneous or hallucinated rules can interfere with deduction.
- Robustness and Sensitivity: RuleMem consistently outperforms all baselines across gpt-4o-mini, gpt-4o, and qwen3-next-80b-a3b-instruct.The RPC admission threshold is most effective at τ = 0.5, while the signal balancing coefficient peaks at α = 0.4; extreme settings either admit unreliable rules, sparsify memory, or reduce performance.
Conclusion
RuleMem transforms agent memory from passive storage into an active tool for retrieving relevant evidence and supporting logical deduction. Experiments indicate that guided recall, explicit reasoning, and RPC filtering address different sources of long-term conversational QA failure.
- RuleMem transforms memory from passive storage into an active guide for evidence retrieval and logical deduction.
- Guided Recall finds logical evidence, while Explicit Reasoning reduces errors in answer generation.
- RPC filters unreliable rules, supporting higher-quality rule memory.